giagrad.Tensor.gelu#

Tensor.gelu() → Tensor[source]#

Creates a new Tensor applying Gaussina Error Linear Unit (Leaky ReLU) function to data. See GELU.

o u t_{i} = d a t a_{i} Φ (d a t a_{i}) = d a t a_{i} \times \frac{1}{2} [1 + erf (\frac{d a t a_{i}}{\sqrt{2}})]

Where $Φ$ is the Gaussian cumulative distribution function.

Examples

>>> t = Tensor.empty(2, 3).uniform(-1, 1)
>>> t
tensor: [[-0.42565832  0.8579072  -0.40772486]
         [ 0.4038496   0.09953032 -0.6694602 ]]
>>> t.gelu()
tensor: [[-0.14268097  0.6901076  -0.1393431 ]
         [ 0.26525608  0.05371065 -0.1684846 ]] fn: GELU