giagrad.Tensor.gelu#

Tensor.gelu() → Tensor[source]#

Creates a new Tensor applying Gaussina Error Linear Unit (Leaky ReLU) function to data. See GELU.

\[out_i = data_i \ \Phi(data_i) = data_i \times \frac{1}{2} \left[1 + \text{erf}(\frac{data_i}{\sqrt{2}})\right]\]

Where \(\Phi\) is the Gaussian cumulative distribution function.

Examples

>>> t = Tensor.empty(2, 3).uniform(-1, 1)
>>> t
tensor: [[-0.42565832  0.8579072  -0.40772486]
         [ 0.4038496   0.09953032 -0.6694602 ]]
>>> t.gelu()
tensor: [[-0.14268097  0.6901076  -0.1393431 ]
         [ 0.26525608  0.05371065 -0.1684846 ]] fn: GELU