giagrad.Tensor.silu#

Tensor.silu(beta=1.0) → Tensor[source]#

Returns a new Tensor with element-wise Sigmoid-Weighted Linear Unit (SiLU) function, also called Swish. See Swish.

For numerical stability SiLU is computed with numpy.logaddexp.

\[out_i = \frac{data_i}{(1 + \exp(\text{beta} \times -data_i))}\]

Parameters:: beta¶ (float) – Hyperparameter for Swish formulation.

Examples

>>> t = Tensor.empty(2, 3).uniform(-10, 10)
>>> t
tensor: [[ 5.4958744   0.13549101 -4.5210676 ]
         [-1.7155124   5.2369795  -7.6546626 ]]
>>> t.silu()
tensor: [[ 5.4734135e+00  7.2327957e-02 -4.8648320e-02]
         [-2.6153007e-01  5.2092857e+00 -3.6252895e-03]] fn: SiLU(beta=1.0)