Scaled Exponential Linear Unit (SELU).

The Scaled Exponential Linear Unit (SELU) activation function is defined as:

• scale * x if x > 0

• scale * alpha * (exp(x) - 1) if x < 0

where alpha and scale are pre-defined constants (alpha = 1.67326324 and scale = 1.05070098).

Basically, the SELU activation function multiplies scale (> 1) with the output of the activation_elu function to ensure a slope larger than one for positive inputs.

The values of alpha and scale are chosen so that the mean and variance of the inputs are preserved between two consecutive layers as long as the weights are initialized correctly (see initializer_lecun_normal()) and the number of input units is "large enough" (see reference paper for more information).

Usage

activation_selu(x)

Arguments

x

Input tensor.

Value

A tensor, the result from applying the activation to the input tensor x.

Notes

• To be used together with initializer_lecun_normal().

• To be used together with the dropout variant layer_alpha_dropout() (legacy, depracated).

Reference

• Klambauer et al., 2017

See also

• https://keras.io/api/layers/activations#selu-function

Other activations:
activation_elu()
activation_exponential()
activation_gelu()
activation_hard_sigmoid()
activation_leaky_relu()
activation_linear()
activation_log_softmax()
activation_mish()
activation_relu()
activation_relu6()
activation_sigmoid()
activation_silu()
activation_softmax()
activation_softplus()
activation_softsign()
activation_tanh()