Which activation function is commonly used in deep learning to introduce non-linearity?

The commonly used activation function in deep learning that introduces non-linearity is the sigmoid activation function. The primary role of an activation function is to apply a non-linear transformation to the output of a neuron, allowing the model to learn complex patterns within the data.

The sigmoid function maps any input value to a value between 0 and 1, which can help model probabilities, particularly in binary classification tasks. It has a characteristic "S" shape, which enables it to handle various ranges of input while still providing non-linearity.

This non-linearity is crucial because, without it, a neural network could only represent linear relationships, similar to linear regression, reducing its capability to learn from complex datasets. The inclusion of activation functions like sigmoid allows networks to approximate functions and learn intricate patterns necessary for effective predictions.

In contrast, the linear activation function does not introduce non-linearity, and the step function can lead to issues such as vanishing gradients, limiting the ability of the network to learn effectively. Weight normalization is primarily a technique for stabilizing the training of neural networks, but it does not serve as an activation function. Therefore, the use of the sigmoid activation function stands out as a significant contributor to the learning capacity of deep networks by providing the necessary