[Machine Learning] Sigmoid function introduction

Last Updated on 2021-06-03 by Clay

Introduction

Sigmoid() function is a mapping function, it will map any variable (In the following content we write the the symbol x) to [0, 1]. And it is often used to be a activation function in neural network layer of Machine Learning.

The most commonly seen application scenario is when we are training the model to do binary classification. We set the last layer of the neural network of the model to have only one neuron, and we input the neural output value into Sigmoid() function. In this way, we will get a value between [0, 1].

What are the benefits of this? Didn't it just mention that what we are going to do is "binary classification" ? We only need to set a threshold, classify all values less than 0.5 as 0, classify all values greater than 0.5 as 1. (In fact, if you want, it doesn't matter the other way around.)

This completes the binary classification.

As for how the model knows which category its output value should be classified into? That is the problem of backward propagation to update the weights of neural networks. This article only focuses on the Sigmoid() function.

The formula of the Sigmoid() function is as follows:

Implement Sigmoid() function

In order to verify that my understanding of the Sigmoid() function is correct, I used Python to implement it here. (Of course you can use other languages)

If you want to repeat the results of my program, you also need to download a package capable of ploting.

sudo pip3 install matplotlib

Below is my code:

# -*- coding: utf-8 -*-
import math
import matplotlib.pyplot as plt

x = []
dx = -20
while dx <= 20:
    x.append(dx)
    dx += 0.1


def sigmoid(x):
    return 1/(1+math.exp(-x))


px = [xv for xv in x]
py = [sigmoid(xv) for xv in x]


plt.plot(px, py)
ax = plt.gca()
ax.spines['right'].set_color('none')
ax.spines['top'].set_color('none')
ax.xaxis.set_ticks_position('bottom')
ax.spines['bottom'].set_position(('data',0))
ax.yaxis.set_ticks_position('left')
ax.spines['left'].set_position(('data',0))
plt.show()

Output:

You can try any range for x, and the output y value is always mapping in [0, 1].

Application

• You can call the "Sigmoid" function in Keras and PyTorch
• If the Sigmoid function output is closed to 0 or 1, sometimes maybe have the "vanishing gradient"

References

• https://arxiv.org/pdf/1811.03378.pdf

Read More

• Sigmoid
• ReLU
• Softmax
• Tanh