Logistic Regression and the Sigmoid Function

Linear regression predicts numbers, but many real questions have yes-or-no answers: will a student pass, is an email spam, will a customer repay a loan? Logistic regression handles these by passing a linear score through the sigmoid function, turning any real number into a probability between 0 and 1. Despite its name, it is a classification method, and it remains one of the most widely used models in statistics and machine learning.

Why Not Just Use a Straight Line?

Suppose we code "pass" as 1 and "fail" as 0 and fit an ordinary regression line. For extreme inputs the line happily predicts values like 1.4 or −0.3, which make no sense as probabilities. We need a function that:

• accepts any real input \(z\),
• always outputs a value strictly between 0 and 1,
• increases smoothly, so that bigger scores mean higher probabilities.

The sigmoid (or logistic) function does exactly this.

The Sigmoid Function

$$ \sigma(z) = \frac{1}{1 + e^{-z}} $$

Some key values show its shape:

• \(\sigma(0) = \frac{1}{1+1} = 0.5\)
• As \(z \to +\infty\), \(e^{-z} \to 0\), so \(\sigma(z) \to 1\).
• As \(z \to -\infty\), \(e^{-z} \to \infty\), so \(\sigma(z) \to 0\).
• Symmetry: \(\sigma(-z) = 1 - \sigma(z)\). For example \(\sigma(2) \approx 0.8808\) and \(\sigma(-2) \approx 0.1192\).

The graph is an S-shaped curve centred at \((0, 0.5)\). Its derivative has a beautifully compact form. Differentiating \((1 + e^{-z})^{-1}\) with the chain rule gives

$$ \sigma'(z) = \frac{e^{-z}}{(1+e^{-z})^2} = \sigma(z)\bigl(1 - \sigma(z)\bigr) $$

This identity makes gradient calculations fast, and it reappears in neural networks.

The Logistic Regression Model

For an input \(x\) (or a vector of features), we compute a linear score \(z = w x + b\) and convert it to a probability:

$$ P(y = 1 \mid x) = \hat{p} = \sigma(wx + b) $$

Log-Odds Interpretation

Rearranging the sigmoid shows what the linear part really models. If \(\hat{p} = \sigma(z)\), then

$$ \ln\!\left(\frac{\hat{p}}{1 - \hat{p}}\right) = z = wx + b $$

The quantity \(\hat{p}/(1-\hat{p})\) is the odds, and its logarithm is the logit. So logistic regression is linear in the log-odds. A one-unit increase in \(x\) adds \(w\) to the log-odds, which multiplies the odds by \(e^{w}\). This is why researchers report odds ratios.

The Decision Boundary

To classify, we usually predict 1 when \(\hat{p} \ge 0.5\). Since \(\sigma(z) = 0.5\) exactly when \(z = 0\), the boundary is the set of points where \(wx + b = 0\). With two features it is a straight line; with many features it is a hyperplane. Logistic regression is therefore a linear classifier.

Worked Example 1: Predicting Exam Success

A model trained on past students gives \(w = 1.2\) and \(b = -3\), where \(x\) is the number of hours of revision per day. Find the probability of passing for 1, 2, 3 and 4 hours, the decision boundary, and the odds ratio.

Step 1: Compute the scores \(z = -3 + 1.2x\):

• \(x = 1\): \(z = -1.8\)
• \(x = 2\): \(z = -0.6\)
• \(x = 3\): \(z = 0.6\)
• \(x = 4\): \(z = 1.8\)

Step 2: Apply the sigmoid. For \(x = 3\):

$$ \hat{p} = \frac{1}{1 + e^{-0.6}} = \frac{1}{1 + 0.5488} = 0.6457 $$

Similarly, \(x = 1\) gives 0.1419, \(x = 2\) gives 0.3543 and \(x = 4\) gives 0.8581. Notice the symmetry: the probabilities for 2 and 3 hours add to 1, as do those for 1 and 4 hours, because their scores are negatives of each other.

Step 3: Decision boundary. Set \(z = 0\):

$$ -3 + 1.2x = 0 \quad\Longrightarrow\quad x = 2.5 \text{ hours} $$

Students revising more than 2.5 hours a day are predicted to pass.

Step 4: Odds ratio. \(e^{1.2} \approx 3.32\). Every extra hour of daily revision multiplies the odds of passing by about 3.3.

Training: Cross-Entropy Loss

We cannot use squared error comfortably here, because combined with the sigmoid it produces a non-convex loss with flat regions. Instead we maximise the likelihood of the observed labels. For one example with true label \(y \in \{0, 1\}\), the probability the model assigns to the correct answer is \(\hat{p}^{\,y}(1-\hat{p})^{1-y}\). Taking the negative logarithm gives the binary cross-entropy (log loss):

$$ L = -\bigl[\,y \ln \hat{p} + (1 - y)\ln(1 - \hat{p})\,\bigr] $$

If \(y = 1\) and the model says \(\hat{p} = 0.6457\), the loss is \(-\ln 0.6457 \approx 0.437\). If it had said 0.01, the loss would be \(-\ln 0.01 \approx 4.61\): confident wrong answers are punished heavily.

The Gradient

Here is the pleasant surprise. Using \(\sigma' = \sigma(1-\sigma)\) and the chain rule, the gradient of the loss with respect to the score simplifies to

$$ \frac{\partial L}{\partial z} = \hat{p} - y $$

and therefore

$$ \frac{\partial L}{\partial w} = (\hat{p} - y)\,x, \qquad \frac{\partial L}{\partial b} = \hat{p} - y $$

The error term is just "prediction minus truth", the same form as in linear regression.

Worked Example 2: One Gradient Descent Step

Current parameters: \(w = 0.5\), \(b = -1\). A training example has \(x = 2\), \(y = 1\). Use learning rate \(\alpha = 0.5\).

Step 1: Forward pass.

$$ z = 0.5(2) - 1 = 0, \qquad \hat{p} = \sigma(0) = 0.5 $$

Step 2: Loss. \(L = -\ln 0.5 \approx 0.693\).

Step 3: Gradients.

$$ \hat{p} - y = 0.5 - 1 = -0.5 $$

$$ \frac{\partial L}{\partial w} = (-0.5)(2) = -1, \qquad \frac{\partial L}{\partial b} = -0.5 $$

Step 4: Update.

$$ w \leftarrow 0.5 - 0.5(-1) = 1.0, \qquad b \leftarrow -1 - 0.5(-0.5) = -0.75 $$

Step 5: Check. The new score is \(1.0(2) - 0.75 = 1.25\), so \(\hat{p} = \sigma(1.25) \approx 0.777\) and the loss falls to \(-\ln 0.777 \approx 0.252\). The model moved towards the correct label, exactly as intended. Unlike linear regression, there is no closed-form solution, so this process (or Newton's method, used by many statistics packages) is repeated until convergence.

Beyond Two Classes

For \(K\) classes, the sigmoid generalises to the softmax function:

$$ P(y = k \mid \mathbf{x}) = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} $$

Each class gets its own score, and softmax converts the scores into probabilities that sum to 1. With \(K = 2\), softmax reduces exactly to the sigmoid.

Did You Know?

• The logistic curve was first studied by the Belgian mathematician Pierre-François Verhulst around the 1840s as a model of population growth, long before machine learning existed.
• A single artificial neuron with a sigmoid activation is mathematically identical to logistic regression.
• Medical researchers often report results of logistic regression as odds ratios, for example "smokers had 2.5 times the odds of the disease".

Key Takeaways

• The sigmoid \(\sigma(z) = 1/(1+e^{-z})\) maps any real score to a probability, and \(\sigma' = \sigma(1-\sigma)\).
• Logistic regression is linear in the log-odds; \(e^{w}\) is the odds ratio for a one-unit increase in \(x\).
• The decision boundary is where \(wx + b = 0\), making it a linear classifier.
• Training minimises binary cross-entropy, whose gradient is simply \((\hat{p} - y)x\).

Need help with classification models, probability derivations or a machine learning project? Amathify's experts and PhD scholars offer step-by-step solutions and assignment help; reach us on WhatsApp or through the Ask a Question form.