How Neural Networks Function and Why Linear Relationships Alone Are Not Enough

Meta description: Ms. Poly uses ReLU and Sigmoid in a simple neural network to model non-linearity and predict 58 exam marks and pass or fail outcomes for beginners.

Tags: ReLU, Sigmoid, neural networks, non-linearity, deep learning

Featured image: abstract friendly illustration of a straight line bending into Rectified Linear Unit and Sigmoid curves, with no real people, no logos, and no text.

This guide is for readers who want to understand non-linearity in neural networks. It follows Ms. Poly, a math teacher introduced in the earlier explainer How Neural Networks Work – Explained Using the Straight Line Equation (y=ax+b). She starts from the assumption that exam marks rise in direct proportion to hours studied, then uses a simple neural network with ReLU and Sigmoid functions to predict marks and pass-or-fail results. The assumed background is basic linear functions and familiarity with deep neural networks.

The practical result is compact: the ReLU-based model predicts 60 marks for a 5-hour study case, close to the actual 58, and the ReLU and Sigmoid model classifies the pass-or-fail result for all students in the example.

Why a straight line cannot fit the marks

A straight line applies one rule across all input values. In the equation y=ax+b, y is the output variable and x is the input variable; a describes how much y changes when x changes, and b is the value of y when x is 0.

For Ms. Poly, y becomes marks scored, x becomes number of hours studied, a becomes the increase in marks for every hour studied, and b becomes the minimum marks every student is capable of scoring without studying. She creates a study plan around one factor: the number of hours studied per week. After a quarterly exam, she interviews her students, records how many hours they studied math per week, and plots the data.

The simple formula does not align with the collected data, which shows that real-life data rarely behaves that directly. Ms. Poly notices a pattern: students who study for only a few hours may not see much improvement. Once they cross a certain point, marks begin to increase with additional study. After another point, marks remain fairly constant.

The relationship can be split into three parts:

• Less than t₁ hours: marks remain constant
• t₁ to t₂ hours: marks increase with study time
• More than t₂ hours: marks remain constant

That means separate equations are needed:

• y = 1st equation, x < t₁
• y = 2nd equation, t₁ <= x < t₂
• y = 3rd equation, x >= t₂

A single straight line is insufficient because the behaviour changes at specific points.

ReLU turns piecewise linear rules into a network layer

To keep the example simple, Ms. Poly first uses two regions:

• Region 1: hours of study < t₁
• Region 2: hours of study >= t₁

In the first region, values hover around a particular level. They do not lie perfectly in a line, but they can be approximated by a line with slope=0, indicating that marks remain roughly the same. In the second region, there is a positive slope, so another straight-line equation is used for the steep upward sloping line.

Individually, these are straight-line equations. Together, the overall relationship is no longer a straight line. When x crosses a certain value, the equation changes. That is a piecewise linear function: different rules for different ranges of input.

This idea appears in ReLU, one of the most widely used activation functions in neural networks. ReLU stands for Rectified Linear Unit. Mathematically:

• y = ReLU(x) = max(0,x)

It keeps positive inputs unchanged and turns negative inputs to zero. The individual piece is linear, but the whole function is non-linear, so it allows a neuron to respond differently across different input ranges.

For the mark-prediction case, Ms. Poly uses an untrained network with manually chosen weights and biases. The input layer receives the number of hours studied, x. The hidden layer applies a linear transformation and ReLU:

• z = w₁x + b₁ = x - 3, with w₁=1 and b₁=-3
• h = ReLU(z) = max(0, z)

The output layer predicts marks after applying a linear transformation to the hidden layer output:

• ŷ = w₂h + b₂ = 10h + 40, with w₂=10 and b₂=40

For a student who studies 5 hours:

• z = 5 - 3 = 2
• h = max(0, 2) = 2
• ŷ = 10(2) + 40 = 60

The network predicts 60 marks, while the actual mark in the dataset is 58.

The actual and predicted marks do not match exactly, but they are close. The marks keep increasing and do not flatten out at the end because the value keeps increasing after the ReLU threshold. This shows how ReLU helps model non-linear relationships in the data.

Sigmoid adds a smooth probability shape

The piecewise linear fit still has a sharp transition from one region to another. Ms. Poly also only considered two regions for simplicity, even though the distribution can be divided into three regions. She draws a smooth curve roughly passing through all the points.

That curve stays almost flat when a student studies up to t₁ hours. From t₁ to t₂, more study hours increase the marks scored, creating a slope. After t₂, it flattens again. The transition is gradual and continuous, unlike the piecewise linear curve, which changes direction using simple straight lines.

This shape can be represented mathematically by the sigmoid function. A sigmoid accepts any real-valued input and maps it to values between 0 and 1. The curve starts near 0, rises quickly in the middle, and settles near 1.

• y = σ(x) = 1 / (1 + exp(-x))

In neural networks, the sigmoid function is useful because it introduces non-linearity while also restricting output between 0 and 1. That makes it suitable when the output needs to represent a probability, such as the likelihood of a student passing an exam. A higher sigmoid output indicates a higher probability of passing, and a lower output indicates a lower probability.

For pass-or-fail prediction, Ms. Poly assumes a student needs a minimum of 50 marks to pass. The network uses:

• Input layer: number of hours studied, x
• Hidden layer: z₁ = w₁x + b₁ = x - 4, with w₁=1 and b₁=-4
• h = ReLU(z₁) = max(0, z₁)
• Output layer: z₂ = w₂h + b₂ = 2h - 1, with w₂=2 and b₂=-1
• p = σ(z₂) = 1 / (1 + exp(-z₂))
• ŷ = Pass, if p >= 0.5
• ŷ = Fail, if p < 0.5

For a student who studies 3 hours:

• z₁ = 3 - 4 = -1
• h = max(0, -1) = 0
• z₂ = 2(0) - 1 = -1
• p = 1 / (1 + exp(-(-1))) ≈ 0.269

Since 0.269 < 0.5, the predicted result is Fail.

For a student who studies 5 hours:

• z₁ = 5 - 4 = 1
• h = max(0, 1) = 1
• z₂ = 2(1) - 1 = 1
• p = 1 / (1 + exp(-1)) ≈ 0.731

Since 0.731 > 0.5, the predicted result is Pass.

Every student is classified correctly in this small example. As study hours increase beyond 4, the predicted score increases, making a pass prediction more likely. The example shows how ReLU transforms the input before Sigmoid maps it to a probability.

Common mistakes and what to watch next

The example shows three common mistakes. First, readers may assume y=ax+b can describe every relationship, but real data can change behaviour at different input ranges. Second, they may expect ReLU to flatten outputs; in the mark model, predicted marks reach 90 at 8 hours because the linear output continues after the ReLU threshold. Third, they may treat the manually chosen weights and biases as a finished model. They are only for illustration.

In practice, Ms. Poly would not choose the parameters herself. A real network learns weights and biases during training by repeatedly comparing predicted values with actual values and adjusting parameters to minimize error. The pass-or-fail case also shows why Sigmoid matters: it turns a numeric signal into a probability between 0 and 1, then lets a threshold decide the final result.

For readers, the takeaway is that non-linearity lets neural networks learn patterns a straight line cannot describe. ReLU and Sigmoid are two simple ways to introduce that behaviour, and they can be combined when a model needs both changing ranges and probability output.