Activation Functions Influence the Behavior of an AI Assistant
Activation functions shape how AI assistants answer business questions; choose internal and output activations based on decision type.
Key Takeaways
- Activation functions shape an AI assistant’s behavior by influencing both internal feature transformations and the format of business outputs.
- Design should start from business question types—yes/no, multiple choice, ranking, or explanation—rather than naming functions like ReLU or GELU.
- Internal activations such as ReLU, GELU, SiLU, and SwiGLU operate inside Transformer layers, while output activations like Sigmoid and Softmax determine the final decision format.
- In a Transformer, Softmax is used in attention and output layers to convert scores into probability distributions, while feed-forward activations control how learned features are processed.
- Matching activation choices to task needs helps prevent architecture errors and supports clearer stakeholder discussions before committing to a model design.
When building an AI assistant, the mathematical choices inside the model—particularly activation functions—directly shape how it handles business questions like "Is this expense allowed?" or "Which department should handle this?" This guide explains where activation functions sit in a Transformer, distinguishes internal from business-output activations, and shows how to match function choices to question types such as yes/no decisions, rankings, multiple choice, and explanations. You will get a practical framework for design discussions with stakeholders before committing to model architecture.
Start with Business Questions, Not Terminology
During work discovery, the first conversation should not begin with ReLU, GELU, or SwiGLU. Instead, identify the mathematical operations your assistant must perform. Sample questions reveal the shape of required outputs:
- "Is this expense allowed?" — yes/no decision
- "Which department should handle this?" — selection from many options
- "What is our leave policy?" — explanation generation
- "Compare Plan A and Plan B" — ranking or comparison
- "What should I do next?" — multi-step recommendation
Some questions need a single answer from many choices (multiple choice); others need several answers simultaneously (checkbox type); still others require ranking from high to low or generating explanations. Categorizing questions this way helps determine both the Transformer architecture and the probable activation functions needed.
Two Different Activation Problems
A useful distinction separates activation functions into two categories:
Internal activation functions control how features transform inside the network. Examples include ReLU, GELU, SiLU, GLU, and SwiGLU. The flow of control is: Inside Transformer → How should internal features be transformed? → ReLU / GELU / SiLU / SwiGLU.
Business-output activation functions control the result of the business decision. Examples include Sigmoid and Softmax. The flow is: Business Output → What kind of answer is required? → Sigmoid / Softmax.
Keeping these separate prevents a common design error: tuning internal activations while ignoring the output layer that actually determines whether the assistant returns a probability, a class, or a ranked list.
A Minimal Transformer Pipeline
A small Transformer written in Python contains these steps: token embeddings, positional encoding, self-attention, feed-forward layers, residual connections, normalization, and output layers. A mini transformer is committed on GitHub for reference.
The core pipeline runs: Tokens → Embedding → Positional Information → Self-Attention → Residual + Normalization → Feed-Forward Network → Residual + Normalization → Output Representation.
Activation-related decisions occur in five places: attention Softmax, feed-forward activation, optional gating, task-specific output activation, and final vocabulary Softmax.
Think of the Transformer as a Feature Factory
For a question such as "Can an employee carry forward unused leave?", the Transformer builds an internal representation with hidden features: employee policy = strong, leave = very strong, carry forward = very strong, payroll = weak, termination = weak, exception rule = medium, eligibility = strong.
Call this hidden representation h. The same representation can feed different business purposes: a yes/no head, an intent head, a retrieval score, and language generation. The Transformer creates reusable contextual features; the business heads decide what those features become. This is why output activation choice matters as much as internal activation choice.
Where Softmax Sits in Attention
The attention block computes Q = XWQ, K = XWK, V = XWV, then AttentionScores = QK^T / sqrt(dk), then AttentionWeights = Softmax(AttentionScores), then Output = AttentionWeights × V.
Attention is a competition: among all tokens, where should attention be distributed? For the sentence "the employee submitted the invoice because it was overdue," the word "it" might attend to invoice at 0.72, employee at 0.09, submitted at 0.07, and overdue at 0.12. These values sum to approximately 1—a probability distribution. Softmax converts raw scores (logits) into that distribution.
An illustrative attention function walks through token embedding plus position, QKV projection, dot-product similarity, causal mask plus Softmax, unmasked attention, and weighted value combination. Each step has a distinct job:
- Token Embedding + Position converts words into numerical vectors and adds positional information, enabling mathematical processing that respects word order.
- Query, Key, Value (QKV) separates what to look for (Q), against which to compare (K), and what information to deliver (V), decoupling retrieval from output delivery.
- Dot-Product Similarity measures compatibility between Query and Key vectors via dot product divided by square root of width, controlling score magnitude.
- Causal Mask + Softmax blocks future positions and converts similarity scores into attention weights summing to 1.
- Unmasked Attention computes attention without future-position masking.
- Weighted Value Combination multiplies Value vectors by attention weights and sums them, producing context-aware representations.
The Feed-Forward Network: What to Do with What You Found
Attention answers "Where should I look?" The feed-forward network answers "What should I do with the information I have now?"
A simple feed-forward block is FFN(x) = W2(ReLU(W1x + b1)) + b2. Here ReLU can be replaced with GELU, SiLU, or the block redesigned using SwiGLU. This is where activation-level experimentation with the FFN belongs.
ReLU for Simpler Problems
In designing an HR assistant, questions like "How many casual leaves are available?", "Can leave be carried forward?", "Who approves travel reimbursement?", and "What is the notice period?" are mostly factual, narrow, structured, and relatively low in ambiguity. For such cases, ReLU is a reasonable baseline: ReLU(x) = max(0, x).
If leave_policy = +3.2, termination = -1.4, salary_revision = -0.8, then after ReLU: leave_policy = 3.2, termination = 0, salary_revision = 0. The intuition: keep useful positive features, suppress the rest. For a low-cost system this may be enough, though it should be tested rather than assumed.
Comparing Activation Functions Numerically
Utility code for sigmoid, relu, gelu, and silu runs over inputs from -3.0 to 3.0. The formulas are:
- ReLU(x) = max(0, x)
- GELU(x) = x * NormalCDF(x)
- SiLU(x) = x * sigmoid(x)
- Sigmoid(x) = 1 / (1 + exp(-x))
The output table shows how each function behaves across the input range:
| Input | ReLU | GELU | SiLU | Sigmoid |
|---|---|---|---|---|
| -3.00 | 0.0000 | -0.0040 | -0.1423 | 0.0474 |
| -2.00 | 0.0000 | -0.0455 | -0.2384 | 0.1192 |
| -1.00 | 0.0000 | -0.1587 | -0.2689 | 0.2689 |
| -0.50 | 0.0000 | -0.1543 | -0.1888 | 0.3775 |
| 0.00 | 0.0000 | 0.0000 | 0.0000 | 0.5000 |
| 0.50 | 0.5000 | 0.3457 | 0.3112 | 0.6225 |
| 1.00 | 1.0000 | 0.8413 | 0.7311 | 0.7311 |
| 2.00 | 2.0000 | 1.9545 | 1.7616 | 0.8808 |
| 3.00 | 3.0000 | 2.9960 | 2.8577 | 0.9526 |
The table makes the trade-offs visible: ReLU hard-clips negatives to zero, GELU and SiLU allow small negative values that preserve gradient signal, and Sigmoid compresses everything into the 0–1 range suitable for probability outputs.
Running a Mini Transformer Exercise
The code exercise uses prompts with targets: "cats drink" targeting "milk," "dogs drink" targeting "water," and "I like" targeting "tea." The mini_transformer module exposes ACTIVATION_NAMES, NEXT_WORDS, count_feed_forward_parameters, format_vector, and run_transformer. For a one-word target, loss is computed as -ln(probability assigned to the target), a direct way to see how activation choice affects the probability the model assigns to the correct next word.
What This Means for Assistant Design
The practical takeaway is a two-layer decision. For internal feature transformation, choose among ReLU, GELU, SiLU, and SwiGLU based on how much negative-value signal you want to preserve and how much compute you can spend. For business outputs, choose Sigmoid for yes/no probabilities and Softmax for selecting among mutually exclusive options or producing attention distributions. Map your stakeholder questions to these output shapes first; the internal activation choice follows from the model architecture and budget. Test rather than assume—the numeric table above shows the functions diverge most in the -1 to 1 range, which is exactly where many business-relevant features live.

Comments (0)