Inside the Mechanics of Large Language Models — Daily AI Post

Meta description: Learn how ChatGPT works: small code, huge weights, token chunks, embeddings, and why it can explain ideas but miss letter counts in plain English.

Tags: ChatGPT, large language models, tokens, embeddings, AI literacy

Featured image: abstract friendly illustration of token blocks forming a glowing answer bubble, no people, no logos, no text.

For readers who use ChatGPT daily but cannot explain it, this explainer gives the practical anatomy: ChatGPT works by pairing a small execution program with a very large collection of learned weights. The program, sometimes described by AI researcher Andrej Karpathy as run.c, is about 500 lines of plain C. The weights, in parameters.bin, hold billions of learned numbers, often dozens of gigabytes, and can be imagined as seventy billion sliders. Together they compress roughly 10 to 15 terabytes of web pages, books, code, research papers, and forum posts into about 140 gigabytes of parameters. That is the practical answer to a question people have been avoiding for years: the system can rephrase angry emails to a landlord, calculate the shelf life of opened oat milk, debug Python code, draft a John Donne style sonnet, explain quantum entanglement through cooking metaphors, and still fail the early 2024 strawberry R test.

The Machine Is Two Files, Not a Mind

Karpathy's definition is useful because it removes the mystique. A large language model is not a live mind connected to a server farm. It is two files. That framing avoids two extremes: claims of sentience and dismissals of pure autocomplete.

run.c does not know Napoleon, bread, or gravity. It loads numbers, multiplies them, adds them, applies simple math functions, and prints a word. parameters.bin carries the knowledge-like structure. During training, no engineer set slider number 4,912,014 to 0.042 so the model remembers that Rome is the capital of Italy. The sliders moved through repeated exposure to text, nudged by statistical corrections, like sand dunes shaped by wind.

Next-word prediction is the crank; the apparent understanding is the emergent gear. Calling the result just autocomplete misses the engineering leap. It is closer to saying a Boeing 787 is just a fast bicycle: the basic idea of next-token guessing sounds simple, but the scale changes what the machine can do.

Pretraining Compresses Patterns, Not Pages

Pretraining takes an enormous pile of human text and turns it into weights. The source text is roughly 10 to 15 terabytes, and the output is about 140 gigabytes, a compression ratio near 100-to-1. Ted Chiang's phrase, a blurry JPEG of the web, fits because image compression preserves patterns while discarding exact pixels.

The result is closer to the world's most educated concussion: it does not recite page 214 of War and Peace, but it can answer that Pierre Bezukhov's marriage was unhappy, that Paris is in France, that dropped wine glasses shatter on tile, and that apologies often follow mistakes. It absorbed the shape of the library, not the library word by word.

Stephen Wolfram's point is related: human language has a surprisingly smooth mathematical structure, an implicit physics of semantic motion, much as Isaac Newton found laws that describe cannonballs and moons.

Tokens Explain the Strawberry Failure

The early 2024 question about the R's in strawberry exposed a deeper design choice. Models do not see letters. A tokenizer chops text into standardized chunks from a vocabulary of roughly 100,000 tokens. Strawberry may become [str][aw][berry], arriving as three integer IDs such as [496, 675, 15717]. The letter R inside berry is hidden, so asking the model to count R's is like asking someone to inspect atoms inside a sealed water glass.

This creates a Swiss cheese capability profile: a model can write a Shakespearean soliloquy about quantum chromodynamics because that uses high-level concepts, yet stumble on letter counting or decimal comparisons because tokens mangle the surface form. The holes are not proof of stupidity; they are scars of the machine's anatomy.

Embeddings Turn Words Into Geography

Computers cannot multiply dog by cat, so the model maps each token into a numeric space. Think of Meaning City, where cat, feline, and kitten live near each other, while depreciation and quarterly earnings sit in a financial district. In modern large language models, that map can have 8,192 dimensions.

The model does not understand in a human sense, but nearby directions in that space can support useful reasoning: predicting the next word of a murder mystery requires inferring who the killer might be, predicting a software patch requires logic, memory, and syntax, and predicting how a doctor answers a frightened patient requires something like empathy, triage, and bedside manner.

What This Means for Everyday Use

A raw pretrained model is a document dreamer, not a helpful assistant. If you ask it, "What is the capital of Australia?" it may list questions about New Zealand, Canada, and South Africa instead of answering Canberra, because the internet contains many quizzes. If you paste a medical inquiry, it may hallucinate a forum thread with broken HTML, fake usernames, and herbal tea arguments. ChatGPT's polished behavior is a trained layer on top of that pattern completion.

For users, the practical takeaway is to treat answers as drafts that can be strong on structure and weak on surface details. The tool can run offline from two files, without Google or Wikipedia, but its competence comes from compressed statistical patterns, not a person reading your prompt. That distinction matters when teams turn the tool into daily infrastructure.