AI fundamentals, part 4: Scaling laws — the economics of intelligence

Part 4from the AI fundamentals series · 8 parts in all

Part 4 is the one practitioners should read twice: the mathematics that decides whether a model costs \$100 or \$100 million. Neural loss follows strikingly regular power laws in parameters, data and compute — discovered empirically and so reliable they've been used to predict the performance of GPT-4 before training it. Understanding them means understanding every modern training decision.

The three-way trade-off

Three knobs, one loss. Roughly (Chinchilla-era formulation):

L(N, D) ≈ E + A/N**alpha + B/D**beta
#      irreducible + param-limited + data-limited terms
# N = parameters, D = training tokens, E = entropy of natural text (~1.7 nats)
# alpha ~ 0.34, beta ~ 0.28 in the Chinchilla paper's fit

Each term shrinks as you grow, with diminishing returns — double the parameters and the parameter-limited term only drops by 2^-0.34 ≈ 0.79×. The art is choosing N and D so neither term dominates.

Chinchilla: the 20-tokens-per-parameter rule

Before 2022, the industry over-parameterized: GPT-3 has 175B parameters trained on only 300B tokens. The Chinchilla study re-ran the optimization and found compute-optimal training scales data and parameters together — about 20 tokens per parameter. GPT-3 should have been trained on 3.5 trillion tokens; a 70B model trained on 1.4T tokens beat it at a fraction of the serving cost. Every serious model since (Llama etc.) is Chinchilla-shaped.

def chinchilla_optimal(compute_budget_flops, tokens_per_param=20):
    # Given a compute budget, what's the compute-optimal model size?
    # Approximate training compute: C ~ 6 * N * D (forward + backward, per token).
    # C = 6*N*D and D = 20*N  =>  C = 120*N^2  =>  N = sqrt(C/120)
    n_params = (compute_budget_flops / (6 * tokens_per_param)) ** 0.5
    return int(n_params), int(tokens_per_param * n_params)

N, D = chinchilla_optimal(1e23)     # a serious-but-plausible budget
print(f"optimal: {N/1e9:.1f}B params on {D/1e9:.1f}B tokens")
# -> ~29B params on ~577B tokens (sanity check: right ballpark for modern open models)

What compute actually costs

def training_cost(n_params, n_tokens, gpu_hour_price=2.0, mfu=0.4):
    # dollars = (6*N*D flops) / (GPU flops/s * utilization) * price/hour
    # MFU = model flops utilization: real clusters hit 0.3-0.5, not datasheet peak.
    h100_flops = 990e12                      # dense BF16, datasheet
    flops = 6 * n_params * n_tokens
    gpu_hours = flops / (h100_flops * mfu) / 3600
    return gpu_hours * gpu_hour_price

print(f"${training_cost(70e9, 1.4e12):,.0f} for a 70B/1.4T-token run")
# -> roughly $400k of compute. Scale N and D 10x and you're at tens of millions.

This is why scaling laws were secret-adjacent for years: the curve is the business plan. It tells you whether the next \$10M buys a meaningfully better model or disappears into the flat part of the curve.

Beyond the optimum: inference matters too

Chinchilla optimizes training compute. If a model will serve a billion requests, its lifetime inference cost can dwarf training — which argues for smaller, longer-trained models (the "overtrained" regime: Llama-3 trains 8B on 15T tokens, 15× past Chinchilla, because serving 8B is many times cheaper than serving 70B). The modern recipe: Chinchilla-optimal for frontier experiments, deliberately overtrained for deployable models.

Emergence — and its skeptics

Some abilities (multi-digit arithmetic, chain-of-thought reasoning) look like they switch on sharply at scale — the "emergence" phenomenon. A careful reading: metrics can make smooth improvements look sudden (getting 9/10 digits right scores zero on "exact match"). The safe engineering conclusion either way: capability curves are predictable in aggregate and unpredictable at the task level — test your own task at the scale you can afford before betting on it.

Next: we leave training and enter inference — how a trained model turns those laws into text, one sampled token at a time, and why that process costs 1000× less than you'd fear but is harder to make fast than you'd think.