From Logistic Regression to AI: The Scaling Laws That Bridge Classical Statistics and Modern Machine Learning

What do a 19th-century statistical method and today's billion-parameter language models have in common? More than you might think.

{{IMAGE:1}}

The Logistic Regression Foundation

Logistic regression, developed in the 1840s, remains one of the most practical statistical tools in existence. Unlike its more famous cousin linear regression, logistic regression predicts probabilities for binary outcomes—will a patient survive? Will a customer buy? Will an email be spam?

The beauty of logistic regression lies in its simplicity and interpretability. Each parameter represents a meaningful relationship, and the model's predictions are transparent. I've seen firsthand how a carefully crafted logistic regression model can be valuable enough for clients to pursue patents—not for the algorithm itself, but for its novel application to specific problems.

The Small Data World

In clinical trials at MD Anderson Cancer Center, we worked with Bayesian logistic regression on datasets of just dozens of patients. This is far from "big data" territory. With limited samples, we couldn't afford complex models. The rule of thumb was clear: you need at least 10 events per parameter (EVP). If your outcome occurs 20% of the time, that's about 50 data points per parameter.

This constraint forces careful thinking. Each parameter must justify its existence through information criteria like AIC or BIC. Statisticians agonize over whether adding that fourth parameter truly improves the model or just overfits the noise.

The Scaling Paradox

Then came the neural network revolution. Suddenly, models with millions—then billions—of parameters were not just feasible but effective. To statisticians trained on small datasets, this seemed insane. How could a model with a billion parameters possibly generalize when we struggled with four?

Yet it works. Not automatically—developing large models remains partly a black art—but the results speak for themselves. Language models can write poetry, code, and answer complex questions. Image generators create photorealistic scenes from text descriptions.

The Scaling Laws Connection

The key insight is that scaling follows predictable patterns. Various scaling laws suggest how model performance improves with more data and parameters. While "Open" AI no longer shares their training details, other models reveal a fascinating pattern: they're trained using roughly 100 tokens per parameter.

This ratio—100 tokens per parameter—is remarkably similar to the 10 events per parameter rule in logistic regression. But the comparison isn't straightforward. In logistic regression, you might have binary inputs (1 bit each) and 64-bit parameters. With 200 samples of 4 binary variables determining 4 parameters, you get about 20 bits of data per parameter bit.

For LLMs, tokens are typically 32-bit numbers, and parameters might be 32-bit values (quantized to 8 bits for inference). With 100 tokens per parameter, that's 400 bits of training data per inference parameter bit.

The ratio of data bits to parameter bits is roughly similar between logistic regression and LLMs. This similarity is surprising, especially given the vast gulf between models with four parameters and those with billions.

Why Scale Works

The success of large models isn't just about having more parameters—it's about what emerges at scale. New phenomena appear that couldn't be anticipated at smaller scales. The model learns hierarchical representations, discovers patterns too subtle for smaller models, and develops capabilities through emergent behavior.

But scale alone isn't enough. The architecture matters, the training process matters, and the data quality matters enormously. Large models are sensitive to initialization, learning rates, and regularization techniques that smaller models can absorb.

The No Man's Land

There's a curious gap between traditional statistical models and modern AI. Models with hundreds or thousands of parameters occupy a sort of no man's land—too complex for careful statistical analysis, too simple to benefit from scale effects. They're neither fish nor fowl.

The transition from careful parameter selection to massive parameter counts represents a fundamental shift in how we approach modeling. Instead of agonizing over each parameter's inclusion, we accept that most parameters will be useless or even harmful, relying on regularization and massive datasets to sort the signal from the noise.

Implications for the Future

This scaling relationship suggests that the gap between classical statistics and modern AI might not be as wide as it appears. The same principles that guide careful model building—balancing complexity against data, using regularization, validating assumptions—still apply, just at a different scale.

As models continue to grow, we may see new scaling laws emerge. Perhaps the 100 tokens per parameter ratio will evolve, or new architectures will change how we think about the data-parameter relationship. The field is still young enough that fundamental discoveries await.

What's clear is that the journey from logistic regression to large language models isn't just about bigger computers or more data. It's about understanding how learning changes as systems scale, and recognizing that sometimes the most powerful insights come from connecting the old with the new.

The statistician who carefully tunes four parameters and the AI researcher training a billion-parameter model are working on the same fundamental problem: extracting signal from noise. They just operate in different regimes of the same mathematical universe.