Article illustration 1

In 1990, psychologist Jacob Cohen lamented a persistent blind spot in research: 'Investigators could not have known how underpowered their research was, as their training had not prepared them to know anything about power.' Despite his efforts to simplify power analysis, the problem endures—especially in tech fields like AI and data science, where underpowered experiments can derail innovation and drain budgets. This isn't just academic nitpicking; it's a crisis of efficiency and credibility that affects everything from A/B testing in software development to clinical trials for medical AI.

What Exactly is Statistical Power?

Statistical power quantifies the probability that an experiment will correctly reject the null hypothesis when it's false. As Carlisle Rainey's primer explains, it answers the question: 'What's the chance my data will detect a real effect?' Power depends on the effect size you're targeting—specifically, the smallest effect of substantive interest (SESOI). For instance, in an A/B test for a new app feature, power determines if you'll spot a meaningful user engagement boost or miss it entirely.

'Power matters because researchers do not want to waste their time, effort, and money,' Rainey notes. A Type II error—failing to reject a false null hypothesis—represents a 'wasted opportunity' after investing in design, data collection, and analysis.

Why Tech Professionals Should Care

For developers and engineers, low power isn't just a statistical faux pas; it's a resource sink. Imagine training a machine learning model for months, only to have an underpowered test overlook its true performance gains. Or consider cybersecurity teams running simulations that miss critical vulnerabilities due to inadequate sample sizes. The fallout extends beyond the lab: readers of research papers often dismiss non-significant results, creating a 'significance filter' that skews published literature toward false positives when power is low. As Rainey argues, 'All relevant information is in the confidence interval,' but power shapes how we interpret 'failed' experiments.

Calculating Power: Practical Rules for Builders

You don't need complex software to estimate power. Rainey offers simple heuristics:
- Aim for 80% power by detecting effects 2.5 times the standard error.
- Achieve 95% power at 3.3 times the standard error.

For a two-group comparison, predict the standard error with:

SE = σ / √n

where σ is the outcome's standard deviation and n is the sample size per group. Use historical data or pilot studies for σ—or a rough estimate like range/4. Regression adjustments can also boost power by reducing standard errors via √(1 - R²), particularly in pre-post designs where adjustments often halve errors.

The Bigger Picture: Power in the Age of AI

Underpowered studies plague tech research, with fields like AI and psychology showing alarmingly low observed power. This leads to 'vicious cycles' where insignificant results are ignored, perpetuating myths and misallocating R&D funds. As Cohen foresaw, the solution starts with education: integrating power analysis into developer workflows ensures experiments are designed to detect meaningful effects. Tools like Bloom's minimum detectable effect framework or Rainey's SESOI approach empower teams to build smarter, not harder.

Embracing power isn't about statistical pedantry—it's about respecting the ingenuity behind every line of code and every dataset. After all, in a world driven by data, the cost of a 'wasted opportunity' is innovation left on the table.

Source: Carlisle Rainey, 'One-Page Primer on Statistical Power' (2025).