Most Landing Page A/B Tests Never Reach Significance. Test Angles, Not Button Colors.

Every A/B testing guide opens the same way. Pick one element. Change the headline. Swap the hero image. Run the green button against the blue one. Wait for significance, keep the winner, compound the gains forever.

It feels like science. At paid-traffic volumes, it is astrology with a dashboard.

The method is not the problem. The arithmetic is. And almost nobody does the arithmetic before they pick the test.

The advice was written for someone else's traffic

The experimentation canon comes from companies whose smallest page gets more sessions in an afternoon than your entire funnel gets in a quarter. At that scale, a two-percent relative lift is detectable in a couple of weeks and worth real money. So the playbook says: change one small thing, isolate it, measure it precisely.

The blog posts that playbook spawned kept everything except the traffic assumption. Your landing page for one ad group gets a few hundred visits a month. Split that across two variants and each arm sees a number that would be a rounding error at the companies the advice was copied from. The advice did not fail you. It was never addressed to you.

Do the arithmetic before the test

Take the standard setup: 95% confidence, 80% power, two variants. The per-variant sample size for comparing two conversion rates lands roughly at n ≈ 16 · p(1−p) / δ², where p is your conversion rate and δ is the lift you want to detect. Assume a page converting at 4% and look at what different test ideas actually cost in traffic:

What you're testing	Lift you need to detect	Visits per variant
Button color, tighter subhead (10% relative)	4.0% → 4.4%	~40,000
Stronger offer framing (25% relative)	4.0% → 5.0%	~7,000
A different angle entirely (50% relative)	4.0% → 6.0%	~1,900
A different offer (100% relative)	4.0% → 8.0%	~600

The row that matters is the first one. A button-color test is a bet on a single-digit relative lift — call it ten percent, generously. At a 4% baseline that is a forty-thousand-visit-per-variant bet. If the ad group sends 500 visits a month, you need over a decade.

Nobody runs a test for a decade. They run it for three weeks, watch the lines cross twice, and ship whichever version was ahead when the meeting happened.

You can check your own numbers in about ten seconds:

import math

def per_variant_n(baseline, target):
    # two-proportion test, 95% confidence, 80% power
    p = (baseline + target) / 2
    d = target - baseline
    return math.ceil(2 * (1.96 + 0.84) ** 2 * p * (1 - p) / d**2)

print(per_variant_n(0.04, 0.044))  # tweak: ~39,000 per variant
print(per_variant_n(0.04, 0.060))  # swing: ~1,900 per variant

Any sample-size calculator will land in the same ballpark. Or paste your baseline and monthly traffic into Claude and ask it directly. The tool doesn't matter. Doing it before the test does.

What an underpowered test actually tells you

Nothing — but it says it confidently.

When a test with 300 visits per variant "reaches significance," what usually happened is that someone checked the dashboard daily and stopped the moment the lines crossed. The measured lift on that winner is inflated, sometimes wildly. You ship it, it regresses to the old number, and a year of disciplined-looking testing produces no cumulative gain at all.

The deeper cost is attention. A micro-testing program occupies the page, the roadmap, and the team's belief that they are "data-driven" — while the big questions about the page never get asked, because the button-color program is sitting where a real test would go.

Work backwards from what you can detect

Flip the order of operations. Don't ask "what should we test?" Ask "what lift could our traffic even detect this quarter?"

That number is your minimum detectable effect. If it comes back at 40% relative — and at paid-traffic volumes it usually does — then the test selection writes itself. You can only afford to test things that could plausibly move conversion by 40% or more.

What clears the bar

• The offer. Demo vs. trial vs. pricing vs. lead magnet. A different ask is the biggest lever on the page.
• The angle. Same product, different argument: pain-led vs. outcome-led vs. proof-led. Your search terms report is the best source of angles — the queries people actually typed before they clicked, as covered in Stop Generating Blog Posts From Keyword Lists. Your Search Terms Report Is a Better Editorial Calendar..
• The match. If the ad promises one thing and the page opens on another, fixing that gap is routinely a large swing, not a marginal one. That is the whole argument of 40 Ad Groups, 3 Landing Pages: A Claude + MCP Pipeline for Real Message Match.

What never clears the bar

Button color. Hero stock photo. Font pairing. "Sign up" vs. "Get started." These are sub-5%-relative bets on a good day, which makes them untestable at your volume. Not unimportant — untestable. There is a difference, and the difference is the whole post.

Big swings used to be the expensive option

The honest reason everyone tested button colors is that it was the only affordable test. A genuinely different page — new argument, new structure, new proof — used to cost a copywriter a week or more. Micro-tests were a rational response to expensive variants: you couldn't afford three different pages, so you ran three shades of one page and called it a program. The traffic math was broken then too, but the alternative was worse.

That constraint is gone. The same pipeline that keeps message match across forty ad groups — ad copy and intent in, brand-locked page out — also means your variant B can be a different bet instead of a recolor. Generate the pain-led page and the proof-led page, publish both with the setup described in The Publish Button Your AI Was Missing, and let the traffic judge a question that was actually worth asking.

A testing calendar sized for paid traffic

1. Compute the MDE first. Baseline, monthly visits, split. If the detectable lift is huge — and it is — write it at the top of the doc so nobody proposes a headline test next sprint.
2. Two variants, never six. A thousand monthly visits split six ways is 167 per arm. That is astrology again, just with more horoscopes.
3. One bet per test. Variant B is one different idea, not seven tweaks at once. You won't know which tweak did it — but at this volume you were never going to run the follow-up isolation tests anyway. What you need is the direction.
4. Pre-commit the sample size. Write the stopping number before launch. No peeking. And no sequential-testing method rescues 300 visits per arm against a 10% lift — the math is the math.
5. Test per intent cluster, not blended traffic. Blending ad groups averages away the effect you're measuring. The intent clusters from your search terms report are the unit to test on.
6. Keep the losers. A decisive loss deletes an angle from your space permanently. Noise deletes nothing. Fewer, bigger tests means the losses are information too.

The constraint moved

The scarce resource in landing-page testing was never statistical sophistication. It was page production cost — and while pages were expensive, you couldn't see that the real budget was traffic all along.

Pages are cheap now. Traffic is not. Spend yours on questions big enough to answer.

The only test worth running at your volume is the one that would embarrass you if you hadn't run it. The button color was never that test. The angle probably is.