The Rho Illusion: Why UK Product Teams Keep Mistaking Coincidence for Conviction

We named this site after a Greek letter, and not for reasons of branding whimsy. Rho is the symbol statistics uses for a correlation coefficient — a number between minus one and one that describes how snugly two variables travel together. It is a modest thing. It does not know what it is measuring. It does not know whether one thing caused the other, or whether both are being dragged around by a third thing nobody bothered to put on the chart.

Somewhere in the last decade, rho got promoted. It stopped being a diagnostic and became a verdict. Product teams now ship features because a chart went up and to the right, and they kill features because a different chart went down and to the right at roughly the same moment. The letter never asked for this responsibility.

A number that doesn't know what it's measuring

Here's the thing about correlation dashboards: they are extremely good at being right about the past and extremely bad at telling you why. If you plot weekly active users against support ticket volume in any UK fintech, you'll get a beautiful line. It means nothing beyond the fact that growth brings friction. But put it in a Monday review deck next to a screenshot of a new onboarding flow, and someone in the room will start drawing arrows.

This is the trap. Not the maths — the theatre around the maths. The dashboard is a stage set, and the moment you present it in a room with a roadmap in it, everyone starts assigning blame and credit.

Spurious by design: the British edition

Classic spurious correlations are good pub material. Ice cream sales and drowning deaths. Nicolas Cage films and swimming pool accidents. But the product-world versions are subtler and far more expensive, because they arrive dressed as insight.

Consider the onboarding email that “reduced churn”. What actually happened: the campaign went out in the same fortnight that the clocks went back, the evenings got darker, and a whole cohort of users started opening apps indoors instead of in a beer garden. Or: the experiment ran over a payday week, and payday is a genuinely enormous confounder for anything involving consumer spending. Or: the app store pushed a new version, review scores rose, and the support team quietly cleared a backlog that had been poisoning your NPS for months.

The confounding variables in UK consumer products are unusually seasonal and unusually strong. Student term dates, bank holidays, the August lull, Black Friday, the January reset, the weekend the entire country is glued to a tournament. If your test window straddles one of those, your lift is partly weather.

Survivorship bias, or why every scale-up case study sounds identical

There's a second-order version of the problem that rarely gets named: the case study itself. UK tech has a well-stocked shelf of “how we did it” posts from scale-ups that made it. Squads read them, recognise the shape of their own company, and copy the practices.

The problem is that nobody publishes the post-mortem from the company that ran the same playbook and died. The surviving cohort is the entire visible dataset. So when a Series C fintech says its weekly ritual of correlation reviews drove growth, you're hearing from a winner describing a process — not from the hundreds of teams that did the same thing and simply got unlucky, or lucky in a different direction, or were carried by a market that happened to expand underneath them.

Treat every scale-up playbook as a sample of one, drawn from a population whose failures are silent. It's not that the advice is wrong. It's that it's an observation with no control group.

The A/B test that wasn't

Most A/B tests in UK product teams are not really experiments. They're observations with a toggle switch.

A few of the usual failure modes: peeking at the dashboard hourly until a 12% lift appears on 400 users, then declaring victory and closing the ticket. Running a test for four days across a bank holiday. Aggregating results across iOS and Android when the effect is real on one platform and mildly negative on the other — Simpson's paradox, wearing a hoodie. Novelty effects that inflate the first week and evaporate by the third. And the quiet one: a test that was underpowered from the start, then interpreted anyway because the meeting was already in the calendar.

None of these are exotic. All of them produce a rho that looks like a signal.

A correlation hygiene checklist

None of this means you should stop looking at correlations. It means you should treat them as smoke alarms, not verdicts. Here's a working checklist for PMs and analysts.

1. Write the hypothesis before you open the query. If the question arrives after the chart, you're pattern-matching, not testing.
2. Pin down the denominator. Per what? Per user, per session, per paying account, per region? Most “conversion is up” claims die here.
3. Ask what else moved that week. Releases, marketing spend, pricing changes, weather, term dates, a competitor outage.
4. Check the sample honestly. n, statistical power, and duration. Two full weekly cycles minimum for anything consumer-facing; a month if you can stomach it.
5. Segment before you aggregate. Platform, acquisition channel, tenure, region. If the effect flips between segments, you have two stories, not one.
6. Pre-commit your stopping rule. Decide in advance when the test ends. Write it down. Then don't look until then.
7. Run a null test. Split a slice of traffic into two identical arms. If you detect a “lift”, your pipeline is the problem.
8. Look for the mechanism. Why would this cause that? If the best explanation you have is “the numbers moved”, you don't have an explanation.
9. Replay it out of sample. Check the same effect on a period the model never saw.
10. Name the confounder you can't rule out. Say it out loud in the meeting. It builds trust and it stops bad decisions.

Rho points; experiments decide

Correlation is a wonderful starting gun. It tells you where to look, which is more than most dashboards manage. What it can't do is tell you what happened, why it happened, or whether it'll happen again on a Tuesday in February with a different cohort of users.

So let rho do its actual job. Let it raise an eyebrow. Then go and run the experiment, find the mechanism, and check it survives contact with a second dataset. The letter is humble. Keep it that way.