The let's-just-test-it problem.
“Let’s just test it” is the most common way thinking gets replaced with motion. The companies that test their way to nowhere have one thing in common. No thesis.
Testing without a thesis is data collection. The team launches a variant. The variant wins or loses. The team reports the result. The result rarely generalizes because the test was not designed to test a hypothesis. It was designed to see what would happen.
A test with a thesis looks different. The team starts with a stated belief about how the audience will respond to a particular change and why. The test is structured to either confirm or refute that belief. The result, win or lose, teaches the team something about the audience. The next test builds on the learning.
Tests without theses do not compound. Each test is a one-off. The team accumulates a backlog of variants that beat the control by two percent and learns nothing about why. By year two the win rate flattens because the team has run out of obvious changes and has no model of which non-obvious changes might work.
The cost is real. Testing programs without theses cost a measurable percentage of marketing labor and produce a measurable percentage of compounding learning, where the compounding number is close to zero.
The alternative is not less testing. The alternative is more thinking before the test. A two-page test plan with a stated thesis, a clear success criterion, and a description of what the team will believe differently if the test wins or loses. Two pages. Most tests do not get them. Most tests should.
The other problem with thesis-free testing is that it disguises strategic problems as tactical ones. A team that cannot move the conversion rate has a tactical problem in the team’s view. From outside the team, it often looks like an audience problem or an offer problem, both strategic. The team keeps testing landing pages while the strategic problem stays unaddressed.
Tests are useful.
Thesis-free testing is decoration.
Write the thesis first.