AI & Search

AI versus human content across 11 markets: what a controlled experiment actually showed

Most teams settle the AI content question with anecdotes. We ran it as an experiment across eleven European markets and ten content types. The answer was neither camp's.

In early 2025 most SEO teams were having the same conversation: can we use AI to produce content at scale without wrecking our rankings? There were plenty of opinions and very little controlled data. This is an account of getting the data properly, across eleven European markets and more than ten content types, inside a large sports media group. The headline is that neither camp was right. AI content did not uniformly underperform, and it did not uniformly match human writing either. What it did depended on content type and market, which is precisely why a single-site test would have produced the wrong answer, in either direction.

In this article
  1. Why the question needed an experiment, not an opinion
  2. The coordination problem comes first
  3. The measurement framework
  4. What the data showed
  5. What this project was really about
  6. If you want to run your own version

Why the question needed an experiment, not an opinion

The organisation had already started using AI-generated content on a few sites, mainly betting tips in the German and Spanish markets, as a cost measure. The early results were ambiguous in the way early results usually are. Some pages did fine. Some did not. Nobody could say whether the AI content itself was the cause, or whether market and content-type differences were drowning out the signal.

Phase two was a full European expansion: the same question, asked across all major markets, with a measurement framework rigorous enough that the findings could survive a senior stakeholder meeting. My role was the Organic Performance side of that: designing the measurement, building the reporting, coordinating market representatives, and making sure the findings were interpretable by people who would use them to make production decisions across a dozen sites.

The coordination problem comes first

Running an experiment across eleven markets is not primarily a data problem. It is a coordination problem. Each market had its own editorial team, its own content calendar, its own prompt habits and its own relationship with the AI tools. Left alone, “AI content” would have meant eleven different things, and the results would have been incomparable by design.

The structure we built had two roles per market. An AI Ambassador was responsible for producing the AI content and managing the prompt workflows. An Organic Performance representative, from my team, was responsible for tracking and reporting the SEO metrics. The two roles were deliberately held by different people.

Separate the people who generate the data from the people who evaluate it. It is the most basic principle of experimental design, and it is the one most often skipped inside companies.

That separation is not about distrust. It is about incentives. Someone who has spent weeks tuning prompts has a stake in the output looking good, and will read ambiguous data generously without ever intending to. Putting measurement in different hands removes the temptation and, just as importantly, removes the suspicion that it might have been there.

The measurement framework

For each market and content type we tracked a core set of metrics from Search Console and Analytics: clicks, impressions, click-through rate, average position, and on-page engagement (time on page and bounce rate). The comparison was AI-generated pages against a matched set of human-written pages on the same sites: same content type, same publication window, similar keyword intent. Matching on intent mattered more than matching on volume; a betting preview and an evergreen explainer are not comparable even when they target similar traffic.

Design elementChoiceWhy
Unit of comparisonAI page vs matched human pageControls for site, content type, time and intent
Search metricsClicks, impressions, CTR, positionVisibility and appeal, separated
Engagement metricsTime on page, bounce rateWhether the page satisfied the click
ReportingOne standard template per marketDenmark and Italy readable side by side
RolesProducer and evaluator separatedRemoves evaluation bias

The standardised reporting template is the unglamorous part that made aggregation possible. Findings from every market arrived in the same shape, with the same definitions and the same windows, so they could be rolled up for senior stakeholders without a translation layer. If you have ever tried to combine eleven spreadsheets built by eleven people, you know why this deserves its own row in the table.

What the data showed

I cannot publish the per-market numbers, and I would be suspicious of anyone who could publish numbers from a client project like this. What I can share is the shape of the findings, because the shape is the useful part.

  • AI content was not uniformly worse. On some content types in some markets, matched AI and human pages were indistinguishable on search metrics.
  • AI content was not uniformly equal either. On other combinations the gap was large, and it showed up first in engagement rather than in rankings. Pages ranked, got clicked, and then failed to hold the reader.
  • Content type was a stronger predictor than market. Highly templated, data-driven formats behaved differently from formats where the value is judgement or local knowledge.
  • Market still mattered. The same content type did not behave the same way everywhere, which is a warning against exporting a conclusion from one language to another.

The most valuable output was not a headline number. It was a decision framework: given a content type, a market and an intent class, should this be produced with AI, produced with AI and edited, or written by a human? That framework is now used to make production decisions across a dozen sites, and it is a better answer than “AI works” or “AI does not work” because it is the only one that is actually true.

A single-market test would have produced a confident, wrong conclusion. This is the general problem with anecdotal evidence inside organisations: it is not that anecdotes are false, it is that they are true of a sample too small to generalise from, and they are reported by the people most motivated to generalise.

What this project was really about

The broader lesson is about running research inside a large organisation without it collapsing into anecdote. There is a strong pull towards “we tried AI on a few pages and traffic went up, so AI works”, and an equally strong pull towards the opposite. Neither tells you anything you can act on.

What does is a controlled comparison, a consistent measurement framework, a clean separation between the people producing and the people measuring, and enough scale to tell signal from noise. It takes longer to set up than a quick test, and the results are messier than a clean headline. But they are true, and they keep being true after the meeting ends.

If you want to run your own version

  1. Define “AI content” precisely and write it down: which model, which prompt workflow, how much human editing. If that definition varies between teams, you are running several experiments and reporting one.
  2. Match pairs on content type, publication window and intent before you look at any performance data.
  3. Track engagement as well as search metrics. Ranking is not the same as satisfying the reader, and the gap opened in engagement first.
  4. Put measurement in different hands from production.
  5. Report every market in the same template, and resist the urge to summarise before you have the full set.
  6. Deliver a decision framework, not a verdict. The question is never “does AI content work” but “where, for what, and with how much editing”.

Leave a comment

Your email stays private. Thoughtful disagreement welcome.