In short
- 31 and 14 and 0
- Three figures on one page of our own report, all three correct, unreadable together.
- Not a calculation error
- This is the annoying kind of error: nothing breaks, no test fails, and the figure is defensible. Only the reader cannot make sense of it.
- One formula
- The fix: the headline score is calculated in one place, and the report and the dashboard both read that one. Two places drift apart sooner or later.
- The depth stayed
- The deep analysis on one engine was not scrapped but renamed to what it is. A measurement you cannot explain should not be removed but put in its place.
What does the study say?
This is not somebody else's publication but a case of our own, and it is here because it answers the question every buyer is really asking: what happens when you are wrong? The report showed 31 out of 100 at the top and, in the table below, 14 out of 100 and 0 out of 100. Nothing had been miscalculated. At the top was one engine that had received all the questions; below were seven engines from a different phase with a different set.
What makes this kind of error dangerous is that nothing breaks. No test fails, no alert goes off, and each figure on its own is defensible. The only one who notices is the reader, and they do not conclude that there are two measurements but that the report is wrong — and therefore that the rest is wrong too. An unexplainable figure costs more trust than a low one.
| What was measured | How | What the study does not say |
|---|---|---|
| Our own report, before the repair | one score at the top from the analysis phase on one engine, and a table below from the engine phase with a different question set | that either figure was miscalculated. Both were correct within their own measurement, and that is exactly why no check ever fired |
| What came of it | one place where the headline score is calculated, read by both the report and the dashboard, plus three rules that belong in every report: who counts, who is weighted, and how much of the market is covered | that this kind of error is now impossible. It only says that this one cannot happen again, and that is all a repair can ever say |
Our own case. The three figures and the reason sit in the comment above the function that calculates the headline score — that comment was written at the time of the repair and is its justification.
Why does this matter to you?
Because it can happen at any supplier and you cannot see it from the outside. So do not ask whether the figure is correct but how it is built: which engines, the same questions, the same method? The moment the answer is that the headline score comes from a different measurement than the table below it, there are two things on one page that do not belong next to each other.
And because the urge to fix something like this quietly is strong. Adjusting the figure until the two match is half a day's work and nobody would have noticed. That is exactly why this story sits here rather than in an internal document: a mistake you publish yourself can no longer be used against you, and it is the only credible way to say that you measure honestly.
What does ceeme do with it?
Since then the headline score is calculated in one place, and the report and the dashboard both read that one. It is the weighted average across the engines measured with the same questions and the same method, weighted by their share in the client's market. Two places computing the same number drift apart sooner or later — that is not pessimism but what happened.
Three rules belong in every report since then, and they are there. Who counts: only engines that actually searched live, because an engine answering from memory measures model knowledge and not visibility. Who is weighted: only engines with a known share in that market; the rest appear as measured-but-not-weighted. And how much of the market is covered — that figure has its own study next to this one.
And the deep analysis has not gone. One engine receiving up to two hundred questions, with brands, sources, sentiment and markets, remains the richest picture there is. It is simply no longer called the score but what it is: the deep analysis on one engine. That is the cheapest repair there is — a measurement you cannot explain should not be removed but put in its place, under the right name.
Frequently asked questions
Why put a mistake of your own on the website?
Because otherwise the promise is worth nothing. We ask clients to believe us when we say a fix did not work, and you only believe that from someone who says it about themselves too. This is the case where it was about us.
How do I know there are no more errors like this?
You do not, and neither do we. What you can check is whether a supplier publishes their method and writes down their own corrections — because an error recorded nowhere is an error that repeats. With us, every figure in the report comes with the engines, the questions and the date, so you can lay a second measurement beside it.
Is the deep analysis worth less, then?
No, it is the richest picture we have — it simply is not a visibility score. One engine receiving up to two hundred questions reveals brands, sources, sentiment and markets you do not see with four engines and thirty questions. The problem was never the measurement but the name above it.
Read on
Free, no account and no card.