Earlier this month one of the providers we track started returning errors instead of answers. Not all of them, and not for long — but long enough that a scan came back with 50 of its 66 queries on that provider missing. The other two providers answered everything.
Here is the uncomfortable part. Nothing about that scan looked broken. It had a score. The score was lower. And a lower score on a chart looks exactly like a brand losing visibility, which is the one interpretation guaranteed to send someone hunting for a problem that doesn’t exist.
That gap — between a number being wrong and a number being plausible — is what this post is about.
TL;DR
- A partial scan says so. Results appear as soon as the scan finishes, marked with how many answers have arrived, so a partial number is never mistaken for a complete one.
- Failures are reported per provider, not per scan. “One provider answered 16 of 66” is diagnostic; “148 of 198” is just alarming.
- Failed queries are re-run automatically for the rest of the scan’s day: the first attempt after ten minutes, then backing off to every two hours, up to eight attempts. Most outages are repaired before anyone opens the dashboard.
- The scan email waits for the final numbers, and a report made while answers are still missing says so.
- Only missing answers are re-run. Never ones that answered unfavourably. This matters more than the rest of the list combined.
- A failed query costs nothing. Credits are spent on answers, not attempts.
- If recovery can’t fix it, you can re-run the failures yourself, or an org admin can remove the scan so it doesn’t sit in your history.
A 21-point drop, and nothing on the chart says it is not real. The obvious next move is to go looking for a content or ranking problem that never happened.
Why a partial scan is worse than a failed one
A scan that fails outright is a nuisance. You see an error, you re-run it, you lose an afternoon. Nobody makes a decision based on it.
A scan that half succeeds is a different animal, because it produces a number, and numbers get acted on. Visibility scores are averages over the answers collected. Take away a third of the answers and the average doesn’t announce that it’s now measuring something else — it just moves. Down, usually, because a brand can’t be mentioned in an answer that was never collected.
So the failure mode isn’t “we lost some data”. It’s “we quietly published a different metric under the same name, on the same axis, next to five points that were measured properly”. Everything downstream inherits it: the trend line, share of voice against competitors, the client report you exported that afternoon.
This is a category-wide problem and mostly an invisible one. Every tool in this space queries providers that fail sometimes; the difference is whether the tool tells you. A single number with no denominator can’t.
What happens now
The scan says what it collected. Status, answers collected against answers requested, and a per-provider breakdown. Two providers at 66 of 66 and one at 16 of 66 is an upstream fault, and it reads as one immediately — where a single aggregate figure would have you checking your prompts.
The provider rejected these requests rather than answering them. Nothing about the prompts changed — the same 66 ran clean on the previous scan, and the other two providers answered all 66 on this one.
Both are available to you, but you rarely need them — automatic recovery has already started.
While a scan is still missing answers, a marker sits beside the scan picker on the dashboard (“190/204 answers · retrying”), with the time of the next attempt. It disappears once the scan is complete.
The failures are re-run without you asking. A transient provider failure is usually repaired before anyone looks. The first attempt comes ten minutes after the scan finishes, and the gap doubles each time (20, 40, 80 minutes) up to every two hours, for up to eight attempts. Only the queries that came back empty are re-run.
The delay is the point. A provider having a bad minute recovers on its own; retrying instantly just collects the same errors again, more expensively.
Recovery used to stop after two attempts. That turned out to be too few: some outages last hours, not minutes, and two attempts in the first half-hour gave up on the day too early. Recovery now keeps going until the end of the scan’s day, because a scan that’s complete by the evening is still a measurement of that day.
One provider stopped answering partway through. 50 of its 66 queries came back as errors, so the scan is marked partial rather than complete. The 50 failures cost nothing: credits are spent on answers, not attempts. The completion email is held until recovery has finished.
The scan email waits for the final numbers. If a scan finishes with answers missing and recovery is going to try again, the completion email is held until the scan is complete or recovery has finished for the day. It’s then sent once, with the final numbers, and if anything is still missing it says how many answers couldn’t be collected.
Reports say what they’re based on. A report, PDF or share link freezes the numbers at the moment it’s made. Create one while a scan is still missing answers and BotScope warns you first; a report made from an incomplete scan carries a line like “Based on 190 of 204 answers”.
You keep the manual controls. Recovery has bounds, and past them it stops. When it does, the failed queries are listed and you can re-run just those — not the whole scan. Asking is never capped, because a person asking has a reason.
And you can delete a scan. Some scans aren’t worth repairing. An org admin can remove one, and it comes out of the charts, the trend line, and any report generated afterwards. We’d rather give you a way to remove a bad measurement than have you explain it to a client every quarter.
The honest limitations
Recovery only re-runs queries that returned nothing. It cannot and does not re-run a query that answered in a way you wouldn’t like. If that distinction sounds like an implementation detail, consider the alternative: a system that retries unfavourable answers is a system that manufactures its own numbers, and every score it has ever produced would be worthless. Missing and unfavourable are different states, and only one of them is eligible.
It costs credits. Recovered answers bill exactly like any other answer — an answer is an answer. Failed attempts cost nothing, and recovery only re-runs the queries that failed, at most eight times and only on the scan’s own day. The worst case is bounded: the answers the scan was supposed to collect in the first place. But it does mean recovery spends money without asking you first, which you should know rather than discover.
There’s no per-project off switch yet. Recovery is on for everyone. If the credit behaviour is a problem for how you work, tell us and we’ll add the setting — it’s a small change and nobody has asked yet.
Some failures are terminal. If a provider is refusing requests for a reason a retry can’t fix, recovery leaves those queries alone rather than collecting the same error eight times. What it no longer does is give up after an attempt that gains nothing: while a provider is struggling, an empty attempt is normal, and stopping there meant abandoning the day after one bad hour.
A recovered answer is collected later than its siblings. Minutes to hours later, which is why recovery stops at the end of the scan’s day (midnight UTC, or three hours after the scan finished if that’s later). Past that, a “repaired” answer would be measuring a different day to the rest of the scan, and we’d rather leave the gap visible than fill it with something that isn’t comparable.
Recovery is not a guarantee. A scan can end up partial anyway. When it does, it stays marked as partial — the mark isn’t removed just because we tried.
FAQs
Do I need to turn this on? No. Automatic recovery is live for every project on every plan.
Will it retry my old partial scans? No. Recovery only works on a scan during its own day, for the comparability reason above. Older partial scans stay as they are, and you can still re-run them manually if you want the answers.
Did my charts have silent holes before this? Partial scans have always been marked partial — that isn’t new. What’s new is the per-provider detail, the automatic repair, and the ability to remove a scan that isn’t worth keeping. Before, a partial scan was something you could see and nothing you could do anything about.
Does a retried scan re-send my completion email? No. Each scan sends one completion email. If recovery is going to run, the email waits until it has finished, so the one you get carries the final numbers rather than the first pass.
Are the recovered answers scored differently? No, but the scoring is incremental: answers that were already scored aren’t re-scored, and recovered ones are added. You don’t get double-counted mentions from a retry.
How do I know whether a scan was repaired? The scan’s diagnostics show the attempts and what each one recovered. If a scan reached full coverage after recovery, it reports as complete, because it is.
Most of the trust in a measurement tool is built in the unglamorous places — what it does when something upstream breaks, and whether it tells you. The providers we track will fail again; that part isn’t up to us. What is up to us is whether a broken scan arrives on your chart looking exactly like bad news.