apps/api/src/store-providers/ parses store responses, so a break is one module, not the application. Historical data is never lost. Collection pauses, it does not roll back.
This page is what to do between the break and the fix.
Symptoms
In the order they appear:- The store canary reports
broken. A scheduled check asks each store whether it still parses onCRON_STORE_CANARY, four times a day by default, an hour before the daily run. It names the store before any user notices a stale number, andstore.parser.brokenpages on the first failing run. - Rankings stop advancing. The most recent date on the keyword monitor stays fixed while the calendar moves.
- The health badge in the header turns amber and names a failed job count.
- Failed jobs accumulate in the queue dashboard at
/admin/queues. - The data derived alerts stay silent. This is the most misleading symptom, and it is worth stating plainly: those alerts are generated from captured data, so when nothing is captured, nothing alerts. The canary is the exception precisely because it asks the store a question of its own instead of reading what was collected. Silence from the rest is not health.
Triage
Three causes look identical from the outside. This tells them apart.1
Read the canary verdict first
The canary has already classified the failure, so read it before reading the failed set:
A verdict of
unreachable is decided by matching the underlying message against a named list, TRANSPORT_MESSAGES in apps/api/src/store-providers/canary/canary-outcome.ts: ECONNRESET, ECONNREFUSED, ETIMEDOUT, EHOSTUNREACH, ENETUNREACH, EPIPE, EAI_AGAIN, ENOTFOUND, UND_ERR, socket hang up, tunneling socket, timeout, fetch failed, 429 and 503. Anything the list does not name is called broken on purpose, because a failure the code cannot name is likelier a shape it has never seen than a network error, and a false broken is corrected by a human within hours while a false unreachable hides a real outage for days.No series at all means the canary has not run yet, which is what a fresh instance looks like before its first scheduled check.2
Read the failure count
failedJobs counts the App Store, Google Play and alert queues together. A non zero value that keeps climbing across daily runs is the signal. A single failure that clears on retry is not an incident.3
Open a failed job and read its message
Go to The store, the provider method and the original message are all there. Note the method: it names the parser to start from.
/admin/queues, choose the failed set, and read failedReason. Every store failure is wrapped the same way, which makes the message the primary evidence:4
Classify it
The canary answers the same question earlier and without a human reading messages:
unreachable is the first row, broken is the second, and a single failing app leaves the canary on ok because the canary probes its own fixture rather than yours.The distinguishing signal for a parser break is that it is confined to one store and the messages are structural rather than transport. Both stores failing at once is almost always rate limiting or a network problem, because Apple and Google do not change their markup on the same night.What the user sees
A confirmed break raises one banner above every page, for every signed in user, fromGET /jobs/store-health. It names the store, states that the cause is on asobeast rather than on the reader’s configuration, says that stored data is untouched and that collection resumes on its own, and links this page.
Three things about its timing are worth knowing before somebody asks why it did not appear.
- It waits for two consecutive failing canary runs. The user facing threshold is
CANARY_CONFIRMATIONS, so at the default schedule a break is announced within twelve hours rather than instantly. One failing run raisesstore.parser.brokenfor the operator and nothing for the user, which is the whole reason both signals exist. - It clears on its own. The banner polls once a minute, so it disappears within a minute of the canary returning
ok. Nobody has to reload. - It replaces the delayed run notice rather than stacking with it. A broken parser causes a delayed run, and reading the symptom and the cause as two problems is worse than reading one. Only the highest severity notice renders.
unreachable verdict deliberately renders nothing. It means the pool or the network, the delayed run notice already describes what the reader can see, and a second message about a cause nobody can act on is noise.
Containment
What to do immediately, without waiting for a fix.- Failed jobs do not block the queue. The natural assumption is the opposite, so it is worth being explicit: each app, keyword and category is its own job. A job that fails is retried with backoff and then set aside, and the rest of the run continues. If Google Play is broken, the App Store still collects that night.
- Nothing is lost. Every position, review and snapshot already captured stays exactly as it was. A break pauses collection.
- Decide whether to pause the daily run. Leaving it on against a broken parser spends rate limit budget on requests that cannot succeed and fills the failed set. If the break is confirmed and a fix is days away, set
CRON_DAILYto a schedule that will not fire, restart the API, and put it back afterwards. If a fix is hours away, leave it alone. - Clear the failed set once resolved. Retry the failed jobs from
/admin/queuesif the day still matters, or discard them if the next daily run will cover it. Retrying a rank check for a date that has passed captures today’s SERP under today’s date, not the missed one.
Recovery
For whoever ships the fix.1
Reproduce outside the application
Confirm whether the fault is the scraper library or the integration:It is opt in and never runs in CI. See Check the parsers directly.
2
Check upstream
Look for an open issue or a release on the library the failing store uses. An upstream fix is usually a dependency bump rather than a code change here.
3
Change one module
Only the provider directory needs to change. Start from the file for the failing store, and the method the failed job named.
4
Ship it with a regression test
Build the test from the captured raw payload, so it fails for the reported reason before the fix and passes after. Commit as
fix(providers).Reprocess history from stored payloads
Raw scraper responses are stored inraw JSON columns precisely so a corrected parser can be re-run over them. Nobody would guess this is available, so it is worth saying: AppSnapshot.raw holds the full store response for every snapshot ever captured.
That means a parser bug which mapped a field wrongly, rather than one which failed outright, can be corrected retroactively. Re-read the stored payload and rewrite the derived columns. A parser that threw captured nothing, so there is nothing to reprocess for those days.
Backfill
Whether the missed days come back, honestly:
The honest summary is that a gap in ranking history stays a gap. Charts will show it. This is inherent to observing a search result rather than being handed a dataset.
Check the parsers directly
One command that answers “is the parser working right now” without running the pipeline or waiting for cron:apps/api/src/store-providers/canary/canary-checks.ts, so a green run here and a green canary mean the same thing. It is the only thing in the repository that talks to a real store, so it is gated twice: it is not part of any test run, and it exits without doing anything unless SMOKE_PROVIDERS=1 is set. CI never runs it, because a build must not depend on a third party.
Related
The daily pipeline and rate limits
Why both workers run at concurrency 1.
Health checks and monitoring
Where the failed job count comes from.