Skip to main content
The self hosted version of this incident is covered in Recover from a broken store parser. The triage and the fix are identical. What changes on the hosted service is the blast radius: every customer stops collecting at the same moment, and they find out from their own dashboards before they hear from you. On the hosted service, communication is the first step, not the last.

Symptoms

Everything in the self hosted runbook, plus:
  • asobeast_workspace_rankings_captured falls to near zero across every workspace that tracks the affected store, while the other store keeps working.
  • daily.run.incomplete pages for many workspaces at once rather than one.
  • Error reports arrive tagged with many different workspaces and one repeated error type.
One workspace failing is not this incident. Every workspace on one store failing is.

Triage

1

Confirm the blast radius before anything else

Count the workspaces reporting daily.run.incomplete. Many workspaces on one store is a parser break. Many workspaces on both stores is the pool, not the parser.
2

Read one failed job

Open /admin/queues, pick the failed set for that store, and read failedReason. It names the store, the provider method and the original message, which is where the fix starts.
3

Reproduce outside the queue

Run the provider smoke check against the affected store. A parse failure reproduces immediately and consistently. A rate limit does not.

Containment

  1. Announce it to every opted in installation first. Edit docs/status.json and push. The docs deployment publishes it at https://docs.asobeast.com/status.json, and every installation that set STORE_STATUS_URL picks it up within the hour, on CRON_STORE_STATUS. Set the affected store to broken, give it a since and a summary a person can read, and leave the healthy store on ok:
    Keep summary under 280 characters, because installations truncate it there. Do not add a url or a version field: the schema carries neither on purpose, and an installation that receives one ignores it. Setting the store back to ok when the fix ships clears the announcement within one poll interval, so remember to make that second edit. Confirm the document is actually served before relying on it, and again after any change to the docs deployment: curl https://docs.asobeast.com/status.json must return the file rather than a 404, because an installation that receives one changes nothing and says nothing.
  2. Tell customers within the hour. Post the incident, and say which store is affected and that stored data is untouched.
  3. Pause that store’s queue. Failed jobs retry with backoff and add load to a pool that cannot use it.
  4. Leave the other store running. Half a product is much better than none, and the stores fail independently by design.
  5. Do not roll back the database. Collection pauses; it never rewrites history.

Recovery

  1. Fix the parser inside apps/api/src/store-providers/. No other module parses store responses, so the change is contained there by design.
  2. Add a regression test built from the response shape that broke, taken from the stored raw payload.
  3. Deploy, unpause the queue, and confirm one workspace captures cleanly before requeueing the rest.
  4. Backfill nothing automatically. Ranking history is a daily series and a synthetic fill would be a lie. Leave the gap and say it is a gap.

Who to tell

  • Customers, first and in public. Name the store, state that stored data is safe, and give the next update time rather than an estimate you cannot keep.
  • Customers again when collection resumes, naming the days that will stay empty.
  • Self hosted users, through the release notes, since the same fix reaches them as an upgrade.
Last modified on August 29, 2026