Skip to main content
Local plan state is a cache of what the billing provider says. Every billing incident is therefore the same shape: the cache and the provider disagree, and a customer is on the wrong side of it. Two failures matter and they are not equally urgent. A customer wrongly locked out is losing something they paid for. A customer wrongly served is costing you capacity. Fix the first one first.

​
Symptoms

  • billing.webhooks.failing pages, from a stored billing event that recorded a failure and still waits for a fix.
  • billing.reconcile.discrepancy arrives when events pile up unprocessed.
  • billing.orphan.events arrives at investigate when an event named a subscription this instance cannot place, and settled as orphaned without retrying. It is almost never a customer. It usually means a drill left a subscription behind, a workspace was erased while Stripe still sent its last events, or another stack shares this sandbox and its checkouts arrive here. Find the subscription named in the settled as orphaned log line. If it belongs to a test customer, delete that customer in its sandbox, which cancels the subscription. If another stack shares the sandbox, move that stack to a sandbox of its own, as described in Billing.
  • billing.orphan.subscriptions arrives at investigate when the last nightly reconciliation found Stripe subscriptions that belong to no workspace. The summary lists up to five ids. Treat each like an orphaned event: a leftover test customer is deleted in its sandbox, and a real customer with no workspace is almost always one whose workspace was erased before erasure cancelled billing, so cancel the subscription in the dashboard and refund.
  • asobeast_billing_events_unprocessed climbs and does not drain.
  • A customer reports a 402 after paying, or keeps full access after cancelling.

​
Triage

The commands run on the host and read a personal API token of the platform operator from ASOBEAST_API_TOKEN. Behind the tunnel overlay, which publishes no host port, replace http://127.0.0.1:3001 with your public origin. The POST requests need a write scoped one. See API tokens.
1

Check the webhook secret before anything else

An unverified endpoint refuses every delivery, so a paid subscription never provisions. If STRIPE_WEBHOOK_SECRET is empty or stale, nothing else you do will hold.
2

Read the workspace

storedPlan is what the database holds. plan is what entitlement resolves right now, which is what the customer actually experiences.
3

Compare with the provider

Open the same customer in the provider dashboard. The provider is the source of truth for the subscription; the database is not.
4

Read why an event stopped

An event that stopped retrying is counted by asobeast_billing_events_failed and keeps its failure text on the stored row. The API log names it with will not be retried until an operator replays it. It is almost always a subscription on a price that carries neither a catalog lookup key nor asobeast_plan metadata. Give the price that metadata in the dashboard, or run pnpm --filter api stripe:catalog so the current prices carry their lookup keys.

​
Containment

Never edit a workspace plan directly in the database. It drifts back at the next reconciliation and you lose the record of why it changed.
  1. Reconcile the one workspace, which corrects local state from the provider and records who did it and why:
  2. Reconciliation looks for a subscription on the workspace customer as well as by the stored id, so a customer who paid and whose webhook never landed is repaired by this step rather than needing the id pasted in by hand.
  3. If many workspaces are wrong, fix the webhook path first. Reconciling everyone against a broken cause just delays the same incident.
  4. Once the cause of a stopped event is fixed, replay it. The replay clears how the event settled, queues it again and records who asked and why:
    Replayed events are safe. Webhook handling is idempotent, and an event older than the state the workspace already holds is ignored. An event that names neither the workspace nor its Stripe customer answers 404.
  5. A customer who deleted their account and is still charged had their workspace erased before erasure cancelled billing. Cancel the subscription in the dashboard and refund the charges since the deletion.

​
Recovery

  1. Confirm the daily reconciliation from CRON_BILLING_RECONCILE runs clean, with asobeast_billing_events_unprocessed back at zero.
  2. Confirm asobeast_billing_events_failed is zero. It counts only events still waiting for a fix, and it pages precisely so that a single failure does not sit unnoticed.
  3. Re-register the webhook endpoint with the provider if the secret changed, and verify with one test delivery.

​
Who to tell

  • The affected customer, directly, the same day. Say what happened, that access is restored, and whether they were charged.
  • Every customer, only when checkout itself was down. A lockout affecting a handful of accounts is handled account by account.
  • Nobody, for a payment that failed for the customer’s own reason. Dunning already emails them a portal link.
Last modified on October 6, 2026