Skip to main content
Every hosted store request leaves through the egress pool. When the pool degrades, collection slows for every customer at once, and the worst version of the failure is the quiet one: endpoints that answer with plausible but wrong results.

​
Symptoms

  • pool.healthy.low pages, or pool.blocked.rising and pool.silent.rising arrive for the same day.
  • asobeast_pool_endpoints_healthy falls while asobeast_pool_endpoints holds steady.
  • asobeast_workspace_rankings_unresolved climbs across many workspaces at once, which is what a silent failure looks like from the customer side.
  • Worker concurrency drops on its own, because it tracks the healthy pool size.

​
Triage

1

Read the pool

GET /admin/proxy-pool reports per store health, per endpoint outcomes and residential spend. The alerts array repeats what fired.
2

Classify the outcome that dominates

3

Separate one store from both

Blocking on one store with the other healthy is a store decision about the range. Both stores failing at once is almost always the provider.

​
Containment

A silent failure writes wrong data. Contain it before you fix it, because every hour it runs adds ranking rows a customer will act on.
  1. Silent failures: pause the affected store’s queue from /admin/queues. No capture is better than a wrong capture.
  2. Blocking: leave the pool to cool down. Blocked endpoints are already parked, and forcing them back accelerates the block.
  3. Rate limiting: reduce the store’s requests per minute and let the current run finish late rather than fail.
  4. Do not raise the residential cap to ride out a datacenter block. Residential egress is metered and priced per gigabyte, and a mixed customer costs several times their subscription there.

​
Recovery

  1. Buy more datacenter endpoints from the provider and let CRON_PROXY_SYNC add them, or run the sync early.
  2. Confirm asobeast_pool_endpoints_healthy recovers above half of asobeast_pool_endpoints for both stores.
  3. If a store blocks the whole range, the options in order of preference are: a different datacenter range, reduced Google Play depth, and residential egress for a small share of traffic with PROXY_RESIDENTIAL_MONTHLY_CAP_USD set.
  4. Delete ranking rows captured during a confirmed silent window. They are worse than a gap.
  5. Unpause the queue and watch one full daily cycle.

​
Who to tell

  • Customers, if collection paused for longer than one daily cycle. Say which store is delayed. A delay that is explained reads as a delay; silence reads as broken data.
  • The proxy provider, with endpoint addresses and outcome counts, for anything that looks like a transport fault.
  • Nobody outside, for a block. It is a normal cost of the business and does not affect customer data.
Last modified on August 24, 2026