pnpm --filter api run bench:target.
What the pipeline costs to orchestrate
This is the code’s own overhead: building the target list, estimating the budget, and enqueueing the whole fan-out with a provider that returns instantly.
Orchestration is not the constraint and never becomes one. Enqueueing 75,100 jobs takes four seconds, against a run that will take days. Budget Redis at roughly 1 KB per queued job.
What the pipeline costs in wall clock
This is arithmetic, not measurement, and it is what you actually wait for. Job count divided by the configured rate limit.
The
target row is the stated benchmark: 20 apps, 200 keywords each, two storefronts, both stores. It finishes inside the night. The stress row does not finish inside a day at all.
Where the day runs out
A run stops finishing inside 24 hours at:
Keywords dominate the count, and a keyword is counted once per market. So the Apple ceiling is roughly 21,600 keyword and market pairs at default limits. Some shapes that reach it:
- 20 apps, 200 keywords each, 5 markets is 20,000 searches, about 22 hours. A sixth market does not fit.
- 50 apps, 200 keywords each, 2 markets is 20,000 searches, the same wall from a different direction.
- 10 apps, 400 keywords each, 5 markets is 20,000 searches again.
The per market multiplier
Adding a storefront for a keyword set you already track doubles the searches for that set. The same phrase in two markets is two keyword rows, checked by two searches, because rankings differ per storefront and one search cannot answer for both. At thetarget scale, dropping from two markets to one halves the App Store projection from 4.5 hours to about 2.3. Adding a third takes it to about 6.7.
This is the single most expensive decision you make, and it is invisible until the run stops finishing. Check it before you add a market, not after.
Check your own numbers
GET /jobs/budget estimates the fan-out from your real data, broken down per store, and the settings page renders the same figures as a budget card. The estimate is verified against the actual fan-out in CI on every pull request, so the number the card shows is the number of jobs the pipeline will enqueue.
utilization is the fraction of a day the run will occupy at your configured limits. Above 1.0 the run cannot finish before the next one starts.
Should you raise the limits?
Usually not.SCRAPE_ITUNES_RPM defaults to 15 against an informal ceiling of roughly 20 requests per minute per IP address. There is headroom, and there is not much of it. Going past what the store tolerates does not produce an error you can plan around: requests start failing, jobs land in the failed set, and in the worst case the store stops answering that IP address for a while. That cost falls on you, not on the project, and the recovery is waiting.
Prefer, in this order:
- Remove keywords you do not act on. Most tracked sets have a long tail nobody reads.
- Remove markets you do not sell in. This is the multiplier.
- Only then consider raising
SCRAPE_ITUNES_RPM, in small steps, watching the failed job count on the health endpoint after each change.
The read path holds up
Query plans were captured at thetarget scale, with 720,000 ranking rows loaded, after ANALYZE:
All three use the existing
("appId", date) index rather than scanning the table, so no index was added. The dashboard stays responsive at the benchmark scale; the collection window is the thing that runs out first.
Related
The daily pipeline and rate limits
Why concurrency is 1 and what the limiter does.
Countries and markets
Why a market multiplies rather than adds.