pnpm --filter api run bench:target.
What the pipeline costs to orchestrate
This is the code’s own overhead: building the target list, estimating the budget, and enqueueing the whole fan-out with a provider that returns instantly.
Orchestration is not the constraint and never becomes one. Enqueueing 75,100 jobs takes four seconds, against a run that will take days. Budget Redis at roughly 1 KB per queued job.
What the pipeline costs in wall clock
These are measurements, taken against the memory ceilings the stack ships rather than against an unconstrained runner. The full record, including how to reproduce them, is inapps/api/test/load/MEASUREMENTS.md.
Both columns count requests, which is what the store sees and what risks throttling. A depth 200 Google Play keyword search costs about eight requests where the Apple equivalent costs one, so a workload split evenly between the stores spends roughly eight times as long on Play as on Apple for the same number of keywords.
The
target row does not finish inside a day. Apple’s share does, comfortably; Play’s does not, by a factor of two. That is the measured answer to whether one host carries 20 apps at 200 keywords across two storefronts and two stores: it does not, and the reason is SCRAPE_GPLAY_RPM, not the machine.
Orchestration is not what you wait for. Building the daily job set at target scale costs 110 ms to price the day and 443 ms to enqueue it, against a window measured in hours.
What the guard does when the day is too small
The daily run does not simply overrun. At 222% pressure it sheds review syncing, and past 150% it sheds app refreshes as well, keeping the keyword captures that are the product. Both were observed in the measurement runs. See The daily pipeline and rate limits.Where the day runs out
A run stops finishing inside 24 hours at:
Keywords dominate the count, and a keyword is counted once per market. What that ceiling buys depends entirely on which store the keywords are on.
Some Apple only shapes that reach the ceiling:
- 20 apps, 200 keywords each, 5 markets is 20,000 searches, about 22 hours. A sixth market does not fit.
- 50 apps, 200 keywords each, 2 markets is 20,000 searches, the same wall from a different direction.
- 10 apps, 400 keywords each, 5 markets is 20,000 searches again.
The per market multiplier
Adding a storefront for a keyword set you already track doubles the searches for that set. The same phrase in two markets is two keyword rows, checked by two searches, because rankings differ per storefront and one search cannot answer for both. At thetarget scale, dropping from two markets to one halves the App Store projection from 4.5 hours to about 2.3. Adding a third takes it to about 6.7.
This is the single most expensive decision you make, and it is invisible until the run stops finishing. Check it before you add a market, not after.
Check your own numbers
GET /jobs/budget estimates the fan-out from your real data, broken down per store, and the settings page renders the same figures as a budget card. The estimate is verified against the actual fan-out in CI on every pull request, so the number the card shows is the number of jobs the pipeline will enqueue.
utilization is the fraction of a day the run will occupy at your configured limits. Above 1.0 the run cannot finish before the next one starts.
completion answers the question a customer actually asks: startsAt is the next scheduled run, hours is how long the work takes at the current capacity, and completesAt is when the run should finish. When an egress proxy pool is configured the capacity behind those numbers is the healthy endpoint count rather than the single host budget, so the projection degrades honestly as endpoints cool down.
quota is present only when BILLING_ENABLED is true. It reports the plan, the apps and keyword markets used against their limits, and overLimitSince once a workspace has been over its keyword limit long enough for daily checks to cover only the first N keyword markets in a stable order.
Operators get the cross-tenant view at GET /api/backend/admin/capacity, which sums demand across every workspace the daily run visits and names the ones consuming most. It is reserved for the platform operator and answers 404 to anyone else, including a customer who owns their own workspace.
Under pressure
When the previous run’s backlog plus today’s demand passes the daily capacity, the run sheds work in a stated order rather than overlapping itself. Category ranks and review sync go first, app refreshes next, and rank checks are never dropped because they are the product. The decision is logged at error level with the pressure figure and what was shed. On demand actions are limited per workspace so one frustrated customer cannot spend the pool. Manual refresh, run daily and keyword scoring are capped per day; suggestion lookups are capped per hour. Over the limit the API answers429 with retryAfterSeconds in the error envelope. Nothing is limited while BILLING_ENABLED is false, so a self hosted instance behaves as before.
What each container is allowed on the host
Everything above bounds how much work the day can do. This bounds what one container can take from the machine while it does it, because on a single host PostgreSQL, Redis, the API and the web app are neighbours rather than isolated tenants. Compose sets a memory ceiling per service and rotates every container log.
The ceilings sum to roughly 1.8 GB, which is why System requirements asks for about 2 GB. The at rest figures come from an idle stack with a small portfolio.
Under load the picture is tighter, and it was measured rather than assumed.
Nothing was OOM killed at any scale, and the largest workload leaves almost no headroom in either service.
storage.headroom.low and queue.memory.high alert at 80 percent and page at 90, so both would have fired on the way to that last row rather than after it.
Why Redis refuses instead of evicting
Redis runs withmaxmemory 192mb and maxmemory-policy noeviction, well under its container ceiling.
BullMQ job state is not a cache. Under an eviction policy Redis would silently discard queued work to make room, and the run would simply not happen with nothing to show for it. Under noeviction a write past the ceiling fails with OOM command not allowed when used memory > 'maxmemory', the API answers 500 on the affected request, and previously queued state stays readable. Loud and recoverable beats quiet and lost.
Do not change this to allkeys-lru for convenience.
Every container log is rotated
Each service writes to thejson-file driver capped at 10 MB across 3 files, so no container can grow past 30 MB of logs and the four together cannot pass 120 MB.
The Compose stack also lowers LOG_LEVEL to log. At debug the API records every request, which on a busy pipeline day fills the rotation window fast enough that yesterday’s evidence is gone before you go looking for it.
The ceilings are watched
Every ceiling above used to be detected only by the symptom it produced, which on a single host means the service is already stopping. The API now measures what it can reach and alerts before that.
The API container cannot see the host filesystem, so it does not pretend to measure free space. It measures the database, which is the thing that grows, and compares it against
DISK_BUDGET_BYTES, which you set to what the partition can spare. Set it and you get a true headroom alert; leave it at 0 and you get no alert rather than a false one.
Redis reports its own ceiling, so queue.memory.high needs no configuration. An unlimited Redis reports maxmemory as 0 and is left alone rather than compared against nothing.
A reading that cannot be taken is left out of the scrape instead of being reported as zero, and one failing collector never takes the rest of /metrics down with it. See Hosted observability.
Which symptom points at which limit
Should you raise the limits?
Usually not.SCRAPE_ITUNES_RPM defaults to 15 against an informal ceiling of roughly 20 requests per minute per IP address. There is headroom, and there is not much of it. Going past what the store tolerates does not produce an error you can plan around: requests start failing, jobs land in the failed set, and in the worst case the store stops answering that IP address for a while. That cost falls on you, not on the project, and the recovery is waiting.
Prefer, in this order:
- Remove keywords you do not act on. Most tracked sets have a long tail nobody reads.
- Remove markets you do not sell in. This is the multiplier.
- Only then consider raising
SCRAPE_ITUNES_RPM, in small steps, watching the failed job count on the health endpoint after each change.
The read path holds up
Query plans were captured at thetarget scale, with 720,000 ranking rows loaded, after ANALYZE:
All three use the existing
("appId", date) index rather than scanning the table, so no index was added. The dashboard stays responsive at the benchmark scale; the collection window is the thing that runs out first.
Related
The daily pipeline and rate limits
Why concurrency is 1 and what the limiter does.
Countries and markets
Why a market multiplies rather than adds.