Structured logs
Logs are JSON on stdout in production and human readable in development, controlled byLOG_LEVEL. Any hosting platform or log service can consume them without extra software.
Every line emitted inside a request or a job carries:
Send
X-Correlation-Id on a request to choose the id yourself, which lets a customer report carry the exact key you need to search on. An id that is missing, too long or carries characters that could forge a log line is replaced with a generated one.
The metrics endpoint
GET /metrics serves the Prometheus text format. It is gated exactly like the queue dashboard: anything other than the entitled platform operator receives 404, never 401 or 403, so the endpoint never confirms it exists. A customer who owns their own workspace is not an operator and receives 404. Reach it through the web origin at /metrics, the same way you reach /admin/queues.
Per workspace, one series per metric:
Instance wide, the pool and the money:
asobeast_pool_endpoints, asobeast_pool_endpoints_healthy, asobeast_pool_success_rate, asobeast_pool_endpoints_blocked, asobeast_pool_endpoints_silent, asobeast_proxy_requests_month, asobeast_residential_spend_usd, asobeast_residential_cap_usd, asobeast_billing_workspaces, asobeast_billing_subscriptions, asobeast_billing_trials_active, asobeast_billing_events_unprocessed, asobeast_billing_events_failed, asobeast_backup_last_completed_seconds, asobeast_backup_max_age_hours, asobeast_database_bytes, asobeast_disk_budget_bytes, asobeast_queue_memory_bytes, asobeast_queue_memory_max_bytes, asobeast_account_mail_attempts, asobeast_store_canary and asobeast_isolation_anomalies.
asobeast_store_canary{store,outcome} carries the parser canary verdict as a label rather than a number, so an alert expression reads outcome="broken" instead of decoding an enum. A store the canary has not answered for yet has no series at all, which is what a fresh instance looks like until the first scheduled run.
The four resource series and the canary are the only ones that can be absent while the instance is healthy. A reading the API could not take is omitted rather than reported as zero, because a zero here reads as an empty database or an idle queue, which is the opposite of what a failed measurement means. One collector failing degrades its own series and nothing else, so a scrape still carries everything the operator needs to see the failure.
Backup freshness is exposed as the completion time rather than as an age, so time() - asobeast_backup_last_completed_seconds graphs it correctly whatever METRICS_CACHE_SECONDS is set to. A cached age would be stale by up to the cache window; a cached timestamp is not.
Cardinality is bounded on purpose. There is one series per workspace per metric, and never one per keyword.
The alert catalogue
Every scrape evaluates the rules below and exposes each firing alert asasobeast_operator_alert{alert,severity} with the value 1. The same alert is written to the log at a matching level, so an instance with no scraper still leaves a trail. Absence of the series means the alert is not firing.
Page immediately
The database being unreachable does not need an alert rule. The scrape itself fails, and so does
/health, which is what an uptime check watches. That check is not part of asobeast and cannot be, because every rule below is evaluated by the API during a scrape of the API. See Uptime probes and the status page for what watches the machine from outside it.
backup.stale is the one rule that watches something outside the API. The backup script writes its completion time to a Redis key when a run finishes, including the offsite upload, and the scrape reads it back. That makes a timer somebody disabled, a disk too full for pg_dump, and a remote that stopped accepting uploads all look the same from here: the age stops moving. It says nothing until BACKUP_MAX_AGE_HOURS declares a window, because an instance that backs up some other way should not be told it is broken, and it stays quiet while dependency.redis.unavailable is firing, because the key it reads lives in the Redis that is down. Set the window to comfortably more than one backup interval, for example 36 for a daily schedule, so a single skipped run is not a page.
Investigate the same day
Review weekly
Error reporting
Both apps report through the official Sentry SDK, each behind its own gate.
When a gate is closed the SDK is never initialized, so nothing is instrumented and nothing is sent.
The API never reports from a self hosted deployment.
BILLING_ENABLED is false there, so ERROR_TRACKING_DSN is ignored even if somebody sets it, and the end to end suite asserts that a deployment with no cloud configuration opens no connection beyond its own database and queue.
The web app has no equivalent third condition, and one thing follows from that. No Compose file sets SENTRY_DSN, so a self hosted deployment reports nothing by default. An operator who sets it anyway is opting their own browsers in, and the reports go to whichever project that dsn names. Nothing reaches the hosted service, because the dsn is never baked into the published image.
The web dsn is read at runtime and handed to the browser by /api/health, the same way STATUS_PAGE_URL reaches it. It is never baked into the image, because this repository publishes one set of images that the hosted service and every self hoster run.
What is reported
Tracing is deliberately off.
GET /metrics already answers the performance questions this instance asks, and spans from a scraping pipeline are high volume for a question nobody is asking.
What a report carries
The error type, a secret scrubbed message, a structured stack with source lines, the workspace and correlation id, and the request method with identifiers replaced by:id.
Breadcrumbs are turned off in both apps. A breadcrumb records the url of every outgoing request, and a store request carries the term a customer tracks in its query string, so the trail that would help debugging is the one place customer data would reach a report.
The same reasoning covers the two places a url reaches an event by another route: the path Next.js hands to onRequestError and the transaction name. Both are masked, so a keyword typed into the spider or the serp filter never leaves the browser’s own machine.
Support tooling
/admin/support answers operational questions about one workspace without browsing what that workspace tracks. It is gated exactly like the queue dashboard and answers 404 to anyone else.
Two rules hold the surface honest:
- Every mutation needs
confirm: trueand a written reason of at least eight characters. Both are stored. - Every access is recorded, reads included, in
SupportAccesswith the operator, the workspace, the action and the reason. That table carries no foreign key, so deleting a workspace or an account leaves the audit trail intact.