how it works

How it works

An agent measures what every operation does and ships summaries. The collector compares each release against the one before it. A model reads the finished candidates and decides which are worth interrupting somebody for.

agent ──► /v1/ingest ──► Postgres ──► detector ──► triage ──► feed
measures    rollups +     sketches +   deploy-     keep/drop   web,
            exemplars     exemplars    anchored    + correlate  MCP

Comparisons are anchored to deploys, not to the clock

“The last thirty minutes against the thirty before” fires constantly, because traffic is seasonal and multimodal and a Tuesday lunch hour does not resemble the hour before it. Comparing release A against release B answers the question somebody actually has — did my change do this? — and sidesteps most of the noise on the way.

This is why SIXTY_RELEASE matters more than any other setting. Without it every measurement lands in one nameless bucket and there is no “before” to compare against. It is picked up automatically on Vercel, Render, Railway, Fly, Heroku and GitHub Actions; anywhere else, set it to the commit SHA.

It is also why the first deploy after an install is quiet. Nothing can be compared until a second release has reported with traffic under it.

Quantiles, not averages

Latency is heavy-tailed and multimodal. A mean does not move enough to catch a real p95 regression, and a standard-deviation threshold fires all night. Every agent ships DDSketch summaries instead: bounded relative error at every quantile, and merging is lossless — so any window you ask for is a union of five-minute buckets with no drift in the numbers.

There are five independent implementations of that sketch — JavaScript, Python, Go, Ruby and PHP — and they write the same bytes. Each is checked against fixtures generated by the JavaScript one, because a bug that is consistent between a writer and a reader in the same language passes every round-trip test there is. A service split across two languages is one history rather than two.

The agent pre-aggregates

It measures every call and ships a summary on an interval, not a record per request. Bandwidth scales with the number of distinct operations rather than with traffic: a service doing 50,000 requests a second across 400 operations sends 400 rows per flush, not 50,000 per second.

A query is credited to the function that issued it

This is the part that turns “the endpoint got slow” into “this function started issuing fourteen queries per call”. Each runtime gets there differently: Node opens a span per exported async function at build time and carries it in AsyncLocalStorage; Python uses contextvars; Go threads an explicit context.Context; Ruby wraps public methods on classes that ask for it.

The failure mode is worth knowing because it is invisible in development. A connection pool that queues a callback runs it later, from the context of whichever other request released a connection — so the query gets credited to a stranger. An idle pool never queues, which is why this only appears once the application is busy: on a real database at concurrency 50, one query per call was attributed correctly 15% of the time. The adapters bind pool callbacks back to the context that created them, which puts it back to 100%.

Statistics detect, a model triages

The detector is tuned for recall and is deliberately a little too sensitive. The model gets finished candidates with the numbers already computed, and decides what is worth interrupting a person for — and correlates, so fourteen drifts that are really one missing index arrive as one thing. Using a model as the detector would cost more and find less.

What it costs the application

An observability agent that harms the application it watches is worse than no agent. That is a measurable claim, so it is measured — and every one of these is reproducible on your own hardware with the benchmark in the package.

The unit that matters is the span rather than the request: an endpoint with one query and three measured functions pays for four of them, against a request that takes milliseconds. What is not free is instrumenting something called hundreds of thousands of times a second — a tight inner loop is the thing to leave alone.

PHP is the outlier and the reason is structural rather than sloppy. A share-nothing worker cannot run a timer, so it has to serialize and hand over its window every single request, where a threaded agent accumulates in memory and serializes once a minute. What the reader waits for is only the measuring — the draining, the merge and the POST all happen after fastcgi_finish_request(), so a collector that is down and takes the full timeout costs them nothing.

It cannot break your application

Every measurement path is wrapped so a failure inside the agent surfaces as nothing at all. A driver’s optional interfaces are mirrored rather than swallowed, which is the difference between a wrapped connection and a broken one: the Go agent implements NamedValueChecker, Hijacker, Flusher and the column-type interfaces because omitting any one of them breaks a real application — WebSockets, server-sent events, an array parameter — and each has a test that fails without it.

When the collector is unreachable the agent keeps measuring into a bounded buffer and drops the oldest window rather than growing. Nothing blocks, nothing retries forever, and the application never finds out. Without a key the agent stays inert and says so once.

What never leaves the process

• No user identity of any kind. No id, no session, no cookie, no IP address. That is a property of the code rather than a setting, which makes “which user saw this” a question this product cannot answer for anybody, including us.
• No values from any query. A statement is normalized to its shape — literals and parameters replaced — before anything is transmitted, and the normalization happens in the process that ran it. MySQL is read by MySQL’s quoting rules rather than Postgres’s, because a double-quoted string is a literal in one and an identifier in the other, and reading one with the other’s rules transmits the literal.
• No raw paths. /users/42/orders becomes /users/:id/orders, and a segment containing an @ is templated whether or not it looks like an id.
• No error text that was not written by a programmer. Messages go through a redactor that keeps identifiers and replaces anything that looks like a value.