Dean.Taxi Blog
/find
Most of the crawls are empty. That's the useful part.

ORCA  · 504 words · 3 min read · by

Most of the crawls are empty. That's the useful part.

I have 19,245 crawls of OpenRouter in a pile. About three quarters of them contain no selected core change. The interesting number is not 19,245. It's the 22,947 field-level events hiding in the other quarter.

This is the bit of ORCA that looks like overengineering until you live with a catalog that twitches every hour.

Three steps, on purpose

The pipeline is deliberately boring:

  1. Crawl — a Convex cron, hourly, fetches the catalog, gzips the raw payload, stamps a crawl_id. Preview deploys do not run this. I learned that the expensive way.
  2. Materialize — project the blob into models / endpoints / providers. The UI never reads the archive. The archive is evidence; the views are disposable.
  3. Diff — consecutive snapshots through json-diff-ts, atomized into a change history. Monitor and Discord both drink from that.

Note

The snapshot code is marked legacy in its own AGENTS.md. That's not a confession, it's a plan. The raw bundles stay. The projections get to be rebuilt.

Retries on the fetch are quadratic, because linear retries just hammer a flapping origin:

delay=attempt2×1000ms

Three attempts. Then we record the failure against that model and exclude it from both sides of the next diff. A failed fetch is not a mass deletion. I will die on this hill.

The skip list is the product

If you diff everything, you get a change feed of clocks. This is the actual design work:

Field Why we skip it
updated_at, unavailable_at A clock is not news.
stats, status Latency jitter every hour. Nobody wants that pinged to Discord.
icon_url CDN churn. The whale does not need a new favicon event.
denormalised names on the endpoint The model didn't change. We copied the label again.

Arrays are the other trap. Naive index comparison on supported_parameters produces enormous groups of fake updates the moment someone shuffles the list. The fix is to tell the differ that membership is the identity:

const DIFF_OPTIONS = {
  keysToSkip: [
    "updated_at",
    "unavailable_at",
    "icon_url",
    "stats",
    "status",
  ],
  embeddedObjKeys: {
    input_modalities: "$value",
    output_modalities: "$value",
    supported_parameters: "$value",
    datacenters: "$value",
  },
  treatTypeChangeAsReplace: false,
}

That's it. That's the whole trick. A reordering is not a change. An add or a remove is.

A high-frequency price period is often three to twelve field updates per crawl — mostly endpoint prices — not a new universe. Adjacent crawls are frequently identical. Sparsity is the point of storing diffs at all.1

What a change looks like

Monitor doesn't show a blob. It shows this:

{
  "path": "pricing.text_input",
  "from": "0.90",
  "to": "0.42"
}

Green down, red up, a percentage if we can make one without lying. Discord gets the same event squeezed into an embed, which is why the keys are cache_read and not text_cache_read. I have opinions about pixels.

Pricing history for GLM 5.2 — thirty providers, none of them agreeing
Pricing history for GLM 5.2 — thirty providers, none of them agreeing

GLM 5.2, thirty days, too many providers. The chart is behind a flag. The skip list is not.

The raw bundles remain the source of truth. A changeset is a convenience on top of evidence we already kept. If the differ is wrong, we still have the gzip. That's the contract: capture what we use as raw artifacts, then build projections we are allowed to throw away.

  1. Reversible changesets — `diff` / `applyChangeset` / `revertChangeset` — are sitting in a notes folder, not in production. The archive does not depend on them. I am allowed to faff. I am not allowed to make the evidence unrecoverable. ↩

Related posts