Most of the crawls are empty. That's the useful part.

I have 19,245 crawls of OpenRouter in a pile. About three quarters of them contain no selected core change. The interesting number is not 19,245. It's the 22,947 field-level events hiding in the other quarter.

This is the bit of ORCA that looks like overengineering until you live with a catalog that twitches every hour.

Three steps, on purpose

The pipeline is deliberately boring:

1. Crawl — a Convex cron, hourly, fetches the catalog, gzips the raw payload, stamps a crawl_id. Preview deploys do not run this. I learned that the expensive way.
2. Materialize — project the blob into models / endpoints / providers. The UI never reads the archive. The archive is evidence; the views are disposable.
3. Diff — consecutive snapshots through json-diff-ts, atomized into a change history. Monitor and Discord both drink from that.

Retries on the fetch are quadratic, because linear retries just hammer a flapping origin:

Three attempts. Then we record the failure against that model and exclude it from both sides of the next diff. A failed fetch is not a mass deletion. I will die on this hill.

The skip list is the product

If you diff everything, you get a change feed of clocks. This is the actual design work:

Arrays are the other trap. Naive index comparison on supported_parameters produces enormous groups of fake updates the moment someone shuffles the list. The fix is to tell the differ that membership is the identity:

const DIFF_OPTIONS = {
  keysToSkip: [
    "updated_at",
    "unavailable_at",
    "icon_url",
    "stats",
    "status",
  ],
  embeddedObjKeys: {
    input_modalities: "$value",
    output_modalities: "$value",
    supported_parameters: "$value",
    datacenters: "$value",
  },
  treatTypeChangeAsReplace: false,
}

That's it. That's the whole trick. A reordering is not a change. An add or a remove is.

A high-frequency price period is often three to twelve field updates per crawl — mostly endpoint prices — not a new universe. Adjacent crawls are frequently identical. Sparsity is the point of storing diffs at all.¹

What a change looks like

Monitor doesn't show a blob. It shows this:

{
  "path": "pricing.text_input",
  "from": "0.90",
  "to": "0.42"
}

Green down, red up, a percentage if we can make one without lying. Discord gets the same event squeezed into an embed, which is why the keys are cache_read and not text_cache_read. I have opinions about pixels.

GLM 5.2, thirty days, too many providers. The chart is behind a flag. The skip list is not.

The raw bundles remain the source of truth. A changeset is a convenience on top of evidence we already kept. If the differ is wrong, we still have the gzip. That's the contract: capture what we use as raw artifacts, then build projections we are allowed to throw away.

---

1. Reversible changesets — `diff` / `applyChangeset` / `revertChangeset` — are sitting in a notes folder, not in production. The archive does not depend on them. I am allowed to faff. I am not allowed to make the evidence unrecoverable. ↩