WI-070: The forge, and a client that cannot be impolite

The catalogue is 333 hand-seeded fixture games. ADR-0016 chose Wikidata (CC0) as the way past that, with Wikipedia (CC BY-SA) for prose and per-field provenance throughout. This is the scaffold that harvesting plugs into.

tools/catalogue-forge is standalone and offline. It never deploys, never runs on a Worker, and nothing in apps/ or packages/ depends on it — it emits seed data behind the existing CatalogueSource seam and stops.

The part that is not scaffolding

ADR-0016 makes scraping conduct binding: honour robots.txt, identify the client, prefer structured endpoints, never work around a block.

Rules like that are followed right up until somebody is debugging at speed and adds a retry. So they are enforced by construction rather than written down and hoped for:

There is no default User-Agent. The constructor throws without one, so there is no code path that is anonymous by omission.
robots.txt is consulted before the request exists. A disallowed path never reaches the network — asking and discarding the answer is still asking.
An unreadable robots.txt fails closed. A 404 means no restrictions; a 500 means we do not know, and not knowing is not permission.
A 401 or 403 is a block. Never retried, and never re-attempted with different headers.
A 429 waits for Retry-After — their number, not ours.
Requests to one host are queued, not timestamped. A timestamp check lets three parallel callers through at once, all seeing the same “last request was long ago”.

In scope

The package, in the gate: lint, typecheck, test.
PoliteClient, and a robots.txt parser good enough to obey.
A CLI that runs — forge help, forge check <url> — because a scaffold that runs is worth more than one that is described.

Definition of done

tools/catalogue-forge builds, lints and tests in the gate.
A client with no User-Agent cannot be constructed.
A disallowed path produces no request.
An unreadable robots.txt denies rather than permits.
A 401/403 is never retried.
A 429 waits the time it was asked for.
Parallel callers to one host still queue.
Each check proven able to fail.
Gate green.

Verification

pnpm --filter @tabletop/catalogue-forge test
pnpm --filter @tabletop/catalogue-forge forge help
pnpm gate

Seeded failures

Seed	Bit
A default User-Agent, so anonymity is one omission away	1
Retry a 403	2
An unreadable `robots.txt` means yes	1
Fetch first, honour `robots.txt` afterwards	1
Rate limit by timestamp instead of a queue	1
Ignore `Retry-After`	1
An empty `Disallow` blocks everything	2

A seed that proved nothing

The first attempt at the robots.txt ordering seed moved the check into the request loop — but before the fetch call, so the request still never happened and the test still passed. It demonstrated that a throw had moved, not that the check works.

Redone so the fetch genuinely happens first, it bites. Worth recording, because a seed that does not reproduce the defect is indistinguishable from a check that works.

Not done here

Harvesting. Wikidata is WI-071, Wikipedia and attribution WI-072, entity resolution and export WI-074. AI normalisation (WI-073) is blocked on WI-H05 — spend is never an agent’s call.

There is no cache layer yet. It belongs with the first real source, where the shape of what is worth caching is known rather than guessed.