ADR-0017: AI-assisted normalisation with confidence gating

Status: Accepted
Date: 2026-07-30
Supersedes: —
Superseded by: —

Context

ADR-0016 establishes Wikidata and Wikipedia as the catalogue sources. Between them they give reliable identity data — title, designers, publishers, year.

They do not give the fields discovery actually runs on. ADR-0006 needs mechanics, weight, and player count; ADR-0003 chose Postgres specifically for array columns of mechanics and tags. Wikidata’s genre property is inconsistent and coarse. Wikipedia has the information but as prose.

There is also an entity-resolution problem that no structured source solves: the same game appears across sources under different titles, editions, and translations.

Decision

Use Claude in tools/catalogue-forge for three jobs, as an offline batch stage — never in the request path:

Taxonomy normalisation — map messy source genres and categories onto a controlled vocabulary of mechanics and themes we own.
Field extraction — pull player count, playing time, and a weight estimate out of Wikipedia prose.
Entity resolution — decide whether two records describe the same game.

Configuration:

Concern	Choice
Model	`claude-opus-5` (1M context, $5/$25 per MTok)
Bulk runs	Message Batches API — 50% cost reduction, async, tolerates the latency
Output shape	Structured outputs — `output_config.format` with a JSON Schema derived from the Zod schemas in `packages/shared`
Prompt caching	Controlled vocabulary and instructions cached as a stable prefix; per-game prose after the breakpoint
Refusal handling	`fallbacks: "default"` with beta `server-side-fallback-2026-07-01`

The gate — this is the substance of the decision:

Every AI-derived field is written with source = 'ai', the model ID, and a confidence score, into the same provenance table as everything else.
AI output never outranks a structured source. If Wikidata has the field, Wikidata wins.
Records below the confidence threshold go to a review queue, not into the catalogue.
Entity-resolution merges are always human-confirmed. A wrong merge destroys two games’ data and is painful to unpick.

Rationale

The AI is doing the one job the structured sources genuinely can’t: turning prose into the typed, enumerated fields the discovery feature queries. That is a narrow, well-specified extraction task, and it is exactly what structured outputs are for — deriving the JSON Schema from the same Zod schemas the API validates against means the model physically cannot emit a shape the database will reject.

claude-opus-5 is the default. Entity resolution in particular is a judgement task where a wrong call is expensive to reverse, and the batch discount plus prompt caching already brings the cost of the bulk stages down substantially. Model tier and effort level are a human’s cost/quality decision, not an agent’s — see the spend gate below.

Batching fits the workload precisely: this is a one-off backfill plus occasional incremental runs, with no latency requirement whatsoever. Paying interactive prices for it would be waste.

The confidence gate exists because the failure mode here is specific and nasty: a plausible, confidently-wrong mechanic tag is worse than a missing one. A missing tag makes a game harder to discover; a wrong tag actively misleads, and it looks identical to a correct one in the UI. Provenance plus a threshold plus a review queue is what keeps that from silently polluting the catalogue.

Consequences

This is the first running cost beyond Apple’s $99. The £0 constraint in docs/architecture.md §1 covers the deployed system; it does not cover a one-off enrichment batch. Every enrichment run is gated on WI-H05 — a human approves the spend and the volume. Agents never trigger a paid batch run.
Cost is bounded and predictable: it scales with catalogue size, not with traffic, and it is paid once per game rather than per request.
packages/shared gains a CatalogueEnrichment schema. The JSON Schema handed to the model is derived from it, so a schema change automatically constrains the model — the same no-duplicate-types rule as everywhere else.
The review queue is a real surface someone has to work through, not a theoretical safeguard. Budget for it.
AI-derived fields must be visibly distinguishable in any admin view. A reviewer needs to know what they’re checking.
Re-running enrichment after a prompt or model change must be idempotent — keyed on game plus model plus prompt version, so a re-run doesn’t duplicate provenance rows.

Alternatives considered

No AI — hand-curate mechanics and weight. Highest quality and zero cost. Rejected on volume: it is per-game manual work across the whole catalogue, and it makes discovery hostage to how many evenings are available.

Classical NLP (keyword matching, TF-IDF over prose). Free and deterministic. Rejected: board game mechanics are described in wildly varying language across articles, and keyword matching produces exactly the confidently-wrong tags this ADR is designed to prevent — without a confidence score to gate on.

AI at request time. Rejected outright. It would put a multi-second, paid, rate-limited external call inside a Worker with a 10ms CPU budget and no spend protection (ADR-0001). Enrichment is a build-time concern.

A cheaper model tier for bulk normalisation. Genuinely reasonable, and the batch discount may make it unnecessary. Left as a tunable for WI-H05 to decide with real volume numbers rather than pre-committed here.