WI-071: Wikidata

Real catalogue data, from the source ADR-0016 chose: CC0, structured, and nobody can withdraw it.

It stopped first

https://query.wikidata.org/robots.txt disallows /sparql to every user-agent. That is the endpoint ADR-0016 named, and ADR-0016 also makes honouring robots.txt binding. Both could not hold, and reconciling them is a judgement about a third party’s terms — a person’s call under agent boundaries. It became ADR-0023, and the answer was to query the service and amend ADR-0016 in writing rather than quietly reinterpret it.

One query was sent before this was noticed. LIMIT 5, identified, through curl rather than PoliteClient — which is exactly how the guard built in WI-070 came to be bypassed. Building a rule into a client does nothing if the next request goes around it.

The exception is not a flag

There is no ignoreRobots: true. A RobotsExemption requires an origin, a path prefix, a reason, and the ADR that authorised it; missing any one and PoliteClient refuses to construct. It covers /sparql on query.wikidata.org and nothing else — /bigdata on the same host is still refused, /sparql on any other host is still refused, and a 403 from the server is still a block, because an exemption is a reading of a directive rather than permission to ignore one.

A parser bug, found by checking it against reality

robots.txt supports * and $; the parser did prefix matching only. Wikidata has:

Disallow: /wiki/Special:EntityData/
Allow:    /wiki/Special:EntityData/*.

Prefix matching reads that as “all of EntityData is forbidden” and refuses Q42.json, a path we are explicitly invited to use. Failing safe is the right way to be wrong; being wrong still costs a source.

Three minutes or three hours

Wikidata’s playing time (P2047) carries a unit, and it is not always minutes. Reading the bare number turns a three-hour game into a three-minute one — wrong by a factor of sixty, entirely plausible on a card, and therefore worse than a missing value because nothing about it invites suspicion. The query asks for the unit; an unrecognised one is dropped rather than guessed.

Two other things the real data forced:

An item with no English label comes back with its Q-id as the label. Shipping that puts Q100272319 on a card, so it is rejected.
GROUP_CONCAT returns an empty string, not an absent binding, when the OPTIONAL matched nothing — and a bare separator when one grouped value is empty. Either way a naive split invents a designer called "".

Rejected records are reported, not dropped. A harvest that quietly discards a tenth of its input looks exactly like one that worked.

Definition of done

Verification

pnpm --filter @tabletop/catalogue-forge test
pnpm --filter @tabletop/catalogue-forge forge harvest --limit 12 --out /tmp/seed.json
pnpm gate

Seeded failures

Seed	Bit
Wildcards dropped from the matcher	2
`$` anchoring dropped	1
Exemptions accepted with no reason or ADR	1
The exemption ignores the path prefix	1
The exemption ignores the origin	1
An exemption also waves through a 403	1
Playing time read without its unit	2
An unknown unit assumed to be minutes	1
No English label ships as a Q-id title	1
An empty concatenated entry becomes a value	1
Provenance stamped on unpopulated fields	1
Wikidata genre passed off as a BGG mechanic	7
Invalid records dropped silently	1

A seed aimed at the wrong line

The empty-list seed removed .filter() and everything still passed: list() returns early on an empty string, so the filter never ran for that case. It is load-bearing only for an interior empty — "Kosmos||Mayfair" — which nothing tested. A test for that case was added, and then the seed bit.

Not done here

Descriptions. Wikidata’s are one-line disambiguators (“2017 boardgame”), not prose. Real descriptions are Wikipedia, under CC BY-SA, with an attribution obligation — WI-072.
Mechanics. Wikidata does not model them.
Loading a snapshot into the database, and reconciling it with the 333 fixture games — WI-074.