Skip to content
Tabletop Tracker

ADR-0023: How the forge reaches Wikidata

ADR-0023: How the forge reaches Wikidata

  • Status: Accepted
  • Date: 2026-08-04
  • Supersedes: —
  • Superseded by: —

Context

ADR-0016 made Wikidata the primary catalogue source: CC0, structured, terms that cannot be withdrawn. It named the SPARQL endpoint as the mechanism and made scraping conduct binding — including honour robots.txt.

While building WI-071, https://query.wikidata.org/robots.txt turned out to read:

User-agent: *
Disallow: /sparql
Disallow: /bigdata

The endpoint ADR-0016 assumed is disallowed to every user-agent, without exception. PoliteClient refuses it, correctly.

One query was sent before this was noticed — LIMIT 5, identified, via curl rather than through PoliteClient, which is how the guard was bypassed. It is recorded here rather than quietly dropped.

What robots.txt permits, checked:

query.wikidata.org/sparqldisallowed
query.wikidata.org/bigdata (Linked Data Fragments)disallowed
www.wikidata.org/w/api.php (Action API)disallowed
www.wikidata.org/wiki/Special:EntityData/Q42.jsonallowed, via Allow: /wiki/Special:EntityData/*.
dumps.wikimedia.orgno robots.txt at all (404) — unrestricted

The decision required

Reading Disallow: /sparql literally forbids every published Wikidata client library, because robots.txt is a crawler-control protocol and the Query Service separately documents programmatic access, publishes rate limits, and asks for a descriptive User-Agent — which is what this project already sends.

So the letter and the evident intent disagree. That is a judgement about a third party’s terms, and agent boundaries puts it with a person, not an agent.

Options

A — Bulk dumps (dumps.wikimedia.org)

What ADR-0016 already says to prefer, and unrestricted.

  • ✅ No conduct question at all.
  • ❌ There is no board-games-only dump. The smallest useful artefact is the truthy N-Triples dump, tens of gigabytes compressed, streamed and filtered in one pass. A day of downloading, once.
  • ❌ Substantial code for a one-off, and awkward to refresh.

B — Special:EntityData/{Q}.json

Explicitly allowed by the wildcard rule.

  • ✅ Unambiguous, cheap, per-entity.
  • ❌ Cannot enumerate. It answers “tell me about Q42”, not “which items are board games” — and without SPARQL there is no way to get that list. Useful for enrichment, useless for discovery.

C — Treat WDQS’s own documented policy as authoritative

Query SPARQL with the descriptive User-Agent and rate limits the service asks for, and record that robots.txt is understood to target crawlers.

  • ✅ The mechanism ADR-0016 chose, and how the ecosystem actually works.
  • ❌ Contradicts a binding rule in an accepted ADR. Needs ADR-0016 amended to say so explicitly, not quietly ignored.

D — A different source entirely

  • ❌ Nothing else in this space is CC0 and structured. This is why ADR-0016 chose Wikidata.

Recommendation

C, with ADR-0016 amended, falling back to A if that reading is not acceptable.

C matches the service’s own instructions and keeps the design. But it must be an explicit, recorded amendment — the value of a binding conduct rule is entirely in not being quietly reinterpreted the first time it costs something.

If C is rejected, A works and is honest; it is slow and heavy, and it is a one-off.

Decision

Option C. The Query Service is used, and ADR-0016 is amended to say so rather than left to be quietly reinterpreted.

Consequences

  • “Honour robots.txt” still stands. The exception is narrow, recorded, and expressed in code as a RobotsExemption with an origin, a path prefix, a reason and this ADR’s number — all four required, or PoliteClient refuses to construct. It is not a flag; there is no ignoreRobots: true.
  • It covers /sparql on query.wikidata.org and nothing else. /bigdata on the same host is still refused, as is /sparql on any other host — both asserted by tests.
  • A block the server actually sends is unaffected. An exemption is a reading of a directive, not permission to ignore a 403. Also a test.
  • The next source that presents this conflict gets its own ADR. The mechanism makes that cheap and makes skipping it obvious.
  • www.wikidata.org/w/api.php is disallowed and has no exemption. Special:EntityData/{Q}.json is permitted outright, via a wildcard Allow the parser now understands.