relaton 3.1.0 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (72) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/command.rb +6 -4
  7. data/lib/relaton/cli/frontend_assets.rb +7 -6
  8. data/lib/relaton/db.rb +31 -14
  9. data/lib/relaton/ecma/bibliography.rb +2 -2
  10. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  11. data/lib/relaton/iala/bibliography.rb +1 -1
  12. data/lib/relaton/iana.rb +1 -1
  13. data/lib/relaton/iec/bibliography.rb +32 -20
  14. data/lib/relaton/iec/data_parser.rb +41 -2
  15. data/lib/relaton/itu/README.adoc +1 -1
  16. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  17. data/lib/relaton/oasis/bibliography.rb +1 -1
  18. data/lib/relaton/ogc/hit_collection.rb +1 -1
  19. data/lib/relaton/ogc.rb +1 -1
  20. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  21. data/lib/relaton/plateau/hit_collection.rb +1 -1
  22. data/lib/relaton/render.rb +17 -0
  23. data/lib/relaton/sae/data_fetcher.rb +5 -0
  24. data/lib/relaton/sae.rb +1 -1
  25. data/lib/relaton/version.rb +1 -1
  26. data/lib/relaton/w3c/bibliography.rb +1 -1
  27. data/lib/relaton/xsf/bibliography.rb +1 -1
  28. data/lib/relaton/xsf.rb +1 -1
  29. data/lib/relaton-render.rb +4 -10
  30. metadata +33 -46
  31. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  32. data/lib/relaton/adobe/CLAUDE.md +0 -104
  33. data/lib/relaton/bib/CLAUDE.md +0 -150
  34. data/lib/relaton/bipm/CLAUDE.md +0 -187
  35. data/lib/relaton/bsi/CLAUDE.md +0 -35
  36. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  37. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  38. data/lib/relaton/cen/CLAUDE.md +0 -219
  39. data/lib/relaton/cie/CLAUDE.md +0 -90
  40. data/lib/relaton/core/CLAUDE.md +0 -140
  41. data/lib/relaton/db/CLAUDE.md +0 -385
  42. data/lib/relaton/doi/CLAUDE.md +0 -105
  43. data/lib/relaton/easc/CLAUDE.md +0 -85
  44. data/lib/relaton/ecma/CLAUDE.md +0 -217
  45. data/lib/relaton/etsi/CLAUDE.md +0 -219
  46. data/lib/relaton/evs/CLAUDE.md +0 -120
  47. data/lib/relaton/gb/CLAUDE.md +0 -47
  48. data/lib/relaton/gost/CLAUDE.md +0 -100
  49. data/lib/relaton/iala/CLAUDE.md +0 -97
  50. data/lib/relaton/iana/CLAUDE.md +0 -105
  51. data/lib/relaton/iec/CLAUDE.md +0 -13
  52. data/lib/relaton/ieee/CLAUDE.md +0 -202
  53. data/lib/relaton/ietf/CLAUDE.md +0 -382
  54. data/lib/relaton/iho/CLAUDE.md +0 -76
  55. data/lib/relaton/index/CLAUDE.md +0 -208
  56. data/lib/relaton/isbn/CLAUDE.md +0 -47
  57. data/lib/relaton/iso/CLAUDE.md +0 -38
  58. data/lib/relaton/itu/CLAUDE.md +0 -685
  59. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  60. data/lib/relaton/jis/CLAUDE.md +0 -64
  61. data/lib/relaton/logger/CLAUDE.md +0 -37
  62. data/lib/relaton/nist/CLAUDE.md +0 -194
  63. data/lib/relaton/oasis/CLAUDE.md +0 -251
  64. data/lib/relaton/ogc/CLAUDE.md +0 -242
  65. data/lib/relaton/oiml/CLAUDE.md +0 -47
  66. data/lib/relaton/omg/CLAUDE.md +0 -137
  67. data/lib/relaton/plateau/CLAUDE.md +0 -126
  68. data/lib/relaton/sae/CLAUDE.md +0 -89
  69. data/lib/relaton/sdo/CLAUDE.md +0 -81
  70. data/lib/relaton/un/CLAUDE.md +0 -91
  71. data/lib/relaton/w3c/CLAUDE.md +0 -369
  72. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,369 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-w3c is a Ruby gem for retrieving and representing W3C Standards bibliographic data using the Relaton model. It is part of the larger Relaton ecosystem of gems. Uses a LutaML-based model architecture under the `Relaton::W3c` namespace.
8
-
9
- ## Common Commands
10
-
11
- ```bash
12
- # Install dependencies
13
- bundle install
14
-
15
- # Run all tests
16
- bundle exec rake spec
17
-
18
- # Run a specific test file
19
- bundle exec rspec spec/relaton/w3c/item_spec.rb
20
-
21
- # Run a specific test by line number
22
- bundle exec rspec spec/relaton/w3c/item_spec.rb:7
23
-
24
- # Lint
25
- bundle exec rubocop
26
-
27
- # Interactive console
28
- bin/console
29
- ```
30
-
31
- ## Architecture
32
-
33
- ### Class Hierarchy
34
-
35
- All classes live under `lib/relaton/w3c/` in the `Relaton::W3c` namespace:
36
-
37
- **Model classes:**
38
- - **`Item`** (`item.rb`) — extends `Bib::Item`, adds W3C `ext` attribute. Base class for both Bibitem and Bibdata.
39
- - **`ItemData`** (`item_data.rb`) — LutaML data model for `Item`
40
- - **`Bibitem`** (`bibitem.rb`) — extends `Item`, includes `Bib::BibitemShared` (XML serialization without `<bibdata>` wrapper)
41
- - **`Bibdata`** (`bibdata.rb`) — extends `Item`, includes `Bib::BibdataShared` (XML serialization with `<bibdata>` wrapper)
42
- - **`Ext`** (`ext.rb`) — extends `Bib::Ext`, adds W3C-specific `doctype` attribute
43
- - **`Doctype`** (`doctype.rb`) — extends `Bib::Doctype`, restricts content to `groupNote` or `technicalReport`
44
-
45
- **Public API:**
46
- - **`Bibliography`** (`bibliography.rb`) — search and retrieve W3C standards from the Relaton index. Parses the reference with `Pubid::W3c` and narrows the index by number before matching; see **Index (`index-v2`, pubid-keyed)**.
47
- - **`Processor`** (`processor.rb`) — extends `Relaton::Core::Processor`, registers the W3C flavor (prefix `W3C`, dataset `w3c-api`)
48
-
49
- **Data fetching:**
50
- - **`DataFetcher`** (`data_fetcher.rb`) — extends `Core::DataFetcher`, fetches all W3C specs via the W3C API. Fetches the specification index with `embed: true` so each spec is realized from the page's embedded payload instead of a per-spec HTTP request, and paginates by page number (only the `fetch` path repopulates `_embedded`, unlike realizing the `next` link). Runs `fetch_spec` across a small thread pool. A SIGINT (Ctrl-C) is handled gracefully — the producer stops queuing and workers stop after their in-flight spec, then the index of everything fetched so far is saved (the prior INT handler is restored afterwards, so the trap doesn't leak into the host process). The crawl refuses to save a truncated dataset — `crawler.rb` wipes `data/` before each run, so saving one commits mass deletions. Three guards enforce that, all raising `CrawlIncompleteError`: an index page that fails after retries or pagination ending before the API's advertised last page (`enqueue_specs`); the governor concluding the crawl is banned (`guard_rate_limited`); and more than `max_throttled_losses` resources dropped to rate limiting (`guard_throttle_budget` — the per-resource counterpart, since pagination can complete cleanly while the documents behind it were quietly hollowed out). A **deliberate** `@interrupted` (Ctrl-C) is exempt: it already means "give me what you have". The worker pool is still drained in an `ensure` so an abort doesn't deadlock. See **Crawler tuning** for the env-var knobs.
51
- - **`DataParser`** (`data_parser.rb`) — converts W3C API spec objects into `Relaton::W3c::Item` instances
52
- - **`Governor`** (`governor.rb`) — the W3C binding of **`Relaton::Core::Governor`**, which is where process-wide, thread-safe rate-limit back-pressure now lives (see Rate limiting & retries). It was promoted out of this flavor when the ITU-R crawler became the second consumer. What stays here is only what W3C knows: `THROTTLE_ERRORS` (429 **and** 403, the lutaml-hal classes core must not depend on) and `ENV_PREFIX = "RELATON_W3C"`. The ladder, the per-round escalation, the jitter, the latched give-up and the `Retry-After` parsing are core's, and `spec/w3c/relaton/w3c/governor_spec.rb` still exercises all of it through this subclass unchanged — which is the promotion's acceptance test.
53
- - **`SafeRealize`** (`safe_realize.rb`) — mixin that, on a terminal error, skips the resource (returns `nil`) so one bad link doesn't abort the crawl, and that routes rate limiting through the `Governor` instead (see Rate limiting & retries). It does not cache successes — that lives upstream.
54
- - **`Docidentifier`** (`docidentifier.rb`) — `Bib::Docidentifier` subclass that parses its `content` into a `Pubid::W3c::Identifier` and exposes it as `#pubid`. See **Index (`index-v2`, pubid-keyed)**.
55
- - **`PubId`** (`pubid.rb`) — the **legacy `index-v1` row shape**, no longer on any lookup or crawl path here. Retained public only so `relaton-data-w3c`'s crawler can keep emitting `index-v1` for released relaton v2 consumers (the `Relaton::Bipm::Id` precedent). Do not add call sites.
56
-
57
- **Utilities:**
58
- - **`Util`** (`util.rb`) — extends `Relaton::Bib::Util`, sets `PROGNAME` for logging
59
-
60
- The entry module is defined in `lib/relaton/w3c.rb` and exposes `grammar_hash`.
61
-
62
- ### Crawler tuning
63
-
64
- `DataFetcher` is tunable via environment variables (read by class methods, so they apply to the whole crawl):
65
-
66
- - **`RELATON_W3C_FETCH_CONCURRENCY`** (default `4`) — number of `fetch_spec` worker threads. Kept conservative so the version-history requests don't burst fast enough to trip the W3C API rate limiter (429s); raise it for a faster run, lower it for debugging or if 429 skips appear.
67
- - **`RELATON_W3C_FETCH_VERSIONS`** (default enabled) — set to `false`/`0`/`no`/`off` for a faster, shallower crawl that emits only the top-level specifications and skips each spec's version-history fan-out (version_history, predecessor/successor versions — the bulk of the API requests). Leave it set (the default) for a complete dataset.
68
- - **`RELATON_W3C_THROTTLE_BASE`** / **`_MAX`** / **`_GIVEUP`** (defaults `60` / `900` / `5` — see `Governor`) — first cooldown, cooldown ceiling, and how many consecutive throttle rounds without a success end the crawl. Non-positive values are ignored. These three are read by the governor on construction **and on each `SafeRealize.reset!`** (i.e. at the start of every `fetch`), so like the others they can be set any time before the crawl rather than before the flavor is autoloaded.
69
- - **`RELATON_W3C_MAX_THROTTLED`** (default `25`) — how many resources may be lost to rate limiting before `fetch` refuses to save the index. `0` is accepted and means "tolerate none".
70
-
71
- `embed: true` (always on) inlines each specification into its index page, so the per-spec realize is served from memory rather than an HTTP request — the largest single reduction in request count.
72
-
73
- ### Rate limiting & retries
74
-
75
- Per-request retries are layered upstream; **pool-wide back-pressure is this
76
- gem's job** (`Governor`), because upstream has no notion of the crawl as a whole.
77
-
78
- - **w3c_api** (>= 0.3.3, pinned in the gemspec) builds its HAL client with
79
- `faraday-retry` for HTTP 403 — the W3C rate-limit signal — plus
80
- connection/timeout errors, and sends an identifying `User-Agent`. Both landed
81
- in 0.3.3 (relaton/w3c_api#22, #23); before it, requests went out as
82
- `Faraday v2.x` (a Cloudflare bot-heuristic trigger, and the most likely reason
83
- a crawl flipped into the banned mode at all) and the 403 retry never fired.
84
- - **lutaml-hal** (>= 0.2.5, declared directly in the gemspec because this flavor
85
- rescues its error classes by name) retries 429 and 5xx with exponential
86
- backoff. `DataFetcher` **shortens** that policy via
87
- `W3cApi::Hal.instance.configure_rate_limiting(UPSTREAM_RATE_LIMITING)`: its
88
- defaults burn ~31 s (1+2+4+8+16) before a 429 ever reaches us, which only
89
- delays the back-pressure that actually works. `max_retries` stays at 5 — one
90
- count covers both 429 and 5xx, and a 5xx is still a permanent skip here.
91
- From 0.3.3 that call rebuilds w3c_api's register itself; **don't** follow it
92
- with `reset_register`, which would leave the register absent from lutaml-hal's
93
- `GlobalRegister`, where `Link#realize` raises.
94
- `max_retry_after` is set too, to `max_delay` (4.0). Left unset, lutaml-hal's
95
- `DEFAULT_MAX_RETRY_AFTER` (300.0) applies and a large `Retry-After` on a 429
96
- parks one thread inside the HTTP layer for up to 5 x 300 s with the governor
97
- hearing nothing — the very thing the compressed chain exists to prevent. The
98
- server's hint is not discarded: `Governor.retry_after` reads it off the same
99
- error and uses it as the **floor** of the pool-wide cooldown, under the
100
- governor's own 900 s cap, so it is honoured once at pool scale. Do **not**
101
- lower `w3c_api`'s `DEFAULT_RETRY_OPTIONS[:max_interval]` or lutaml-hal's
102
- `DEFAULT_MAX_RETRY_AFTER` instead — the first turns a polite server hint into
103
- an immediate give-up, and either imposes a crawl-shaped number on every
104
- consumer of those gems.
105
-
106
- **`Governor` (`governor.rb`) is the important piece.** `api.w3.org` is behind
107
- Cloudflare, whose rate limiting is **bimodal**: a run either never trips it or
108
- trips it and stays banned for hours. Second-scale backoff cannot climb out of
109
- that, and per-worker backoff is worse than useless — the retries are what keep
110
- the limiter engaged. So all workers share one cooldown:
111
-
112
- - Any thread seeing a 429 opens/extends it; every other thread observes it in
113
- `#wait`. Cooldowns escalate **per round** (60 s → 120 → … → 900 s cap), where a
114
- "round" is one open cooldown — four workers tripping the same limiter must not
115
- fast-forward the ladder four steps.
116
- - A `Retry-After` header raises the floor but never lowers our own ladder, and is
117
- itself capped (a hostile value must not park a worker for an hour). Both RFC
118
- 9110 forms are honoured: delta-seconds, and an HTTP-date parsed properly
119
- (digit-scanning one would pick up the day-of-month).
120
- - A success calls `#succeeded!`, which clears the cooldown and **decays**
121
- (halves) the penalty rather than resetting it, so a flapping limiter still
122
- climbs. **Only a realize that actually went to the network counts** — with a
123
- `parent_resource`, lutaml-hal serves the object out of the page's `_embedded`
124
- payload and issues no request (`link.rb`: *"Priority 1: check embedded content
125
- first"*). That is the common path, one per specification, and counting those
126
- would reset the ladder between every pair of version-history 429s, so the
127
- governor could never reach its threshold.
128
- - After `GIVE_UP_AFTER` consecutive rounds with no success (~15 min) the crawl is
129
- declared banned: `#exhausted?` goes true, `#wait` stops blocking so shutdown is
130
- immediate, and `fetch` aborts. The abort message reads `#give_up_rounds`, the
131
- round count **latched** at that moment — `#throttle_rounds` is live and
132
- `#succeeded!` resets it, so a straggler success made a give-up at round 5 print
133
- as *"gave up after 2 consecutive backoffs"*.
134
- - The mutex is never held across a sleep, and wake-ups are jittered so workers
135
- don't resume in lockstep.
136
-
137
- **Giving up stops asking, not just waiting.** `#wait` returning 0.0 once latched
138
- makes shutdown prompt, but it does not stop a request. `#stopping?` is consulted
139
- only by the producer loop and at `queue.pop`, so an in-flight spec used to run
140
- its whole fan-out — `fetch_versions` plus `DataParser`'s seven realize sites —
141
- against a host that had just banned it. The 2026-08-31 crawl (CI run
142
- 33435903404) spent its last **11 minutes** doing exactly that, which extends the
143
- Cloudflare ban into the next run; 95 of its 105 `Throttled out` warnings say
144
- `after 1 attempt(s)`, which in `record_throttle` can only mean `gave_up`.
145
-
146
- The guard is one line at the top of `SafeRealize#realize`
147
- (`return nil if SafeRealize.governor.exhausted?`) because that is the **single
148
- choke point** every crawl request passes through — `fetch_specifications_page`
149
- is the only other one, and it already returns nil on give-up. One check
150
- therefore covers every fan-out site, including any added later, and every site
151
- already tolerates `nil` (the `SafeRealize` contract; `data_parser_spec.rb`
152
- proves it per site). Per-site checks in `data_parser.rb` would also need a stop
153
- flag plumbed into `DataParser`, which holds no reference to the fetcher.
154
-
155
- It checks `governor.exhausted?`, **not** `#stopping?`, and
156
- `DataFetcher#abandoned?` names that distinction. The two stop reasons want
157
- opposite things of a spec already in flight: a **Ctrl-C** saves the index, so
158
- the spec must finish or a half-parsed document is written; a **latched give-up**
159
- discards everything (`guard_rate_limited` raises), so there is nothing to gain
160
- from parsing, writing or crawling one more document.
161
-
162
- `SafeRealize` sits on top of it and makes the distinction that matters: **a
163
- throttle is not a broken resource.** `Governor::THROTTLE_ERRORS` is the single
164
- list of what counts — `TooManyRequestsError` **and `ForbiddenError`**, because
165
- api.w3.org signals rate limiting with 403 as well as 429 (w3c_api's whole retry
166
- layer is built around that). Blacklisting a 403 would be the same bug as
167
- blacklisting a 429. It retries a rate-limited realize
168
- `THROTTLE_ATTEMPTS` times behind the shared cooldown and records a give-up in
169
- `SafeRealize.throttled` — *never* in `skipped`. An href that a later attempt does
170
- realize is removed from `throttled` again, so a briefly-throttled-then-recovered
171
- resource doesn't count against the budget for a document that is in fact present.
172
- `fetch_specifications_page` is the exception to `THROTTLE_ATTEMPTS`: an index page
173
- retries until the *governor* gives up, since a 3-attempt cap would pre-empt the
174
- escalation ladder and report a pagination fault instead of the real reason.
175
- Everything else is unchanged:
176
- `NotFoundError` and other terminal `Lutaml::Hal::Error`s (403, 5xx) still go into
177
- `skipped` (a `Concurrent::Map`) so a broken link isn't re-fetched for every
178
- reference, and network errors are remembered nowhere so a later reference can try
179
- again. 5xx deliberately stays a permanent skip — the handful of persistent
180
- per-resource 500s a healthy crawl sees are broken records, and routing them
181
- through the governor would open a pool-wide cooldown for each.
182
-
183
- Successful objects are cached by **w3c_api** (lutaml-hal caches realized objects
184
- keyed by URL, thread-safely as of lutaml-hal 0.2.1), so `SafeRealize` doesn't
185
- cache them.
186
-
187
- > **Why this exists.** The 2026-08-19 `relaton-data-w3c` crawl ran **4 h 56 m**
188
- > (healthy runs take ~1 h 30 m), permanently blacklisted **1,412** rate-limited
189
- > resources, and only then aborted on a failed index page. Every 429 warning in
190
- > that log is ~31 s apart — one exhausted lutaml-hal chain — i.e. *every* request
191
- > for four hours was rate-limited. Had the last index page happened to succeed,
192
- > a dataset missing 1,412 documents would have been committed.
193
-
194
- ### Index (`index-v2`, pubid-keyed)
195
-
196
- `INDEXFILE` is the pubid-backed **`index-v2`**: rows are `Pubid::W3c::Identifier`
197
- leaves serialized as `_type: pubid:w3c:*` with a `number` (the document slug) and
198
- an optional verbatim `date`.
199
-
200
- ```yaml
201
- - :id:
202
- _type: pubid:w3c:recommendation
203
- number: xml-names
204
- date: '20091208'
205
- :file: data/rec-xml-names-20091208.yaml
206
- ```
207
-
208
- **Why.** `Relaton::Index::Type#candidates_by_number` sorts and binary-searches
209
- every row on `id.root.number.to_s`. The old `index-v1` stored plain hashes and
210
- was built without `pubid_class:`, so `FileIO#sorted` stayed false and every
211
- lookup scanned all 17,287 rows. Keyed on the pubid slug the corpus splits into
212
- **1,972 buckets**, the largest holding 101 rows.
213
-
214
- **The pubid contract.** `Pubid::W3c::Identifier` keeps the slug in `number`.
215
- It was called `code` until pubid #339 (merged 2026-08-28, `181fd4b9`), which
216
- renamed it with **no alias** — read `number`, never `code`. The root `Gemfile`
217
- pins pubid to `main`, so a checkout whose lock predates that merge still has the
218
- old attribute: run `bundle update pubid`, not `bundle install`, because an
219
- already-locked git source does not refloat.
220
-
221
- **Where the consumer reads it: the Pages machine index (relaton#189 pilot).**
222
- `Bibliography#index` is `find_or_create(:W3C, pages_url: PAGES_URL,
223
- pubid_class: ...)`, not the `index-v2.zip` under `SOURCE`. A parsed query reads
224
- `index/manifest.json` and the one `index/shard-NNNNN.json` of its slug from
225
- `https://relaton.github.io/relaton-data-w3c/`, in memory for 24 h — three
226
- requests for a lookup, including the document, in place of a 180 KB zip. A Pages
227
- failure raises `Relaton::RequestError`, with no fallback to the zip. Only the
228
- case-insensitive `#loose_match?` fallback reads every row (the Pages copy of the
229
- zip), and only after an exact miss. Documents still come from `SOURCE`.
230
- `get`/`search` accept a parsed `Pubid::W3c::Identifier` as well as a String, so
231
- `Relaton::Db` can hand over the pubid it parsed (#189 change 2).
232
-
233
- **The Pages copy lags the data repo.** The site is rebuilt by the `Deploy`
234
- workflow, not by the crawler's push, so a document the crawler committed after
235
- the last deploy is not in any shard yet. On 2026-09-23 the Pages index had
236
- 17,396 rows against 17,444 in `v2/index-v2.zip`: the 48 missing rows
237
- (`webxrlayers-1`, `rdf12-xml`, …) were committed at 20:01, after the 18:54
238
- deploy. Until the next deploy those references answer "not found".
239
-
240
- **Producer and consumer must stay in step.** Both pass
241
- `pubid_class: ::Pubid::W3c::Identifier` — `DataFetcher#index` and
242
- `Bibliography#index` (which deserializes the shard rows with it). (`Processor#remove_index_file` passes `url: true` and
243
- `file:` only: the delete never reads the index, see
244
- `lib/relaton/index/CLAUDE.md`.) On the producer side
245
- `#index_primary` stores the pubid **object**: `Relaton::Index::FileIO#save` only
246
- serializes to the `_type:` shape when the value is an instance of the configured
247
- `pubid_class`, so handing it a hash writes a v1-shaped file under a v2 name,
248
- silently, with no narrowing. On the consumer side, omitting `pubid_class:` leaves
249
- the rows as raw hashes and `FileIO#sorted` false, so `Type#search` stops
250
- narrowing — also silently.
251
-
252
- **How a reference is matched.** `Bibliography#best_match` passes the parsed
253
- pubid to `Index::Type#search`, which is what enables the bsearch; a block alone
254
- scans everything. It then selects with pubid's asymmetric subset match
255
- (`Pubid::SubsetMatch`) — the reference on the left, a component it omits matches
256
- any value:
257
-
258
- ```ruby
259
- rows = index.search(pubid) { |r| pubid === r[:id] }
260
- ```
261
-
262
- `date` is W3C's only optional component, so an undated `REC-xml-names` still
263
- finds the dated row and a dated reference reaches only its own. The maturity
264
- level is **not** a wildcard: it is the identifier's class, and `===` requires
265
- the same class, so `WD-`, `REC-` and a bare slug never match each other — the
266
- contract the bespoke `PubId#==` had for its `stage`/`type`. Measured over the
267
- whole fixture index (17,303 rows, 1,214,578 comparisons): the subset match and
268
- the old `matches?(…, ignore: [:date])` agree on every pair. Newest edition wins, with the file path breaking ties because
269
- undated rows all score 0 and the index sort is not stable.
270
-
271
- Two things the narrowing cannot do, both handled around it:
272
-
273
- - **Case.** The bsearch key is case-sensitive; `PubId#==` compared its slug with
274
- `casecmp?`. When the narrowed range is empty, `#loose_match?` repeats the
275
- comparison case-insensitively over a full scan (the BIPM `search_index`
276
- precedent).
277
- - **Queries that are not identifiers.** `#parse_ref`/`#normalize_ref` absorb a
278
- URL (`https://www.w3.org/TR/xml-names/`) and a leading `TR-`/`TR/`, which the
279
- bespoke regex accepted. A URL is not an identifier and `TR` is a path segment
280
- of one rather than a maturity level, so neither belongs in a pubid grammar —
281
- they stay here. The publisher prefix is added when absent. A reference pubid
282
- still rejects is a **miss, not an error**: `parse_ref` returns nil outside the
283
- transport rescue, so it never becomes a `Relaton::RequestError`.
284
-
285
- **An unparseable id is an ERROR, never a warning.** `Docidentifier#parse` logs at
286
- `Util.error` and leaves `#pubid` nil — it must not raise, or an already-published
287
- record would stop deserializing. `DataFetcher#index_primary` then records the
288
- failure in `@errors` **keyed by the id, not the output path** (the id is the
289
- defect; one broken id reaching several files must still report once), and the
290
- inherited `Core::DataFetcher#report_errors` logs it and raises a tracked GitHub
291
- issue. The row is skipped rather than indexed unparsed: `Relaton::Index` rejects
292
- the *whole* index if one row fails to deserialize, so one bad record must cost
293
- one document, never every lookup. The data file is still written — unindexed,
294
- not lost.
295
-
296
- `Core::DataFetcher#report_errors` treats a **String** `@errors` value as the
297
- message itself; a boolean still means "this field failed for every record" and
298
- renders as `Failed to fetch <key>`. That is what lets this flavor report a
299
- specific per-document failure without overriding `report_errors`.
300
-
301
- ### Key Dependencies
302
-
303
- - **relaton-bib** (~> 2.2.0) — provides base `Bib::Item`, `Bib::Ext`, `Bib::Doctype` and serialization mixins (LutaML model layer)
304
- - **relaton-core** — provides base `Core::Processor` and `Core::DataFetcher`
305
- - **relaton-index** — index-based search for bibliographic references; also unpacks the index zip at runtime
306
- - **pubid** — `Pubid::W3c::Identifier` backs `Docidentifier` and the `index-v2` rows
307
- - **w3c_api** (~> 0.3.3) — W3C API (HAL/REST) client used by `DataFetcher` to retrieve specifications; owns the `User-Agent`, the 403 retry, and the object cache. The 0.3.3 floor is load-bearing, not cosmetic — see **Rate limiting & retries**.
308
- - **lutaml-hal** (~> 0.2, >= 0.2.5) — HAL layer beneath w3c_api. Declared directly because this flavor rescues `Lutaml::Hal::*` error classes by name; `ForbiddenError` only exists from 0.2.5, and w3c_api's own `~> 0.2.1` would resolve happily to an older one and NameError at rescue time. 0.2.5 also made `Client#get`'s last-response bookkeeping thread-local (lutaml/lutaml-hal#21), which matters directly to the worker pool.
309
-
310
- The W3C data is fetched entirely through `w3c_api`; the older RDF/SPARQL/scraping stack (linkeddata, rdf, sparql, shex, mechanize, …) has been removed.
311
-
312
- ### Schema Validation
313
-
314
- XML output is validated against RelaxNG grammars in `grammars/`:
315
- - `relaton-w3c-compile.rng` — top-level compiled grammar (includes all others)
316
- - `relaton-w3c.rng` — W3C-specific overrides (DocumentType restrictions)
317
- - `basicdoc.rng`, `biblio.rng`, `biblio-standoc.rng` — shared base schemas
318
-
319
- Tests use [Jing](https://github.com/jing-trang/jing-trang) for RelaxNG validation.
320
-
321
- ### Test Structure
322
-
323
- Tests use RSpec with:
324
- - **Round-trip tests** — YAML/XML → object → YAML/XML, verifying lossless serialization
325
- - **Schema validation** — XML output validated against `grammars/relaton-w3c-compile.rng`
326
- - **VCR** — recorded HTTP cassettes in `spec/vcr_cassettes/` (never auto-rerecorded; re-record deliberately)
327
- - **WebMock** — disables external HTTP in tests
328
-
329
- Test fixtures live in `spec/fixtures/` (YAML and XML files).
330
-
331
- ## Style
332
-
333
- - Follows [Ribose OSS Ruby style guide](https://github.com/riboseinc/oss-guides) via RuboCop
334
- - Target Ruby version: 3.1
335
- - RuboCop config inherits from remote Ribose guide; Rails cops disabled
336
-
337
- ## CI
338
-
339
- GitHub Actions workflows (auto-generated by Cimas) delegate to shared workflows in `relaton/support`:
340
- - `rake.yml` — runs tests on push to main and PRs
341
- - `release.yml` — gem versioning and publishing to RubyGems
342
-
343
- ## Testing
344
-
345
- - **Index fixture:** `spec/fixtures/index-v2.zip` — the published index verbatim, seeded into the `Relaton::Index` pool by `spec/support/webmock.rb`. Refresh it from the live published `index-v2.zip`; there is no `rake spec:update_index` task in this repo.
346
- - It is built **with `pubid_class:`**. Without it the rows stay raw hashes and `Type#search` silently stops narrowing, so the suite would pass while exercising something the runtime never does.
347
- - It is re-seeded in **`before(:each)`**, not only `before(:suite)`: `Index::Pool#type` replaces the pooled entry whenever `actual?` says no, and `DataFetcher#index` asks for the same `:W3C` type with `file:` but no `url:`. `before(:suite)` alone leaves a later example searching a producer index. IANA hit this first.
348
- - The zip name is read from `Relaton::W3c::INDEXFILE`, but **lazily** — `spec_helper` loads `support/` before the flavor, so touching the constant at file-body time NameErrors.
349
- - **The HAL object cache is process-global, and the suite clears it per example.**
350
- `W3cApi::Hal` is a Singleton holding one `Lutaml::Hal::ModelRegister` with an
351
- in-memory URL cache, registered in the `Lutaml::Hal::GlobalRegister` Singleton.
352
- `W3cApi::Client.new` carries no state, so a fresh client per example still
353
- reads that one cache. Left warm it chains the examples together:
354
- `DataParser#parse` realizes `@spec.links.specification` for the `editionOf`
355
- relation, so the `vcr: "webrtc-20241008"` examples fetch
356
- `https://api.w3.org/specifications/webrtc` and the later `vcr: "webrtc"`
357
- examples then issue no request at all. A cassette recorded in that state holds
358
- only what the cache was missing, and replays only in that one example order —
359
- which is how `spec/w3c/vcr_cassettes/webrtc.yml` came to lack the very request
360
- its own examples make. `spec/w3c/support/hal_cache.rb` clears the cache in a
361
- `before` hook so each example issues its own requests. **A cassette
362
- re-recorded without that reset is not reusable.**
363
- - **`json` must stay below 3 (`relaton.gemspec`).** json 3.0.0 made the
364
- `JSON.parse` options keyword-only; Faraday's JSON response middleware — the
365
- one `Lutaml::Hal::Client#create_connection` installs — still passes them
366
- positionally, up to and including faraday 2.14.3. With json 3 every
367
- api.w3.org response raises `ArgumentError: wrong number of arguments (given 2,
368
- expected 1)`, surfaced as `Lutaml::Hal::ParsingError`, and this flavor fetches
369
- nothing. `spec/w3c/relaton/w3c/faraday_json_spec.rb` guards the constraint.
@@ -1,189 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## What is relaton-xsf?
6
-
7
- A Ruby gem for bibliographic retrieval of XMPP XEP (XMPP Extension Protocol) specifications. Part of the Relaton family of gems. Fetches data from https://xmpp.org/extensions/refs/ and the relaton-data-xsf GitHub repository.
8
-
9
- ## Commands
10
-
11
- - `bundle exec rake spec:xsf` — run this flavor's suite (from the repo root)
12
- - `bundle exec rake spec:update_index_xsf` — refresh
13
- `spec/xsf/fixtures/index-v2.zip` from the published index
14
- - `bundle exec rubocop` — lint
15
- - `bundle exec rubocop -a` — lint with auto-fix
16
- - `bin/console` — interactive console with gem loaded
17
-
18
- ## Architecture
19
-
20
- Namespace: `Relaton::Xsf` (under `lib/relaton/xsf/`). Branch `lutaml-integration` uses the new nested namespace (not the old `RelatonXsf`).
21
-
22
- Key classes and their base classes from relaton-core:
23
-
24
- | Class | Base | Role |
25
- |---|---|---|
26
- | `Processor` | `Relaton::Core::Processor` | Plugin entry point for relaton registry |
27
- | `Bibliography` | Module (extends self) | Search & get interface; **parses the reference into a pubid** (`parse_ref`) |
28
- | `HitCollection` | `Relaton::Core::HitCollection` | Collection of search results |
29
- | `Hit` | `Relaton::Core::Hit` | Single result; lazy-loads YAML from GitHub |
30
- | `DataFetcher` | `Relaton::Core::DataFetcher` | Crawls xmpp.org, parses BibXML, saves docs |
31
- | `Docidentifier` | `Relaton::Bib::Docidentifier` | Exposes the id as a `Pubid::Xsf::Identifier` via `#pubid` |
32
- | `Item` / `Bibitem` / `Bibdata` | `Relaton::Bib::Item` | Bibliographic item models (lutaml-model based) |
33
-
34
- ## Index (`index-v2`, pubid-keyed)
35
-
36
- `INDEXFILE` is the pubid-backed `index-v2`: rows are `Pubid::Xsf::Identifier`
37
- hashes (`_type: pubid:xsf:xep` with `number`). The consumer
38
- (`HitCollection#index`) and the producer (`DataFetcher#index`) pass
39
- `pubid_class: ::Pubid::Xsf::Identifier`. Omitting it on the **producer**
40
- writes v1-shaped rows under a v2 name, silently (`FileIO#save` calls `to_hash`
41
- only for instances of `pubid_class`); omitting it on the **consumer** leaves
42
- the rows raw hashes with `FileIO#sorted` false, so every lookup scans all 520.
43
- `Processor#remove_index_file` passes `url: true` and `file:` only: the delete
44
- never reads the index (see `lib/relaton/index/CLAUDE.md`).
45
-
46
- An XEP identifier is a publisher and a number, nothing else — no edition, no
47
- date, no part, and one row per XEP (520 rows, ids unique). That makes this the
48
- simplest flavor in the sweep: there is no selection order to preserve, and the
49
- reference states every component it has, so pubid's asymmetric subset match
50
- (`ref === row[:id]`, `Pubid::SubsetMatch`) is exact equality after narrowing.
51
-
52
- ### The index guard, and the two rows that are pages rather than XEPs
53
-
54
- `DataFetcher#add_to_index` parses the docid and, on failure, records
55
- `@errors[docid]` — which `report_errors` turns into a GitHub issue at the end of
56
- the crawl (the 3GPP/ECMA precedent) — and skips the row. The data file is still
57
- written, so a document that cannot be indexed is unindexed, never lost.
58
-
59
- The guard is not decoration. One unparseable row does not fail that row:
60
- `Relaton::Index` declares the **whole file** corrupt, deletes it, and hands back
61
- an **empty** index — measured, 519 good rows plus one unparseable id loads as
62
- **0 rows**, and the only trace is two INFO lines (`Wrong structure of file …`,
63
- `Considering … corrupt, removing it`). Every XSF lookup then fails silently.
64
-
65
- Two published rows are pages rather than XEPs — the XMPP repository's `README`
66
- and its `xep-xxxx` template, which the crawl mints records for because they sit
67
- in the same listing. pubid accepts them as the literal numbers `README` and
68
- `xxxx`, so they carry into `index-v2` like any other row and resolve normally.
69
- It stayed strict about everything else: `XEP banana` and a typo such as
70
- `XEP 00O1` are still rejected, which is what keeps an unparseable reference a
71
- warning rather than a silent miss.
72
-
73
- Nothing in the published corpus trips the guard today, so a recorded error means
74
- upstream grew a shape pubid does not know — exactly when an issue is worth
75
- filing.
76
-
77
- ### Where the reference becomes an identifier
78
-
79
- `Bibliography.parse_ref` does it, not `HitCollection`. That split is not
80
- cosmetic: `Type#search_candidates` narrows only when its search argument is
81
- **not** a `String`, so the string has to become an identifier at the entry
82
- point. `HitCollection` therefore receives a `Pubid::Xsf::Identifier` (or nil,
83
- when the reference could not be parsed) and does nothing but the index lookup.
84
- `Core::HitCollection` documents `ref` as `[String, Pubid]`, so this is the
85
- interface working as intended.
86
-
87
- ### Pubid-backed docidentifier
88
-
89
- `Relaton::Xsf::Docidentifier` parses its `content` into a
90
- `Pubid::Xsf::Identifier` kept in `@pubid`, while the lutaml `content` attribute
91
- stays a plain string. Parsing is soft — a missing gem or non-XSF content leaves
92
- `@pubid` nil rather than raising during deserialization. `Item` narrows the
93
- inherited `docidentifier` attribute to it.
94
-
95
- It is **purely additive**: `to_yaml` and `to_xml` are byte-identical before and
96
- after, so no published document or index needs regenerating.
97
-
98
- **It deliberately implements none of `remove_part!` / `remove_date!` /
99
- `to_all_parts!`.** An XEP identifier is a publisher and a number and nothing
100
- else, so there is genuinely nothing for any of them to strip, and
101
- `Bib::Docidentifier` defaults all three to no-ops for exactly that case. Empty
102
- overrides would assert a flavor rule that does not exist. Compare
103
- `Relaton::Ogc::Docidentifier`, which overrides `remove_date!` because OGC really
104
- does carry a revision.
105
-
106
- ### Lookup
107
-
108
- `HitCollection#search` follows the ETSI/W3C/OGC idiom:
109
-
110
- - **Pass the pubid, not the string.** `Type#search_candidates` narrows only
111
- when the argument is not a `String`, so the plain reference this flavor used
112
- to pass disabled the binary search however the index was built.
113
- `pubid_class:` alone fixes nothing; both had to change together.
114
- - **Matching is exact.** The old `index.search(ref)` compared a **substring**
115
- of the rendered id, so a bare `001` answered with **11** documents
116
- (`XEP 0001` and every `XEP 001x`) and `Bibliography#get` took `.first` —
117
- a truncated reference silently resolved to whichever sorted first. It now
118
- returns nothing.
119
- - **An unrecognized reference raises.** Like ISO, ETSI and 3GPP, the parse
120
- error propagates: relaton-cli rescues `Pubid::Errors::Error` and renders
121
- `"..." is not a recognized standards identifier`
122
- (`gems/relaton-cli/lib/relaton/cli/command.rb:324`,
123
- `subcommand_collection.rb:134`), and `Db#fetch` logs it through the
124
- `StandardError` arm at `lib/relaton/db.rb:122`. `Pubid::Errors::ParseError`
125
- includes the marker module `Pubid::Errors::Error`, which is what makes the
126
- CLI's rescue work for every flavor.
127
-
128
- Rescuing into a warning and an empty result — which several flavors still do
129
- — collapses "this identifier is malformed" into "no such document", and a
130
- caller cannot tell them apart. The parse also happens **before**
131
- `HitCollection` is constructed, so it is never relabelled as the
132
- `Relaton::RequestError` that collection's own rescue raises.
133
- - **Three reference forms are normalized** in `Bibliography#normalize_ref`, because
134
- `Pubid::Xsf` accepts only the canonical `XEP 0001`:
135
-
136
- | form | before | after |
137
- |---|---|---|
138
- | `XEP 0001` | resolves | resolves |
139
- | `0001` (bare) | resolves (substring) | resolves — the token is added |
140
- | `XEP-0001` | **no match** | resolves — the spelling xmpp.org itself uses |
141
- | `xep 0001` | no match | resolves — the token match is case-insensitive |
142
-
143
- The hyphen and case forms are new support, not preserved behaviour: the
144
- substring match compared against `XEP 0001`, which has a space.
145
-
146
- ### Both indexes, one crawl
147
-
148
- `relaton-data-xsf` publishes **both** from a single crawl: the `index-v2` this
149
- fetcher writes, and its own `index-v1` built by `build_index_v1.rb` for released
150
- relaton v2 consumers. Both carry the same 520 rows — building v1 from `data/`
151
- was originally what kept `XEP README` and `XEP xxxx` available there while they
152
- were unindexable in v2, and pubid accepting them has since removed that
153
- asymmetry.
154
-
155
- Verified against the published `index-v2`: 520 rows, all deserialize, sorted,
156
- **0** keying on `""`, `XEP 0001` narrows to 1 of 520, and every accepted
157
- spelling resolves to a file that returns HTTP 200.
158
-
159
- Data flow: `Processor#get` → `Bibliography.get` → `HitCollection.search` → `Hit#item` → fetches YAML → `Relaton::Bib::Item.from_yaml`
160
-
161
- DataFetcher flow: Crawls `https://xmpp.org/extensions/refs/`, parses each XML ref via `Relaton::Bib::Converter::BibXml.to_item`, sets `ext.flavor = "xsf"`, saves to disk.
162
-
163
- Constants: `INDEXFILE = "index-v2"`, `GHDATA_URL` points to relaton-data-xsf `v2` branch.
164
-
165
- ## Testing
166
-
167
- - **Index fixture:** `spec/xsf/fixtures/index-v2.zip` is seeded into the
168
- `Relaton::Index` pool by `spec/xsf/support/webmock.rb` — see **Index
169
- (`index-v2`, pubid-keyed)** above for why it is re-seeded per example and
170
- built with `pubid_class:`. Refresh it with `rake spec:update_index_xsf`.
171
- - RSpec with VCR cassettes (`spec/vcr_cassettes/`) for HTTP interactions
172
- - WebMock disables all external network connections
173
- - Fixtures in `spec/fixtures/` (item.yaml, bibdata.xml, bibitem.xml)
174
- - Round-trip tests verify YAML→Item→YAML and XML→Item→XML fidelity
175
- - `DataFetcher` is lazily required — specs that test it must `require "relaton/xsf/data_fetcher"` explicitly
176
- - Same for `Processor` — `require "relaton/xsf/processor"`
177
-
178
- ## Key dependencies
179
-
180
- - `relaton-core` — abstract base classes (Processor, HitCollection, Hit, DataFetcher)
181
- - `relaton-bib` — bibliographic models, XML/YAML serialization (lutaml-model based)
182
- - `relaton-index` — index management for quick document lookups
183
- - `mechanize` — HTTP fetching and HTML parsing
184
-
185
- ## Style
186
-
187
- - RuboCop with relaton shared config (inherits from riboseinc/oss-guides)
188
- - Target Ruby version: 3.1
189
- - Logging via `Relaton::Xsf::Util` (extends `Relaton::Bib::Util`, PROGNAME = "relaton-xsf")