relaton 3.1.0 → 3.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +5 -3
- data/lib/relaton/3gpp.rb +1 -1
- data/lib/relaton/bib/model/docidentifier.rb +66 -5
- data/lib/relaton/calconnect/hit_collection.rb +1 -1
- data/lib/relaton/cli/command.rb +6 -4
- data/lib/relaton/cli/frontend_assets.rb +7 -6
- data/lib/relaton/db.rb +31 -14
- data/lib/relaton/ecma/bibliography.rb +2 -2
- data/lib/relaton/etsi/data_fetcher.rb +1 -1
- data/lib/relaton/iala/bibliography.rb +1 -1
- data/lib/relaton/iana.rb +1 -1
- data/lib/relaton/iec/bibliography.rb +32 -20
- data/lib/relaton/iec/data_parser.rb +41 -2
- data/lib/relaton/itu/README.adoc +1 -1
- data/lib/relaton/itu/data_crawler_r.rb +1 -1
- data/lib/relaton/oasis/bibliography.rb +1 -1
- data/lib/relaton/ogc/hit_collection.rb +1 -1
- data/lib/relaton/ogc.rb +1 -1
- data/lib/relaton/plateau/handbook_parser.rb +31 -2
- data/lib/relaton/plateau/hit_collection.rb +1 -1
- data/lib/relaton/render.rb +17 -0
- data/lib/relaton/sae/data_fetcher.rb +5 -0
- data/lib/relaton/sae.rb +1 -1
- data/lib/relaton/version.rb +1 -1
- data/lib/relaton/w3c/bibliography.rb +1 -1
- data/lib/relaton/xsf/bibliography.rb +1 -1
- data/lib/relaton/xsf.rb +1 -1
- data/lib/relaton-render.rb +4 -10
- metadata +33 -46
- data/lib/relaton/3gpp/CLAUDE.md +0 -218
- data/lib/relaton/adobe/CLAUDE.md +0 -104
- data/lib/relaton/bib/CLAUDE.md +0 -150
- data/lib/relaton/bipm/CLAUDE.md +0 -187
- data/lib/relaton/bsi/CLAUDE.md +0 -35
- data/lib/relaton/calconnect/CLAUDE.md +0 -177
- data/lib/relaton/ccsds/CLAUDE.md +0 -74
- data/lib/relaton/cen/CLAUDE.md +0 -219
- data/lib/relaton/cie/CLAUDE.md +0 -90
- data/lib/relaton/core/CLAUDE.md +0 -140
- data/lib/relaton/db/CLAUDE.md +0 -385
- data/lib/relaton/doi/CLAUDE.md +0 -105
- data/lib/relaton/easc/CLAUDE.md +0 -85
- data/lib/relaton/ecma/CLAUDE.md +0 -217
- data/lib/relaton/etsi/CLAUDE.md +0 -219
- data/lib/relaton/evs/CLAUDE.md +0 -120
- data/lib/relaton/gb/CLAUDE.md +0 -47
- data/lib/relaton/gost/CLAUDE.md +0 -100
- data/lib/relaton/iala/CLAUDE.md +0 -97
- data/lib/relaton/iana/CLAUDE.md +0 -105
- data/lib/relaton/iec/CLAUDE.md +0 -13
- data/lib/relaton/ieee/CLAUDE.md +0 -202
- data/lib/relaton/ietf/CLAUDE.md +0 -382
- data/lib/relaton/iho/CLAUDE.md +0 -76
- data/lib/relaton/index/CLAUDE.md +0 -208
- data/lib/relaton/isbn/CLAUDE.md +0 -47
- data/lib/relaton/iso/CLAUDE.md +0 -38
- data/lib/relaton/itu/CLAUDE.md +0 -685
- data/lib/relaton/jcgm/CLAUDE.md +0 -107
- data/lib/relaton/jis/CLAUDE.md +0 -64
- data/lib/relaton/logger/CLAUDE.md +0 -37
- data/lib/relaton/nist/CLAUDE.md +0 -194
- data/lib/relaton/oasis/CLAUDE.md +0 -251
- data/lib/relaton/ogc/CLAUDE.md +0 -242
- data/lib/relaton/oiml/CLAUDE.md +0 -47
- data/lib/relaton/omg/CLAUDE.md +0 -137
- data/lib/relaton/plateau/CLAUDE.md +0 -126
- data/lib/relaton/sae/CLAUDE.md +0 -89
- data/lib/relaton/sdo/CLAUDE.md +0 -81
- data/lib/relaton/un/CLAUDE.md +0 -91
- data/lib/relaton/w3c/CLAUDE.md +0 -369
- data/lib/relaton/xsf/CLAUDE.md +0 -189
data/lib/relaton/w3c/CLAUDE.md
DELETED
|
@@ -1,369 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-w3c is a Ruby gem for retrieving and representing W3C Standards bibliographic data using the Relaton model. It is part of the larger Relaton ecosystem of gems. Uses a LutaML-based model architecture under the `Relaton::W3c` namespace.
|
|
8
|
-
|
|
9
|
-
## Common Commands
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
# Install dependencies
|
|
13
|
-
bundle install
|
|
14
|
-
|
|
15
|
-
# Run all tests
|
|
16
|
-
bundle exec rake spec
|
|
17
|
-
|
|
18
|
-
# Run a specific test file
|
|
19
|
-
bundle exec rspec spec/relaton/w3c/item_spec.rb
|
|
20
|
-
|
|
21
|
-
# Run a specific test by line number
|
|
22
|
-
bundle exec rspec spec/relaton/w3c/item_spec.rb:7
|
|
23
|
-
|
|
24
|
-
# Lint
|
|
25
|
-
bundle exec rubocop
|
|
26
|
-
|
|
27
|
-
# Interactive console
|
|
28
|
-
bin/console
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
## Architecture
|
|
32
|
-
|
|
33
|
-
### Class Hierarchy
|
|
34
|
-
|
|
35
|
-
All classes live under `lib/relaton/w3c/` in the `Relaton::W3c` namespace:
|
|
36
|
-
|
|
37
|
-
**Model classes:**
|
|
38
|
-
- **`Item`** (`item.rb`) — extends `Bib::Item`, adds W3C `ext` attribute. Base class for both Bibitem and Bibdata.
|
|
39
|
-
- **`ItemData`** (`item_data.rb`) — LutaML data model for `Item`
|
|
40
|
-
- **`Bibitem`** (`bibitem.rb`) — extends `Item`, includes `Bib::BibitemShared` (XML serialization without `<bibdata>` wrapper)
|
|
41
|
-
- **`Bibdata`** (`bibdata.rb`) — extends `Item`, includes `Bib::BibdataShared` (XML serialization with `<bibdata>` wrapper)
|
|
42
|
-
- **`Ext`** (`ext.rb`) — extends `Bib::Ext`, adds W3C-specific `doctype` attribute
|
|
43
|
-
- **`Doctype`** (`doctype.rb`) — extends `Bib::Doctype`, restricts content to `groupNote` or `technicalReport`
|
|
44
|
-
|
|
45
|
-
**Public API:**
|
|
46
|
-
- **`Bibliography`** (`bibliography.rb`) — search and retrieve W3C standards from the Relaton index. Parses the reference with `Pubid::W3c` and narrows the index by number before matching; see **Index (`index-v2`, pubid-keyed)**.
|
|
47
|
-
- **`Processor`** (`processor.rb`) — extends `Relaton::Core::Processor`, registers the W3C flavor (prefix `W3C`, dataset `w3c-api`)
|
|
48
|
-
|
|
49
|
-
**Data fetching:**
|
|
50
|
-
- **`DataFetcher`** (`data_fetcher.rb`) — extends `Core::DataFetcher`, fetches all W3C specs via the W3C API. Fetches the specification index with `embed: true` so each spec is realized from the page's embedded payload instead of a per-spec HTTP request, and paginates by page number (only the `fetch` path repopulates `_embedded`, unlike realizing the `next` link). Runs `fetch_spec` across a small thread pool. A SIGINT (Ctrl-C) is handled gracefully — the producer stops queuing and workers stop after their in-flight spec, then the index of everything fetched so far is saved (the prior INT handler is restored afterwards, so the trap doesn't leak into the host process). The crawl refuses to save a truncated dataset — `crawler.rb` wipes `data/` before each run, so saving one commits mass deletions. Three guards enforce that, all raising `CrawlIncompleteError`: an index page that fails after retries or pagination ending before the API's advertised last page (`enqueue_specs`); the governor concluding the crawl is banned (`guard_rate_limited`); and more than `max_throttled_losses` resources dropped to rate limiting (`guard_throttle_budget` — the per-resource counterpart, since pagination can complete cleanly while the documents behind it were quietly hollowed out). A **deliberate** `@interrupted` (Ctrl-C) is exempt: it already means "give me what you have". The worker pool is still drained in an `ensure` so an abort doesn't deadlock. See **Crawler tuning** for the env-var knobs.
|
|
51
|
-
- **`DataParser`** (`data_parser.rb`) — converts W3C API spec objects into `Relaton::W3c::Item` instances
|
|
52
|
-
- **`Governor`** (`governor.rb`) — the W3C binding of **`Relaton::Core::Governor`**, which is where process-wide, thread-safe rate-limit back-pressure now lives (see Rate limiting & retries). It was promoted out of this flavor when the ITU-R crawler became the second consumer. What stays here is only what W3C knows: `THROTTLE_ERRORS` (429 **and** 403, the lutaml-hal classes core must not depend on) and `ENV_PREFIX = "RELATON_W3C"`. The ladder, the per-round escalation, the jitter, the latched give-up and the `Retry-After` parsing are core's, and `spec/w3c/relaton/w3c/governor_spec.rb` still exercises all of it through this subclass unchanged — which is the promotion's acceptance test.
|
|
53
|
-
- **`SafeRealize`** (`safe_realize.rb`) — mixin that, on a terminal error, skips the resource (returns `nil`) so one bad link doesn't abort the crawl, and that routes rate limiting through the `Governor` instead (see Rate limiting & retries). It does not cache successes — that lives upstream.
|
|
54
|
-
- **`Docidentifier`** (`docidentifier.rb`) — `Bib::Docidentifier` subclass that parses its `content` into a `Pubid::W3c::Identifier` and exposes it as `#pubid`. See **Index (`index-v2`, pubid-keyed)**.
|
|
55
|
-
- **`PubId`** (`pubid.rb`) — the **legacy `index-v1` row shape**, no longer on any lookup or crawl path here. Retained public only so `relaton-data-w3c`'s crawler can keep emitting `index-v1` for released relaton v2 consumers (the `Relaton::Bipm::Id` precedent). Do not add call sites.
|
|
56
|
-
|
|
57
|
-
**Utilities:**
|
|
58
|
-
- **`Util`** (`util.rb`) — extends `Relaton::Bib::Util`, sets `PROGNAME` for logging
|
|
59
|
-
|
|
60
|
-
The entry module is defined in `lib/relaton/w3c.rb` and exposes `grammar_hash`.
|
|
61
|
-
|
|
62
|
-
### Crawler tuning
|
|
63
|
-
|
|
64
|
-
`DataFetcher` is tunable via environment variables (read by class methods, so they apply to the whole crawl):
|
|
65
|
-
|
|
66
|
-
- **`RELATON_W3C_FETCH_CONCURRENCY`** (default `4`) — number of `fetch_spec` worker threads. Kept conservative so the version-history requests don't burst fast enough to trip the W3C API rate limiter (429s); raise it for a faster run, lower it for debugging or if 429 skips appear.
|
|
67
|
-
- **`RELATON_W3C_FETCH_VERSIONS`** (default enabled) — set to `false`/`0`/`no`/`off` for a faster, shallower crawl that emits only the top-level specifications and skips each spec's version-history fan-out (version_history, predecessor/successor versions — the bulk of the API requests). Leave it set (the default) for a complete dataset.
|
|
68
|
-
- **`RELATON_W3C_THROTTLE_BASE`** / **`_MAX`** / **`_GIVEUP`** (defaults `60` / `900` / `5` — see `Governor`) — first cooldown, cooldown ceiling, and how many consecutive throttle rounds without a success end the crawl. Non-positive values are ignored. These three are read by the governor on construction **and on each `SafeRealize.reset!`** (i.e. at the start of every `fetch`), so like the others they can be set any time before the crawl rather than before the flavor is autoloaded.
|
|
69
|
-
- **`RELATON_W3C_MAX_THROTTLED`** (default `25`) — how many resources may be lost to rate limiting before `fetch` refuses to save the index. `0` is accepted and means "tolerate none".
|
|
70
|
-
|
|
71
|
-
`embed: true` (always on) inlines each specification into its index page, so the per-spec realize is served from memory rather than an HTTP request — the largest single reduction in request count.
|
|
72
|
-
|
|
73
|
-
### Rate limiting & retries
|
|
74
|
-
|
|
75
|
-
Per-request retries are layered upstream; **pool-wide back-pressure is this
|
|
76
|
-
gem's job** (`Governor`), because upstream has no notion of the crawl as a whole.
|
|
77
|
-
|
|
78
|
-
- **w3c_api** (>= 0.3.3, pinned in the gemspec) builds its HAL client with
|
|
79
|
-
`faraday-retry` for HTTP 403 — the W3C rate-limit signal — plus
|
|
80
|
-
connection/timeout errors, and sends an identifying `User-Agent`. Both landed
|
|
81
|
-
in 0.3.3 (relaton/w3c_api#22, #23); before it, requests went out as
|
|
82
|
-
`Faraday v2.x` (a Cloudflare bot-heuristic trigger, and the most likely reason
|
|
83
|
-
a crawl flipped into the banned mode at all) and the 403 retry never fired.
|
|
84
|
-
- **lutaml-hal** (>= 0.2.5, declared directly in the gemspec because this flavor
|
|
85
|
-
rescues its error classes by name) retries 429 and 5xx with exponential
|
|
86
|
-
backoff. `DataFetcher` **shortens** that policy via
|
|
87
|
-
`W3cApi::Hal.instance.configure_rate_limiting(UPSTREAM_RATE_LIMITING)`: its
|
|
88
|
-
defaults burn ~31 s (1+2+4+8+16) before a 429 ever reaches us, which only
|
|
89
|
-
delays the back-pressure that actually works. `max_retries` stays at 5 — one
|
|
90
|
-
count covers both 429 and 5xx, and a 5xx is still a permanent skip here.
|
|
91
|
-
From 0.3.3 that call rebuilds w3c_api's register itself; **don't** follow it
|
|
92
|
-
with `reset_register`, which would leave the register absent from lutaml-hal's
|
|
93
|
-
`GlobalRegister`, where `Link#realize` raises.
|
|
94
|
-
`max_retry_after` is set too, to `max_delay` (4.0). Left unset, lutaml-hal's
|
|
95
|
-
`DEFAULT_MAX_RETRY_AFTER` (300.0) applies and a large `Retry-After` on a 429
|
|
96
|
-
parks one thread inside the HTTP layer for up to 5 x 300 s with the governor
|
|
97
|
-
hearing nothing — the very thing the compressed chain exists to prevent. The
|
|
98
|
-
server's hint is not discarded: `Governor.retry_after` reads it off the same
|
|
99
|
-
error and uses it as the **floor** of the pool-wide cooldown, under the
|
|
100
|
-
governor's own 900 s cap, so it is honoured once at pool scale. Do **not**
|
|
101
|
-
lower `w3c_api`'s `DEFAULT_RETRY_OPTIONS[:max_interval]` or lutaml-hal's
|
|
102
|
-
`DEFAULT_MAX_RETRY_AFTER` instead — the first turns a polite server hint into
|
|
103
|
-
an immediate give-up, and either imposes a crawl-shaped number on every
|
|
104
|
-
consumer of those gems.
|
|
105
|
-
|
|
106
|
-
**`Governor` (`governor.rb`) is the important piece.** `api.w3.org` is behind
|
|
107
|
-
Cloudflare, whose rate limiting is **bimodal**: a run either never trips it or
|
|
108
|
-
trips it and stays banned for hours. Second-scale backoff cannot climb out of
|
|
109
|
-
that, and per-worker backoff is worse than useless — the retries are what keep
|
|
110
|
-
the limiter engaged. So all workers share one cooldown:
|
|
111
|
-
|
|
112
|
-
- Any thread seeing a 429 opens/extends it; every other thread observes it in
|
|
113
|
-
`#wait`. Cooldowns escalate **per round** (60 s → 120 → … → 900 s cap), where a
|
|
114
|
-
"round" is one open cooldown — four workers tripping the same limiter must not
|
|
115
|
-
fast-forward the ladder four steps.
|
|
116
|
-
- A `Retry-After` header raises the floor but never lowers our own ladder, and is
|
|
117
|
-
itself capped (a hostile value must not park a worker for an hour). Both RFC
|
|
118
|
-
9110 forms are honoured: delta-seconds, and an HTTP-date parsed properly
|
|
119
|
-
(digit-scanning one would pick up the day-of-month).
|
|
120
|
-
- A success calls `#succeeded!`, which clears the cooldown and **decays**
|
|
121
|
-
(halves) the penalty rather than resetting it, so a flapping limiter still
|
|
122
|
-
climbs. **Only a realize that actually went to the network counts** — with a
|
|
123
|
-
`parent_resource`, lutaml-hal serves the object out of the page's `_embedded`
|
|
124
|
-
payload and issues no request (`link.rb`: *"Priority 1: check embedded content
|
|
125
|
-
first"*). That is the common path, one per specification, and counting those
|
|
126
|
-
would reset the ladder between every pair of version-history 429s, so the
|
|
127
|
-
governor could never reach its threshold.
|
|
128
|
-
- After `GIVE_UP_AFTER` consecutive rounds with no success (~15 min) the crawl is
|
|
129
|
-
declared banned: `#exhausted?` goes true, `#wait` stops blocking so shutdown is
|
|
130
|
-
immediate, and `fetch` aborts. The abort message reads `#give_up_rounds`, the
|
|
131
|
-
round count **latched** at that moment — `#throttle_rounds` is live and
|
|
132
|
-
`#succeeded!` resets it, so a straggler success made a give-up at round 5 print
|
|
133
|
-
as *"gave up after 2 consecutive backoffs"*.
|
|
134
|
-
- The mutex is never held across a sleep, and wake-ups are jittered so workers
|
|
135
|
-
don't resume in lockstep.
|
|
136
|
-
|
|
137
|
-
**Giving up stops asking, not just waiting.** `#wait` returning 0.0 once latched
|
|
138
|
-
makes shutdown prompt, but it does not stop a request. `#stopping?` is consulted
|
|
139
|
-
only by the producer loop and at `queue.pop`, so an in-flight spec used to run
|
|
140
|
-
its whole fan-out — `fetch_versions` plus `DataParser`'s seven realize sites —
|
|
141
|
-
against a host that had just banned it. The 2026-08-31 crawl (CI run
|
|
142
|
-
33435903404) spent its last **11 minutes** doing exactly that, which extends the
|
|
143
|
-
Cloudflare ban into the next run; 95 of its 105 `Throttled out` warnings say
|
|
144
|
-
`after 1 attempt(s)`, which in `record_throttle` can only mean `gave_up`.
|
|
145
|
-
|
|
146
|
-
The guard is one line at the top of `SafeRealize#realize`
|
|
147
|
-
(`return nil if SafeRealize.governor.exhausted?`) because that is the **single
|
|
148
|
-
choke point** every crawl request passes through — `fetch_specifications_page`
|
|
149
|
-
is the only other one, and it already returns nil on give-up. One check
|
|
150
|
-
therefore covers every fan-out site, including any added later, and every site
|
|
151
|
-
already tolerates `nil` (the `SafeRealize` contract; `data_parser_spec.rb`
|
|
152
|
-
proves it per site). Per-site checks in `data_parser.rb` would also need a stop
|
|
153
|
-
flag plumbed into `DataParser`, which holds no reference to the fetcher.
|
|
154
|
-
|
|
155
|
-
It checks `governor.exhausted?`, **not** `#stopping?`, and
|
|
156
|
-
`DataFetcher#abandoned?` names that distinction. The two stop reasons want
|
|
157
|
-
opposite things of a spec already in flight: a **Ctrl-C** saves the index, so
|
|
158
|
-
the spec must finish or a half-parsed document is written; a **latched give-up**
|
|
159
|
-
discards everything (`guard_rate_limited` raises), so there is nothing to gain
|
|
160
|
-
from parsing, writing or crawling one more document.
|
|
161
|
-
|
|
162
|
-
`SafeRealize` sits on top of it and makes the distinction that matters: **a
|
|
163
|
-
throttle is not a broken resource.** `Governor::THROTTLE_ERRORS` is the single
|
|
164
|
-
list of what counts — `TooManyRequestsError` **and `ForbiddenError`**, because
|
|
165
|
-
api.w3.org signals rate limiting with 403 as well as 429 (w3c_api's whole retry
|
|
166
|
-
layer is built around that). Blacklisting a 403 would be the same bug as
|
|
167
|
-
blacklisting a 429. It retries a rate-limited realize
|
|
168
|
-
`THROTTLE_ATTEMPTS` times behind the shared cooldown and records a give-up in
|
|
169
|
-
`SafeRealize.throttled` — *never* in `skipped`. An href that a later attempt does
|
|
170
|
-
realize is removed from `throttled` again, so a briefly-throttled-then-recovered
|
|
171
|
-
resource doesn't count against the budget for a document that is in fact present.
|
|
172
|
-
`fetch_specifications_page` is the exception to `THROTTLE_ATTEMPTS`: an index page
|
|
173
|
-
retries until the *governor* gives up, since a 3-attempt cap would pre-empt the
|
|
174
|
-
escalation ladder and report a pagination fault instead of the real reason.
|
|
175
|
-
Everything else is unchanged:
|
|
176
|
-
`NotFoundError` and other terminal `Lutaml::Hal::Error`s (403, 5xx) still go into
|
|
177
|
-
`skipped` (a `Concurrent::Map`) so a broken link isn't re-fetched for every
|
|
178
|
-
reference, and network errors are remembered nowhere so a later reference can try
|
|
179
|
-
again. 5xx deliberately stays a permanent skip — the handful of persistent
|
|
180
|
-
per-resource 500s a healthy crawl sees are broken records, and routing them
|
|
181
|
-
through the governor would open a pool-wide cooldown for each.
|
|
182
|
-
|
|
183
|
-
Successful objects are cached by **w3c_api** (lutaml-hal caches realized objects
|
|
184
|
-
keyed by URL, thread-safely as of lutaml-hal 0.2.1), so `SafeRealize` doesn't
|
|
185
|
-
cache them.
|
|
186
|
-
|
|
187
|
-
> **Why this exists.** The 2026-08-19 `relaton-data-w3c` crawl ran **4 h 56 m**
|
|
188
|
-
> (healthy runs take ~1 h 30 m), permanently blacklisted **1,412** rate-limited
|
|
189
|
-
> resources, and only then aborted on a failed index page. Every 429 warning in
|
|
190
|
-
> that log is ~31 s apart — one exhausted lutaml-hal chain — i.e. *every* request
|
|
191
|
-
> for four hours was rate-limited. Had the last index page happened to succeed,
|
|
192
|
-
> a dataset missing 1,412 documents would have been committed.
|
|
193
|
-
|
|
194
|
-
### Index (`index-v2`, pubid-keyed)
|
|
195
|
-
|
|
196
|
-
`INDEXFILE` is the pubid-backed **`index-v2`**: rows are `Pubid::W3c::Identifier`
|
|
197
|
-
leaves serialized as `_type: pubid:w3c:*` with a `number` (the document slug) and
|
|
198
|
-
an optional verbatim `date`.
|
|
199
|
-
|
|
200
|
-
```yaml
|
|
201
|
-
- :id:
|
|
202
|
-
_type: pubid:w3c:recommendation
|
|
203
|
-
number: xml-names
|
|
204
|
-
date: '20091208'
|
|
205
|
-
:file: data/rec-xml-names-20091208.yaml
|
|
206
|
-
```
|
|
207
|
-
|
|
208
|
-
**Why.** `Relaton::Index::Type#candidates_by_number` sorts and binary-searches
|
|
209
|
-
every row on `id.root.number.to_s`. The old `index-v1` stored plain hashes and
|
|
210
|
-
was built without `pubid_class:`, so `FileIO#sorted` stayed false and every
|
|
211
|
-
lookup scanned all 17,287 rows. Keyed on the pubid slug the corpus splits into
|
|
212
|
-
**1,972 buckets**, the largest holding 101 rows.
|
|
213
|
-
|
|
214
|
-
**The pubid contract.** `Pubid::W3c::Identifier` keeps the slug in `number`.
|
|
215
|
-
It was called `code` until pubid #339 (merged 2026-08-28, `181fd4b9`), which
|
|
216
|
-
renamed it with **no alias** — read `number`, never `code`. The root `Gemfile`
|
|
217
|
-
pins pubid to `main`, so a checkout whose lock predates that merge still has the
|
|
218
|
-
old attribute: run `bundle update pubid`, not `bundle install`, because an
|
|
219
|
-
already-locked git source does not refloat.
|
|
220
|
-
|
|
221
|
-
**Where the consumer reads it: the Pages machine index (relaton#189 pilot).**
|
|
222
|
-
`Bibliography#index` is `find_or_create(:W3C, pages_url: PAGES_URL,
|
|
223
|
-
pubid_class: ...)`, not the `index-v2.zip` under `SOURCE`. A parsed query reads
|
|
224
|
-
`index/manifest.json` and the one `index/shard-NNNNN.json` of its slug from
|
|
225
|
-
`https://relaton.github.io/relaton-data-w3c/`, in memory for 24 h — three
|
|
226
|
-
requests for a lookup, including the document, in place of a 180 KB zip. A Pages
|
|
227
|
-
failure raises `Relaton::RequestError`, with no fallback to the zip. Only the
|
|
228
|
-
case-insensitive `#loose_match?` fallback reads every row (the Pages copy of the
|
|
229
|
-
zip), and only after an exact miss. Documents still come from `SOURCE`.
|
|
230
|
-
`get`/`search` accept a parsed `Pubid::W3c::Identifier` as well as a String, so
|
|
231
|
-
`Relaton::Db` can hand over the pubid it parsed (#189 change 2).
|
|
232
|
-
|
|
233
|
-
**The Pages copy lags the data repo.** The site is rebuilt by the `Deploy`
|
|
234
|
-
workflow, not by the crawler's push, so a document the crawler committed after
|
|
235
|
-
the last deploy is not in any shard yet. On 2026-09-23 the Pages index had
|
|
236
|
-
17,396 rows against 17,444 in `v2/index-v2.zip`: the 48 missing rows
|
|
237
|
-
(`webxrlayers-1`, `rdf12-xml`, …) were committed at 20:01, after the 18:54
|
|
238
|
-
deploy. Until the next deploy those references answer "not found".
|
|
239
|
-
|
|
240
|
-
**Producer and consumer must stay in step.** Both pass
|
|
241
|
-
`pubid_class: ::Pubid::W3c::Identifier` — `DataFetcher#index` and
|
|
242
|
-
`Bibliography#index` (which deserializes the shard rows with it). (`Processor#remove_index_file` passes `url: true` and
|
|
243
|
-
`file:` only: the delete never reads the index, see
|
|
244
|
-
`lib/relaton/index/CLAUDE.md`.) On the producer side
|
|
245
|
-
`#index_primary` stores the pubid **object**: `Relaton::Index::FileIO#save` only
|
|
246
|
-
serializes to the `_type:` shape when the value is an instance of the configured
|
|
247
|
-
`pubid_class`, so handing it a hash writes a v1-shaped file under a v2 name,
|
|
248
|
-
silently, with no narrowing. On the consumer side, omitting `pubid_class:` leaves
|
|
249
|
-
the rows as raw hashes and `FileIO#sorted` false, so `Type#search` stops
|
|
250
|
-
narrowing — also silently.
|
|
251
|
-
|
|
252
|
-
**How a reference is matched.** `Bibliography#best_match` passes the parsed
|
|
253
|
-
pubid to `Index::Type#search`, which is what enables the bsearch; a block alone
|
|
254
|
-
scans everything. It then selects with pubid's asymmetric subset match
|
|
255
|
-
(`Pubid::SubsetMatch`) — the reference on the left, a component it omits matches
|
|
256
|
-
any value:
|
|
257
|
-
|
|
258
|
-
```ruby
|
|
259
|
-
rows = index.search(pubid) { |r| pubid === r[:id] }
|
|
260
|
-
```
|
|
261
|
-
|
|
262
|
-
`date` is W3C's only optional component, so an undated `REC-xml-names` still
|
|
263
|
-
finds the dated row and a dated reference reaches only its own. The maturity
|
|
264
|
-
level is **not** a wildcard: it is the identifier's class, and `===` requires
|
|
265
|
-
the same class, so `WD-`, `REC-` and a bare slug never match each other — the
|
|
266
|
-
contract the bespoke `PubId#==` had for its `stage`/`type`. Measured over the
|
|
267
|
-
whole fixture index (17,303 rows, 1,214,578 comparisons): the subset match and
|
|
268
|
-
the old `matches?(…, ignore: [:date])` agree on every pair. Newest edition wins, with the file path breaking ties because
|
|
269
|
-
undated rows all score 0 and the index sort is not stable.
|
|
270
|
-
|
|
271
|
-
Two things the narrowing cannot do, both handled around it:
|
|
272
|
-
|
|
273
|
-
- **Case.** The bsearch key is case-sensitive; `PubId#==` compared its slug with
|
|
274
|
-
`casecmp?`. When the narrowed range is empty, `#loose_match?` repeats the
|
|
275
|
-
comparison case-insensitively over a full scan (the BIPM `search_index`
|
|
276
|
-
precedent).
|
|
277
|
-
- **Queries that are not identifiers.** `#parse_ref`/`#normalize_ref` absorb a
|
|
278
|
-
URL (`https://www.w3.org/TR/xml-names/`) and a leading `TR-`/`TR/`, which the
|
|
279
|
-
bespoke regex accepted. A URL is not an identifier and `TR` is a path segment
|
|
280
|
-
of one rather than a maturity level, so neither belongs in a pubid grammar —
|
|
281
|
-
they stay here. The publisher prefix is added when absent. A reference pubid
|
|
282
|
-
still rejects is a **miss, not an error**: `parse_ref` returns nil outside the
|
|
283
|
-
transport rescue, so it never becomes a `Relaton::RequestError`.
|
|
284
|
-
|
|
285
|
-
**An unparseable id is an ERROR, never a warning.** `Docidentifier#parse` logs at
|
|
286
|
-
`Util.error` and leaves `#pubid` nil — it must not raise, or an already-published
|
|
287
|
-
record would stop deserializing. `DataFetcher#index_primary` then records the
|
|
288
|
-
failure in `@errors` **keyed by the id, not the output path** (the id is the
|
|
289
|
-
defect; one broken id reaching several files must still report once), and the
|
|
290
|
-
inherited `Core::DataFetcher#report_errors` logs it and raises a tracked GitHub
|
|
291
|
-
issue. The row is skipped rather than indexed unparsed: `Relaton::Index` rejects
|
|
292
|
-
the *whole* index if one row fails to deserialize, so one bad record must cost
|
|
293
|
-
one document, never every lookup. The data file is still written — unindexed,
|
|
294
|
-
not lost.
|
|
295
|
-
|
|
296
|
-
`Core::DataFetcher#report_errors` treats a **String** `@errors` value as the
|
|
297
|
-
message itself; a boolean still means "this field failed for every record" and
|
|
298
|
-
renders as `Failed to fetch <key>`. That is what lets this flavor report a
|
|
299
|
-
specific per-document failure without overriding `report_errors`.
|
|
300
|
-
|
|
301
|
-
### Key Dependencies
|
|
302
|
-
|
|
303
|
-
- **relaton-bib** (~> 2.2.0) — provides base `Bib::Item`, `Bib::Ext`, `Bib::Doctype` and serialization mixins (LutaML model layer)
|
|
304
|
-
- **relaton-core** — provides base `Core::Processor` and `Core::DataFetcher`
|
|
305
|
-
- **relaton-index** — index-based search for bibliographic references; also unpacks the index zip at runtime
|
|
306
|
-
- **pubid** — `Pubid::W3c::Identifier` backs `Docidentifier` and the `index-v2` rows
|
|
307
|
-
- **w3c_api** (~> 0.3.3) — W3C API (HAL/REST) client used by `DataFetcher` to retrieve specifications; owns the `User-Agent`, the 403 retry, and the object cache. The 0.3.3 floor is load-bearing, not cosmetic — see **Rate limiting & retries**.
|
|
308
|
-
- **lutaml-hal** (~> 0.2, >= 0.2.5) — HAL layer beneath w3c_api. Declared directly because this flavor rescues `Lutaml::Hal::*` error classes by name; `ForbiddenError` only exists from 0.2.5, and w3c_api's own `~> 0.2.1` would resolve happily to an older one and NameError at rescue time. 0.2.5 also made `Client#get`'s last-response bookkeeping thread-local (lutaml/lutaml-hal#21), which matters directly to the worker pool.
|
|
309
|
-
|
|
310
|
-
The W3C data is fetched entirely through `w3c_api`; the older RDF/SPARQL/scraping stack (linkeddata, rdf, sparql, shex, mechanize, …) has been removed.
|
|
311
|
-
|
|
312
|
-
### Schema Validation
|
|
313
|
-
|
|
314
|
-
XML output is validated against RelaxNG grammars in `grammars/`:
|
|
315
|
-
- `relaton-w3c-compile.rng` — top-level compiled grammar (includes all others)
|
|
316
|
-
- `relaton-w3c.rng` — W3C-specific overrides (DocumentType restrictions)
|
|
317
|
-
- `basicdoc.rng`, `biblio.rng`, `biblio-standoc.rng` — shared base schemas
|
|
318
|
-
|
|
319
|
-
Tests use [Jing](https://github.com/jing-trang/jing-trang) for RelaxNG validation.
|
|
320
|
-
|
|
321
|
-
### Test Structure
|
|
322
|
-
|
|
323
|
-
Tests use RSpec with:
|
|
324
|
-
- **Round-trip tests** — YAML/XML → object → YAML/XML, verifying lossless serialization
|
|
325
|
-
- **Schema validation** — XML output validated against `grammars/relaton-w3c-compile.rng`
|
|
326
|
-
- **VCR** — recorded HTTP cassettes in `spec/vcr_cassettes/` (never auto-rerecorded; re-record deliberately)
|
|
327
|
-
- **WebMock** — disables external HTTP in tests
|
|
328
|
-
|
|
329
|
-
Test fixtures live in `spec/fixtures/` (YAML and XML files).
|
|
330
|
-
|
|
331
|
-
## Style
|
|
332
|
-
|
|
333
|
-
- Follows [Ribose OSS Ruby style guide](https://github.com/riboseinc/oss-guides) via RuboCop
|
|
334
|
-
- Target Ruby version: 3.1
|
|
335
|
-
- RuboCop config inherits from remote Ribose guide; Rails cops disabled
|
|
336
|
-
|
|
337
|
-
## CI
|
|
338
|
-
|
|
339
|
-
GitHub Actions workflows (auto-generated by Cimas) delegate to shared workflows in `relaton/support`:
|
|
340
|
-
- `rake.yml` — runs tests on push to main and PRs
|
|
341
|
-
- `release.yml` — gem versioning and publishing to RubyGems
|
|
342
|
-
|
|
343
|
-
## Testing
|
|
344
|
-
|
|
345
|
-
- **Index fixture:** `spec/fixtures/index-v2.zip` — the published index verbatim, seeded into the `Relaton::Index` pool by `spec/support/webmock.rb`. Refresh it from the live published `index-v2.zip`; there is no `rake spec:update_index` task in this repo.
|
|
346
|
-
- It is built **with `pubid_class:`**. Without it the rows stay raw hashes and `Type#search` silently stops narrowing, so the suite would pass while exercising something the runtime never does.
|
|
347
|
-
- It is re-seeded in **`before(:each)`**, not only `before(:suite)`: `Index::Pool#type` replaces the pooled entry whenever `actual?` says no, and `DataFetcher#index` asks for the same `:W3C` type with `file:` but no `url:`. `before(:suite)` alone leaves a later example searching a producer index. IANA hit this first.
|
|
348
|
-
- The zip name is read from `Relaton::W3c::INDEXFILE`, but **lazily** — `spec_helper` loads `support/` before the flavor, so touching the constant at file-body time NameErrors.
|
|
349
|
-
- **The HAL object cache is process-global, and the suite clears it per example.**
|
|
350
|
-
`W3cApi::Hal` is a Singleton holding one `Lutaml::Hal::ModelRegister` with an
|
|
351
|
-
in-memory URL cache, registered in the `Lutaml::Hal::GlobalRegister` Singleton.
|
|
352
|
-
`W3cApi::Client.new` carries no state, so a fresh client per example still
|
|
353
|
-
reads that one cache. Left warm it chains the examples together:
|
|
354
|
-
`DataParser#parse` realizes `@spec.links.specification` for the `editionOf`
|
|
355
|
-
relation, so the `vcr: "webrtc-20241008"` examples fetch
|
|
356
|
-
`https://api.w3.org/specifications/webrtc` and the later `vcr: "webrtc"`
|
|
357
|
-
examples then issue no request at all. A cassette recorded in that state holds
|
|
358
|
-
only what the cache was missing, and replays only in that one example order —
|
|
359
|
-
which is how `spec/w3c/vcr_cassettes/webrtc.yml` came to lack the very request
|
|
360
|
-
its own examples make. `spec/w3c/support/hal_cache.rb` clears the cache in a
|
|
361
|
-
`before` hook so each example issues its own requests. **A cassette
|
|
362
|
-
re-recorded without that reset is not reusable.**
|
|
363
|
-
- **`json` must stay below 3 (`relaton.gemspec`).** json 3.0.0 made the
|
|
364
|
-
`JSON.parse` options keyword-only; Faraday's JSON response middleware — the
|
|
365
|
-
one `Lutaml::Hal::Client#create_connection` installs — still passes them
|
|
366
|
-
positionally, up to and including faraday 2.14.3. With json 3 every
|
|
367
|
-
api.w3.org response raises `ArgumentError: wrong number of arguments (given 2,
|
|
368
|
-
expected 1)`, surfaced as `Lutaml::Hal::ParsingError`, and this flavor fetches
|
|
369
|
-
nothing. `spec/w3c/relaton/w3c/faraday_json_spec.rb` guards the constraint.
|
data/lib/relaton/xsf/CLAUDE.md
DELETED
|
@@ -1,189 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## What is relaton-xsf?
|
|
6
|
-
|
|
7
|
-
A Ruby gem for bibliographic retrieval of XMPP XEP (XMPP Extension Protocol) specifications. Part of the Relaton family of gems. Fetches data from https://xmpp.org/extensions/refs/ and the relaton-data-xsf GitHub repository.
|
|
8
|
-
|
|
9
|
-
## Commands
|
|
10
|
-
|
|
11
|
-
- `bundle exec rake spec:xsf` — run this flavor's suite (from the repo root)
|
|
12
|
-
- `bundle exec rake spec:update_index_xsf` — refresh
|
|
13
|
-
`spec/xsf/fixtures/index-v2.zip` from the published index
|
|
14
|
-
- `bundle exec rubocop` — lint
|
|
15
|
-
- `bundle exec rubocop -a` — lint with auto-fix
|
|
16
|
-
- `bin/console` — interactive console with gem loaded
|
|
17
|
-
|
|
18
|
-
## Architecture
|
|
19
|
-
|
|
20
|
-
Namespace: `Relaton::Xsf` (under `lib/relaton/xsf/`). Branch `lutaml-integration` uses the new nested namespace (not the old `RelatonXsf`).
|
|
21
|
-
|
|
22
|
-
Key classes and their base classes from relaton-core:
|
|
23
|
-
|
|
24
|
-
| Class | Base | Role |
|
|
25
|
-
|---|---|---|
|
|
26
|
-
| `Processor` | `Relaton::Core::Processor` | Plugin entry point for relaton registry |
|
|
27
|
-
| `Bibliography` | Module (extends self) | Search & get interface; **parses the reference into a pubid** (`parse_ref`) |
|
|
28
|
-
| `HitCollection` | `Relaton::Core::HitCollection` | Collection of search results |
|
|
29
|
-
| `Hit` | `Relaton::Core::Hit` | Single result; lazy-loads YAML from GitHub |
|
|
30
|
-
| `DataFetcher` | `Relaton::Core::DataFetcher` | Crawls xmpp.org, parses BibXML, saves docs |
|
|
31
|
-
| `Docidentifier` | `Relaton::Bib::Docidentifier` | Exposes the id as a `Pubid::Xsf::Identifier` via `#pubid` |
|
|
32
|
-
| `Item` / `Bibitem` / `Bibdata` | `Relaton::Bib::Item` | Bibliographic item models (lutaml-model based) |
|
|
33
|
-
|
|
34
|
-
## Index (`index-v2`, pubid-keyed)
|
|
35
|
-
|
|
36
|
-
`INDEXFILE` is the pubid-backed `index-v2`: rows are `Pubid::Xsf::Identifier`
|
|
37
|
-
hashes (`_type: pubid:xsf:xep` with `number`). The consumer
|
|
38
|
-
(`HitCollection#index`) and the producer (`DataFetcher#index`) pass
|
|
39
|
-
`pubid_class: ::Pubid::Xsf::Identifier`. Omitting it on the **producer**
|
|
40
|
-
writes v1-shaped rows under a v2 name, silently (`FileIO#save` calls `to_hash`
|
|
41
|
-
only for instances of `pubid_class`); omitting it on the **consumer** leaves
|
|
42
|
-
the rows raw hashes with `FileIO#sorted` false, so every lookup scans all 520.
|
|
43
|
-
`Processor#remove_index_file` passes `url: true` and `file:` only: the delete
|
|
44
|
-
never reads the index (see `lib/relaton/index/CLAUDE.md`).
|
|
45
|
-
|
|
46
|
-
An XEP identifier is a publisher and a number, nothing else — no edition, no
|
|
47
|
-
date, no part, and one row per XEP (520 rows, ids unique). That makes this the
|
|
48
|
-
simplest flavor in the sweep: there is no selection order to preserve, and the
|
|
49
|
-
reference states every component it has, so pubid's asymmetric subset match
|
|
50
|
-
(`ref === row[:id]`, `Pubid::SubsetMatch`) is exact equality after narrowing.
|
|
51
|
-
|
|
52
|
-
### The index guard, and the two rows that are pages rather than XEPs
|
|
53
|
-
|
|
54
|
-
`DataFetcher#add_to_index` parses the docid and, on failure, records
|
|
55
|
-
`@errors[docid]` — which `report_errors` turns into a GitHub issue at the end of
|
|
56
|
-
the crawl (the 3GPP/ECMA precedent) — and skips the row. The data file is still
|
|
57
|
-
written, so a document that cannot be indexed is unindexed, never lost.
|
|
58
|
-
|
|
59
|
-
The guard is not decoration. One unparseable row does not fail that row:
|
|
60
|
-
`Relaton::Index` declares the **whole file** corrupt, deletes it, and hands back
|
|
61
|
-
an **empty** index — measured, 519 good rows plus one unparseable id loads as
|
|
62
|
-
**0 rows**, and the only trace is two INFO lines (`Wrong structure of file …`,
|
|
63
|
-
`Considering … corrupt, removing it`). Every XSF lookup then fails silently.
|
|
64
|
-
|
|
65
|
-
Two published rows are pages rather than XEPs — the XMPP repository's `README`
|
|
66
|
-
and its `xep-xxxx` template, which the crawl mints records for because they sit
|
|
67
|
-
in the same listing. pubid accepts them as the literal numbers `README` and
|
|
68
|
-
`xxxx`, so they carry into `index-v2` like any other row and resolve normally.
|
|
69
|
-
It stayed strict about everything else: `XEP banana` and a typo such as
|
|
70
|
-
`XEP 00O1` are still rejected, which is what keeps an unparseable reference a
|
|
71
|
-
warning rather than a silent miss.
|
|
72
|
-
|
|
73
|
-
Nothing in the published corpus trips the guard today, so a recorded error means
|
|
74
|
-
upstream grew a shape pubid does not know — exactly when an issue is worth
|
|
75
|
-
filing.
|
|
76
|
-
|
|
77
|
-
### Where the reference becomes an identifier
|
|
78
|
-
|
|
79
|
-
`Bibliography.parse_ref` does it, not `HitCollection`. That split is not
|
|
80
|
-
cosmetic: `Type#search_candidates` narrows only when its search argument is
|
|
81
|
-
**not** a `String`, so the string has to become an identifier at the entry
|
|
82
|
-
point. `HitCollection` therefore receives a `Pubid::Xsf::Identifier` (or nil,
|
|
83
|
-
when the reference could not be parsed) and does nothing but the index lookup.
|
|
84
|
-
`Core::HitCollection` documents `ref` as `[String, Pubid]`, so this is the
|
|
85
|
-
interface working as intended.
|
|
86
|
-
|
|
87
|
-
### Pubid-backed docidentifier
|
|
88
|
-
|
|
89
|
-
`Relaton::Xsf::Docidentifier` parses its `content` into a
|
|
90
|
-
`Pubid::Xsf::Identifier` kept in `@pubid`, while the lutaml `content` attribute
|
|
91
|
-
stays a plain string. Parsing is soft — a missing gem or non-XSF content leaves
|
|
92
|
-
`@pubid` nil rather than raising during deserialization. `Item` narrows the
|
|
93
|
-
inherited `docidentifier` attribute to it.
|
|
94
|
-
|
|
95
|
-
It is **purely additive**: `to_yaml` and `to_xml` are byte-identical before and
|
|
96
|
-
after, so no published document or index needs regenerating.
|
|
97
|
-
|
|
98
|
-
**It deliberately implements none of `remove_part!` / `remove_date!` /
|
|
99
|
-
`to_all_parts!`.** An XEP identifier is a publisher and a number and nothing
|
|
100
|
-
else, so there is genuinely nothing for any of them to strip, and
|
|
101
|
-
`Bib::Docidentifier` defaults all three to no-ops for exactly that case. Empty
|
|
102
|
-
overrides would assert a flavor rule that does not exist. Compare
|
|
103
|
-
`Relaton::Ogc::Docidentifier`, which overrides `remove_date!` because OGC really
|
|
104
|
-
does carry a revision.
|
|
105
|
-
|
|
106
|
-
### Lookup
|
|
107
|
-
|
|
108
|
-
`HitCollection#search` follows the ETSI/W3C/OGC idiom:
|
|
109
|
-
|
|
110
|
-
- **Pass the pubid, not the string.** `Type#search_candidates` narrows only
|
|
111
|
-
when the argument is not a `String`, so the plain reference this flavor used
|
|
112
|
-
to pass disabled the binary search however the index was built.
|
|
113
|
-
`pubid_class:` alone fixes nothing; both had to change together.
|
|
114
|
-
- **Matching is exact.** The old `index.search(ref)` compared a **substring**
|
|
115
|
-
of the rendered id, so a bare `001` answered with **11** documents
|
|
116
|
-
(`XEP 0001` and every `XEP 001x`) and `Bibliography#get` took `.first` —
|
|
117
|
-
a truncated reference silently resolved to whichever sorted first. It now
|
|
118
|
-
returns nothing.
|
|
119
|
-
- **An unrecognized reference raises.** Like ISO, ETSI and 3GPP, the parse
|
|
120
|
-
error propagates: relaton-cli rescues `Pubid::Errors::Error` and renders
|
|
121
|
-
`"..." is not a recognized standards identifier`
|
|
122
|
-
(`gems/relaton-cli/lib/relaton/cli/command.rb:324`,
|
|
123
|
-
`subcommand_collection.rb:134`), and `Db#fetch` logs it through the
|
|
124
|
-
`StandardError` arm at `lib/relaton/db.rb:122`. `Pubid::Errors::ParseError`
|
|
125
|
-
includes the marker module `Pubid::Errors::Error`, which is what makes the
|
|
126
|
-
CLI's rescue work for every flavor.
|
|
127
|
-
|
|
128
|
-
Rescuing into a warning and an empty result — which several flavors still do
|
|
129
|
-
— collapses "this identifier is malformed" into "no such document", and a
|
|
130
|
-
caller cannot tell them apart. The parse also happens **before**
|
|
131
|
-
`HitCollection` is constructed, so it is never relabelled as the
|
|
132
|
-
`Relaton::RequestError` that collection's own rescue raises.
|
|
133
|
-
- **Three reference forms are normalized** in `Bibliography#normalize_ref`, because
|
|
134
|
-
`Pubid::Xsf` accepts only the canonical `XEP 0001`:
|
|
135
|
-
|
|
136
|
-
| form | before | after |
|
|
137
|
-
|---|---|---|
|
|
138
|
-
| `XEP 0001` | resolves | resolves |
|
|
139
|
-
| `0001` (bare) | resolves (substring) | resolves — the token is added |
|
|
140
|
-
| `XEP-0001` | **no match** | resolves — the spelling xmpp.org itself uses |
|
|
141
|
-
| `xep 0001` | no match | resolves — the token match is case-insensitive |
|
|
142
|
-
|
|
143
|
-
The hyphen and case forms are new support, not preserved behaviour: the
|
|
144
|
-
substring match compared against `XEP 0001`, which has a space.
|
|
145
|
-
|
|
146
|
-
### Both indexes, one crawl
|
|
147
|
-
|
|
148
|
-
`relaton-data-xsf` publishes **both** from a single crawl: the `index-v2` this
|
|
149
|
-
fetcher writes, and its own `index-v1` built by `build_index_v1.rb` for released
|
|
150
|
-
relaton v2 consumers. Both carry the same 520 rows — building v1 from `data/`
|
|
151
|
-
was originally what kept `XEP README` and `XEP xxxx` available there while they
|
|
152
|
-
were unindexable in v2, and pubid accepting them has since removed that
|
|
153
|
-
asymmetry.
|
|
154
|
-
|
|
155
|
-
Verified against the published `index-v2`: 520 rows, all deserialize, sorted,
|
|
156
|
-
**0** keying on `""`, `XEP 0001` narrows to 1 of 520, and every accepted
|
|
157
|
-
spelling resolves to a file that returns HTTP 200.
|
|
158
|
-
|
|
159
|
-
Data flow: `Processor#get` → `Bibliography.get` → `HitCollection.search` → `Hit#item` → fetches YAML → `Relaton::Bib::Item.from_yaml`
|
|
160
|
-
|
|
161
|
-
DataFetcher flow: Crawls `https://xmpp.org/extensions/refs/`, parses each XML ref via `Relaton::Bib::Converter::BibXml.to_item`, sets `ext.flavor = "xsf"`, saves to disk.
|
|
162
|
-
|
|
163
|
-
Constants: `INDEXFILE = "index-v2"`, `GHDATA_URL` points to relaton-data-xsf `v2` branch.
|
|
164
|
-
|
|
165
|
-
## Testing
|
|
166
|
-
|
|
167
|
-
- **Index fixture:** `spec/xsf/fixtures/index-v2.zip` is seeded into the
|
|
168
|
-
`Relaton::Index` pool by `spec/xsf/support/webmock.rb` — see **Index
|
|
169
|
-
(`index-v2`, pubid-keyed)** above for why it is re-seeded per example and
|
|
170
|
-
built with `pubid_class:`. Refresh it with `rake spec:update_index_xsf`.
|
|
171
|
-
- RSpec with VCR cassettes (`spec/vcr_cassettes/`) for HTTP interactions
|
|
172
|
-
- WebMock disables all external network connections
|
|
173
|
-
- Fixtures in `spec/fixtures/` (item.yaml, bibdata.xml, bibitem.xml)
|
|
174
|
-
- Round-trip tests verify YAML→Item→YAML and XML→Item→XML fidelity
|
|
175
|
-
- `DataFetcher` is lazily required — specs that test it must `require "relaton/xsf/data_fetcher"` explicitly
|
|
176
|
-
- Same for `Processor` — `require "relaton/xsf/processor"`
|
|
177
|
-
|
|
178
|
-
## Key dependencies
|
|
179
|
-
|
|
180
|
-
- `relaton-core` — abstract base classes (Processor, HitCollection, Hit, DataFetcher)
|
|
181
|
-
- `relaton-bib` — bibliographic models, XML/YAML serialization (lutaml-model based)
|
|
182
|
-
- `relaton-index` — index management for quick document lookups
|
|
183
|
-
- `mechanize` — HTTP fetching and HTML parsing
|
|
184
|
-
|
|
185
|
-
## Style
|
|
186
|
-
|
|
187
|
-
- RuboCop with relaton shared config (inherits from riboseinc/oss-guides)
|
|
188
|
-
- Target Ruby version: 3.1
|
|
189
|
-
- Logging via `Relaton::Xsf::Util` (extends `Relaton::Bib::Util`, PROGNAME = "relaton-xsf")
|