relaton 3.1.1 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (67) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/frontend_assets.rb +7 -6
  7. data/lib/relaton/db.rb +31 -14
  8. data/lib/relaton/ecma/bibliography.rb +2 -2
  9. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  10. data/lib/relaton/iala/bibliography.rb +1 -1
  11. data/lib/relaton/iana.rb +1 -1
  12. data/lib/relaton/iec/data_parser.rb +41 -2
  13. data/lib/relaton/itu/README.adoc +1 -1
  14. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  15. data/lib/relaton/oasis/bibliography.rb +1 -1
  16. data/lib/relaton/ogc/hit_collection.rb +1 -1
  17. data/lib/relaton/ogc.rb +1 -1
  18. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  19. data/lib/relaton/plateau/hit_collection.rb +1 -1
  20. data/lib/relaton/sae.rb +1 -1
  21. data/lib/relaton/version.rb +1 -1
  22. data/lib/relaton/w3c/bibliography.rb +1 -1
  23. data/lib/relaton/xsf/bibliography.rb +1 -1
  24. data/lib/relaton/xsf.rb +1 -1
  25. metadata +32 -46
  26. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  27. data/lib/relaton/adobe/CLAUDE.md +0 -104
  28. data/lib/relaton/bib/CLAUDE.md +0 -150
  29. data/lib/relaton/bipm/CLAUDE.md +0 -187
  30. data/lib/relaton/bsi/CLAUDE.md +0 -35
  31. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  32. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  33. data/lib/relaton/cen/CLAUDE.md +0 -219
  34. data/lib/relaton/cie/CLAUDE.md +0 -90
  35. data/lib/relaton/core/CLAUDE.md +0 -140
  36. data/lib/relaton/db/CLAUDE.md +0 -385
  37. data/lib/relaton/doi/CLAUDE.md +0 -105
  38. data/lib/relaton/easc/CLAUDE.md +0 -85
  39. data/lib/relaton/ecma/CLAUDE.md +0 -217
  40. data/lib/relaton/etsi/CLAUDE.md +0 -219
  41. data/lib/relaton/evs/CLAUDE.md +0 -120
  42. data/lib/relaton/gb/CLAUDE.md +0 -47
  43. data/lib/relaton/gost/CLAUDE.md +0 -100
  44. data/lib/relaton/iala/CLAUDE.md +0 -97
  45. data/lib/relaton/iana/CLAUDE.md +0 -105
  46. data/lib/relaton/iec/CLAUDE.md +0 -13
  47. data/lib/relaton/ieee/CLAUDE.md +0 -202
  48. data/lib/relaton/ietf/CLAUDE.md +0 -382
  49. data/lib/relaton/iho/CLAUDE.md +0 -76
  50. data/lib/relaton/index/CLAUDE.md +0 -208
  51. data/lib/relaton/isbn/CLAUDE.md +0 -47
  52. data/lib/relaton/iso/CLAUDE.md +0 -38
  53. data/lib/relaton/itu/CLAUDE.md +0 -685
  54. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  55. data/lib/relaton/jis/CLAUDE.md +0 -64
  56. data/lib/relaton/logger/CLAUDE.md +0 -37
  57. data/lib/relaton/nist/CLAUDE.md +0 -194
  58. data/lib/relaton/oasis/CLAUDE.md +0 -251
  59. data/lib/relaton/ogc/CLAUDE.md +0 -242
  60. data/lib/relaton/oiml/CLAUDE.md +0 -47
  61. data/lib/relaton/omg/CLAUDE.md +0 -137
  62. data/lib/relaton/plateau/CLAUDE.md +0 -126
  63. data/lib/relaton/sae/CLAUDE.md +0 -91
  64. data/lib/relaton/sdo/CLAUDE.md +0 -81
  65. data/lib/relaton/un/CLAUDE.md +0 -91
  66. data/lib/relaton/w3c/CLAUDE.md +0 -369
  67. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,382 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## What This Gem Does
6
-
7
- The `Relaton::Ietf` flavor fetches and parses IETF bibliographic data (RFCs, Internet-Drafts, BCPs, FYIs, STDs) into the Relaton data model. Models are `lutaml-model`-based.
8
-
9
- ## Commands
10
-
11
- The suite runs from its own directory (`spec/ietf/`), as every flavor suite does.
12
-
13
- ```bash
14
- bundle exec rake spec:ietf # run this flavor's suite
15
- cd spec/ietf && bundle exec rspec -I . relaton/ietf/bibxml_parser_spec.rb # single file
16
- cd spec/ietf && bundle exec rspec -I . relaton/ietf/rfc/entry_spec.rb:19 # single example
17
- cd spec/ietf && bundle exec rspec -I . -e "creates primary docid" # by description
18
- bundle exec rubocop # lint
19
- ```
20
-
21
- ## Architecture
22
-
23
- ### Data Model (Lutaml::Model)
24
-
25
- All models use `Lutaml::Model::Serializable` with `attribute` declarations and `xml do ... end` blocks for serialization. The IETF classes extend base `Relaton::Bib` classes:
26
-
27
- - `Relaton::Ietf::ItemData < Bib::ItemData` — core bibliographic data
28
- - `Relaton::Ietf::Item < Bib::Item` — adds `ext: Ext` attribute
29
- - `Relaton::Ietf::Bibdata < Item` / `Bibitem < Item` — include shared serialization concerns
30
- - `Relaton::Ietf::Ext` — IETF extensions: `doctype`, `flavor`, `stream`, `area`, `ipr`, `pi`
31
- - `Relaton::Ietf::Doctype` — types: `"rfc"`, `"internet-draft"`
32
-
33
- ### Namespace Resolution
34
-
35
- Converter classes use `Bib::NamespaceHelper` which resolves `namespace` by taking the first two segments of the class name. For `Relaton::Ietf::BibXMLParser::FromRfcxml`, `namespace` returns `Relaton::Ietf`, so `namespace::ItemData` → `Relaton::Ietf::ItemData`, `namespace::Ext` → `Relaton::Ietf::Ext`, etc.
36
-
37
- ### Key Flows
38
-
39
- 1. **Single document lookup**: `Processor#get` → `Bibliography.get(code)` → `Scraper.scrape_page` → parses the reference with `Pubid::Ietf` → searches the one combined `index-v2` from `relaton-data-ietf` → fetches that row's YAML; a versioned draft the index lacks is read from bib.ietf.org (see below)
40
-
41
- 2. **Bulk data fetching**: `DataFetcher` extends `Relaton::Core::DataFetcher` with three datasets:
42
- - `ietf-rfcsubseries` / `ietf-rfc-entries`: parse `rfc-index.xml` via `Rfc::Index` / `Rfc::Entry#to_item`
43
- - `ietf-internet-drafts`: parse local BibXML files via `BibXMLParser.parse`
44
-
45
- 3. **BibXML parsing** (`BibXMLParser` module):
46
- - `parse(xml)` — parses `<reference>` elements via `FromRfcxml` converter
47
- - `parse_rfc(xml)` — parses `<rfc>` root documents via `FromRfc` converter
48
- - Converters inherit from `Bib::Converter::BibXml::FromRfcxml` with IETF overrides for contributor/person/org handling
49
-
50
- 4. **RFC Index parsing** (`Rfc::Entry#to_item`): converts RFC editor index entries (BCP/FYI/STD/RFC) into `ItemData`, handling subseries vs full RFC entries differently
51
-
52
- ### Converter Inheritance Chain
53
-
54
- ```
55
- Bib::Converter::BibXml::FromRfcxml # base: handles <reference> generically
56
- └── Ietf::BibXMLParser::FromRfcxml # IETF overrides: publisher, org recognition, person names
57
- └── Ietf::BibXMLParser::FromRfc # <rfc> root: uses doc_name instead of anchor, no ref-level series_info
58
- ```
59
-
60
- `FromRfc` must override all methods that access `@reference.anchor`, `@reference.target`, `@reference.format`, or `@reference.series_info` since `Rfcxml::V3::Rfc` lacks these attributes (unlike `Rfcxml::V3::Reference`).
61
-
62
- ## Unified index (relaton#109)
63
-
64
- Producer and consumer are both on the pubid `index-v2` now. `DataFetcher` writes
65
- it; `Scraper` reads one combined index from `relaton-data-ietf` — replacing the
66
- three per-type `index-v1` files (`:RFC`/`:RSS`/`:IDS`) it used to read from
67
- `relaton-data-{rfcs,rfcsubseries,ids}`. Those three keep publishing `index-v1`
68
- untouched, for released relatons.
69
-
70
- `INDEXFILE` is a single constant again (`index-v2`); the `INDEXFILE_V2` that
71
- existed while the two halves were split is gone.
72
-
73
- Decisions taken (2026-08-18 ... 2026-08-24):
74
-
75
- | | |
76
- |---|---|
77
- | Index structure | **unchanged** - `:id` + `:file` only. No `docids[]`, no summary fields; a third key invalidates the whole index for every released gem |
78
- | `:id` shape | structured pubid hash (`Pubid::Ietf::Identifier#to_hash`) |
79
- | Row per record | keyed by the record's **primary** docidentifier. A sub-series id resolves to its own container record, *not* to a constituent RFC - 42 of 365 sub-series include more than one RFC; consumers follow `relation: includes` |
80
- | DOI ids | omitted - not `pubid:ietf`-representable, and one unparseable row kills the index. DOIs are served by the `doi` flavor |
81
- | Who builds it | `DataFetcher`, during the crawl - not a separate pass over the written records, which would be a second answer to "what is this record's id" |
82
- | Three type repos | untouched: they pin `relaton/relaton-ietf`, not this monorepo, so they keep emitting `index-v1` |
83
-
84
- **Resolved 2026-08-26.** `relaton-data-ietf`'s RFC and sub-series records were
85
- schema-invalid — missing `ext.doctype`, and RFC committee organizations carrying
86
- `subdivision` with no `name` — because that repo converted the `ietf-tools`
87
- v1.2.3 mirrors rather than crawling. It now runs `Relaton::Ietf::DataFetcher`
88
- directly, which synthesises `doctype`, `status`, `stream`, the organization names
89
- and the resolved WG titles. The two `pending` examples in `ietf_spec.rb` reported
90
- `FIXED` on the first run against the new corpus and have been removed.
91
-
92
- Two consequences of that migration to know about: record filenames are now
93
- lowercase *and* unpadded (`data/RFC8341.yaml` -> `data/rfc8341.yaml`,
94
- `data/STD0066.yaml` -> `data/std66.yaml`), since `output_file` derives them from
95
- the docnumber; and the record content changed enough that the three XML fixtures
96
- had to be regenerated.
97
-
98
- ### The consumer: `Scraper` queries with parsed identifiers
99
-
100
- ```ruby
101
- Relaton::Index.find_or_create(
102
- :IETF, url: "#{IETF}#{INDEXFILE}.zip", file: "#{INDEXFILE}.yaml",
103
- pubid_class: ::Pubid::Ietf::Identifier
104
- )
105
- ```
106
-
107
- `parse_id` turns the reference into a `Pubid::Ietf::Identifier` before searching,
108
- and **that is not cosmetic**: `Type#search_candidates` narrows only for non-String
109
- queries, so a String falls into `match_item`'s `item[:id].to_s.include?(id)`, which
110
- renders every pubid in the index on every lookup. Measured against the live
111
- 177k-row index: **~43 s per reference with a String, sub-millisecond with a parsed
112
- id.** A String query also returns *nothing* if `pubid_class:` is omitted, since the
113
- rows would then be raw hashes.
114
-
115
- `parse_id` also normalises the Internet-Draft spellings - `I-D.<slug>` and
116
- `I-D <slug>` - onto the `draft-...` form, adding the `draft-` stem when absent
117
- (`I-D.ietf-quic-transport`, the bibxml anchor and the `docnumber` records carry).
118
- The old plain-string index matched that by substring; matching is exact now.
119
- A reference pubid cannot parse returns nil, so `Bibliography.get "CN 8341"` logs
120
- "Not found." rather than raising.
121
-
122
- ### An index miss on a versioned draft reads bib.ietf.org (relaton-ietf#125)
123
-
124
- The data repo crawls the rsync `bibxml-ids` snapshot, and that snapshot keeps
125
- only some versions of each draft: `draft-ietf-vcarddav-vcardrev` has only `-22`,
126
- so `-06` was "Not found." When `fetch_doc` misses and the query is a
127
- `Pubid::Ietf::Identifiers::InternetDraft` **with a version**, `fetch_bibxml` reads
128
- `https://bib.ietf.org/public/rfc/bibxml-ids/reference.I-D.<draft>-NN.xml`
129
- (`BIBXML_IDS`). That service resolves any version the datatracker knows and
130
- answers 404 otherwise; the datatracker's own `doc/bibxml3/...` URL answers 404
131
- for these old versions. The parse is the crawler's: `BibXMLParser.to_utf8`
132
- (the CP1252 repair, shared with `DataFetcher#read_bibxml`), `BibXMLParser.parse`,
133
- and the version from the identifier, as `process_singleton` sets it.
134
-
135
- Only a 404 is "not found". Any other non-200 (429, 5xx) raises
136
- `Relaton::RequestError`, so `Db#net_retry` retries it and `Db::Cache` does not
137
- store an outage as a not-found row. A 200 whose body does not parse logs a
138
- warning and returns nil; that rescue names the parse errors and is not
139
- `StandardError`, so a transport error still becomes `RequestError`.
140
-
141
- RFCs, sub-series and unversioned drafts do **not** fall back: the index holds
142
- all of them, so a miss is a real "not found". A record read this way has no
143
- `updates`/`updatedBy` neighbours — the crawl links those, and one fetch cannot.
144
- Old bibxml carries no `target`, so such a record has a `TXT` URI and no `src`.
145
-
146
- ### The producer: `DataFetcher` writes `index-v2`
147
-
148
- `DataFetcher` builds the index during the crawl, from the identifier it already
149
- holds for each record — `record_index_entry` parses the primary docidentifier and
150
- `add_or_update`s the pubid:
151
-
152
- ```ruby
153
- Relaton::Index.find_or_create(
154
- :IETF, file: "#{INDEXFILE}.yaml", pubid_class: ::Pubid::Ietf::Identifier
155
- )
156
- ```
157
-
158
- **Both halves are load-bearing.** `FileIO#save` serialises an id to its `_type:`
159
- hash only when it is an instance of the configured `pubid_class`, so passing a
160
- String to a `pubid_class:` index writes a v1-shaped file under a v2 name — no
161
- error, just an index that never narrows.
162
-
163
- Three properties worth keeping:
164
-
165
- - **Key on the primary docidentifier**, which `serialize_and_write` already
166
- picks. Not the first entry: draft records carry an `I-D.foo` anchor id (not a
167
- pubid grammar) and their unversioned series id, which is the *primary* of a
168
- separate aggregator record — keying on either collides on one `id.to_s` and
169
- `add_or_update` silently overwrites `:file`.
170
- - **DOIs stay out.** `10.17487/RFC3986` is not a `pubid:ietf` identifier, and one
171
- unparseable row invalidates the whole index. The primary is never the DOI, so
172
- this is already right — don't "improve" it into indexing every docidentifier.
173
- - **Sub-series key to their own record.** `STD 66` → the STD container, not a
174
- constituent RFC: 42 of the 365 published sub-series include more than one, so
175
- there is no single document to alias to. Consumers follow `relation: includes`.
176
-
177
- A record whose id pubid rejects is written but not indexed, warned about, and
178
- tallied for a summary at the end of the crawl. The load is all-or-nothing —
179
- `deserialize_id` raises on the first bad id and `load_index` then rejects the
180
- entire index — so one malformed upstream record must cost one document, never
181
- every lookup. All 176,862 published ids parse today; this guards against drift.
182
-
183
- **Blast radius, worth re-checking before touching this.** The six data repos
184
- feeding today's released consumers do *not* use this monorepo: `ietf-tools/*` pin
185
- `relaton/relaton-ietf` @ `main` (`RelatonIetf::DataFetcher`), and
186
- `relaton/relaton-data-{rfcs,rfcsubseries,ids}` pin it @ `lutaml-integration` —
187
- both branches carry their own `INDEXFILE = "index-v1"` and their own fetcher. Only
188
- `relaton/relaton-data-ietf` runs this one. So the producer flip needs no
189
- dual-write and no cutover window.
190
-
191
- ### Still outstanding
192
-
193
- Retiring `relaton-data-{rfcs,rfcsubseries,ids}`'s `index-v1` files, and their
194
- `data-index/configs.yml` rows, waits until a relaton carrying this change has
195
- shipped — see `/work/HANDOFFS/relaton__support__ietf-index-config-and-cron.md`.
196
-
197
- ## Filename collisions: workers write, the parent settles the names
198
-
199
- `Core::DataFetcher#output_file` sanitizes `.`, `,`, `/`, `:`, `(`, `)`, `-` and
200
- whitespace all to `-`, so two **distinct** docids can want one path. This flavor
201
- is the only one that writes from forked processes, so it is the only one where
202
- that could destroy a record without a word.
203
-
204
- It did. In the first production crawl of `relaton-data-ietf`,
205
- `data/draft-ietf-6man-hbh-processing.yaml` — the aggregator for a draft with 21
206
- versions — was written and then overwritten by a singleton built from the
207
- malformed sibling id `draft-ietf-6man-hbh-processing-`, which sanitizes to the
208
- same name. The published corpus came out 177,362 index rows over 177,361 files,
209
- and nothing in the log said so. Exactly one such group exists in that corpus;
210
- the mechanism, not the blast radius, is the problem.
211
-
212
- Three pieces close it, and all three are load-bearing:
213
-
214
- 1. **`@cross_process = true`, set in `fetch_ieft_internet_drafts` before the
215
- first `parallelize`** so the forked children inherit it. It tells
216
- `Core::DataFetcher#write_unique` that an existing file may belong to a peer
217
- and must never be overwritten.
218
- 2. **`serialize_and_write` writes through `write_unique`** and reports two extra
219
- keys: `docid` (the id the file was written under) and `plain_file` (the name
220
- it would have taken uncontested). `docid` is **not** `index_id` — `id` falls
221
- back through `docnumber` and `formattedref`, so on the RFC path the two
222
- differ (`RFC0001` vs `RFC 1`, i.e. stems `rfc0001` and `rfc-1`). Keying the
223
- reconciliation on `index_id` renames records to the wrong names.
224
- 3. **`reconcile_output_files`, in the parent, before `record_index_entry`.** A
225
- worker only learns that a path was taken, never which clashing docid deserves
226
- it, so left alone the winner would follow the race and the two filenames would
227
- swap between crawls. The parent sees every docid, so it decides: within a
228
- group of records that wanted one path, the alphabetically first docid keeps
229
- it and the rest take their digest variant.
230
-
231
- Details of (3) that are easy to get wrong:
232
-
233
- - It runs over **every** group, not just clashing ones. A lone record that fell
234
- back to a digest path must get its plain name back, or the published filename
235
- churns and the old file is orphaned. `next if r[:file] == target` makes that
236
- free for ~100% of records.
237
- - **Losers move first.** A loser may be sitting *on* the plain path; moving the
238
- winner there first would destroy it.
239
- - Two results for the **same** docid are collapsed, not split: the stray file is
240
- deleted and one warning is emitted, which is what the crawl did before. The
241
- stray may already be **gone** — the fallback name is keyed on the docid, so two
242
- workers that both lost the race to the plain path land on one name and the
243
- first of them renames it onto the target. The result must follow the document
244
- to the target anyway, or the index gets a row pointing at nothing.
245
- - Sorting *unique docids* leaves no tie for Ruby's unstable `sort_by` to resolve
246
- differently between runs.
247
- - A failed rename warns and carries on. A multi-hour crawl must not die at the
248
- last step.
249
-
250
- `report_collisions` restates the total once at the end of the crawl, beside
251
- `report_unparsed` and `report_unindexed` — one line for a crawl that writes
252
- ~177k records.
253
-
254
- The sequential paths (`fetch_ieft_rfcsubseries`, `fetch_ieft_rfcs`) get no
255
- reconciliation pass and need none: one process, so `unique_output_file` is
256
- already deterministic there. They do now emit a digest filename for a genuine
257
- cross-docid collision instead of overwriting, which follows `rfc-index`
258
- iteration order — stable for a given input.
259
-
260
- ## Thin records inherit from their constituents
261
-
262
- Two builders produce records for documents that have **no upstream source of
263
- their own**, and both would otherwise publish without date, doctype or source:
264
-
265
- | Builder | File | Inherits from |
266
- |---|---|---|
267
- | `to_subseries_item` | `rfc/entry.rb` | the newest RFC in its `is-also` |
268
- | `build_unversioned_doc` | `data_fetcher.rb` | the newest version in `sorted` |
269
-
270
- This is **not** a parsing gap to go fix in the XML mapping. A sub-series entry in
271
- `rfc-index.xml` is only a pointer:
272
-
273
- ```xml
274
- <bcp-entry>
275
- <doc-id>BCP0003</doc-id>
276
- <is-also><doc-id>RFC1915</doc-id></is-also>
277
- </bcp-entry>
278
- ```
279
-
280
- The RFC Editor publishes no date, title, author or status for it, because a
281
- sub-series *has* no metadata of its own — it is a label on one or more RFCs.
282
- Unversioned draft aggregators are further still from an upstream document: they
283
- are synthesised from the versions found on disk.
284
-
285
- Neither site needs a second pass or another read of the corpus — `rfc_index` and
286
- `sorted` already hold the constituents at build time. (A downstream
287
- implementation in `relaton-data-ietf`'s crawler did use a second pass over
288
- written files, because its groups could span source repos; that constraint does
289
- not exist here.)
290
-
291
- **Constituent lookup is normalised on both sides, deliberately.**
292
- `Entry.squish` folds case and strips whitespace and dots. Relation targets and the
293
- docids they reference disagree in the published corpus (`dyndNS` vs `dyndns`),
294
- which left 28 documents undated downstream on a strict lookup — and, in
295
- `build_relations`, silently downgraded a full constituent bibitem to a minimal
296
- one.
297
-
298
- `rfc_index` is therefore **keyed by `Entry.squish(doc_id)`, and the caller builds
299
- it** — `fetch_ieft_rfcsubseries` does so once for the crawl. Normalising inside
300
- `Entry` instead would rebuild a ~9,800-row table for each of the 367 sub-series
301
- (measured: ~161 MB retained, ~6 s), all of it the same table.
302
-
303
- Why it matters beyond tidiness: the Pages index sorts by date, so undated records
304
- sort as one undifferentiated block, and a record with no `ext.doctype` renders
305
- with no document type at all.
306
-
307
- Inherited values are `dup`'d rather than shared with the constituent's own bib —
308
- aliasing them would make a later edit to the aggregator mutate the `-NN` record.
309
-
310
- Verified against the live `rfc-index.xml`: **367/367 sub-series dated**, 367/367
311
- with full constituent bibitems, in 1.4 s; `BCP 3` → 1996-02, `STD 51` → 1994-07,
312
- `STD 66` → 2005-01 (RFC 3986's date).
313
-
314
- ## bibxml encoding: transcode, don't scrub, don't skip
315
-
316
- Some bibxml files declare `encoding='UTF-8'` and carry **Windows-1252** bytes —
317
- smart quotes and accented Latin letters. `File.read(encoding: "UTF-8")` only
318
- *tags* a string; it validates nothing, so those reached lutaml as invalid UTF-8
319
- and `Rfcxml::V3::Reference.from_xml` raised `Lutaml::Model::InvalidFormatError`.
320
-
321
- `read_bibxml` transcodes them through `BibXMLParser.to_utf8`
322
- (`bibxml_parser.rb`, shared with the scraper's bib.ietf.org fallback). Three things make that the
323
- right call rather than `scrub`:
324
-
325
- - **All of them decode losslessly as CP1252** — 125 of the ~122k published files
326
- are affected, 0 unmappable.
327
- - **The bytes present prove CP1252, not Latin-1.** `0x91`–`0x94` (`‘ ’ “ ”`) are
328
- control characters in Latin-1.
329
- - **`scrub` is lossy.** It replaces each byte with `U+FFFD`: `client’s` becomes
330
- `client�s`, `Muñoz` becomes `Mu�oz`. Author surnames are among the
331
- affected fields. Verified on a real file — scrub and transcode yield the same
332
- *length*, so a length check will not catch the damage.
333
-
334
- Don't copy `lib/relaton/bipm/si_brochure_parser.rb:33-35`, whose
335
- `force_encoding` guard is a no-op. The `scrub` idiom at
336
- `lib/relaton/itu/data_crawler_r.rb:588-599` is right *there* because it repairs
337
- scraped HTML with no recoverable original; here there is one.
338
-
339
- ### Why the rescue is per file, and why the tally is in the parent
340
-
341
- `parse_bibxml` rescues `StandardError` — not a narrow list. lutaml raises
342
- `InvalidFormatError` on bad bytes, but the converter also runs regexes over
343
- parsed text (`parse_surname_initials` and friends), which raise
344
- `ArgumentError: invalid byte sequence` on anything that slips through.
345
-
346
- It has to be **inside** the worker. The drafts path runs under `Parallel.map`,
347
- which discards every result from the pass when a worker raises — so one bad file
348
- out of ~167k meant no index at all, plus the already-written records orphaned on
349
- disk, because `record_index_entry` and `index.save` only run after both passes
350
- return. A rescue around `parallelize` would salvage "the crawl didn't die", not
351
- "the other 166,999 files got indexed".
352
-
353
- The skip therefore rides back as `{ unparsed: path, error: }` and is counted in
354
- the parent, the way `parse_pubid`'s result rides back as `result[:pubid]` — a
355
- counter incremented in a worker process is lost. `report_unparsed` warns once
356
- with a count and a sample, not once per file.
357
-
358
- A failed version is dropped from `sorted` **before** `link_neighbor_relations`
359
- and `build_unversioned_doc` run, since both dereference `entry[:bib]`. Knock-on
360
- to accept: neighbours then link across the gap, and if the dropped file was the
361
- newest, the aggregator inherits from the previous version. With CP1252 recovery
362
- that should approach zero — which is the argument for recovering over skipping.
363
-
364
- Verified end to end on 41,408 real draft files (16 invalid UTF-8): the crawl
365
- completes in 57 s, writes 46,461 records and 46,461 index rows, **0 skipped**,
366
- and the affected files come through with their real characters —
367
- `\x93Access Node Contr` → `“Access Node Contr`.
368
-
369
- ## Testing Patterns
370
-
371
- - **Index fixture:** `spec/ietf/fixtures/ietf-index-v2.zip` - the combined index, copied **wholesale** from `relaton-data-ietf` rather than curated, since a subset goes stale and surfaces as a 404 on the document fetch. `spec/ietf/support/webmock.rb` deserialises it into pubid ids, sorts by the narrowing key, sets `sorted = true` (without which every lookup is a ~40 s scan) and pre-loads it into the pool as `:IETF` in `before(:suite)`; `spec/ietf/support/vcr.rb` ignores requests ending `index-v2.zip`. Refresh by re-downloading that zip. There is **no** `rake spec:update_index` task in this repo.
372
- - **VCR cassettes** in `spec/ietf/vcr_cassettes/` record HTTP interactions; tests use `vcr: "cassette_name"` metadata
373
- - **WebMock** disables net connections by default
374
- - **Fixtures** in `spec/ietf/fixtures/` — XML/YAML expected outputs; many tests auto-generate fixtures on first run (`File.write file, xml unless File.exist? file`)
375
- - **An oracle for `read_bibxml` must read binary.** The method is byte-verbatim (`File.binread`), so comparing it against `File.read(path, encoding: "UTF-8")` compares two different strings **on Windows only**: Ruby opens a file in text mode there and collapses CRLF to LF. The repo declares no `.gitattributes` and the GitHub Windows runner image defaults to `core.autocrlf=true`, so a fixture stored with LF is checked out CRLF — which failed all three `windows-latest` jobs while every Linux and macOS job passed. Use `File.read(path, mode: "rb", encoding: "UTF-8")`. macOS cannot reproduce this by checking the fixture out as CRLF, because its text-mode read does not translate; emulate Windows explicitly with `File.read(path, encoding: "UTF-8", universal_newline: true)`. The two `keeps CRLF line endings verbatim` examples pin the contract with `binwrite` temp files, independent of the checkout — one per branch, since the CP1252 repair rebuilds the string run by run and is where a line ending is likeliest to be lost. Note that a byte-equality oracle alone cannot catch a dropped `force_encoding`: `rfc.xml` is pure ASCII, and Ruby compares two ASCII-only strings as equal across encodings, so the example asserts `encoding` separately.
376
- - **Schema validation** via `ruby-jing` against the shared RNG grammars in the repo-root `grammar/` (`Jing.new "../../grammar/relaton-ietf-compile.rng"`, relative to the suite's CWD `spec/ietf/`)
377
- - **Shared examples** for org/person parsing in `bibxml_parser_spec.rb` (`parse_org`, `parse_person`)
378
- - **`canon`** gem used for XML comparison (`be_equivalent_to` matcher)
379
-
380
- ## RuboCop
381
-
382
- Inherits from `rubose/rubocop-rubose` (Ribose OSS config). Target Ruby 3.1. Rails cops disabled.
@@ -1,76 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## What This Is
6
-
7
- relaton-iho is a Ruby gem that fetches and serializes IHO (International Hydrographic Organization) standards metadata. It is a "flavor" gem in the Relaton ecosystem, extending relaton-bib with IHO-specific bibliographic data models.
8
-
9
- ## Common Commands
10
-
11
- ```bash
12
- bundle exec rspec # run all tests
13
- bundle exec rspec spec/relaton/iho/item_spec.rb # run a single spec file
14
- bundle exec rspec spec/relaton/iho/item_spec.rb:15 # run a single example by line
15
- bundle exec rubocop # lint
16
- bundle exec rubocop -a # lint with auto-fix
17
- bundle exec rake install # install gem locally
18
- ```
19
-
20
- ## Architecture
21
-
22
- ### Flavor Pattern
23
-
24
- This gem follows the Relaton flavor pattern: it subclasses base classes from `relaton-bib` under the `Relaton::Iho` namespace. The key mechanism is `NamespaceHelper` (from relaton-bib), which dynamically resolves the module namespace from the class name. This means `Relaton::Iho::ItemData#to_xml` automatically delegates to `Relaton::Iho::Bibdata.to_xml` (not `Relaton::Bib::Bibdata`).
25
-
26
- Core model classes and their base classes:
27
- - `Item < Bib::Item` — main bibliographic item, uses `model ItemData`
28
- - `ItemData < Bib::ItemData` — data container enabling namespace-aware `to_xml`/`to_yaml`/`to_json`
29
- - `ItemBase < Item` — stripped-down item for use inside relations (no id, schema_version, fetched, ext)
30
- - `Bibitem < Item` — XML bibitem serialization (includes `Bib::BibitemShared`)
31
- - `Bibdata < Item` — XML bibdata serialization (includes `Bib::BibdataShared`)
32
- - `Relation < Bib::Relation` — overrides `bibitem` attribute to use `Iho::ItemBase`
33
- - `Doctype < Bib::Doctype` — IHO document type vocabulary
34
- - `Ext < Bib::Ext` — IHO-specific extension data (doctype, commentperiod)
35
- - `Processor < Core::Processor` — Relaton processor for registry integration
36
- - `HashParserV1` — converts v1-format YAML/Hash data to v2 model objects (required on demand, not auto-loaded)
37
- - `Bibliography` — fetches IHO data from relaton-data-iho GitHub repo
38
-
39
- All model classes use Lutaml::Model for declarative attribute/serialization definitions.
40
-
41
- ### HashParserV1
42
-
43
- `HashParserV1` (in `hash_parser_v1.rb`) handles conversion of legacy v1 data to the current model. It overrides the base `Bib::HashParserV1` to handle IHO-specific structures:
44
- - **editorialgroup**: IHO v1 data has nested committee arrays (`editorialgroup: [[{committee: ...}, ...]]`). Each committee is converted to a `contributor` with `role type="author"` / `description="committee"`, organization "International Hydrographic Organization"/"IHO", and nested `subdivision` elements for sub-committees.
45
- - **commentperiod**: Moved from top-level into `ext`.
46
- - Factory methods (`bib_item`, `create_doctype`, `create_relation`) return IHO-specific classes.
47
-
48
- This module is required on demand (`require "relaton/iho/hash_parser_v1"`) and not auto-loaded by the main `relaton/iho` entry point.
49
-
50
- ### Load Order Constraint
51
-
52
- In `item.rb`, `require_relative "relation"` is placed **after** the `Item` class definition because `Relation` → `ItemBase` → `Item` creates a circular dependency if loaded before `Item` exists.
53
-
54
- ### Data Retrieval
55
-
56
- `Bibliography.search`/`.get` fetches YAML from the relaton-data-iho GitHub repo via `Relaton::Index`, then deserializes into model objects.
57
-
58
- **Editions and series (the pubid 2.0.5 component model, relaton#440).** The pubid attribute is `edition` (dotted `a.b.c`, `IHO B-12 3.0.0`; the pre-440 name `version` is gone). A reference with no edition answers with the **latest**: `recency_key` compares the segments as integers (`2.10.0` after `2.9.0`). It used to be `min_by` on the string, which returned the oldest.
59
-
60
- **Serial locators are volumes (relaton#440 unified the year locators of relaton#407).** The yearbook year IS a volume: `IHO P-5 Vol. 2020` is canonical, the bare-year faces (`P-5 2020`, `P-5/2020`) parse as legacy aliases into the same `volume`, and the IHR is `P-1 Vol. 30` with `Issue 1` (no parens) or the short slash form `P-1/30`. A bare series query answers with the latest volume/issue. **Components can publish separately and carry their own edition:** `IHO S-66 Suppl 1 2.0.0` gives `supplement_edition`, `S-100 Part S-102 1.0.0` gives `part_edition`, annexes likewise — and the publication years in parens (`S-52 Annex A 4.0.3 (2021)`) are `*_year` metadata, not identity: a stated year pins the record, an omitted one wildcards. Matching (`pubid_match?`) treats a discriminator the query omits as a wildcard: `IHO P-1 Vol. 30` matches any issue of volume 30. The comparison runs through pubid's `#exclude` — excluding `:volume` nils the `issue`, excluding an `:appendix` nils its `appendix_subnumber` (scalar clusters) — the way BIPM's `stem_keys` does. The subdivisions (part/appendix/annex/supplement) stay strict: this flavor cannot use the `===` subset match, because pubid does not declare `subset_strict` for IHO's subdivisions and a nil `part` would read as "any part" (`IHO S-100` would answer with a part row).
61
-
62
- **The edition reaches the index only through the committed row.** A record prints `B-12` and keeps its edition in `edition:`, so `relaton index` cannot read a version from the docid. `MachineIndex#refine` (relaton-cli) takes the committed `index-v3.yaml` row when it adds only a key in `REFINABLE_KEYS` to the parse — `version`/`edition`/`volume`/`year`/`issue` (relaton#407 added the year/issue pair for the series rows). Before that, the Pages index (generated 2026-10-29) had no `version` in any of its 325 rows, so every versioned reference answered not-found (relaton-data-iho#20). The suite did not see it: `spec/fixtures/index-v3.zip` is the committed index, which has versions. The suite's fixture carries the #48-era re-keyed series rows migrated to the #440 keys (`version`→`edition`/`<component>_edition`, year locators→`volume`; `support/migrate_index_fixture.rb` did the one-shot migration) — refresh the fixture from the live published zip once relaton-data-iho round 2 republishes, and reconcile any file-name drift then.
63
-
64
- ### Test Infrastructure
65
-
66
- - **VCR cassettes** (`spec/vcr_cassettes/`): recorded HTTP responses for offline testing
67
- - **RelaxNG validation**: specs validate XML output against `grammars/relaton-iho-compile.rng` using ruby-jing
68
- - **Fixtures** (`spec/fixtures/`): reference XML/YAML documents for round-trip serialization tests
69
-
70
- ### IHO-Specific Document Types
71
-
72
- policy-and-procedures, best-practices, supporting-document, report, legal, directives, proposal, standard
73
-
74
- ## Testing
75
-
76
- - **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iho.