relaton 3.1.1 → 3.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +5 -3
- data/lib/relaton/3gpp.rb +1 -1
- data/lib/relaton/bib/converter/csl/from_csl.rb +195 -0
- data/lib/relaton/bib/converter/csl.rb +13 -1
- data/lib/relaton/bib/model/docidentifier.rb +66 -5
- data/lib/relaton/calconnect/hit_collection.rb +1 -1
- data/lib/relaton/cli/command.rb +3 -1
- data/lib/relaton/cli/frontend_assets.rb +7 -6
- data/lib/relaton/cli/subcommand_collection.rb +5 -1
- data/lib/relaton/core/request_error.rb +18 -0
- data/lib/relaton/db/config.rb +9 -1
- data/lib/relaton/db/workers_pool.rb +11 -1
- data/lib/relaton/db.rb +160 -58
- data/lib/relaton/ecma/bibliography.rb +2 -2
- data/lib/relaton/etsi/data_fetcher.rb +16 -6
- data/lib/relaton/iala/bibliography.rb +1 -1
- data/lib/relaton/iana.rb +1 -1
- data/lib/relaton/iec/data_parser.rb +41 -2
- data/lib/relaton/itu/README.adoc +1 -1
- data/lib/relaton/itu/data_crawler_r.rb +1 -1
- data/lib/relaton/jis/README.adoc +2 -0
- data/lib/relaton/jis/data_fetcher.rb +74 -13
- data/lib/relaton/jis/hit_collection.rb +2 -1
- data/lib/relaton/jis/processor.rb +1 -1
- data/lib/relaton/jis/scraper.rb +46 -5
- data/lib/relaton/oasis/bibliography.rb +1 -1
- data/lib/relaton/ogc/hit_collection.rb +1 -1
- data/lib/relaton/ogc.rb +1 -1
- data/lib/relaton/plateau/data_fetcher.rb +10 -1
- data/lib/relaton/plateau/handbook_parser.rb +38 -2
- data/lib/relaton/plateau/hit_collection.rb +20 -5
- data/lib/relaton/sae.rb +1 -1
- data/lib/relaton/version.rb +1 -1
- data/lib/relaton/w3c/bibliography.rb +1 -1
- data/lib/relaton/xsf/bibliography.rb +1 -1
- data/lib/relaton/xsf.rb +1 -1
- metadata +35 -48
- data/lib/relaton/3gpp/CLAUDE.md +0 -218
- data/lib/relaton/adobe/CLAUDE.md +0 -104
- data/lib/relaton/bib/CLAUDE.md +0 -150
- data/lib/relaton/bipm/CLAUDE.md +0 -187
- data/lib/relaton/bsi/CLAUDE.md +0 -35
- data/lib/relaton/calconnect/CLAUDE.md +0 -177
- data/lib/relaton/ccsds/CLAUDE.md +0 -74
- data/lib/relaton/cen/CLAUDE.md +0 -219
- data/lib/relaton/cie/CLAUDE.md +0 -90
- data/lib/relaton/core/CLAUDE.md +0 -140
- data/lib/relaton/db/CLAUDE.md +0 -385
- data/lib/relaton/doi/CLAUDE.md +0 -105
- data/lib/relaton/easc/CLAUDE.md +0 -85
- data/lib/relaton/ecma/CLAUDE.md +0 -217
- data/lib/relaton/etsi/CLAUDE.md +0 -219
- data/lib/relaton/evs/CLAUDE.md +0 -120
- data/lib/relaton/gb/CLAUDE.md +0 -47
- data/lib/relaton/gost/CLAUDE.md +0 -100
- data/lib/relaton/iala/CLAUDE.md +0 -97
- data/lib/relaton/iana/CLAUDE.md +0 -105
- data/lib/relaton/iec/CLAUDE.md +0 -13
- data/lib/relaton/ieee/CLAUDE.md +0 -202
- data/lib/relaton/ietf/CLAUDE.md +0 -382
- data/lib/relaton/iho/CLAUDE.md +0 -76
- data/lib/relaton/index/CLAUDE.md +0 -208
- data/lib/relaton/isbn/CLAUDE.md +0 -47
- data/lib/relaton/iso/CLAUDE.md +0 -38
- data/lib/relaton/itu/CLAUDE.md +0 -685
- data/lib/relaton/jcgm/CLAUDE.md +0 -107
- data/lib/relaton/jis/CLAUDE.md +0 -64
- data/lib/relaton/logger/CLAUDE.md +0 -37
- data/lib/relaton/nist/CLAUDE.md +0 -194
- data/lib/relaton/oasis/CLAUDE.md +0 -251
- data/lib/relaton/ogc/CLAUDE.md +0 -242
- data/lib/relaton/oiml/CLAUDE.md +0 -47
- data/lib/relaton/omg/CLAUDE.md +0 -137
- data/lib/relaton/plateau/CLAUDE.md +0 -126
- data/lib/relaton/sae/CLAUDE.md +0 -91
- data/lib/relaton/sdo/CLAUDE.md +0 -81
- data/lib/relaton/un/CLAUDE.md +0 -91
- data/lib/relaton/w3c/CLAUDE.md +0 -369
- data/lib/relaton/xsf/CLAUDE.md +0 -189
data/lib/relaton/ietf/CLAUDE.md
DELETED
|
@@ -1,382 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## What This Gem Does
|
|
6
|
-
|
|
7
|
-
The `Relaton::Ietf` flavor fetches and parses IETF bibliographic data (RFCs, Internet-Drafts, BCPs, FYIs, STDs) into the Relaton data model. Models are `lutaml-model`-based.
|
|
8
|
-
|
|
9
|
-
## Commands
|
|
10
|
-
|
|
11
|
-
The suite runs from its own directory (`spec/ietf/`), as every flavor suite does.
|
|
12
|
-
|
|
13
|
-
```bash
|
|
14
|
-
bundle exec rake spec:ietf # run this flavor's suite
|
|
15
|
-
cd spec/ietf && bundle exec rspec -I . relaton/ietf/bibxml_parser_spec.rb # single file
|
|
16
|
-
cd spec/ietf && bundle exec rspec -I . relaton/ietf/rfc/entry_spec.rb:19 # single example
|
|
17
|
-
cd spec/ietf && bundle exec rspec -I . -e "creates primary docid" # by description
|
|
18
|
-
bundle exec rubocop # lint
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
## Architecture
|
|
22
|
-
|
|
23
|
-
### Data Model (Lutaml::Model)
|
|
24
|
-
|
|
25
|
-
All models use `Lutaml::Model::Serializable` with `attribute` declarations and `xml do ... end` blocks for serialization. The IETF classes extend base `Relaton::Bib` classes:
|
|
26
|
-
|
|
27
|
-
- `Relaton::Ietf::ItemData < Bib::ItemData` — core bibliographic data
|
|
28
|
-
- `Relaton::Ietf::Item < Bib::Item` — adds `ext: Ext` attribute
|
|
29
|
-
- `Relaton::Ietf::Bibdata < Item` / `Bibitem < Item` — include shared serialization concerns
|
|
30
|
-
- `Relaton::Ietf::Ext` — IETF extensions: `doctype`, `flavor`, `stream`, `area`, `ipr`, `pi`
|
|
31
|
-
- `Relaton::Ietf::Doctype` — types: `"rfc"`, `"internet-draft"`
|
|
32
|
-
|
|
33
|
-
### Namespace Resolution
|
|
34
|
-
|
|
35
|
-
Converter classes use `Bib::NamespaceHelper` which resolves `namespace` by taking the first two segments of the class name. For `Relaton::Ietf::BibXMLParser::FromRfcxml`, `namespace` returns `Relaton::Ietf`, so `namespace::ItemData` → `Relaton::Ietf::ItemData`, `namespace::Ext` → `Relaton::Ietf::Ext`, etc.
|
|
36
|
-
|
|
37
|
-
### Key Flows
|
|
38
|
-
|
|
39
|
-
1. **Single document lookup**: `Processor#get` → `Bibliography.get(code)` → `Scraper.scrape_page` → parses the reference with `Pubid::Ietf` → searches the one combined `index-v2` from `relaton-data-ietf` → fetches that row's YAML; a versioned draft the index lacks is read from bib.ietf.org (see below)
|
|
40
|
-
|
|
41
|
-
2. **Bulk data fetching**: `DataFetcher` extends `Relaton::Core::DataFetcher` with three datasets:
|
|
42
|
-
- `ietf-rfcsubseries` / `ietf-rfc-entries`: parse `rfc-index.xml` via `Rfc::Index` / `Rfc::Entry#to_item`
|
|
43
|
-
- `ietf-internet-drafts`: parse local BibXML files via `BibXMLParser.parse`
|
|
44
|
-
|
|
45
|
-
3. **BibXML parsing** (`BibXMLParser` module):
|
|
46
|
-
- `parse(xml)` — parses `<reference>` elements via `FromRfcxml` converter
|
|
47
|
-
- `parse_rfc(xml)` — parses `<rfc>` root documents via `FromRfc` converter
|
|
48
|
-
- Converters inherit from `Bib::Converter::BibXml::FromRfcxml` with IETF overrides for contributor/person/org handling
|
|
49
|
-
|
|
50
|
-
4. **RFC Index parsing** (`Rfc::Entry#to_item`): converts RFC editor index entries (BCP/FYI/STD/RFC) into `ItemData`, handling subseries vs full RFC entries differently
|
|
51
|
-
|
|
52
|
-
### Converter Inheritance Chain
|
|
53
|
-
|
|
54
|
-
```
|
|
55
|
-
Bib::Converter::BibXml::FromRfcxml # base: handles <reference> generically
|
|
56
|
-
└── Ietf::BibXMLParser::FromRfcxml # IETF overrides: publisher, org recognition, person names
|
|
57
|
-
└── Ietf::BibXMLParser::FromRfc # <rfc> root: uses doc_name instead of anchor, no ref-level series_info
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
`FromRfc` must override all methods that access `@reference.anchor`, `@reference.target`, `@reference.format`, or `@reference.series_info` since `Rfcxml::V3::Rfc` lacks these attributes (unlike `Rfcxml::V3::Reference`).
|
|
61
|
-
|
|
62
|
-
## Unified index (relaton#109)
|
|
63
|
-
|
|
64
|
-
Producer and consumer are both on the pubid `index-v2` now. `DataFetcher` writes
|
|
65
|
-
it; `Scraper` reads one combined index from `relaton-data-ietf` — replacing the
|
|
66
|
-
three per-type `index-v1` files (`:RFC`/`:RSS`/`:IDS`) it used to read from
|
|
67
|
-
`relaton-data-{rfcs,rfcsubseries,ids}`. Those three keep publishing `index-v1`
|
|
68
|
-
untouched, for released relatons.
|
|
69
|
-
|
|
70
|
-
`INDEXFILE` is a single constant again (`index-v2`); the `INDEXFILE_V2` that
|
|
71
|
-
existed while the two halves were split is gone.
|
|
72
|
-
|
|
73
|
-
Decisions taken (2026-08-18 ... 2026-08-24):
|
|
74
|
-
|
|
75
|
-
| | |
|
|
76
|
-
|---|---|
|
|
77
|
-
| Index structure | **unchanged** - `:id` + `:file` only. No `docids[]`, no summary fields; a third key invalidates the whole index for every released gem |
|
|
78
|
-
| `:id` shape | structured pubid hash (`Pubid::Ietf::Identifier#to_hash`) |
|
|
79
|
-
| Row per record | keyed by the record's **primary** docidentifier. A sub-series id resolves to its own container record, *not* to a constituent RFC - 42 of 365 sub-series include more than one RFC; consumers follow `relation: includes` |
|
|
80
|
-
| DOI ids | omitted - not `pubid:ietf`-representable, and one unparseable row kills the index. DOIs are served by the `doi` flavor |
|
|
81
|
-
| Who builds it | `DataFetcher`, during the crawl - not a separate pass over the written records, which would be a second answer to "what is this record's id" |
|
|
82
|
-
| Three type repos | untouched: they pin `relaton/relaton-ietf`, not this monorepo, so they keep emitting `index-v1` |
|
|
83
|
-
|
|
84
|
-
**Resolved 2026-08-26.** `relaton-data-ietf`'s RFC and sub-series records were
|
|
85
|
-
schema-invalid — missing `ext.doctype`, and RFC committee organizations carrying
|
|
86
|
-
`subdivision` with no `name` — because that repo converted the `ietf-tools`
|
|
87
|
-
v1.2.3 mirrors rather than crawling. It now runs `Relaton::Ietf::DataFetcher`
|
|
88
|
-
directly, which synthesises `doctype`, `status`, `stream`, the organization names
|
|
89
|
-
and the resolved WG titles. The two `pending` examples in `ietf_spec.rb` reported
|
|
90
|
-
`FIXED` on the first run against the new corpus and have been removed.
|
|
91
|
-
|
|
92
|
-
Two consequences of that migration to know about: record filenames are now
|
|
93
|
-
lowercase *and* unpadded (`data/RFC8341.yaml` -> `data/rfc8341.yaml`,
|
|
94
|
-
`data/STD0066.yaml` -> `data/std66.yaml`), since `output_file` derives them from
|
|
95
|
-
the docnumber; and the record content changed enough that the three XML fixtures
|
|
96
|
-
had to be regenerated.
|
|
97
|
-
|
|
98
|
-
### The consumer: `Scraper` queries with parsed identifiers
|
|
99
|
-
|
|
100
|
-
```ruby
|
|
101
|
-
Relaton::Index.find_or_create(
|
|
102
|
-
:IETF, url: "#{IETF}#{INDEXFILE}.zip", file: "#{INDEXFILE}.yaml",
|
|
103
|
-
pubid_class: ::Pubid::Ietf::Identifier
|
|
104
|
-
)
|
|
105
|
-
```
|
|
106
|
-
|
|
107
|
-
`parse_id` turns the reference into a `Pubid::Ietf::Identifier` before searching,
|
|
108
|
-
and **that is not cosmetic**: `Type#search_candidates` narrows only for non-String
|
|
109
|
-
queries, so a String falls into `match_item`'s `item[:id].to_s.include?(id)`, which
|
|
110
|
-
renders every pubid in the index on every lookup. Measured against the live
|
|
111
|
-
177k-row index: **~43 s per reference with a String, sub-millisecond with a parsed
|
|
112
|
-
id.** A String query also returns *nothing* if `pubid_class:` is omitted, since the
|
|
113
|
-
rows would then be raw hashes.
|
|
114
|
-
|
|
115
|
-
`parse_id` also normalises the Internet-Draft spellings - `I-D.<slug>` and
|
|
116
|
-
`I-D <slug>` - onto the `draft-...` form, adding the `draft-` stem when absent
|
|
117
|
-
(`I-D.ietf-quic-transport`, the bibxml anchor and the `docnumber` records carry).
|
|
118
|
-
The old plain-string index matched that by substring; matching is exact now.
|
|
119
|
-
A reference pubid cannot parse returns nil, so `Bibliography.get "CN 8341"` logs
|
|
120
|
-
"Not found." rather than raising.
|
|
121
|
-
|
|
122
|
-
### An index miss on a versioned draft reads bib.ietf.org (relaton-ietf#125)
|
|
123
|
-
|
|
124
|
-
The data repo crawls the rsync `bibxml-ids` snapshot, and that snapshot keeps
|
|
125
|
-
only some versions of each draft: `draft-ietf-vcarddav-vcardrev` has only `-22`,
|
|
126
|
-
so `-06` was "Not found." When `fetch_doc` misses and the query is a
|
|
127
|
-
`Pubid::Ietf::Identifiers::InternetDraft` **with a version**, `fetch_bibxml` reads
|
|
128
|
-
`https://bib.ietf.org/public/rfc/bibxml-ids/reference.I-D.<draft>-NN.xml`
|
|
129
|
-
(`BIBXML_IDS`). That service resolves any version the datatracker knows and
|
|
130
|
-
answers 404 otherwise; the datatracker's own `doc/bibxml3/...` URL answers 404
|
|
131
|
-
for these old versions. The parse is the crawler's: `BibXMLParser.to_utf8`
|
|
132
|
-
(the CP1252 repair, shared with `DataFetcher#read_bibxml`), `BibXMLParser.parse`,
|
|
133
|
-
and the version from the identifier, as `process_singleton` sets it.
|
|
134
|
-
|
|
135
|
-
Only a 404 is "not found". Any other non-200 (429, 5xx) raises
|
|
136
|
-
`Relaton::RequestError`, so `Db#net_retry` retries it and `Db::Cache` does not
|
|
137
|
-
store an outage as a not-found row. A 200 whose body does not parse logs a
|
|
138
|
-
warning and returns nil; that rescue names the parse errors and is not
|
|
139
|
-
`StandardError`, so a transport error still becomes `RequestError`.
|
|
140
|
-
|
|
141
|
-
RFCs, sub-series and unversioned drafts do **not** fall back: the index holds
|
|
142
|
-
all of them, so a miss is a real "not found". A record read this way has no
|
|
143
|
-
`updates`/`updatedBy` neighbours — the crawl links those, and one fetch cannot.
|
|
144
|
-
Old bibxml carries no `target`, so such a record has a `TXT` URI and no `src`.
|
|
145
|
-
|
|
146
|
-
### The producer: `DataFetcher` writes `index-v2`
|
|
147
|
-
|
|
148
|
-
`DataFetcher` builds the index during the crawl, from the identifier it already
|
|
149
|
-
holds for each record — `record_index_entry` parses the primary docidentifier and
|
|
150
|
-
`add_or_update`s the pubid:
|
|
151
|
-
|
|
152
|
-
```ruby
|
|
153
|
-
Relaton::Index.find_or_create(
|
|
154
|
-
:IETF, file: "#{INDEXFILE}.yaml", pubid_class: ::Pubid::Ietf::Identifier
|
|
155
|
-
)
|
|
156
|
-
```
|
|
157
|
-
|
|
158
|
-
**Both halves are load-bearing.** `FileIO#save` serialises an id to its `_type:`
|
|
159
|
-
hash only when it is an instance of the configured `pubid_class`, so passing a
|
|
160
|
-
String to a `pubid_class:` index writes a v1-shaped file under a v2 name — no
|
|
161
|
-
error, just an index that never narrows.
|
|
162
|
-
|
|
163
|
-
Three properties worth keeping:
|
|
164
|
-
|
|
165
|
-
- **Key on the primary docidentifier**, which `serialize_and_write` already
|
|
166
|
-
picks. Not the first entry: draft records carry an `I-D.foo` anchor id (not a
|
|
167
|
-
pubid grammar) and their unversioned series id, which is the *primary* of a
|
|
168
|
-
separate aggregator record — keying on either collides on one `id.to_s` and
|
|
169
|
-
`add_or_update` silently overwrites `:file`.
|
|
170
|
-
- **DOIs stay out.** `10.17487/RFC3986` is not a `pubid:ietf` identifier, and one
|
|
171
|
-
unparseable row invalidates the whole index. The primary is never the DOI, so
|
|
172
|
-
this is already right — don't "improve" it into indexing every docidentifier.
|
|
173
|
-
- **Sub-series key to their own record.** `STD 66` → the STD container, not a
|
|
174
|
-
constituent RFC: 42 of the 365 published sub-series include more than one, so
|
|
175
|
-
there is no single document to alias to. Consumers follow `relation: includes`.
|
|
176
|
-
|
|
177
|
-
A record whose id pubid rejects is written but not indexed, warned about, and
|
|
178
|
-
tallied for a summary at the end of the crawl. The load is all-or-nothing —
|
|
179
|
-
`deserialize_id` raises on the first bad id and `load_index` then rejects the
|
|
180
|
-
entire index — so one malformed upstream record must cost one document, never
|
|
181
|
-
every lookup. All 176,862 published ids parse today; this guards against drift.
|
|
182
|
-
|
|
183
|
-
**Blast radius, worth re-checking before touching this.** The six data repos
|
|
184
|
-
feeding today's released consumers do *not* use this monorepo: `ietf-tools/*` pin
|
|
185
|
-
`relaton/relaton-ietf` @ `main` (`RelatonIetf::DataFetcher`), and
|
|
186
|
-
`relaton/relaton-data-{rfcs,rfcsubseries,ids}` pin it @ `lutaml-integration` —
|
|
187
|
-
both branches carry their own `INDEXFILE = "index-v1"` and their own fetcher. Only
|
|
188
|
-
`relaton/relaton-data-ietf` runs this one. So the producer flip needs no
|
|
189
|
-
dual-write and no cutover window.
|
|
190
|
-
|
|
191
|
-
### Still outstanding
|
|
192
|
-
|
|
193
|
-
Retiring `relaton-data-{rfcs,rfcsubseries,ids}`'s `index-v1` files, and their
|
|
194
|
-
`data-index/configs.yml` rows, waits until a relaton carrying this change has
|
|
195
|
-
shipped — see `/work/HANDOFFS/relaton__support__ietf-index-config-and-cron.md`.
|
|
196
|
-
|
|
197
|
-
## Filename collisions: workers write, the parent settles the names
|
|
198
|
-
|
|
199
|
-
`Core::DataFetcher#output_file` sanitizes `.`, `,`, `/`, `:`, `(`, `)`, `-` and
|
|
200
|
-
whitespace all to `-`, so two **distinct** docids can want one path. This flavor
|
|
201
|
-
is the only one that writes from forked processes, so it is the only one where
|
|
202
|
-
that could destroy a record without a word.
|
|
203
|
-
|
|
204
|
-
It did. In the first production crawl of `relaton-data-ietf`,
|
|
205
|
-
`data/draft-ietf-6man-hbh-processing.yaml` — the aggregator for a draft with 21
|
|
206
|
-
versions — was written and then overwritten by a singleton built from the
|
|
207
|
-
malformed sibling id `draft-ietf-6man-hbh-processing-`, which sanitizes to the
|
|
208
|
-
same name. The published corpus came out 177,362 index rows over 177,361 files,
|
|
209
|
-
and nothing in the log said so. Exactly one such group exists in that corpus;
|
|
210
|
-
the mechanism, not the blast radius, is the problem.
|
|
211
|
-
|
|
212
|
-
Three pieces close it, and all three are load-bearing:
|
|
213
|
-
|
|
214
|
-
1. **`@cross_process = true`, set in `fetch_ieft_internet_drafts` before the
|
|
215
|
-
first `parallelize`** so the forked children inherit it. It tells
|
|
216
|
-
`Core::DataFetcher#write_unique` that an existing file may belong to a peer
|
|
217
|
-
and must never be overwritten.
|
|
218
|
-
2. **`serialize_and_write` writes through `write_unique`** and reports two extra
|
|
219
|
-
keys: `docid` (the id the file was written under) and `plain_file` (the name
|
|
220
|
-
it would have taken uncontested). `docid` is **not** `index_id` — `id` falls
|
|
221
|
-
back through `docnumber` and `formattedref`, so on the RFC path the two
|
|
222
|
-
differ (`RFC0001` vs `RFC 1`, i.e. stems `rfc0001` and `rfc-1`). Keying the
|
|
223
|
-
reconciliation on `index_id` renames records to the wrong names.
|
|
224
|
-
3. **`reconcile_output_files`, in the parent, before `record_index_entry`.** A
|
|
225
|
-
worker only learns that a path was taken, never which clashing docid deserves
|
|
226
|
-
it, so left alone the winner would follow the race and the two filenames would
|
|
227
|
-
swap between crawls. The parent sees every docid, so it decides: within a
|
|
228
|
-
group of records that wanted one path, the alphabetically first docid keeps
|
|
229
|
-
it and the rest take their digest variant.
|
|
230
|
-
|
|
231
|
-
Details of (3) that are easy to get wrong:
|
|
232
|
-
|
|
233
|
-
- It runs over **every** group, not just clashing ones. A lone record that fell
|
|
234
|
-
back to a digest path must get its plain name back, or the published filename
|
|
235
|
-
churns and the old file is orphaned. `next if r[:file] == target` makes that
|
|
236
|
-
free for ~100% of records.
|
|
237
|
-
- **Losers move first.** A loser may be sitting *on* the plain path; moving the
|
|
238
|
-
winner there first would destroy it.
|
|
239
|
-
- Two results for the **same** docid are collapsed, not split: the stray file is
|
|
240
|
-
deleted and one warning is emitted, which is what the crawl did before. The
|
|
241
|
-
stray may already be **gone** — the fallback name is keyed on the docid, so two
|
|
242
|
-
workers that both lost the race to the plain path land on one name and the
|
|
243
|
-
first of them renames it onto the target. The result must follow the document
|
|
244
|
-
to the target anyway, or the index gets a row pointing at nothing.
|
|
245
|
-
- Sorting *unique docids* leaves no tie for Ruby's unstable `sort_by` to resolve
|
|
246
|
-
differently between runs.
|
|
247
|
-
- A failed rename warns and carries on. A multi-hour crawl must not die at the
|
|
248
|
-
last step.
|
|
249
|
-
|
|
250
|
-
`report_collisions` restates the total once at the end of the crawl, beside
|
|
251
|
-
`report_unparsed` and `report_unindexed` — one line for a crawl that writes
|
|
252
|
-
~177k records.
|
|
253
|
-
|
|
254
|
-
The sequential paths (`fetch_ieft_rfcsubseries`, `fetch_ieft_rfcs`) get no
|
|
255
|
-
reconciliation pass and need none: one process, so `unique_output_file` is
|
|
256
|
-
already deterministic there. They do now emit a digest filename for a genuine
|
|
257
|
-
cross-docid collision instead of overwriting, which follows `rfc-index`
|
|
258
|
-
iteration order — stable for a given input.
|
|
259
|
-
|
|
260
|
-
## Thin records inherit from their constituents
|
|
261
|
-
|
|
262
|
-
Two builders produce records for documents that have **no upstream source of
|
|
263
|
-
their own**, and both would otherwise publish without date, doctype or source:
|
|
264
|
-
|
|
265
|
-
| Builder | File | Inherits from |
|
|
266
|
-
|---|---|---|
|
|
267
|
-
| `to_subseries_item` | `rfc/entry.rb` | the newest RFC in its `is-also` |
|
|
268
|
-
| `build_unversioned_doc` | `data_fetcher.rb` | the newest version in `sorted` |
|
|
269
|
-
|
|
270
|
-
This is **not** a parsing gap to go fix in the XML mapping. A sub-series entry in
|
|
271
|
-
`rfc-index.xml` is only a pointer:
|
|
272
|
-
|
|
273
|
-
```xml
|
|
274
|
-
<bcp-entry>
|
|
275
|
-
<doc-id>BCP0003</doc-id>
|
|
276
|
-
<is-also><doc-id>RFC1915</doc-id></is-also>
|
|
277
|
-
</bcp-entry>
|
|
278
|
-
```
|
|
279
|
-
|
|
280
|
-
The RFC Editor publishes no date, title, author or status for it, because a
|
|
281
|
-
sub-series *has* no metadata of its own — it is a label on one or more RFCs.
|
|
282
|
-
Unversioned draft aggregators are further still from an upstream document: they
|
|
283
|
-
are synthesised from the versions found on disk.
|
|
284
|
-
|
|
285
|
-
Neither site needs a second pass or another read of the corpus — `rfc_index` and
|
|
286
|
-
`sorted` already hold the constituents at build time. (A downstream
|
|
287
|
-
implementation in `relaton-data-ietf`'s crawler did use a second pass over
|
|
288
|
-
written files, because its groups could span source repos; that constraint does
|
|
289
|
-
not exist here.)
|
|
290
|
-
|
|
291
|
-
**Constituent lookup is normalised on both sides, deliberately.**
|
|
292
|
-
`Entry.squish` folds case and strips whitespace and dots. Relation targets and the
|
|
293
|
-
docids they reference disagree in the published corpus (`dyndNS` vs `dyndns`),
|
|
294
|
-
which left 28 documents undated downstream on a strict lookup — and, in
|
|
295
|
-
`build_relations`, silently downgraded a full constituent bibitem to a minimal
|
|
296
|
-
one.
|
|
297
|
-
|
|
298
|
-
`rfc_index` is therefore **keyed by `Entry.squish(doc_id)`, and the caller builds
|
|
299
|
-
it** — `fetch_ieft_rfcsubseries` does so once for the crawl. Normalising inside
|
|
300
|
-
`Entry` instead would rebuild a ~9,800-row table for each of the 367 sub-series
|
|
301
|
-
(measured: ~161 MB retained, ~6 s), all of it the same table.
|
|
302
|
-
|
|
303
|
-
Why it matters beyond tidiness: the Pages index sorts by date, so undated records
|
|
304
|
-
sort as one undifferentiated block, and a record with no `ext.doctype` renders
|
|
305
|
-
with no document type at all.
|
|
306
|
-
|
|
307
|
-
Inherited values are `dup`'d rather than shared with the constituent's own bib —
|
|
308
|
-
aliasing them would make a later edit to the aggregator mutate the `-NN` record.
|
|
309
|
-
|
|
310
|
-
Verified against the live `rfc-index.xml`: **367/367 sub-series dated**, 367/367
|
|
311
|
-
with full constituent bibitems, in 1.4 s; `BCP 3` → 1996-02, `STD 51` → 1994-07,
|
|
312
|
-
`STD 66` → 2005-01 (RFC 3986's date).
|
|
313
|
-
|
|
314
|
-
## bibxml encoding: transcode, don't scrub, don't skip
|
|
315
|
-
|
|
316
|
-
Some bibxml files declare `encoding='UTF-8'` and carry **Windows-1252** bytes —
|
|
317
|
-
smart quotes and accented Latin letters. `File.read(encoding: "UTF-8")` only
|
|
318
|
-
*tags* a string; it validates nothing, so those reached lutaml as invalid UTF-8
|
|
319
|
-
and `Rfcxml::V3::Reference.from_xml` raised `Lutaml::Model::InvalidFormatError`.
|
|
320
|
-
|
|
321
|
-
`read_bibxml` transcodes them through `BibXMLParser.to_utf8`
|
|
322
|
-
(`bibxml_parser.rb`, shared with the scraper's bib.ietf.org fallback). Three things make that the
|
|
323
|
-
right call rather than `scrub`:
|
|
324
|
-
|
|
325
|
-
- **All of them decode losslessly as CP1252** — 125 of the ~122k published files
|
|
326
|
-
are affected, 0 unmappable.
|
|
327
|
-
- **The bytes present prove CP1252, not Latin-1.** `0x91`–`0x94` (`‘ ’ “ ”`) are
|
|
328
|
-
control characters in Latin-1.
|
|
329
|
-
- **`scrub` is lossy.** It replaces each byte with `U+FFFD`: `client’s` becomes
|
|
330
|
-
`client�s`, `Muñoz` becomes `Mu�oz`. Author surnames are among the
|
|
331
|
-
affected fields. Verified on a real file — scrub and transcode yield the same
|
|
332
|
-
*length*, so a length check will not catch the damage.
|
|
333
|
-
|
|
334
|
-
Don't copy `lib/relaton/bipm/si_brochure_parser.rb:33-35`, whose
|
|
335
|
-
`force_encoding` guard is a no-op. The `scrub` idiom at
|
|
336
|
-
`lib/relaton/itu/data_crawler_r.rb:588-599` is right *there* because it repairs
|
|
337
|
-
scraped HTML with no recoverable original; here there is one.
|
|
338
|
-
|
|
339
|
-
### Why the rescue is per file, and why the tally is in the parent
|
|
340
|
-
|
|
341
|
-
`parse_bibxml` rescues `StandardError` — not a narrow list. lutaml raises
|
|
342
|
-
`InvalidFormatError` on bad bytes, but the converter also runs regexes over
|
|
343
|
-
parsed text (`parse_surname_initials` and friends), which raise
|
|
344
|
-
`ArgumentError: invalid byte sequence` on anything that slips through.
|
|
345
|
-
|
|
346
|
-
It has to be **inside** the worker. The drafts path runs under `Parallel.map`,
|
|
347
|
-
which discards every result from the pass when a worker raises — so one bad file
|
|
348
|
-
out of ~167k meant no index at all, plus the already-written records orphaned on
|
|
349
|
-
disk, because `record_index_entry` and `index.save` only run after both passes
|
|
350
|
-
return. A rescue around `parallelize` would salvage "the crawl didn't die", not
|
|
351
|
-
"the other 166,999 files got indexed".
|
|
352
|
-
|
|
353
|
-
The skip therefore rides back as `{ unparsed: path, error: }` and is counted in
|
|
354
|
-
the parent, the way `parse_pubid`'s result rides back as `result[:pubid]` — a
|
|
355
|
-
counter incremented in a worker process is lost. `report_unparsed` warns once
|
|
356
|
-
with a count and a sample, not once per file.
|
|
357
|
-
|
|
358
|
-
A failed version is dropped from `sorted` **before** `link_neighbor_relations`
|
|
359
|
-
and `build_unversioned_doc` run, since both dereference `entry[:bib]`. Knock-on
|
|
360
|
-
to accept: neighbours then link across the gap, and if the dropped file was the
|
|
361
|
-
newest, the aggregator inherits from the previous version. With CP1252 recovery
|
|
362
|
-
that should approach zero — which is the argument for recovering over skipping.
|
|
363
|
-
|
|
364
|
-
Verified end to end on 41,408 real draft files (16 invalid UTF-8): the crawl
|
|
365
|
-
completes in 57 s, writes 46,461 records and 46,461 index rows, **0 skipped**,
|
|
366
|
-
and the affected files come through with their real characters —
|
|
367
|
-
`\x93Access Node Contr` → `“Access Node Contr`.
|
|
368
|
-
|
|
369
|
-
## Testing Patterns
|
|
370
|
-
|
|
371
|
-
- **Index fixture:** `spec/ietf/fixtures/ietf-index-v2.zip` - the combined index, copied **wholesale** from `relaton-data-ietf` rather than curated, since a subset goes stale and surfaces as a 404 on the document fetch. `spec/ietf/support/webmock.rb` deserialises it into pubid ids, sorts by the narrowing key, sets `sorted = true` (without which every lookup is a ~40 s scan) and pre-loads it into the pool as `:IETF` in `before(:suite)`; `spec/ietf/support/vcr.rb` ignores requests ending `index-v2.zip`. Refresh by re-downloading that zip. There is **no** `rake spec:update_index` task in this repo.
|
|
372
|
-
- **VCR cassettes** in `spec/ietf/vcr_cassettes/` record HTTP interactions; tests use `vcr: "cassette_name"` metadata
|
|
373
|
-
- **WebMock** disables net connections by default
|
|
374
|
-
- **Fixtures** in `spec/ietf/fixtures/` — XML/YAML expected outputs; many tests auto-generate fixtures on first run (`File.write file, xml unless File.exist? file`)
|
|
375
|
-
- **An oracle for `read_bibxml` must read binary.** The method is byte-verbatim (`File.binread`), so comparing it against `File.read(path, encoding: "UTF-8")` compares two different strings **on Windows only**: Ruby opens a file in text mode there and collapses CRLF to LF. The repo declares no `.gitattributes` and the GitHub Windows runner image defaults to `core.autocrlf=true`, so a fixture stored with LF is checked out CRLF — which failed all three `windows-latest` jobs while every Linux and macOS job passed. Use `File.read(path, mode: "rb", encoding: "UTF-8")`. macOS cannot reproduce this by checking the fixture out as CRLF, because its text-mode read does not translate; emulate Windows explicitly with `File.read(path, encoding: "UTF-8", universal_newline: true)`. The two `keeps CRLF line endings verbatim` examples pin the contract with `binwrite` temp files, independent of the checkout — one per branch, since the CP1252 repair rebuilds the string run by run and is where a line ending is likeliest to be lost. Note that a byte-equality oracle alone cannot catch a dropped `force_encoding`: `rfc.xml` is pure ASCII, and Ruby compares two ASCII-only strings as equal across encodings, so the example asserts `encoding` separately.
|
|
376
|
-
- **Schema validation** via `ruby-jing` against the shared RNG grammars in the repo-root `grammar/` (`Jing.new "../../grammar/relaton-ietf-compile.rng"`, relative to the suite's CWD `spec/ietf/`)
|
|
377
|
-
- **Shared examples** for org/person parsing in `bibxml_parser_spec.rb` (`parse_org`, `parse_person`)
|
|
378
|
-
- **`canon`** gem used for XML comparison (`be_equivalent_to` matcher)
|
|
379
|
-
|
|
380
|
-
## RuboCop
|
|
381
|
-
|
|
382
|
-
Inherits from `rubose/rubocop-rubose` (Ribose OSS config). Target Ruby 3.1. Rails cops disabled.
|
data/lib/relaton/iho/CLAUDE.md
DELETED
|
@@ -1,76 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## What This Is
|
|
6
|
-
|
|
7
|
-
relaton-iho is a Ruby gem that fetches and serializes IHO (International Hydrographic Organization) standards metadata. It is a "flavor" gem in the Relaton ecosystem, extending relaton-bib with IHO-specific bibliographic data models.
|
|
8
|
-
|
|
9
|
-
## Common Commands
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
bundle exec rspec # run all tests
|
|
13
|
-
bundle exec rspec spec/relaton/iho/item_spec.rb # run a single spec file
|
|
14
|
-
bundle exec rspec spec/relaton/iho/item_spec.rb:15 # run a single example by line
|
|
15
|
-
bundle exec rubocop # lint
|
|
16
|
-
bundle exec rubocop -a # lint with auto-fix
|
|
17
|
-
bundle exec rake install # install gem locally
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Architecture
|
|
21
|
-
|
|
22
|
-
### Flavor Pattern
|
|
23
|
-
|
|
24
|
-
This gem follows the Relaton flavor pattern: it subclasses base classes from `relaton-bib` under the `Relaton::Iho` namespace. The key mechanism is `NamespaceHelper` (from relaton-bib), which dynamically resolves the module namespace from the class name. This means `Relaton::Iho::ItemData#to_xml` automatically delegates to `Relaton::Iho::Bibdata.to_xml` (not `Relaton::Bib::Bibdata`).
|
|
25
|
-
|
|
26
|
-
Core model classes and their base classes:
|
|
27
|
-
- `Item < Bib::Item` — main bibliographic item, uses `model ItemData`
|
|
28
|
-
- `ItemData < Bib::ItemData` — data container enabling namespace-aware `to_xml`/`to_yaml`/`to_json`
|
|
29
|
-
- `ItemBase < Item` — stripped-down item for use inside relations (no id, schema_version, fetched, ext)
|
|
30
|
-
- `Bibitem < Item` — XML bibitem serialization (includes `Bib::BibitemShared`)
|
|
31
|
-
- `Bibdata < Item` — XML bibdata serialization (includes `Bib::BibdataShared`)
|
|
32
|
-
- `Relation < Bib::Relation` — overrides `bibitem` attribute to use `Iho::ItemBase`
|
|
33
|
-
- `Doctype < Bib::Doctype` — IHO document type vocabulary
|
|
34
|
-
- `Ext < Bib::Ext` — IHO-specific extension data (doctype, commentperiod)
|
|
35
|
-
- `Processor < Core::Processor` — Relaton processor for registry integration
|
|
36
|
-
- `HashParserV1` — converts v1-format YAML/Hash data to v2 model objects (required on demand, not auto-loaded)
|
|
37
|
-
- `Bibliography` — fetches IHO data from relaton-data-iho GitHub repo
|
|
38
|
-
|
|
39
|
-
All model classes use Lutaml::Model for declarative attribute/serialization definitions.
|
|
40
|
-
|
|
41
|
-
### HashParserV1
|
|
42
|
-
|
|
43
|
-
`HashParserV1` (in `hash_parser_v1.rb`) handles conversion of legacy v1 data to the current model. It overrides the base `Bib::HashParserV1` to handle IHO-specific structures:
|
|
44
|
-
- **editorialgroup**: IHO v1 data has nested committee arrays (`editorialgroup: [[{committee: ...}, ...]]`). Each committee is converted to a `contributor` with `role type="author"` / `description="committee"`, organization "International Hydrographic Organization"/"IHO", and nested `subdivision` elements for sub-committees.
|
|
45
|
-
- **commentperiod**: Moved from top-level into `ext`.
|
|
46
|
-
- Factory methods (`bib_item`, `create_doctype`, `create_relation`) return IHO-specific classes.
|
|
47
|
-
|
|
48
|
-
This module is required on demand (`require "relaton/iho/hash_parser_v1"`) and not auto-loaded by the main `relaton/iho` entry point.
|
|
49
|
-
|
|
50
|
-
### Load Order Constraint
|
|
51
|
-
|
|
52
|
-
In `item.rb`, `require_relative "relation"` is placed **after** the `Item` class definition because `Relation` → `ItemBase` → `Item` creates a circular dependency if loaded before `Item` exists.
|
|
53
|
-
|
|
54
|
-
### Data Retrieval
|
|
55
|
-
|
|
56
|
-
`Bibliography.search`/`.get` fetches YAML from the relaton-data-iho GitHub repo via `Relaton::Index`, then deserializes into model objects.
|
|
57
|
-
|
|
58
|
-
**Editions and series (the pubid 2.0.5 component model, relaton#440).** The pubid attribute is `edition` (dotted `a.b.c`, `IHO B-12 3.0.0`; the pre-440 name `version` is gone). A reference with no edition answers with the **latest**: `recency_key` compares the segments as integers (`2.10.0` after `2.9.0`). It used to be `min_by` on the string, which returned the oldest.
|
|
59
|
-
|
|
60
|
-
**Serial locators are volumes (relaton#440 unified the year locators of relaton#407).** The yearbook year IS a volume: `IHO P-5 Vol. 2020` is canonical, the bare-year faces (`P-5 2020`, `P-5/2020`) parse as legacy aliases into the same `volume`, and the IHR is `P-1 Vol. 30` with `Issue 1` (no parens) or the short slash form `P-1/30`. A bare series query answers with the latest volume/issue. **Components can publish separately and carry their own edition:** `IHO S-66 Suppl 1 2.0.0` gives `supplement_edition`, `S-100 Part S-102 1.0.0` gives `part_edition`, annexes likewise — and the publication years in parens (`S-52 Annex A 4.0.3 (2021)`) are `*_year` metadata, not identity: a stated year pins the record, an omitted one wildcards. Matching (`pubid_match?`) treats a discriminator the query omits as a wildcard: `IHO P-1 Vol. 30` matches any issue of volume 30. The comparison runs through pubid's `#exclude` — excluding `:volume` nils the `issue`, excluding an `:appendix` nils its `appendix_subnumber` (scalar clusters) — the way BIPM's `stem_keys` does. The subdivisions (part/appendix/annex/supplement) stay strict: this flavor cannot use the `===` subset match, because pubid does not declare `subset_strict` for IHO's subdivisions and a nil `part` would read as "any part" (`IHO S-100` would answer with a part row).
|
|
61
|
-
|
|
62
|
-
**The edition reaches the index only through the committed row.** A record prints `B-12` and keeps its edition in `edition:`, so `relaton index` cannot read a version from the docid. `MachineIndex#refine` (relaton-cli) takes the committed `index-v3.yaml` row when it adds only a key in `REFINABLE_KEYS` to the parse — `version`/`edition`/`volume`/`year`/`issue` (relaton#407 added the year/issue pair for the series rows). Before that, the Pages index (generated 2026-10-29) had no `version` in any of its 325 rows, so every versioned reference answered not-found (relaton-data-iho#20). The suite did not see it: `spec/fixtures/index-v3.zip` is the committed index, which has versions. The suite's fixture carries the #48-era re-keyed series rows migrated to the #440 keys (`version`→`edition`/`<component>_edition`, year locators→`volume`; `support/migrate_index_fixture.rb` did the one-shot migration) — refresh the fixture from the live published zip once relaton-data-iho round 2 republishes, and reconcile any file-name drift then.
|
|
63
|
-
|
|
64
|
-
### Test Infrastructure
|
|
65
|
-
|
|
66
|
-
- **VCR cassettes** (`spec/vcr_cassettes/`): recorded HTTP responses for offline testing
|
|
67
|
-
- **RelaxNG validation**: specs validate XML output against `grammars/relaton-iho-compile.rng` using ruby-jing
|
|
68
|
-
- **Fixtures** (`spec/fixtures/`): reference XML/YAML documents for round-trip serialization tests
|
|
69
|
-
|
|
70
|
-
### IHO-Specific Document Types
|
|
71
|
-
|
|
72
|
-
policy-and-procedures, best-practices, supporting-document, report, legal, directives, proposal, standard
|
|
73
|
-
|
|
74
|
-
## Testing
|
|
75
|
-
|
|
76
|
-
- **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iho.
|