relaton 3.0.0.pre.alpha.17 → 3.0.0.pre.alpha.18

data/lib/relaton/ieee/CLAUDE.md

@@ -0,0 +1,176 @@
+# CLAUDE.md
+
+## Development
+
+- `bundle install` — install dependencies
+- `bundle exec rake spec` — run tests
+- `bundle exec rubocop` — lint
+
+## Index (pubid-structured `index-v2`)
+
+IEEE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`
+in `lib/relaton/ieee.rb`): each row's `:id` is a `Pubid::Ieee::Identifier`
+serialized to the `_type: pubid:ieee:*` (lutaml) form, keyed on the index via
+`pubid_class: ::Pubid::Ieee::Identifier`.
+
+- **Write side.** Unlike the doc-fetching pipeline, the index is built by a
+  **separate step**: `Relaton::Ieee::DataFetcher.build_index(dir:, index_file:)`
+  (`data_fetcher.rb`). It globs the per-document `data/*.yaml` files, extracts each
+  primary non-trademark IEEE docidentifier, parses it via `.pubid` (a guarded
+  `::Pubid::Ieee::Identifier.parse` + `from_hash(to_hash) == to_hash` round-trip
+  check), `add_or_update`s the object, and `save`s. Ids pubid can't parse/round-trip
+  are skipped; a per-file skip warning **and** a final
+  `IEEE index-v2.yaml: N/M indexed, K skipped (P% coverage)` line are logged so the
+  loss is never silent. `DataFetcher.fetch` (doc generation) is unchanged — the
+  data repo (`relaton-data-ieee`) calls `build_index` after obtaining `data/`.
+- **Read side.** `bibliography.rb#search` passes `pubid_class:` to `find_or_create`,
+  parses the query with `parse_pubid` (falling back to the raw String on failure),
+  passes the pubid to `index.search` for number-narrowing, and picks the row by
+  `row[:id].to_s`. `processor.rb#remove_index_file` passes `url: true` and `file:`
+  only: the delete never reads the index (see `lib/relaton/index/CLAUDE.md`).
+
+## Identifier resolution (`RawbibIdParser`, pubid-first)
+
+`RawbibIdParser.parse(normtitle, stdnumber)` turns a raw IEEE catalog
+`normtitle` into a **`::Pubid::Ieee::Identifier`** (`idams_parser.rb`/prefilter
+consume `.to_s` / `.to_s(trademark: true)` / docnumber). It is **pubid-first**:
+pubid parses ~93.5% of normtitles directly; the bespoke regex case statement
+(`#parse_fallback`, ~430 lines) only normalizes the messy remainder into a
+canonical string pubid then parses (→ 98.7% pubid objects, 100% coverage).
+
+- **Faithfulness guard.** `#parse` always renders the bespoke reference
+  (`parse_fallback(...).to_id`) and only accepts a pubid parse when it kept every
+  non-date digit (`#faithful?`, a date-stripped **digit** multiset — digit-level
+  so a benign re-split like `P1476/D4` vs `P14764` counts as faithful while a real
+  loss like `/D3.0`→`/D.0` does not). If no pubid parse is faithful it returns the
+  private fallback `Renderer` (the old bespoke id string) — so there is **zero
+  silent digit loss**. ~1.3% of docs keep the `Renderer`.
+- **An id must have a digit in its number.** The last two fallback branches
+  ("publisher, number" and the `else` that uses `stdnumber`) accept a number
+  only when it has a digit. Without that rule the first two words of any title
+  became an id (`A Call to Action …` → `A Call`), and a category `stdnumber`
+  became one too (`White Paper` → `IEEE White Paper`). Since the id is the
+  output filename, 176 records shared 42 files and 134 were overwritten. Now
+  `parse` returns nil for them, and `DataFetcher#parse_entry` skips them with
+  the `PubID parse error` warning. A real standard with a title-like
+  `normtitle` falls through to `stdnumber` and keeps its id
+  (`IEEE Standard for Ethernet …` / `802.3bm-2015` → `IEEE Std 802.3bm-2015`);
+  16 such standards were hidden by the old branch. **The digit rule alone is
+  not enough** — see `#no_standard_number?` under **Fetch robustness**.
+- **No public `PubId` class.** The old `Relaton::Ieee::PubId` is gone; its
+  rendering survives only as the private `RawbibIdParser::Renderer` used by the
+  fallback. pubid (`::Pubid::Ieee::Identifier`) is the identifier abstraction.
+- **Coordination with pubid.** IEEE ids now render in pubid's canonical form
+  (`/D-5`→`/D5`, `/Amd 5-2012` with a space, `802.16-2012 - Redline`, trademark
+  `™`/`®` attached to the document **number**, full corrigendum expansion). This
+  required a batch of pubid fixes (draft-verbatim, redline, revision incl.
+  numbered `RevN`, edition, trademark, historical formats, update_codes one-offs).
+  `Core::DataFetcher#output_file` also sanitizes commas so pubid's `", Mar 2011"`
+  dates don't leak into filenames. The fallback `Renderer`'s suffix strings are
+  **not** the canonical spellings and shouldn't be "fixed" to match: they are the
+  input to `to_id`, which pubid re-parses, so `/D-N`, `/R-N`, `/E-N` are
+  deliberately the *parseable* forms pubid canonicalizes to `/DN` etc. The
+  **trademark** rendering is the exception that must track pubid: `IdamsParser`
+  emits the `scope: trademark` docidentifier with `to_s(trademark: true)` on
+  whatever `parse` returned, so the `Renderer`'s ®/™ rule
+  (`Renderer::Id::REGISTERED_SERIES`) mirrors pubid#322 — series read off the
+  *number* with a leading project `P` stripped, `8802` ungated, everything else
+  gated on IEEE being a (co)publisher. Otherwise a `P802.x` draft would print ™
+  down the fallback path and ® down the pubid path. The one
+  exception is `amd_to_s`, which emits the **spaced** `/Amd N`: pubid parses both
+  spellings, so the spaced one costs nothing and keeps the rendering canonical
+  for the ~1.3% of ids that miss the faithfulness guard and keep the `Renderer`
+  string. Changing any of these also moves the `extract_index_entry` dedup key in
+  `data_fetcher.rb`.
+
+## Fetch robustness (no silent corpus loss)
+
+Amendment docnumbers can be pathologically long — pubid's `to_s` embeds the full
+"(Amendment to … as amended by …)" clause (300+ chars). Two guards keep one such
+id from silently halving the crawl (see the `data_fetcher.rb` history):
+
+- **A record with no standard number is skipped.** `#no_standard_number?` is
+  true when `publicationsubtype` is in `NON_STANDARD_SUBTYPES` (`Whitepapers`,
+  `Research Documents`) **and** `stdnumber` has no digit. IEEE gives such a
+  document no designation and cites it by title, so every id built from it is
+  a title fragment. The digit rule in `RawbibIdParser` cannot catch all of
+  them, because pubid parses a digit out of the title itself
+  (`IEEE 802 Nendica Report: …` → `IEEE Std 802`, three documents; `IEEE 3D
+  Body Processing …` → `IEEE Std 3D`, three more). Both conditions are
+  required, so a record of such a subtype that does carry a number keeps it
+  (`Amendment 1:2011 to SMPTE ST 430-9:2008`). The check runs in **two**
+  places: `#parse_entry` (with a warning) and `#extract_index_entry` — the
+  prefilter groups files by docnumber and keeps the newest of each group, so a
+  white paper that reads as `IEEE Std 802` would otherwise join the real
+  standard's group and push its file out. Measured on the raw corpus: 157
+  records skipped, and no kept document has a digit-less id. The subtype match
+  is exact, which the data supports — all 570,744 raw files carry one of three
+  spellings (`Standard Docs` 564,611, `Whitepapers` 5,240,
+  `Research Documents` 891), so a new spelling is the one thing to watch for.
+- **Bounded filenames.** `Core::DataFetcher#output_file` caps the basename at
+  `MAX_BASENAME_BYTES` (255): when the sanitized id exceeds the OS limit it
+  byte-truncates (`byteslice(…).scrub("")`, valid UTF-8) and appends a 12-char
+  SHA1 of the **full** docid — bounded, unique, and **deterministic** so
+  `save_doc`/`reconcile_staged_outputs`/`read_bib` all resolve the same name.
+  Without this, `File.write` raised `Errno::ENAMETOOLONG`.
+- **Per-document commit guard.** Both the serial (`run_shard`) and parallel
+  (`spawn_batch` fork) paths route through one `commit_entry` helper that wraps
+  `commit_doc` in a `rescue` (logs via `Util.error`, continues). A single bad doc
+  no longer aborts a serial crawl, and a forked worker still reaches its
+  `File.binwrite(state_path, …)` so `merge_state_files` sees its `saved_writes`
+  and `reconcile_staged_outputs` doesn't delete the whole batch's staged files.
+  Reconcile also logs the straggler-cleanup count so an anomaly isn't silent.
+- **A title-fragment id can still slip past `#no_standard_number?`.**
+  `#no_standard_number?` only fires for a `publicationsubtype` in
+  `NON_STANDARD_SUBTYPES`, and a generic mid-tier `RawbibIdParser` fallback
+  branch meant for a real catalog number (e.g. "publisher, number, part",
+  built for `"IEEE 802.3"`) can still read a short, numbered title as one
+  (`"WEB 3.0: The Evolution of ..."` → `WEB 3.0`) for a subtype spelling that
+  list doesn't cover. `RawbibIdParser.fabricated_title_id?(pid, title,
+  isbn_present)` is a second, subtype-agnostic guard on the *resolved id*
+  itself, called from both `IdamsParser#pubid` (full parse, `title` from
+  `@doc.btitle` — distinct from `normtitle`, which for a genuine standard
+  holds its designation, not prose) and `DataFetcher#extract_index_entry`
+  (cheap-parse prefilter, `title` the *last* `<title>` in the raw XML — the
+  first is always the top-level designation, present even for a real
+  standard). It must run in both places for the same reason
+  `#no_standard_number?` does: the prefilter groups files by docnumber before
+  the full parse ever runs, so a fabricated id left un-nilled there still
+  merges a whitepaper's group with an unrelated document's and can silently
+  drop a file from the corpus.
+  **The check is on the id's *class*, not (only) the title prefix:** it
+  requires `pid.is_a?(Renderer)` — i.e. the unvalidated bespoke fallback
+  string, never a real `::Pubid::Ieee::Identifier` that pubid's own grammar
+  actually parsed — **and** the title is a literal prefix of `pid.to_s`
+  **and** the document carries a real ISBN. Title-prefix and ISBN alone are
+  not enough: measured over 9,091 successfully-parsed raw-corpus records, 7
+  genuine, correctly-resolved standards have a title that literally begins
+  with their own real designation and also carry an ISBN (`"IEEE Std
+  1636.99-2013, IEEE Standard for ..."`) — an earlier version of this guard
+  without the `Renderer` restriction nulled out their real ids. All 7 resolve
+  to a real `Pubid::Ieee::Identifier`, never a `Renderer`, so the class check
+  is what tells them apart from the 25 known fabricated ids in a
+  `relaton-data-ieee` audit, all of which are `Renderer`s. A flagged record is
+  treated like any other unparseable id: `docnumber` is nil and
+  `DataFetcher#parse_entry` skips it with the `PubID parse error` warning.
+
+## Testing
+
+- **Framework:** RSpec with VCR cassettes and WebMock.
+- **Index fixture:** `spec/ieee/fixtures/index-v2.zip` (a small structured index for
+  the read-spec targets) is deserialized through `::Pubid::Ieee::Identifier` and
+  pre-loaded into the `Relaton::Index` pool in `before(:suite)`
+  (`spec/ieee/support/webmock.rb`, NIST recipe). `index_builder_spec.rb` covers
+  `build_index` (structured rows + skip logging).
+  When a document 404s after a cassette refresh, the fixture has gone stale
+  against the republished index: copy the matching row **wholesale** (`:id:` *and*
+  `:file:`) out of the live `…/relaton-data-ieee/refs/heads/v2/index-v2.zip`,
+  rewrite the zip with the single `index-v2.yaml` entry, and **delete the cassette**
+  so it re-records — `record: :once` will not re-record a *changed* request URI, it
+  raises `UnhandledHTTPRequestError` instead.
+- **Expected-XML fixtures self-heal only when absent.** `ieee_spec.rb` and
+  `idams_parser_spec.rb` write `fixtures/<name>.xml` with `unless File.exist?`, so
+  reconciling a pubid rendering change means deleting the fixture, re-running, and
+  reviewing the diff — not hand-editing it.
+- **VCR cassettes:** `spec/ieee/vcr_cassettes/` — `index-v2.zip` downloads are
+  ignored by VCR (handled by the fixture).

data/lib/relaton/ieee/README.adoc

@@ -0,0 +1,131 @@
+= Relaton::Ieee
+
+RelatonIeee is a Ruby gem that implements the https://github.com/relaton/relaton-model-ieee#ieee-bibliographic-item-model[IeeeBibliographicItem model].
+
+You can use it to retrieve metadata of IEEE Standards from standards.ieee.org, and access such metadata through the `Relaton::Ieee::ItemData` object.
+
+== Usage
+
+=== Search for a standard using keywords
+
+[source,ruby]
+----
+require 'relaton/ieee'
+=> true
+
+item = Relaton::Ieee::Bibliography.search("IEEE Std 528-2019")
+=> #<Relaton::Ieee::ItemData:0x000000012891cf70
+...
+----
+
+=== XML serialization
+
+[source,ruby]
+----
+item.to_xml
+ => "<bibitem id="IEEEStd5282019" type="standard" schema-version="v1.4.1">
+      <fetched>2026-01-30</fetched>
+      <title type="main">IEEE Standard for Inertial Sensor Terminology</title>
+      <uri type="src">https://ieeexplore.ieee.org/document/8863799</uri>
+      <docidentifier type="IEEE" primary="true">IEEE Std 528-2019</docidentifier>
+      ...
+    </bibitem>"
+----
+
+With argument `bibdata: true` it ouputs XML wrapped by `bibdata` element and adds flavor `ext` element.
+
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.4.1">
+      <fetched>2026-01-30</fetched>
+      <title type="main">IEEE Standard for Inertial Sensor Terminology</title>
+      <uri type="src">https://ieeexplore.ieee.org/document/8863799</uri>
+      <docidentifier type="IEEE" primary="true">IEEE Std 528-2019</docidentifier>
+      ...
+      <ext schema-version="v1.0.1">
+        <doctype>standard</doctype>
+        <flavor>ieee</flavor>
+        ...
+      </ext>
+    </bibdata>"
+----
+
+=== Get document by reference
+[source,ruby]
+----
+Relaton::Ieee::Bibliography.get("IEEE Std 528-2019")
+[relaton-ieee] INFO: (IEEE Std 528-2019) Fetching from Relaton repository ...
+[relaton-ieee] INFO: (IEEE Std 528-2019) Found: `IEEE Std 528-2019`
+=> #<Relaton::Ieee::ItemData:0x00000001288b9380
+ ...
+----
+
+=== Typed source links
+
+Each IEEE document has `src` type link.
+
+[source,ruby]
+----
+item.source.first.type
+=> "src"
+
+item.source.first.content
+=> "https://ieeexplore.ieee.org/document/8863799"
+----
+
+=== Create bibliographic item from XML
+[source,ruby]
+----
+Relaton::Ieee::Item.from_xml File.read "spec/fixtures/ieee_528_2019.xml"
+ => #<Relaton::Ieee::ItemData:0x0000000128815a00
+ ...
+----
+
+=== Create bibliographic item from YAML
+[source,ruby]
+----
+ Relaton::Ieee::Item.from_yaml File.read('spec/fixtures/item.yaml')
+=> #<Relaton::Ieee::ItemData:0x0000000122f70c88
+ ...
+----
+
+=== Fetch data
+
+There is an IEEE dataset https://github.com/relaton/ieee-rawbib which can be converted into BibXML/BibYAML formats. The dataset needs to be placed into the local directory.
+
+The method `Relaton::Ieee::DataFetcher.fetch(source, output: "data", format: "yaml")` converts all the documents from the local `ieee-rawbib` directory and saves them to the `./data` folder in YAML format.
+Arguments:
+
+- `source` - a name of data source (default 'ieee-rawbib').
+- `output` - folder to save documents (default './data').
+- `format` - the format in which the documents are saved. Possible formats are: `yaml`, `xml` (default `yaml`).
+
+[source,ruby]
+----
+Relaton::Ieee::DataFetcher.fetch
+Started at: 2021-09-24 17:55:07 +0200
+Stopped at: 2021-09-24 17:57:30 +0200
+Done in: 143 sec.
+=> nil
+----
+
+=== Logging
+
+RelatonIeee uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To update the index test fixture (used by tests), run `rake spec:update_index`. This downloads the latest `index-v1.zip` from the https://github.com/relaton/relaton-data-ieee[relaton-data-ieee] repository.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton_ieee.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/relaton/ietf/CLAUDE.md

@@ -0,0 +1,357 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Gem Does
+
+The `Relaton::Ietf` flavor fetches and parses IETF bibliographic data (RFCs, Internet-Drafts, BCPs, FYIs, STDs) into the Relaton data model. Models are `lutaml-model`-based.
+
+## Commands
+
+The suite runs from its own directory (`spec/ietf/`), as every flavor suite does.
+
+```bash
+bundle exec rake spec:ietf                                      # run this flavor's suite
+cd spec/ietf && bundle exec rspec -I . relaton/ietf/bibxml_parser_spec.rb  # single file
+cd spec/ietf && bundle exec rspec -I . relaton/ietf/rfc/entry_spec.rb:19   # single example
+cd spec/ietf && bundle exec rspec -I . -e "creates primary docid"          # by description
+bundle exec rubocop                                             # lint
+```
+
+## Architecture
+
+### Data Model (Lutaml::Model)
+
+All models use `Lutaml::Model::Serializable` with `attribute` declarations and `xml do ... end` blocks for serialization. The IETF classes extend base `Relaton::Bib` classes:
+
+- `Relaton::Ietf::ItemData < Bib::ItemData` — core bibliographic data
+- `Relaton::Ietf::Item < Bib::Item` — adds `ext: Ext` attribute
+- `Relaton::Ietf::Bibdata < Item` / `Bibitem < Item` — include shared serialization concerns
+- `Relaton::Ietf::Ext` — IETF extensions: `doctype`, `flavor`, `stream`, `area`, `ipr`, `pi`
+- `Relaton::Ietf::Doctype` — types: `"rfc"`, `"internet-draft"`
+
+### Namespace Resolution
+
+Converter classes use `Bib::NamespaceHelper` which resolves `namespace` by taking the first two segments of the class name. For `Relaton::Ietf::BibXMLParser::FromRfcxml`, `namespace` returns `Relaton::Ietf`, so `namespace::ItemData` → `Relaton::Ietf::ItemData`, `namespace::Ext` → `Relaton::Ietf::Ext`, etc.
+
+### Key Flows
+
+1. **Single document lookup**: `Processor#get` → `Bibliography.get(code)` → `Scraper.scrape_page` → parses the reference with `Pubid::Ietf` → searches the one combined `index-v2` from `relaton-data-ietf` → fetches that row's YAML
+
+2. **Bulk data fetching**: `DataFetcher` extends `Relaton::Core::DataFetcher` with three datasets:
+   - `ietf-rfcsubseries` / `ietf-rfc-entries`: parse `rfc-index.xml` via `Rfc::Index` / `Rfc::Entry#to_item`
+   - `ietf-internet-drafts`: parse local BibXML files via `BibXMLParser.parse`
+
+3. **BibXML parsing** (`BibXMLParser` module):
+   - `parse(xml)` — parses `<reference>` elements via `FromRfcxml` converter
+   - `parse_rfc(xml)` — parses `<rfc>` root documents via `FromRfc` converter
+   - Converters inherit from `Bib::Converter::BibXml::FromRfcxml` with IETF overrides for contributor/person/org handling
+
+4. **RFC Index parsing** (`Rfc::Entry#to_item`): converts RFC editor index entries (BCP/FYI/STD/RFC) into `ItemData`, handling subseries vs full RFC entries differently
+
+### Converter Inheritance Chain
+
+```
+Bib::Converter::BibXml::FromRfcxml    # base: handles <reference> generically
+  └── Ietf::BibXMLParser::FromRfcxml  # IETF overrides: publisher, org recognition, person names
+        └── Ietf::BibXMLParser::FromRfc  # <rfc> root: uses doc_name instead of anchor, no ref-level series_info
+```
+
+`FromRfc` must override all methods that access `@reference.anchor`, `@reference.target`, `@reference.format`, or `@reference.series_info` since `Rfcxml::V3::Rfc` lacks these attributes (unlike `Rfcxml::V3::Reference`).
+
+## Unified index (relaton#109)
+
+Producer and consumer are both on the pubid `index-v2` now. `DataFetcher` writes
+it; `Scraper` reads one combined index from `relaton-data-ietf` — replacing the
+three per-type `index-v1` files (`:RFC`/`:RSS`/`:IDS`) it used to read from
+`relaton-data-{rfcs,rfcsubseries,ids}`. Those three keep publishing `index-v1`
+untouched, for released relatons.
+
+`INDEXFILE` is a single constant again (`index-v2`); the `INDEXFILE_V2` that
+existed while the two halves were split is gone.
+
+Decisions taken (2026-08-18 ... 2026-08-24):
+
+| | |
+|---|---|
+| Index structure | **unchanged** - `:id` + `:file` only. No `docids[]`, no summary fields; a third key invalidates the whole index for every released gem |
+| `:id` shape | structured pubid hash (`Pubid::Ietf::Identifier#to_hash`) |
+| Row per record | keyed by the record's **primary** docidentifier. A sub-series id resolves to its own container record, *not* to a constituent RFC - 42 of 365 sub-series include more than one RFC; consumers follow `relation: includes` |
+| DOI ids | omitted - not `pubid:ietf`-representable, and one unparseable row kills the index. DOIs are served by the `doi` flavor |
+| Who builds it | `DataFetcher`, during the crawl - not a separate pass over the written records, which would be a second answer to "what is this record's id" |
+| Three type repos | untouched: they pin `relaton/relaton-ietf`, not this monorepo, so they keep emitting `index-v1` |
+
+**Resolved 2026-08-26.** `relaton-data-ietf`'s RFC and sub-series records were
+schema-invalid — missing `ext.doctype`, and RFC committee organizations carrying
+`subdivision` with no `name` — because that repo converted the `ietf-tools`
+v1.2.3 mirrors rather than crawling. It now runs `Relaton::Ietf::DataFetcher`
+directly, which synthesises `doctype`, `status`, `stream`, the organization names
+and the resolved WG titles. The two `pending` examples in `ietf_spec.rb` reported
+`FIXED` on the first run against the new corpus and have been removed.
+
+Two consequences of that migration to know about: record filenames are now
+lowercase *and* unpadded (`data/RFC8341.yaml` -> `data/rfc8341.yaml`,
+`data/STD0066.yaml` -> `data/std66.yaml`), since `output_file` derives them from
+the docnumber; and the record content changed enough that the three XML fixtures
+had to be regenerated.
+
+### The consumer: `Scraper` queries with parsed identifiers
+
+```ruby
+Relaton::Index.find_or_create(
+  :IETF, url: "#{IETF}#{INDEXFILE}.zip", file: "#{INDEXFILE}.yaml",
+         pubid_class: ::Pubid::Ietf::Identifier
+)
+```
+
+`parse_id` turns the reference into a `Pubid::Ietf::Identifier` before searching,
+and **that is not cosmetic**: `Type#search_candidates` narrows only for non-String
+queries, so a String falls into `match_item`'s `item[:id].to_s.include?(id)`, which
+renders every pubid in the index on every lookup. Measured against the live
+177k-row index: **~43 s per reference with a String, sub-millisecond with a parsed
+id.** A String query also returns *nothing* if `pubid_class:` is omitted, since the
+rows would then be raw hashes.
+
+`parse_id` also normalises the Internet-Draft spellings - `I-D.<slug>` and
+`I-D <slug>` - onto the `draft-...` form, adding the `draft-` stem when absent
+(`I-D.ietf-quic-transport`, the bibxml anchor and the `docnumber` records carry).
+The old plain-string index matched that by substring; matching is exact now.
+A reference pubid cannot parse returns nil, so `Bibliography.get "CN 8341"` logs
+"Not found." rather than raising.
+
+### The producer: `DataFetcher` writes `index-v2`
+
+`DataFetcher` builds the index during the crawl, from the identifier it already
+holds for each record — `record_index_entry` parses the primary docidentifier and
+`add_or_update`s the pubid:
+
+```ruby
+Relaton::Index.find_or_create(
+  :IETF, file: "#{INDEXFILE}.yaml", pubid_class: ::Pubid::Ietf::Identifier
+)
+```
+
+**Both halves are load-bearing.** `FileIO#save` serialises an id to its `_type:`
+hash only when it is an instance of the configured `pubid_class`, so passing a
+String to a `pubid_class:` index writes a v1-shaped file under a v2 name — no
+error, just an index that never narrows.
+
+Three properties worth keeping:
+
+- **Key on the primary docidentifier**, which `serialize_and_write` already
+  picks. Not the first entry: draft records carry an `I-D.foo` anchor id (not a
+  pubid grammar) and their unversioned series id, which is the *primary* of a
+  separate aggregator record — keying on either collides on one `id.to_s` and
+  `add_or_update` silently overwrites `:file`.
+- **DOIs stay out.** `10.17487/RFC3986` is not a `pubid:ietf` identifier, and one
+  unparseable row invalidates the whole index. The primary is never the DOI, so
+  this is already right — don't "improve" it into indexing every docidentifier.
+- **Sub-series key to their own record.** `STD 66` → the STD container, not a
+  constituent RFC: 42 of the 365 published sub-series include more than one, so
+  there is no single document to alias to. Consumers follow `relation: includes`.
+
+A record whose id pubid rejects is written but not indexed, warned about, and
+tallied for a summary at the end of the crawl. The load is all-or-nothing —
+`deserialize_id` raises on the first bad id and `load_index` then rejects the
+entire index — so one malformed upstream record must cost one document, never
+every lookup. All 176,862 published ids parse today; this guards against drift.
+
+**Blast radius, worth re-checking before touching this.** The six data repos
+feeding today's released consumers do *not* use this monorepo: `ietf-tools/*` pin
+`relaton/relaton-ietf` @ `main` (`RelatonIetf::DataFetcher`), and
+`relaton/relaton-data-{rfcs,rfcsubseries,ids}` pin it @ `lutaml-integration` —
+both branches carry their own `INDEXFILE = "index-v1"` and their own fetcher. Only
+`relaton/relaton-data-ietf` runs this one. So the producer flip needs no
+dual-write and no cutover window.
+
+### Still outstanding
+
+Retiring `relaton-data-{rfcs,rfcsubseries,ids}`'s `index-v1` files, and their
+`data-index/configs.yml` rows, waits until a relaton carrying this change has
+shipped — see `/work/HANDOFFS/relaton__support__ietf-index-config-and-cron.md`.
+
+## Filename collisions: workers write, the parent settles the names
+
+`Core::DataFetcher#output_file` sanitizes `.`, `,`, `/`, `:`, `(`, `)`, `-` and
+whitespace all to `-`, so two **distinct** docids can want one path. This flavor
+is the only one that writes from forked processes, so it is the only one where
+that could destroy a record without a word.
+
+It did. In the first production crawl of `relaton-data-ietf`,
+`data/draft-ietf-6man-hbh-processing.yaml` — the aggregator for a draft with 21
+versions — was written and then overwritten by a singleton built from the
+malformed sibling id `draft-ietf-6man-hbh-processing-`, which sanitizes to the
+same name. The published corpus came out 177,362 index rows over 177,361 files,
+and nothing in the log said so. Exactly one such group exists in that corpus;
+the mechanism, not the blast radius, is the problem.
+
+Three pieces close it, and all three are load-bearing:
+
+1. **`@cross_process = true`, set in `fetch_ieft_internet_drafts` before the
+   first `parallelize`** so the forked children inherit it. It tells
+   `Core::DataFetcher#write_unique` that an existing file may belong to a peer
+   and must never be overwritten.
+2. **`serialize_and_write` writes through `write_unique`** and reports two extra
+   keys: `docid` (the id the file was written under) and `plain_file` (the name
+   it would have taken uncontested). `docid` is **not** `index_id` — `id` falls
+   back through `docnumber` and `formattedref`, so on the RFC path the two
+   differ (`RFC0001` vs `RFC 1`, i.e. stems `rfc0001` and `rfc-1`). Keying the
+   reconciliation on `index_id` renames records to the wrong names.
+3. **`reconcile_output_files`, in the parent, before `record_index_entry`.** A
+   worker only learns that a path was taken, never which clashing docid deserves
+   it, so left alone the winner would follow the race and the two filenames would
+   swap between crawls. The parent sees every docid, so it decides: within a
+   group of records that wanted one path, the alphabetically first docid keeps
+   it and the rest take their digest variant.
+
+Details of (3) that are easy to get wrong:
+
+- It runs over **every** group, not just clashing ones. A lone record that fell
+  back to a digest path must get its plain name back, or the published filename
+  churns and the old file is orphaned. `next if r[:file] == target` makes that
+  free for ~100% of records.
+- **Losers move first.** A loser may be sitting *on* the plain path; moving the
+  winner there first would destroy it.
+- Two results for the **same** docid are collapsed, not split: the stray file is
+  deleted and one warning is emitted, which is what the crawl did before. The
+  stray may already be **gone** — the fallback name is keyed on the docid, so two
+  workers that both lost the race to the plain path land on one name and the
+  first of them renames it onto the target. The result must follow the document
+  to the target anyway, or the index gets a row pointing at nothing.
+- Sorting *unique docids* leaves no tie for Ruby's unstable `sort_by` to resolve
+  differently between runs.
+- A failed rename warns and carries on. A multi-hour crawl must not die at the
+  last step.
+
+`report_collisions` restates the total once at the end of the crawl, beside
+`report_unparsed` and `report_unindexed` — one line for a crawl that writes
+~177k records.
+
+The sequential paths (`fetch_ieft_rfcsubseries`, `fetch_ieft_rfcs`) get no
+reconciliation pass and need none: one process, so `unique_output_file` is
+already deterministic there. They do now emit a digest filename for a genuine
+cross-docid collision instead of overwriting, which follows `rfc-index`
+iteration order — stable for a given input.
+
+## Thin records inherit from their constituents
+
+Two builders produce records for documents that have **no upstream source of
+their own**, and both would otherwise publish without date, doctype or source:
+
+| Builder | File | Inherits from |
+|---|---|---|
+| `to_subseries_item` | `rfc/entry.rb` | the newest RFC in its `is-also` |
+| `build_unversioned_doc` | `data_fetcher.rb` | the newest version in `sorted` |
+
+This is **not** a parsing gap to go fix in the XML mapping. A sub-series entry in
+`rfc-index.xml` is only a pointer:
+
+```xml
+<bcp-entry>
+  <doc-id>BCP0003</doc-id>
+  <is-also><doc-id>RFC1915</doc-id></is-also>
+</bcp-entry>
+```
+
+The RFC Editor publishes no date, title, author or status for it, because a
+sub-series *has* no metadata of its own — it is a label on one or more RFCs.
+Unversioned draft aggregators are further still from an upstream document: they
+are synthesised from the versions found on disk.
+
+Neither site needs a second pass or another read of the corpus — `rfc_index` and
+`sorted` already hold the constituents at build time. (A downstream
+implementation in `relaton-data-ietf`'s crawler did use a second pass over
+written files, because its groups could span source repos; that constraint does
+not exist here.)
+
+**Constituent lookup is normalised on both sides, deliberately.**
+`Entry.squish` folds case and strips whitespace and dots. Relation targets and the
+docids they reference disagree in the published corpus (`dyndNS` vs `dyndns`),
+which left 28 documents undated downstream on a strict lookup — and, in
+`build_relations`, silently downgraded a full constituent bibitem to a minimal
+one.
+
+`rfc_index` is therefore **keyed by `Entry.squish(doc_id)`, and the caller builds
+it** — `fetch_ieft_rfcsubseries` does so once for the crawl. Normalising inside
+`Entry` instead would rebuild a ~9,800-row table for each of the 367 sub-series
+(measured: ~161 MB retained, ~6 s), all of it the same table.
+
+Why it matters beyond tidiness: the Pages index sorts by date, so undated records
+sort as one undifferentiated block, and a record with no `ext.doctype` renders
+with no document type at all.
+
+Inherited values are `dup`'d rather than shared with the constituent's own bib —
+aliasing them would make a later edit to the aggregator mutate the `-NN` record.
+
+Verified against the live `rfc-index.xml`: **367/367 sub-series dated**, 367/367
+with full constituent bibitems, in 1.4 s; `BCP 3` → 1996-02, `STD 51` → 1994-07,
+`STD 66` → 2005-01 (RFC 3986's date).
+
+## bibxml encoding: transcode, don't scrub, don't skip
+
+Some bibxml files declare `encoding='UTF-8'` and carry **Windows-1252** bytes —
+smart quotes and accented Latin letters. `File.read(encoding: "UTF-8")` only
+*tags* a string; it validates nothing, so those reached lutaml as invalid UTF-8
+and `Rfcxml::V3::Reference.from_xml` raised `Lutaml::Model::InvalidFormatError`.
+
+`read_bibxml` transcodes them (`data_fetcher.rb`). Three things make that the
+right call rather than `scrub`:
+
+- **All of them decode losslessly as CP1252** — 125 of the ~122k published files
+  are affected, 0 unmappable.
+- **The bytes present prove CP1252, not Latin-1.** `0x91`–`0x94` (`‘ ’ “ ”`) are
+  control characters in Latin-1.
+- **`scrub` is lossy.** It replaces each byte with `U+FFFD`: `client’s` becomes
+  `client�s`, `Muñoz` becomes `Mu�oz`. Author surnames are among the
+  affected fields. Verified on a real file — scrub and transcode yield the same
+  *length*, so a length check will not catch the damage.
+
+Don't copy `lib/relaton/bipm/si_brochure_parser.rb:33-35`, whose
+`force_encoding` guard is a no-op. The `scrub` idiom at
+`lib/relaton/itu/data_crawler_r.rb:588-599` is right *there* because it repairs
+scraped HTML with no recoverable original; here there is one.
+
+### Why the rescue is per file, and why the tally is in the parent
+
+`parse_bibxml` rescues `StandardError` — not a narrow list. lutaml raises
+`InvalidFormatError` on bad bytes, but the converter also runs regexes over
+parsed text (`parse_surname_initials` and friends), which raise
+`ArgumentError: invalid byte sequence` on anything that slips through.
+
+It has to be **inside** the worker. The drafts path runs under `Parallel.map`,
+which discards every result from the pass when a worker raises — so one bad file
+out of ~167k meant no index at all, plus the already-written records orphaned on
+disk, because `record_index_entry` and `index.save` only run after both passes
+return. A rescue around `parallelize` would salvage "the crawl didn't die", not
+"the other 166,999 files got indexed".
+
+The skip therefore rides back as `{ unparsed: path, error: }` and is counted in
+the parent, the way `parse_pubid`'s result rides back as `result[:pubid]` — a
+counter incremented in a worker process is lost. `report_unparsed` warns once
+with a count and a sample, not once per file.
+
+A failed version is dropped from `sorted` **before** `link_neighbor_relations`
+and `build_unversioned_doc` run, since both dereference `entry[:bib]`. Knock-on
+to accept: neighbours then link across the gap, and if the dropped file was the
+newest, the aggregator inherits from the previous version. With CP1252 recovery
+that should approach zero — which is the argument for recovering over skipping.
+
+Verified end to end on 41,408 real draft files (16 invalid UTF-8): the crawl
+completes in 57 s, writes 46,461 records and 46,461 index rows, **0 skipped**,
+and the affected files come through with their real characters —
+`\x93Access Node Contr` → `“Access Node Contr`.
+
+## Testing Patterns
+
+- **Index fixture:** `spec/ietf/fixtures/ietf-index-v2.zip` - the combined index, copied **wholesale** from `relaton-data-ietf` rather than curated, since a subset goes stale and surfaces as a 404 on the document fetch. `spec/ietf/support/webmock.rb` deserialises it into pubid ids, sorts by the narrowing key, sets `sorted = true` (without which every lookup is a ~40 s scan) and pre-loads it into the pool as `:IETF` in `before(:suite)`; `spec/ietf/support/vcr.rb` ignores requests ending `index-v2.zip`. Refresh by re-downloading that zip. There is **no** `rake spec:update_index` task in this repo.
+- **VCR cassettes** in `spec/ietf/vcr_cassettes/` record HTTP interactions; tests use `vcr: "cassette_name"` metadata
+- **WebMock** disables net connections by default
+- **Fixtures** in `spec/ietf/fixtures/` — XML/YAML expected outputs; many tests auto-generate fixtures on first run (`File.write file, xml unless File.exist? file`)
+- **An oracle for `read_bibxml` must read binary.** The method is byte-verbatim (`File.binread`), so comparing it against `File.read(path, encoding: "UTF-8")` compares two different strings **on Windows only**: Ruby opens a file in text mode there and collapses CRLF to LF. The repo declares no `.gitattributes` and the GitHub Windows runner image defaults to `core.autocrlf=true`, so a fixture stored with LF is checked out CRLF — which failed all three `windows-latest` jobs while every Linux and macOS job passed. Use `File.read(path, mode: "rb", encoding: "UTF-8")`. macOS cannot reproduce this by checking the fixture out as CRLF, because its text-mode read does not translate; emulate Windows explicitly with `File.read(path, encoding: "UTF-8", universal_newline: true)`. The two `keeps CRLF line endings verbatim` examples pin the contract with `binwrite` temp files, independent of the checkout — one per branch, since the CP1252 repair rebuilds the string run by run and is where a line ending is likeliest to be lost. Note that a byte-equality oracle alone cannot catch a dropped `force_encoding`: `rfc.xml` is pure ASCII, and Ruby compares two ASCII-only strings as equal across encodings, so the example asserts `encoding` separately.
+- **Schema validation** via `ruby-jing` against the shared RNG grammars in the repo-root `grammar/` (`Jing.new "../../grammar/relaton-ietf-compile.rng"`, relative to the suite's CWD `spec/ietf/`)
+- **Shared examples** for org/person parsing in `bibxml_parser_spec.rb` (`parse_org`, `parse_person`)
+- **`equivalent-xml`** gem used for XML comparison (`be_equivalent_to` matcher)
+
+## RuboCop
+
+Inherits from `rubose/rubocop-rubose` (Ribose OSS config). Target Ruby 3.1. Rails cops disabled.