relaton 3.0.0.pre.alpha.17 → 3.0.0.pre.alpha.19

data/lib/relaton/ietf/README.adoc

@@ -0,0 +1,167 @@
+= Relaton::Ietf
+
+_Formerly known as_ `rfcbib`.
+
+Relaton::Ietf is a Ruby gem that searches and fetches standards from the https://www.ietf.org[Internet Engineering Task Force (IETF)], including RFCs.
+
+The standards scrape form https://www.rfc-editor.org/search/rfc_search.php
+
+== Usage
+
+=== Fetching documents
+
+[source,ruby]
+----
+require 'relaton/ietf'
+=> true
+
+# Fetch RFC document
+item = Relaton::Ietf::Bibliography.get 'IETF RFC 8341'
+relaton-ietf] INFO: (IETF RFC 8341) Fetching from Relaton repository ...
+[relaton-ietf] INFO: (IETF RFC 8341) Found: `RFC 8341`
+=> #<Relaton::Ietf::ItemData:0x0000000125608cb0
+...
+
+# Fetch Internet-Draft document
+Relaton::Ietf::Bibliography.get 'IETF I-D.draft-abarth-cake-01'
+[relaton-ietf] INFO: (IETF I-D.draft-abarth-cake-01) Fetching from Relaton repository ...
+[relaton-ietf] INFO: (IETF I-D.draft-abarth-cake-01) Found: `draft-abarth-cake-01`
+=> #<Relaton::Ietf::ItemData:0x000000012141fdd0
+...
+
+# Return nil if a document doesn't exist.
+Relaton::Ietf::Bibliography.get 'IETF 1111'
+[relaton-ietf] INFO: (IETF 1111) Fetching from Relaton repository ...
+[relaton-ietf] INFO: (IETF 1111) Not found.
+=> nil
+----
+
+=== Serialization
+
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="RFC8341" type="standard" schema-version="v1.4.1">
+      <fetched>2026-02-18</fetched>
+      <title type="main">Network Configuration Access Control Model</title>
+      <uri type="src">https://www.rfc-editor.org/info/rfc8341</uri>
+      <docidentifier type="IETF" primary="true">RFC 8341</docidentifier>
+      ....
+    </bibitem>"
+----
+With `bibdata: true` option XML output is wrapped with `bibdata` element and `ext` element added.
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.4.1">
+      <fetched>2026-02-18</fetched>
+      <title type="main">Network Configuration Access Control Model</title>
+      <uri type="src">https://www.rfc-editor.org/info/rfc8341</uri>
+      <docidentifier type="IETF" primary="true">RFC 8341</docidentifier>
+      ...
+      <ext schema-version="v1.0.1">
+        <doctype>rfc</doctype>
+        <flavor>ietf</flavor>
+        <stream>IETF</stream>
+      </ext>
+    </bibdata>"
+----
+
+=== Typed source links
+
+IETF documents may have `src`, `xml`, and `doi` source link types.
+
+* `src` - web publication
+* `xml` - BibXML publication
+* `doi` - DOI reference
+
+[source,ruby]
+----
+item.source.first.type
+=> "src"
+item.source.first.content
+=> "https://www.rfc-editor.org/info/rfc8341"
+----
+
+=== Parse a file locally
+
+[source,ruby]
+----
+item = Relaton::Ietf::BibXMLParser.parse_rfc File.read("spec/ietf/fixtures/rfc.xml")
+=> #<Relaton::Ietf::ItemData:0x000000012913e4d8
+...
+----
+
+=== Create bibliographic item from YAML
+[source,ruby]
+----
+Relaton::Ietf::Item.from_yaml File.read("spec/ietf/fixtures/item.yaml")
+=> #<Relaton::Ietf::ItemData:0x0000000127edf508
+...
+----
+
+=== Fetch data
+
+There are IETF datasets that can be converted into RelatonXML/BibXML/BibYAML formats:
+
+- `ietf-rfcsubseries` - https://www.rfc-editor.org/rfc-index.xml (`<bcp-entry>`, `<fyi-entry>`, `<std-entry>`)
+- `ietf-internet-drafts` - convert files from local directory `./bibxml-ids`. Use `rsync -avcizxL rsync.ietf.org::bibxml-ids ./bibxml-ids` command to fetch the files.
+- `ietf-rfc-entries` - https://www.rfc-editor.org/rfc-index.xml (`<rfc-entry>`)
+
+The method `Relaton::Ietf::DataFetcher.fetch(source, output: "data", format: "yaml")` converts all the documents from the dataset and saves them to the `./data` folder in YAML format.
+
+Arguments:
+
+- `source` - dataset name (`ietf-rfcsubseries` or `ietf-internet-drafts`)
+- `output` - folder to save documents (default './data').
+- `format` - the format in which the documents are saved. Possible formats are: `yaml`, `xml`, `bibxml` (default `yaml`).
+
+[source,ruby]
+----
+Relaton::Ietf::DataFetcher.fetch "ietf-internet-drafts"
+Started at: 2021-12-17 10:23:20 +0100
+Stopped at: 2021-12-17 10:29:19 +0100
+Done in: 1360 sec.
+=> nil
+----
+
+=== The index
+
+Lookups read one combined index — `index-v2.zip` from
+https://github.com/relaton/relaton-data-ietf[relaton-data-ietf], covering RFCs,
+the RFC sub-series and Internet-Drafts in ~177k rows. `DataFetcher` writes the same
+shape as it crawls, one row per record keyed by that record's primary
+docidentifier, serialised as a `Pubid::Ietf::Identifier`:
+
+[source,yaml]
+----
+- :id:
+    _type: pubid:ietf:rfc
+    number: '3986'
+  :file: data/rfc3986.yaml
+----
+
+A record whose identifier `Pubid::Ietf` cannot parse is still written, but is left
+out of the index and logged — a single unparseable row would otherwise make the
+whole index unloadable for every consumer. The total is restated when the crawl
+finishes.
+
+Note sub-series ids key to their own record: `STD 66` resolves to the STD document,
+whose `relation: includes` names the RFC. There is no single RFC to alias to in
+general — 42 of the 365 sub-series include more than one.
+
+=== Logging
+
+Relaton::Ietf uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+The suite uses one cached index fixture, `spec/ietf/fixtures/ietf-index-v2.zip`, copied wholesale from https://github.com/relaton/relaton-data-ietf[relaton-data-ietf]. Refresh it by re-downloading that repo's `index-v2.zip` when a document lookup starts 404ing — a stale index names files that no longer exist, which surfaces as a 404 rather than as an index error.
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/relaton-ietf.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/relaton/iho/CLAUDE.md

@@ -0,0 +1,70 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What This Is
+
+relaton-iho is a Ruby gem that fetches and serializes IHO (International Hydrographic Organization) standards metadata. It is a "flavor" gem in the Relaton ecosystem, extending relaton-bib with IHO-specific bibliographic data models.
+
+## Common Commands
+
+```bash
+bundle exec rspec                        # run all tests
+bundle exec rspec spec/relaton/iho/item_spec.rb  # run a single spec file
+bundle exec rspec spec/relaton/iho/item_spec.rb:15  # run a single example by line
+bundle exec rubocop                      # lint
+bundle exec rubocop -a                   # lint with auto-fix
+bundle exec rake install                 # install gem locally
+```
+
+## Architecture
+
+### Flavor Pattern
+
+This gem follows the Relaton flavor pattern: it subclasses base classes from `relaton-bib` under the `Relaton::Iho` namespace. The key mechanism is `NamespaceHelper` (from relaton-bib), which dynamically resolves the module namespace from the class name. This means `Relaton::Iho::ItemData#to_xml` automatically delegates to `Relaton::Iho::Bibdata.to_xml` (not `Relaton::Bib::Bibdata`).
+
+Core model classes and their base classes:
+- `Item < Bib::Item` — main bibliographic item, uses `model ItemData`
+- `ItemData < Bib::ItemData` — data container enabling namespace-aware `to_xml`/`to_yaml`/`to_json`
+- `ItemBase < Item` — stripped-down item for use inside relations (no id, schema_version, fetched, ext)
+- `Bibitem < Item` — XML bibitem serialization (includes `Bib::BibitemShared`)
+- `Bibdata < Item` — XML bibdata serialization (includes `Bib::BibdataShared`)
+- `Relation < Bib::Relation` — overrides `bibitem` attribute to use `Iho::ItemBase`
+- `Doctype < Bib::Doctype` — IHO document type vocabulary
+- `Ext < Bib::Ext` — IHO-specific extension data (doctype, commentperiod)
+- `Processor < Core::Processor` — Relaton processor for registry integration
+- `HashParserV1` — converts v1-format YAML/Hash data to v2 model objects (required on demand, not auto-loaded)
+- `Bibliography` — fetches IHO data from relaton-data-iho GitHub repo
+
+All model classes use Lutaml::Model for declarative attribute/serialization definitions.
+
+### HashParserV1
+
+`HashParserV1` (in `hash_parser_v1.rb`) handles conversion of legacy v1 data to the current model. It overrides the base `Bib::HashParserV1` to handle IHO-specific structures:
+- **editorialgroup**: IHO v1 data has nested committee arrays (`editorialgroup: [[{committee: ...}, ...]]`). Each committee is converted to a `contributor` with `role type="author"` / `description="committee"`, organization "International Hydrographic Organization"/"IHO", and nested `subdivision` elements for sub-committees.
+- **commentperiod**: Moved from top-level into `ext`.
+- Factory methods (`bib_item`, `create_doctype`, `create_relation`) return IHO-specific classes.
+
+This module is required on demand (`require "relaton/iho/hash_parser_v1"`) and not auto-loaded by the main `relaton/iho` entry point.
+
+### Load Order Constraint
+
+In `item.rb`, `require_relative "relation"` is placed **after** the `Item` class definition because `Relation` → `ItemBase` → `Item` creates a circular dependency if loaded before `Item` exists.
+
+### Data Retrieval
+
+`Bibliography.search`/`.get` fetches YAML from the relaton-data-iho GitHub repo via `Relaton::Index`, then deserializes into model objects.
+
+### Test Infrastructure
+
+- **VCR cassettes** (`spec/vcr_cassettes/`): recorded HTTP responses for offline testing
+- **RelaxNG validation**: specs validate XML output against `grammars/relaton-iho-compile.rng` using ruby-jing
+- **Fixtures** (`spec/fixtures/`): reference XML/YAML documents for round-trip serialization tests
+
+### IHO-Specific Document Types
+
+policy-and-procedures, best-practices, supporting-document, report, legal, directives, proposal, standard
+
+## Testing
+
+- **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iho.

data/lib/relaton/iho/README.adoc

@@ -0,0 +1,119 @@
+= Relaton for IHO documents
+
+Relaton::Iho is a Ruby gem that implements the https://github.com/metanorma/metanorma-model-iso#iso-bibliographic-item[IsoBibliographicItem model].
+
+It currently retrieves metadata of IHO Standards from https://github.com/relaton/relaton-data-iho.
+
+== Usage
+
+=== Search for a standard using keywords
+
+Reference can be specified with or without an edition. References without an edition will return the latest edition of the standard. References with an edition will return the standard with the specified edition.
+
+[source,ruby]
+----
+require 'relaton/iho'
+=> true
+
+# Search for a standard using a reference without an edition
+item = Relaton::Iho::Bibliography.search("IHO B-11")
+[relaton-iho] INFO: (IHO B-11) Fetching from Relaton repository ...
+[relaton-iho] INFO: (IHO B-11) Found: `B-11`
+=> #<Relaton::Iho::ItemData:0x0000000121f85aa8...
+
+# Search for a standard using a reference with an edition
+item = Relaton::Iho::Bibliography.search("IHO B-11 1.0.0")
+[relaton-iho] INFO: (IHO B-11 1.0.0) Fetching from Relaton repository ...
+[relaton-iho] INFO: (IHO B-11 1.0.0) Found: `B-11`
+=> #<Relaton::Iho::ItemData:0x0000000125b5bb90
+...
+----
+
+=== XML serialization
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="B11" type="standard" schema-version="v1.4.1">
+      <fetched>2026-02-20</fetched>
+      <title language="en" type="main">IHO-IOC GEBCO Cook Book</title>
+      <title language="fr" type="main">Livre de recettes GEBCO OHI-COI</title>
+      <uri type="pdf">https://www.star.nesdis.noaa.gov/socd/lsa/GEBCO_Cookbook/documents/CookBook_20191031.pdf</uri>
+      <docidentifier type="IHO" primary="true">B-11</docidentifier>
+      ...
+    </bibitem>"
+----
+With argument `bibdata: true` it outputs XML wrapped by `bibdata` element and adds flavor `ext` element.
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.4.1">
+      <fetched>2026-02-20</fetched>
+      <title language="en" type="main">IHO-IOC GEBCO Cook Book</title>
+      <title language="fr" type="main">Livre de recettes GEBCO OHI-COI</title>
+      <uri type="pdf">https://www.star.nesdis.noaa.gov/socd/lsa/GEBCO_Cookbook/documents/CookBook_20191031.pdf</uri>
+      <docidentifier type="IHO" primary="true">B-11</docidentifier>
+      ...
+      <ext schema-version="v1.0.0">
+        <flavor>iho</flavor>
+      </ext>
+    </bibdata>"
+----
+
+=== Get code
+[source,ruby]
+----
+Relaton::Iho::Bibliography.get "IHO B-11"
+[relaton-iho] INFO: (IHO B-11) Fetching from Relaton repository ...
+[relaton-iho] INFO: (IHO B-11) Found: `B-11`
+=> #<Relaton::Iho::ItemData:0x0000000125f530e0
+...
+----
+
+=== Typed links
+
+Each IHO document has `html`, `website`, or `pdf` type link.
+
+[source,ruby]
+----
+item.source.first.type
+=> "pdf"
+
+item.source.first.content
+=> "https://www.star.nesdis.noaa.gov/socd/lsa/GEBCO_Cookbook/documents/CookBook_20191031.pdf"
+----
+
+=== Create bibliographic item from XML
+[source,ruby]
+----
+Relaton::Iho::Item.from_xml File.read('spec/fixtures/bibdata.xml')
+=> #<Relaton::Iho::ItemData:0x0000000125f56920
+...
+----
+
+=== Create bibliographic item from YAML
+[source,ruby]
+----
+Relaton::Iho::Item.from_yaml File.read('spec/fixtures/item.yaml')
+=> #<Relaton::Iho::ItemData:0x000000012274ac98
+...
+----
+
+=== Logging
+
+RelatonIho uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To update the index test fixture (used by tests), run `rake spec:update_index`. This downloads the latest `index-v1.zip` from the https://github.com/relaton/relaton-data-iho[relaton-data-iho] repository.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to https://rubygems.org[rubygems.org].
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton-iho.
+
+== License
+
+The gem is available as open source under the terms of the https://opensource.org/licenses/MIT[MIT License].

data/lib/relaton/index/CLAUDE.md

@@ -0,0 +1,207 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+relaton-index is a Ruby gem that provides indexing and searching of Relaton document references. It maps document identifiers to file paths, supporting both local index creation (for publishing) and remote index consumption (downloading from URLs with 24-hour caching).
+
+## Commands
+
+```bash
+# Run all tests (default rake task)
+rake spec
+
+# Run linting
+rake rubocop
+
+# Run specific test file
+bundle exec rspec spec/relaton/type_spec.rb
+
+# Run specific test by name
+bundle exec rspec spec/relaton/file_io_spec.rb -e "fetch_and_save"
+
+# Install dependencies
+bin/setup
+
+# Interactive console
+bin/console
+```
+
+## Architecture
+
+### Core Classes (all under `Relaton::Index` module in `lib/relaton/index/`)
+
+- **`Relaton::Index`** (module, `lib/relaton/index.rb`) — Static API entry point. Delegates to Pool and Config. Main methods: `find_or_create`, `close`, `configure`.
+
+- **Pool** — Object pool that caches Type instances by document type (`:ISO`, `:IEC`, `:IHO`, etc.). Reuses existing indexes if parameters match, recreates if they change.
+
+- **Type** — Represents one index for a document type. Holds an array of `{id:, file:}` hashes. Provides `add_or_update`, `search` (subset match, string substring match, or block), and `save`.
+
+- **FileIO** — Handles reading/writing/downloading index files. Three modes based on `@url`: string URL (download and cache to `~/.relaton/{type}/`), `true` (read local file from `~/.relaton/{type}/`), `nil` (read from current directory). Uses class-level Mutex for thread-safe downloads. Validates index format on load.
+
+- **ShardSource** (`shard_source.rb`) — Reads the machine index on a data repo's Pages site (relaton#189): `index/manifest.json`, then the one `index/shard-NNNNN.json` at `Zlib.crc32(id.root.number.to_s) % shards`. Memory only, 24 h TTL, one `Mutex`. Built by `Type` when `find_or_create` gets `pages_url:`. Reuses `FileIO#deserialize_id`/`#deserialize_pubid` through a file-less `FileIO`.
+
+- **FileStorage** — Storage abstraction module with `ctime`, `read`, `write`, `remove`. Can be replaced via `Config.storage=` for custom backends (e.g., S3).
+
+- **Config** — Global configuration: `storage`, `storage_dir`, `filename` (default: "index.yaml").
+
+### Data Flow
+
+1. `Relaton::Index.find_or_create(:TYPE, url:, file:, pubid_class:, pages_url:)` → Pool looks up or creates Type (`id_keys:` is accepted for one release with a deprecation warning, then ignored)
+2. Type lazily loads index via FileIO on first access
+3. FileIO either reads local YAML or downloads ZIP from URL, extracts, validates format
+4. Search matches against `:id` field (string comparison via `include?` or custom block)
+5. `save` writes index as YAML to local file
+
+With `pages_url:` (W3C, the #189 pilot) steps 2–3 change: a non-String query
+goes to `ShardSource#rows` and reads one shard, and `Type#index` (a String query,
+a block-only search) is `ShardSource#whole_index`, the `<index>.zip` that the
+manifest names, read from the Pages site. No `FileIO#read`, no disk cache.
+
+### What `#search` matches by default
+
+Search is narrow, then match. The **match** step, for two identifiers, is
+pubid's asymmetric subset match `id === item[:id]` (`Pubid::SubsetMatch`): the
+query is the reference, and a component it leaves nil or empty matches any
+value. That is what a search by reference means, so a flavor whose optional
+components are true wildcards passes **no block** — OASIS, W3C, XSF, IALA,
+OGC, ECMA, 3GPP, CalConnect, GOST and Plateau each call
+`index.search(pubid)`. A component that a flavor reads as "none" when nil is
+declared `subset_strict` in pubid (pubid#408), so `===` compares it exactly.
+
+A nil is a wildcard here, not "the document has none". A caller that needs
+**exact** equality says so with `search(id, exact: true)`, which selects the
+rows with `item[:id] == id`. The binary-search narrowing still runs first.
+These callers do:
+
+| caller | why |
+|---|---|
+| `Relaton::Ccsds::HitCollection#rows` (edition branch) | pubid's CCSDS `suffix` is a wildcard, so `CCSDS 101.0-B-4` would also reach the historical `101.0-B-4-S` — 260 such pairs in the index fixture. (The omitted language reached the translations too, until pubid#408 made `language` `subset_strict`.) |
+| `Relaton::Ietf::Scraper#fetch_doc` | a draft slug omits the version, so `draft-foo` would match every `draft-foo-NN` and `.first` would pick one — 20,513 such pairs in the first 40k fixture rows |
+| `Relaton::Gost::Bibliography#rows` (dated citation) | a dated citation names one edition; it replaces the old `query.matches?(row)` with no ignore list, which is `==` |
+| `Relaton::Plateau::HitCollection#find` (reference with an edition) | a reference with an edition names that one row, as the old `row[:id] == ref` block did |
+| `Relaton::Oiml::Bibliography#best_row` (a Bulletin) | a Bulletin's year, issue and sequence are its locator, so `OIML Bulletin 1960` is the volume and not every article of 1960 |
+
+A String query keeps the substring match it always had, in both directions;
+with `exact: true` it is compared with `==`. A block is a custom predicate for
+any other rule, and `search` raises `ArgumentError` if it gets both a block and
+`exact: true`.
+
+### The narrowing key — one expression, its call sites
+
+Search is two-stage: narrow, then match. Narrowing binary-searches the index for
+the run of entries whose **base document number** equals the query's, using the key
+`id.root.number.to_s` (`#root` walks a supplement/amendment `.base` chain, so a
+document and its wrappers share one key and cluster together).
+
+**Narrowing only happens for non-String queries on a `pubid_class:` index** —
+`search_candidates` loads the raw side first (that load is what settles
+`@file_io.sorted`), then narrows. A String query scans the whole index
+*and* matches via `item[:id].to_s.include?(id)`, which renders every pubid in it.
+So a flavor on a `pubid_class:` index must query with parsed identifiers; querying
+it with strings is slower than the plain-string index it replaced.
+
+That expression is written out in these places, and they must all agree or
+bsearch silently returns the wrong slice:
+
+| file | method | role |
+|---|---|---|
+| `type.rb` | `candidates_by_number` | the query's key |
+| `type.rb` | `bsearch_left` / `bsearch_right` | bounds, over the **precomputed keys array** |
+| `type.rb` | `narrowing_key` | add-time key (root-guarded for String ids) |
+| `file_io.rb` | `deserialize_pubid` | build-time sort of the materialized rows |
+| `file_io.rb` | `warn_unless_sorted` | sortedness check |
+| `file_io.rb` | `sort_structured_index` / `save_raw` | save-time sort |
+
+### Raw rows and the sidecar (relaton#242 stopgap)
+
+`FileIO#read_raw` returns `[keys, rows]`: the rows as plain `to_hash` hashes —
+no pubid objects — with a parallel array of precomputed root-number keys, sorted.
+A **Marshal sidecar** (`<index>.yaml.ms`, `SIDECAR_VERSION`-stamped) holds them,
+so a repeat load skips the YAML parse and the one-time full materialization that
+computes the keys (objects exist only during that build; the raw hash cannot
+walk `.base` itself). The sidecar is authoritative while it is at least as new
+as the yaml — the yaml is not parsed until the sidecar is stale or unreadable —
+and `save`/`remove` delete it. A parsed search materializes pubid objects only
+for its bucket (`FileIO#materialize`); the whole graph materializes only through
+`Type#index`, a String query, or a block. `add_or_update` stores raw rows
+(`to_hash` when the id is a `pubid_class` instance, verbatim otherwise) with an
+O(1) raw-side dedup by id-hash equality. Numbers: the 79,993-row ISO index is
+~311 MB of raw rows instead of a ~675 MB object graph; only the searched bucket
+is materialized at search time.
+
+**Consequence for pubid flavors:** once a flavor does query with parsed ids, an
+identifier family whose `number` is nil keys every row to `""`, so they collapse
+into one bucket and the bsearch buys nothing. It fails *silently* — results stay
+correct, only speed drops. IETF Internet-Drafts are the worked example (see
+`docs/data-repository-format.adoc`, "Two obligations a pubid-backed index
+carries"); the fix belongs upstream, giving the family a real `number`, rather than
+special-casing the key here.
+
+### Index Format
+
+YAML array of hashes with `:id` (string or structured hash) and `:file` (path string). Supports backward compatibility with old string-based format and newer pubid object format. The full data-repository/index specification (schema, `:id` v1/v2 shapes, publishing + GitHub Pages contract) lives in `docs/data-repository-format.adoc` at the repo root.
+
+### Flavor `INDEXFILE` convention
+
+Each flavor names its published index via a single constant in its top-level
+`lib/relaton/<flavor>.rb`:
+
+```ruby
+INDEXFILE = "index-vN".freeze   # base name only — no extension
+```
+
+**Rules:**
+- The constant is spelled **`INDEXFILE`** (one word, no underscore) and holds the
+  **base name without extension** — `"index-v1"`, `"index-v2"`, etc.
+- Call sites append the extension: `"#{INDEXFILE}.yaml"` for the local file passed
+  as `file:`, `"#{INDEXFILE}.zip"` for the published artifact appended to the
+  consumer `url:`. Never embed `.yaml`/`.zip` in the constant, and never hardcode
+  `index-vN.zip` at a call site — derive it from `INDEXFILE` so a version bump is
+  a one-line change.
+- The version number encodes the index **structure** (`v1` = plain-string `:id`,
+  `v2` = pubid-hash `:id`); bumping it lets the previous gem line keep reading the
+  old file while the new structure is published under a new name.
+
+### What each flavor call site passes
+
+A pubid flavor calls `find_or_create` from three places, and they do not need
+the same arguments:
+
+| call site | `url:` | `pubid_class:` |
+|---|---|---|
+| producer — `DataFetcher#index` | none (`file:` is CWD-relative) | **yes** — `FileIO#save` calls `to_hash` only for instances of it |
+| consumer — `Bibliography`/`HitCollection#index` | the published `.zip` | **yes** — without it the rows stay raw hashes and `Type#search` stops narrowing |
+| consumer on the Pages shards (W3C) | none; **`pages_url:`** | **yes** — a shard row is deserialized with it |
+| delete — `Processor#remove_index_file` | **`true`** | **no** |
+
+The delete path is `Type#remove_file` → `FileIO#remove` →
+`FileStorage.remove(file)`. `FileIO#file` is built from `url`, the type's
+directory and `file:` only, and the delete never reads or deserializes the
+index, so `pubid_class:` has no effect on it. `url:` does: without it `file` is
+the bare name, so a `Db#clear` on an empty pool deletes `./index-vN.yaml` in the
+working directory and leaves `~/.relaton/<type>/` in place. OASIS shipped that
+bug until `url: true` was added. `spec/index/relaton/type_spec.rb` and
+`spec/oasis/relaton/oasis/processor_spec.rb` guard both facts.
+
+### Key Design Decisions
+
+- **The shard path does not fall back.** A shard 404 is "not found" (`[]`), a
+  transport failure raises `Relaton::RequestError` so `Db#net_retry` retries it,
+  and an unreadable manifest/shard raises `Relaton::Index::Error`. Falling back to
+  the monolith would turn a Pages outage into a silent full download.
+  `Relaton::RequestError` is declared in `lib/relaton/core/request_error.rb`
+  (`Relaton::Core::RequestError` is an alias). `relaton/index` requires that one file,
+  not all of `relaton/core` or `relaton/bib`.
+- **`Type#actual?` compares `pages_url`.** Without it `Pool#type` would keep
+  serving a zip-backed `Type` to a caller that asks for the shards, or the
+  reverse. A spec fixture that seeds the pool must answer `actual?` for
+  `pages_url:` as well (see `spec/w3c/support/webmock.rb`).
+- **`Type#search` does not look at a loaded `@index` in shard mode.** A parsed
+  query always goes to its shard, so the 24 h TTL applies; `@index`, once
+  loaded, is not refreshed.
+- Remote indexes cached for 24 hours at `~/.relaton/{type}/index.yaml`
+- Thread safety via `@@mutex` in FileIO prevents concurrent downloads of the same file
+- Pubid deserialization is optional — when `pubid_class` is provided, string IDs are converted to structured objects
+- Index format validation checks for required `:id` and `:file` keys, with automatic recovery (re-download or removal) on corruption