relaton 3.1.1 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (67) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/frontend_assets.rb +7 -6
  7. data/lib/relaton/db.rb +31 -14
  8. data/lib/relaton/ecma/bibliography.rb +2 -2
  9. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  10. data/lib/relaton/iala/bibliography.rb +1 -1
  11. data/lib/relaton/iana.rb +1 -1
  12. data/lib/relaton/iec/data_parser.rb +41 -2
  13. data/lib/relaton/itu/README.adoc +1 -1
  14. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  15. data/lib/relaton/oasis/bibliography.rb +1 -1
  16. data/lib/relaton/ogc/hit_collection.rb +1 -1
  17. data/lib/relaton/ogc.rb +1 -1
  18. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  19. data/lib/relaton/plateau/hit_collection.rb +1 -1
  20. data/lib/relaton/sae.rb +1 -1
  21. data/lib/relaton/version.rb +1 -1
  22. data/lib/relaton/w3c/bibliography.rb +1 -1
  23. data/lib/relaton/xsf/bibliography.rb +1 -1
  24. data/lib/relaton/xsf.rb +1 -1
  25. metadata +32 -46
  26. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  27. data/lib/relaton/adobe/CLAUDE.md +0 -104
  28. data/lib/relaton/bib/CLAUDE.md +0 -150
  29. data/lib/relaton/bipm/CLAUDE.md +0 -187
  30. data/lib/relaton/bsi/CLAUDE.md +0 -35
  31. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  32. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  33. data/lib/relaton/cen/CLAUDE.md +0 -219
  34. data/lib/relaton/cie/CLAUDE.md +0 -90
  35. data/lib/relaton/core/CLAUDE.md +0 -140
  36. data/lib/relaton/db/CLAUDE.md +0 -385
  37. data/lib/relaton/doi/CLAUDE.md +0 -105
  38. data/lib/relaton/easc/CLAUDE.md +0 -85
  39. data/lib/relaton/ecma/CLAUDE.md +0 -217
  40. data/lib/relaton/etsi/CLAUDE.md +0 -219
  41. data/lib/relaton/evs/CLAUDE.md +0 -120
  42. data/lib/relaton/gb/CLAUDE.md +0 -47
  43. data/lib/relaton/gost/CLAUDE.md +0 -100
  44. data/lib/relaton/iala/CLAUDE.md +0 -97
  45. data/lib/relaton/iana/CLAUDE.md +0 -105
  46. data/lib/relaton/iec/CLAUDE.md +0 -13
  47. data/lib/relaton/ieee/CLAUDE.md +0 -202
  48. data/lib/relaton/ietf/CLAUDE.md +0 -382
  49. data/lib/relaton/iho/CLAUDE.md +0 -76
  50. data/lib/relaton/index/CLAUDE.md +0 -208
  51. data/lib/relaton/isbn/CLAUDE.md +0 -47
  52. data/lib/relaton/iso/CLAUDE.md +0 -38
  53. data/lib/relaton/itu/CLAUDE.md +0 -685
  54. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  55. data/lib/relaton/jis/CLAUDE.md +0 -64
  56. data/lib/relaton/logger/CLAUDE.md +0 -37
  57. data/lib/relaton/nist/CLAUDE.md +0 -194
  58. data/lib/relaton/oasis/CLAUDE.md +0 -251
  59. data/lib/relaton/ogc/CLAUDE.md +0 -242
  60. data/lib/relaton/oiml/CLAUDE.md +0 -47
  61. data/lib/relaton/omg/CLAUDE.md +0 -137
  62. data/lib/relaton/plateau/CLAUDE.md +0 -126
  63. data/lib/relaton/sae/CLAUDE.md +0 -91
  64. data/lib/relaton/sdo/CLAUDE.md +0 -81
  65. data/lib/relaton/un/CLAUDE.md +0 -91
  66. data/lib/relaton/w3c/CLAUDE.md +0 -369
  67. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,47 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-gb retrieves Chinese National Standards (Guobiao — `GB`, `GB/T`, `GB/Z`, plus social-org and sector standards) using the Relaton model. It scrapes several Chinese government portals (`openstd.samr.gov.cn`, `ttbz.org.cn`, `std.gov.cn`) with Mechanize. Depends on relaton-iso for ISO-based item/extension models, and on Chinese-specific classifiers (`cnccs`, `gb-agencies`).
8
-
9
- ## Development
10
-
11
- ```bash
12
- bundle exec rake # default task → rspec
13
- bundle exec rspec spec/relaton/gb/bibliography_spec.rb # single file
14
- ```
15
-
16
- (RuboCop is configured via the shared `.hound.yml`; the monorepo root config applies otherwise.)
17
-
18
- ## Architecture
19
-
20
- Namespace: `Relaton::Gb`. Note: GB places its model classes at the top of `lib/relaton/gb/` (`item.rb`, `bibitem.rb`, `bibdata.rb`, `ext.rb`, …) rather than under a `model/` subdir.
21
-
22
- Retrieval flow:
23
-
24
- 1. **Bibliography** (`lib/relaton/gb/bibliography.rb`) — `get`/`search` dispatch by prefix: `GB*` → `GbScraper` (openstd.samr.gov.cn), `T/` → `TScraper` (ttbz.org.cn), others → `SecScraper` (std.gov.cn).
25
- 2. **Scrapers** (`scraper.rb` + `gb_scraper.rb` / `t_scraper.rb` / `sec_scraper.rb`) — scrape the relevant portal for hits, then scrape the detail page into an `ItemData`.
26
- 3. **HitCollection / Hit** (`hit_collection.rb`, `hit.rb`, extend the `Relaton::Core` bases) — `Hit#item` lazy-loads the document; a hit holds `pid`, `docref`, `release_date`, `status`.
27
- 4. **Item / ItemData / Ext** (`item.rb`, `item_data.rb`, `ext.rb`) — `Item` extends `Bib::Item`; `Ext` carries GB-specific fields (`gbtype`, `ccs`, `stagename`, `plannumber`). Supporting models: `gb_type.rb`, `ccs.rb` (Chinese classification), `stage_name.rb`, `committee.rb`, `structured_identifier.rb`.
28
- 5. **Processor** (`lib/relaton/gb/processor.rb`) — registry integration; `@prefix = "CN"`, `@defaultprefix = %r{^(GB|GB/T|GB/Z) }`, `@pubid_flavor = :Gb`. Lazy-`require_relative`s `../gb` in its methods.
29
-
30
- ## Identifiers (pubid)
31
-
32
- GB parses identifiers with `Pubid::Gb::Identifier`. It publishes no index, so it has no `INDEXFILE` and no `pubid_class:` (the BSI/CEN shape, see the root `CLAUDE.md`). The attributes it reads are `publisher`, `mandate`, `number`, `part`, `year` and `all_parts`.
33
-
34
- - **Query side raises.** `Bibliography.get` parses the caller's code without a rescue, so a reference that is not a GB identifier raises `Pubid::Errors::ParseError`. `get` also takes the `Pubid::Gb::Identifier` that `Relaton::Db` parsed (relaton#205) and never mutates it: an `AllParts` pubid (or a String `(all parts)`) sets `all_parts` and unwraps its identifier, and every change after that is on the `exclude(:year)` copy. The year comes from the argument or from `pubid.year`; `all_parts` sets `part = "1"` on the copy to find part 1. `search` still routes on the raw string (`GB|GJ|GS` → `GbScraper`, `T/…` → `TScraper`, else `SecScraper`), so the portal for each reference and the cassettes are unchanged.
35
- - **Hit filter.** `search_filter` parses each `hit.docref` (rescued: a docref is data; an unparseable one drops the hit) and keeps the hits that `query.matches?`. It ignores `:year` only when the query has no year. It **never** ignores `:part`: `GB/T 20223` and `GB/T 20223.1` are two documents, and `gb_spec.rb` asserts that `GB/T 20223-2006` does not return `GB/T 20223.1-2006`. This is a deliberate departure from the migration hand-off, which proposed to ignore a part the query omits. It also stops `GB/T 1.1` from matching `GB/T 1.10`, which the old substring compare allowed.
36
- - **`Docidentifier`** (`docidentifier.rb`) follows the CEN/OMG shape: `content=` parses into `@pubid` (rescued, data side), and the mutators replace the pubid with an `exclude` copy and write back through `store_content`. `to_all_parts!` excludes part and year, sets `all_parts`, and pubid renders the ` (all parts)` suffix itself (and parses it back). A nil pubid makes every mutator a no-op.
37
- - **Scrapers** (`scraper.rb`) read the docref through a rescued `docref_pubid`. `get_prefix` keys `yaml/prefixes.yaml` with the first segment of `publisher` (a `Pubid::Components::Publisher`, so read it as `publisher.to_s`; `T/GZAEPI` → `T`). An unknown key gives nil, and `get_gbtype` keeps a nil prefix rather than raising — a portal docref is data. `get_mandate` maps `mandate` `T` → recommended, `Z` → guidelines, nil → mandatory. `get_contributors` passes `publisher.to_s` to `gb_agencies`, which names the same organization for `GB` and `GB/T`. `parse_docref` returns `[id without part and year, part, year]`; the first value is `plannumber`.
38
- - **Pubid traps.** `Pubid::Gb` accepts any upper-case publisher code (`ISO 123` parses), so use `foo` or `GB/T` for a parse-error example. `year` is a reader over `date` (no `year=`); set it with `date = Pubid::Components::Date.new(year:)`. Local (`DB11/T`) forms do not parse. pubid PR #388 (`dcea7380`) renamed `publisher_code` to the `publisher` component, added the year to `to_hash`, and made `GBn` parse; a pubid older than that breaks this flavor. pubid renders an em dash (`T/GZAEPI 001—2018`) as a hyphen; the fixture `tgzaepi_001_2018.xml` still records the em dash, and its spec is pending until pubid decides.
39
- - **Prefix register.** `@pubid_flavor = :Gb` sources `Relaton.prefix_flavor` from `Pubid::Gb.prefixes`, so `GB`, `GB/T`, `JB/T`, `T/` … resolve to `Relaton::Gb`. `CN` is not a pubid prefix, so `prefix_flavor("CN")` is `[]`; no caller in this repo used it, and `Db` routing still reads `@prefix`. `HB` resolves to `[Gb, Nist, Bsi]`, pinned in `spec/relaton/prefix_flavor_spec.rb`.
40
-
41
- ## External dependencies
42
-
43
- `cnccs ~> 0.1.1`, `gb-agencies ~> 0.0.1`, `mechanize ~> 2.10`, `csv ~> 3.0`, `relaton-core`, `relaton-iso`.
44
-
45
- ## Testing
46
-
47
- RSpec with WebMock + VCR (cassettes in `spec/vcr_cassettes/`). Network is blocked via WebMock. SimpleCov and canon are loaded via `spec/support/`. No index fixture — GB is scraped, not indexed; `spec/fixtures/` holds XML/YAML round-trip data.
@@ -1,100 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with the
4
- `Relaton::Gost` flavor.
5
-
6
- ## Project Overview
7
-
8
- `Relaton::Gost` retrieves **GOST** standards — the interstate/national standards of the
9
- former-Soviet space (Государственный стандарт), published by Rosstandart and the
10
- Euro-Asian Council for Standardization (EASC/МГС). It is **index-backed** (no scraping):
11
- it searches a pre-built **pubid-structured** index via `Relaton::Index` and fetches
12
- per-document YAML from the `relaton/relaton-data-gost` GitHub repo. The model layer mirrors
13
- the `Relaton::Easc` sibling; the pubid-backed retrieval mirrors `Relaton::Oiml`.
14
-
15
- ## Architecture
16
-
17
- Namespace: `Relaton::Gost`. Retrieval flow:
18
-
19
- 1. **Bibliography** (`bibliography.rb`) — `get(code, year, opts)` parses the citation with
20
- `Pubid::Gost.parse`, looks it up in the **pubid-structured** index, fetches the matching
21
- `row[:file]` YAML over `Net::HTTP` (`Item.from_yaml`, raising `Relaton::RequestError` on
22
- non-200), and stamps `fetched`. `search` picks the **latest edition** (`max_by { year }`)
23
- among number-matched candidates; `get` then strips the year for an **undated** citation via
24
- `to_most_recent_reference` (so `GOST 1.0` renders undated but resolves the newest edition),
25
- while a dated citation (`GOST 1.0-92`) or explicit `year` pins that edition. Mirrors
26
- `lib/relaton/oiml/bibliography.rb`.
27
- 2. **Index** — `Relaton::Index.find_or_create(:gost, url: "#{ENDPOINT}index-v2.zip",
28
- file: "index-v2.yaml", pubid_class: ::Pubid::Gost::Identifier)`. Each row's `:id` is a
29
- `_type: pubid:gost:{interstate,national}-standard` hash that `Relaton::Index` rebuilds into
30
- a `Pubid::Gost` identifier via `from_hash`; passing the parsed pubid to `#search` lets it
31
- binary-search candidates by number (see `Bibliography#rows`). `INDEXFILE =
32
- "index-v2"` is in `lib/relaton/gost.rb`.
33
- 3. **Item / ItemData / Ext** (`item.rb`, `item_data.rb`, `ext.rb`) — `Item` extends
34
- `Bib::Item`, re-declares `ext` to the typed `Gost::Ext` and `docidentifier` to the pubid
35
- `Gost::Docidentifier`. `Ext`'s GOST fields (`urn`, `webpage`, `ics_code`, `developer`,
36
- `keywords` [collection], `pages`, `designation_original`) round-trip natively through YAML
37
- **and** XML (mapped in both `xml` and `key_value`). (The live dataset uses `urn`/`webpage`/
38
- `ics_code`/`pages`/`designation_original`; `developer`/`keywords` are declared but unused.)
39
- 4. **Doctype** (`doctype.rb`) — `TYPES = %w[interstate national preliminary methodological]`
40
- (GOST, GOST R, PST, OD categories). Carried in the inherited `content` attribute.
41
- 5. **Docidentifier** (`docidentifier.rb`) — wraps a `Pubid::Gost` identifier (`attr_reader
42
- :pubid`); `content=` re-parses to keep `@pubid` in sync, and `remove_date!` nil-s the pubid
43
- `year` and re-renders `content` (GOST carries the edition as `year`, not `date`) so
44
- `to_most_recent_reference` yields `GOST R 34.12`. Mirrors `Oiml::Docidentifier`.
45
- 6. **Processor** (`processor.rb`) — registry integration; `@short = :relaton_gost`,
46
- `@prefix = "GOST"`, `@idtype = "GOST"`, and `@defaultprefix = %r{^(?:GOST|ГОСТ)\b}` so **both
47
- the Latin `GOST` and Cyrillic `ГОСТ` surface forms route here** (the `\b` stops it swallowing
48
- longer tokens). `remove_index_file` passes `url: true` and `file:` only — the delete never
49
- reads the index, so it needs no `pubid_class` (see `lib/relaton/index/CLAUDE.md`). Every method touching a
50
- flavor constant lazy-`require_relative`s `../gost` first (the lazy-registry invariant;
51
- `spec/relaton/lazy_loading_spec.rb` guards it).
52
-
53
- There are no scrapers — everything comes from the curated index + GitHub YAML.
54
-
55
- ## Pubid
56
-
57
- The flavor is **pubid-backed** via `Pubid::Gost` (metanorma/pubid#108), on the `pubid` `main`
58
- branch the root `Gemfile` already pins (for JCGM). `Bibliography#rows` selects a dated
59
- citation with `index.search(query, exact: true)` (pubid `==`) and an undated one with the
60
- default subset match `query === row`: the omitted year matches any value, so it matches every
61
- edition sharing subtype + number. `===` requires the same class, so interstate and national
62
- stay distinct, and `1.1` does not match `1.10`. pubid declares `copublisher` strict for GOST
63
- (pubid#408), so an omitted copublisher means "none". Measured over the full
64
- `relaton-data-gost` index (37,508 rows): `===` and the old `matches?(…, ignore: [:year])` agree
65
- on all 94,279 pairs. The
66
- `relaton-data-gost` `index-v2.yaml` was built with this **same** pubid, so the rows deserialize
67
- (a mismatched pubid would make `Relaton::Index` reject the whole index — cf. the root
68
- `CLAUDE.md` JCGM note). The pinned release `2.0.0.pre.alpha.13` carries `Pubid::Gost` and
69
- `subset_strict`; any lower pin must too.
70
-
71
- ## Dataset
72
-
73
- `relaton/relaton-data-gost` is the live data repo behind `ENDPOINT` (a real pubid-structured
74
- `index-v2.yaml` + `data/*.yaml`). The specs run fully offline against a small committed subset
75
- (see Testing).
76
-
77
- ## Testing
78
-
79
- `spec/gost/` — self-contained, run via `bundle exec rake spec:gost` (or `cd spec/gost &&
80
- bundle exec rspec -I . .`). Coverage:
81
-
82
- - `ext_spec.rb`, `item_spec.rb` — YAML round-trip of the typed ext fields.
83
- - `bibliography_spec.rb` — retrieval against real records copied from `relaton-data-gost`:
84
- the pubid-structured index fixture `fixtures/index-v2.zip` is pre-loaded into the
85
- `Relaton::Index` pool in `before(:suite)` (with `pubid_class`), and `fixtures/data/*.yaml`
86
- are served by WebMock (`support/webmock.rb`) — **no live network**. Covers dated/undated/
87
- Cyrillic lookups, undated→latest-edition, year-pinning, not-found (`nil`), and non-200
88
- (`Relaton::RequestError`). Fixture docs: interstate `GOST 1.0` (editions 2015 + 92) and
89
- national `GOST R 34.12-2015`.
90
- - `processor_spec.rb` — processor shape + Latin/Cyrillic prefix routing.
91
-
92
- To refresh the fixture, copy the wanted rows from `relaton-data-gost/index-v2.yaml` into
93
- `fixtures/index-v2.yaml`, re-zip it to `fixtures/index-v2.zip`, and copy the matching
94
- `data/*.yaml` files.
95
-
96
- There is **no RelaxNG grammar** for GOST and no XML-schema validation spec: the metanorma
97
- document model (`metanorma-model-iso/grammars/`) defines no GOST grammar, so there is nothing
98
- to copy into `grammar/` and nothing to validate `to_xml` against. This matches the sibling
99
- `easc` flavor (also grammar-less). If an official GOST grammar is ever added upstream, copy the
100
- `<flavor>{,-compile}.rng` pair into `grammar/` and add a Jing-validation spec then.
@@ -1,97 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Architecture
4
-
5
- **Pubid-backed docidentifier.** `Relaton::Iala::Docidentifier`
6
- (`docidentifier.rb`, `< Bib::Docidentifier`) parses its `content` into a
7
- `Pubid::Iala::Identifier` kept in `@pubid`, while the lutaml `content` attribute
8
- stays a plain string for serialization. Parsing is **soft**: `content=` lazily
9
- `require "pubid"` + `require "pubid/iala"` (pubid core must load first for
10
- `Pubid::PrefixesSupport`) and rescues `LoadError`/`StandardError`, so a missing
11
- pubid gem or non-IALA/unparseable content leaves `@pubid` nil rather than raising.
12
-
13
- It implements the base class's abstract `remove_part!` / `remove_date!` /
14
- `to_all_parts!` by mutating the pubid and re-rendering via `refresh_content!`
15
- (which writes through the aliased `store_content` — the inherited string setter —
16
- so in-place mutations aren't clobbered by a re-parse). IALA's identifier models
17
- only `publisher`/`number`/`edition`/`language`, so the mapping is IALA-specific:
18
-
19
- - **`remove_date!` → clears `edition`.** IALA carries no date component; `edition`
20
- is its version discriminator, so this yields the version-agnostic ("most
21
- recent") reference, e.g. `IALA S1070 Ed 2.0` → `IALA S1070`.
22
- - **`remove_part!` → clears the (unused) `part`/`subpart` attributes.** A no-op
23
- for the rendered string today: IALA folds any subpart inline into `number`
24
- (e.g. `0103-1`), so the part is not a separable component. Implemented anyway
25
- so it never raises, and it starts working automatically if pubid-iala ever
26
- models `part` separately.
27
- - **`to_all_parts!` → `remove_part!` + `remove_date!`, then wraps `@pubid` in
28
- pubid's `AllParts`.** `content` stays the edition-stripped string — IALA's
29
- own renderer has no "(all parts)" marker, and `content` is cached from the
30
- pre-wrap pubid on purpose. `#pubid` itself, read directly, now answers
31
- `all_parts? == true` and renders WITH pubid's generic marker (`#to_s`),
32
- since it's the wrapper. Mirrors `lib/relaton/iec/model/docidentifier.rb`.
33
-
34
- All three no-op safely when `@pubid` is nil, so `Bib::ItemData#to_all_parts` /
35
- `#to_most_recent_reference` never raise on IALA items.
36
-
37
- **Index (`index-v2`, pubid-keyed).** `INDEXFILE` is the pubid-backed
38
- `index-v2` published by `relaton-data-iala`: 701 rows of
39
- `_type: pubid:iala:*` with `number`/`edition`/`language`. The consumer
40
- (`Bibliography#index`) passes
41
- `pubid_class: ::Pubid::Iala::Identifier` — that is what makes `Relaton::Index`
42
- deserialize the rows into identifiers, sort them by `id.root.number`, and let
43
- `Type#search` bsearch. Omitting it leaves the rows as raw hashes and
44
- `FileIO#sorted` false, so every lookup scans all 701 rows, silently. That was
45
- the state before this flavor adopted the W3C pattern; the data repo already
46
- published a pubid-shaped `index-v2`, only the consumer was unwired.
47
- `Processor#remove_index_file` passes `url: true` and `file:` only: the delete
48
- never reads the index (see `lib/relaton/index/CLAUDE.md`).
49
-
50
- `Bibliography#best_match` follows the ETSI/W3C idiom:
51
-
52
- - **Pass the pubid, not the string.** `Type#search_candidates` narrows only when
53
- the argument is not a `String`, so passing the reference text — which this
54
- flavor used to do — disables the binary search however the index was built.
55
- - **Selection is pubid's asymmetric subset match** (`Pubid::SubsetMatch`):
56
- `pubid === r[:id]`, the reference on the left. A component the reference omits
57
- matches any value, so a bare `IALA M0001` finds the `Ed 9.0 (E)` row, while
58
- `IALA R0106 Ed 2.1 (F)` reaches only its own. `edition` and `language` are
59
- IALA's only optional components.
60
- - **The document type is never a wildcard.** It is the identifier's class, and
61
- `===` requires the same class. 17 of the 309 index numbers are shared by two
62
- types (`R1001` and `C1001` are different documents), so they stay apart. An
63
- `Annex` row carries its base identifier, which `===` compares with its own
64
- subset match.
65
- - **Newest edition wins, deterministically.** Editions are dotted versions and
66
- must be compared segment-by-segment as integers: `"10.0"` is newer than
67
- `"9.0"` but loses as a string. No published edition has a segment above 9
68
- today, so this is latent — the same bug already fixed for ETSI. Ties break on
69
- the edition text and then the file path, because the index sort is not stable.
70
- Among rows of one edition the language-neutral record wins: it is the base
71
- document, the others are translations.
72
- - **An unparseable reference raises.** `parse_ref` lets
73
- `Pubid::Errors::ParseError` propagate, and the full-scan substring fallback is
74
- gone with it: a substring match answered an ambiguous reference with whichever
75
- row sorted first, instead of telling the caller the reference is not an
76
- identifier.
77
-
78
- `require "pubid"` sits in `lib/relaton/iala.rb` (the IANA/IHO form), so pubid
79
- loads with the flavor. `Processor#remove_index_file` names no pubid class, so the
80
- cold `Db#clear` path that `spec/relaton/lazy_loading_spec.rb` guards does not
81
- depend on it. `Docidentifier` keeps its own
82
- lazy, `LoadError`-rescuing require: it is reached by deserialization, which must
83
- not depend on the flavor entry file having been loaded.
84
-
85
- ## Testing
86
-
87
- - `bundle exec rake spec:iala` — run the IALA suite.
88
- - Specs live in `spec/iala/relaton/iala/` and run self-contained (see the root
89
- `CLAUDE.md` Testing section).
90
- - `spec/iala/fixtures/index-v2.zip` is a 19-row curated subset of the published
91
- index, copied row-for-row so the stored shapes stay verbatim. It covers the
92
- matching rules above: a multi-edition document (`R0101`, editions `2` and
93
- `3.0`), a four-language document (`R0106`), a number shared by two types
94
- (`R1001` / `C1001`), a sub-part folded into the number (`C0103-1`), and the
95
- language-neutral/translation split. `spec/iala/support/webmock.rb` seeds it
96
- into the `Relaton::Index` pool **with `pubid_class:`** — without that the
97
- suite would pass while exercising something the runtime never does.
@@ -1,105 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Development
4
-
5
- - `bundle install` — install dependencies
6
- - `bundle exec rake spec:iana` — run this flavor's tests
7
- (rubocop is configured via the root `.rubocop.yml` but is not in the bundle)
8
-
9
- ## What this flavor is
10
-
11
- IANA entries are protocol **registries**, not numbered standards. An identifier
12
- is a hierarchical registry slug capped at one "/":
13
-
14
- ```
15
- IANA _6lowpan-parameters
16
- IANA _6lowpan-parameters/lowpan_nhc
17
- ```
18
-
19
- `Parser#docnumber` builds the bare slug (`registry[/sub-registry]`) by joining a
20
- nested registry's XML `id` to its parent's; `Core::DataFetcher#output_file`
21
- turns that into the document filename (`data/_6lowpan-parameters-lowpan_nhc.yaml`).
22
-
23
- `output_file` collapses `/` and `-` to the same character, so two distinct
24
- registries can want one file — `rpki/signed-objects` and `rpki-signed-objects`
25
- do, the only such pair in the 3405 published rows. `save_doc` therefore writes
26
- through `Core::DataFetcher#unique_output_file`, which hands the second one a
27
- digest-suffixed path instead of overwriting the first; the warning stays so the
28
- clash is visible in the crawl log. Before that fix the corpus had 3405 index
29
- rows over 3404 files, and one id resolved to the wrong document.
30
-
31
- ## Index (`index-v2`, pubid-keyed)
32
-
33
- `INDEXFILE = "index-v2"` (`lib/relaton/iana.rb`). Rows carry
34
- `Pubid::Iana::Identifiers::Registry` identifiers, serialized as:
35
-
36
- ```yaml
37
- - :id:
38
- _type: pubid:iana:registry
39
- sub_registry: lowpan_nhc # absent for a top-level registry
40
- number: _6lowpan-parameters
41
- :file: data/_6lowpan-parameters-lowpan_nhc.yaml
42
- ```
43
-
44
- `number` holds the **top-level** slug — not a document number, because an IANA
45
- identifier has none (digits only ever appear glued inside a slug token:
46
- `sip-parameters/sip-parameters-13`, `idna-tables-11.0.0`). It is the index key:
47
- `Relaton::Index::Type#candidates_by_number` bsearches on `id.root.number.to_s`,
48
- so a registry and all of its sub-registries share one bucket (655 buckets over
49
- the 3405 published rows; median 3, max 73 — `pcep`) and `Bibliography#pubid_match?`
50
- separates them. `registry` is a **derived reader** over `number`, so it is not a
51
- serialized key.
52
-
53
- The pair that must stay in step, both passing
54
- `pubid_class: ::Pubid::Iana::Identifier`:
55
-
56
- | role | file |
57
- |---|---|
58
- | producer | `data_fetcher.rb` — `#index` + the guarded `#add_to_index` |
59
- | consumer | `bibliography.rb` — `#index`, `#parse_ref`, `#pubid_match?` |
60
-
61
- `processor.rb#remove_index_file` passes `url: true` and `file:` only: the delete
62
- never reads the index (see `lib/relaton/index/CLAUDE.md`).
63
-
64
- `#add_to_index` rescues and warns rather than raising: `Relaton::Index` rejects
65
- the **whole** index if one row fails to deserialize, so an unparseable slug has
66
- to be skipped at write time instead of aborting the crawl.
67
-
68
- ### Why a stale index is loud, not silent
69
-
70
- A row serialized before `number` existed (`{_type, registry, sub_registry}`)
71
- deserializes with `number` **nil and no error at all**: lutaml ignores unknown
72
- keys, and `Index::FileIO#id_supported?` skips its `to_hash`/`from_hash`
73
- validation for a concrete subclass — which every IANA id is. pubid therefore
74
- fails at *render* time instead: `Identifier#require_number!` raises
75
- `ArgumentError` ("… regenerate the index") from `to_s`, `to_urn` and
76
- `to_mr_string`. **If you see that error the index is stale — rebuild it, don't
77
- work around it.** `spec/iana/relaton/iana_spec.rb`'s "the pubid index-v2
78
- fixture" block is the tripwire for the fixture.
79
-
80
- ### Migration note (v1 → v2)
81
-
82
- `index-v1` was keyed on the bare slug string, so `Index::Type#search` matched by
83
- **substring** and `min_by { |i| i[:id] }` picked the shortest hit — `IANA rpki`
84
- also matched `rpki/signed-objects`. v2 matches the identifier exactly, so a
85
- partial reference no longer resolves.
86
-
87
- This flavor no longer produces `index-v1`. `relaton-data-iana`'s own
88
- `crawler.rb` derives it from `index-v2` (a plain YAML transform:
89
- `sub_registry ? "#{number}/#{sub_registry}" : number`) so released gem lines
90
- keep resolving.
91
-
92
- ## Testing
93
-
94
- - **Framework:** RSpec with VCR cassettes and WebMock.
95
- - **Index fixture:** `spec/iana/fixtures/index-v2.zip` (all 3405 published rows)
96
- is deserialized through `pubid_class` once per process and **re-seeded into
97
- the `Relaton::Index` pool before every example** — see
98
- `spec/iana/support/webmock.rb` for why (the producer-side `find_or_create`
99
- evicts it otherwise). Its `actual?` override claims only the consumer
100
- (`url:`) lookup.
101
- - **VCR cassettes:** `spec/iana/vcr_cassettes/` — index downloads are ignored by
102
- VCR (`support/vcr.rb`), since the fixture answers them.
103
- - The XML round-trip specs validate against `grammar/relaton-iana-compile.rng`
104
- with Jing, which needs a **Java runtime**; without one they fail with
105
- `Jing::ExecutionError`, unrelated to any code change.
@@ -1,13 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Development
4
-
5
- - `bundle install` — install dependencies
6
- - `bundle exec rake spec` — run tests
7
- - `bundle exec rubocop` — lint
8
-
9
- ## Testing
10
-
11
- - **Framework:** RSpec with VCR cassettes and WebMock
12
- - **Index fixture:** `spec/fixtures/index-v2.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iec. The fixture is the lean pubid-v2 `to_hash` index (`_type`-tagged rows); rows are deserialized via `Pubid::Iec::Identifier.from_hash` and validated by round-trip in relaton-index (no `id_keys` allowlist).
13
- - **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).
@@ -1,202 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Development
4
-
5
- - `bundle install` — install dependencies
6
- - `bundle exec rake spec` — run tests
7
- - `bundle exec rubocop` — lint
8
-
9
- ## Index (pubid-structured `index-v2`)
10
-
11
- IEEE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`
12
- in `lib/relaton/ieee.rb`): each row's `:id` is a `Pubid::Ieee::Identifier`
13
- serialized to the `_type: pubid:ieee:*` (lutaml) form, keyed on the index via
14
- `pubid_class: ::Pubid::Ieee::Identifier`.
15
-
16
- - **Write side.** Unlike the doc-fetching pipeline, the index is built by a
17
- **separate step**: `Relaton::Ieee::DataFetcher.build_index(dir:, index_file:)`
18
- (`data_fetcher.rb`). It globs the per-document `data/*.yaml` files, extracts each
19
- primary non-trademark IEEE docidentifier, parses it via `.pubid` (a guarded
20
- `::Pubid::Ieee::Identifier.parse` + `from_hash(to_hash) == to_hash` round-trip
21
- check), `add_or_update`s the object, and `save`s. Ids pubid can't parse/round-trip
22
- are skipped; a per-file skip warning **and** a final
23
- `IEEE index-v2.yaml: N/M indexed, K skipped (P% coverage)` line are logged so the
24
- loss is never silent. `DataFetcher.fetch` (doc generation) is unchanged — the
25
- data repo (`relaton-data-ieee`) calls `build_index` after obtaining `data/`.
26
- - **Read side.** `bibliography.rb#search` passes `pubid_class:` to `find_or_create`,
27
- parses the query with `parse_pubid` (falling back to the raw String on failure)
28
- and passes the pubid to `index.search` for number-narrowing. A parsed query
29
- selects its rows by **pubid equality with the year aside**
30
- (`id.exclude(:year) == pubid.exclude(:year)`), so a redline, a draft, a
31
- corrigendum and another number (`1003.10` for `1003.1`) never match. A dated
32
- query (its own year, or the `year` argument) takes that year; an **undated**
33
- query takes the **latest** edition, and `get` returns it as
34
- `to_most_recent_reference` (undated IEEE docids, no date, an `instanceOf`
35
- relation to the edition; `opts[:keep_year]` keeps it dated) — relaton-ieee#29.
36
- Before this, a substring match + `min_by(to_s)` answered `IEEE 528` with the
37
- oldest edition, `528-1994`. When equality finds nothing (an unparseable query,
38
- or one of the ~4% of forms whose year pubid keeps elsewhere, e.g.
39
- `IEEE Std 1003.1-1988/INT-1992`, the corrigendum's legacy base year), the old
40
- substring match is the fallback. `processor.rb#remove_index_file` passes
41
- `url: true` and `file:` only: the delete never reads the index (see
42
- `lib/relaton/index/CLAUDE.md`).
43
- - **Why not `pubid === row`.** pubid declares no `subset_strict` attribute for
44
- IEEE, so a nil `parts`, `prefix`, `draft` or `draft_status` in the query is a
45
- wildcard: over the published index (12,616 rows) `===` let `IEEE 802.3` match
46
- `802.3.1-2011` and `IEEE 754` match `IEEE Unapproved P754/D9.0`. Switch to
47
- `index.search(pubid)` once pubid makes those strict, and re-measure `===`
48
- against the equality over the full index first.
49
- - **`Relaton::Ieee::Docidentifier`.** `Item` holds this class (the producer
50
- creates it too: `IdamsParser#docidentifier`, `DataFetcher`), because
51
- `Bib::Docidentifier#remove_date!` is a no-op. Its `remove_date!` re-renders an
52
- `IEEE` docid through pubid without the year (`to_s(trademark: true)` for the
53
- `scope: trademark` one) and leaves an id **with no year** untouched: a
54
- re-render would change a draft's spelling (`P802.5t™/D-2.5` →
55
- `P802.5t®/D2.5`). `ItemData#to_most_recent_reference` drops the ISBN and
56
- DOI, which name one edition.
57
-
58
- ## Identifier resolution (`RawbibIdParser`, pubid-first)
59
-
60
- `RawbibIdParser.parse(normtitle, stdnumber)` turns a raw IEEE catalog
61
- `normtitle` into a **`::Pubid::Ieee::Identifier`** (`idams_parser.rb`/prefilter
62
- consume `.to_s` / `.to_s(trademark: true)` / docnumber). It is **pubid-first**:
63
- pubid parses ~93.5% of normtitles directly; the bespoke regex case statement
64
- (`#parse_fallback`, ~430 lines) only normalizes the messy remainder into a
65
- canonical string pubid then parses (→ 98.7% pubid objects, 100% coverage).
66
-
67
- - **Faithfulness guard.** `#parse` always renders the bespoke reference
68
- (`parse_fallback(...).to_id`) and only accepts a pubid parse when it kept every
69
- non-date digit (`#faithful?`, a date-stripped **digit** multiset — digit-level
70
- so a benign re-split like `P1476/D4` vs `P14764` counts as faithful while a real
71
- loss like `/D3.0`→`/D.0` does not). If no pubid parse is faithful it returns the
72
- private fallback `Renderer` (the old bespoke id string) — so there is **zero
73
- silent digit loss**. ~1.3% of docs keep the `Renderer`.
74
- - **An id must have a digit in its number.** The last two fallback branches
75
- ("publisher, number" and the `else` that uses `stdnumber`) accept a number
76
- only when it has a digit. Without that rule the first two words of any title
77
- became an id (`A Call to Action …` → `A Call`), and a category `stdnumber`
78
- became one too (`White Paper` → `IEEE White Paper`). Since the id is the
79
- output filename, 176 records shared 42 files and 134 were overwritten. Now
80
- `parse` returns nil for them, and `DataFetcher#parse_entry` skips them with
81
- the `PubID parse error` warning. A real standard with a title-like
82
- `normtitle` falls through to `stdnumber` and keeps its id
83
- (`IEEE Standard for Ethernet …` / `802.3bm-2015` → `IEEE Std 802.3bm-2015`);
84
- 16 such standards were hidden by the old branch. **The digit rule alone is
85
- not enough** — see `#no_standard_number?` under **Fetch robustness**.
86
- - **No public `PubId` class.** The old `Relaton::Ieee::PubId` is gone; its
87
- rendering survives only as the private `RawbibIdParser::Renderer` used by the
88
- fallback. pubid (`::Pubid::Ieee::Identifier`) is the identifier abstraction.
89
- - **Coordination with pubid.** IEEE ids now render in pubid's canonical form
90
- (`/D-5`→`/D5`, `/Amd 5-2012` with a space, `802.16-2012 - Redline`, trademark
91
- `™`/`®` attached to the document **number**, full corrigendum expansion). This
92
- required a batch of pubid fixes (draft-verbatim, redline, revision incl.
93
- numbered `RevN`, edition, trademark, historical formats, update_codes one-offs).
94
- `Core::DataFetcher#output_file` also sanitizes commas so pubid's `", Mar 2011"`
95
- dates don't leak into filenames. The fallback `Renderer`'s suffix strings are
96
- **not** the canonical spellings and shouldn't be "fixed" to match: they are the
97
- input to `to_id`, which pubid re-parses, so `/D-N`, `/R-N`, `/E-N` are
98
- deliberately the *parseable* forms pubid canonicalizes to `/DN` etc. The
99
- **trademark** rendering is the exception that must track pubid: `IdamsParser`
100
- emits the `scope: trademark` docidentifier with `to_s(trademark: true)` on
101
- whatever `parse` returned, so the `Renderer`'s ®/™ rule
102
- (`Renderer::Id::REGISTERED_SERIES`) mirrors pubid#322 — series read off the
103
- *number* with a leading project `P` stripped, `8802` ungated, everything else
104
- gated on IEEE being a (co)publisher. Otherwise a `P802.x` draft would print ™
105
- down the fallback path and ® down the pubid path. The one
106
- exception is `amd_to_s`, which emits the **spaced** `/Amd N`: pubid parses both
107
- spellings, so the spaced one costs nothing and keeps the rendering canonical
108
- for the ~1.3% of ids that miss the faithfulness guard and keep the `Renderer`
109
- string. Changing any of these also moves the `extract_index_entry` dedup key in
110
- `data_fetcher.rb`.
111
-
112
- ## Fetch robustness (no silent corpus loss)
113
-
114
- Amendment docnumbers can be pathologically long — pubid's `to_s` embeds the full
115
- "(Amendment to … as amended by …)" clause (300+ chars). Two guards keep one such
116
- id from silently halving the crawl (see the `data_fetcher.rb` history):
117
-
118
- - **A record with no standard number is skipped.** `#no_standard_number?` is
119
- true when `publicationsubtype` is in `NON_STANDARD_SUBTYPES` (`Whitepapers`,
120
- `Research Documents`) **and** `stdnumber` has no digit. IEEE gives such a
121
- document no designation and cites it by title, so every id built from it is
122
- a title fragment. The digit rule in `RawbibIdParser` cannot catch all of
123
- them, because pubid parses a digit out of the title itself
124
- (`IEEE 802 Nendica Report: …` → `IEEE Std 802`, three documents; `IEEE 3D
125
- Body Processing …` → `IEEE Std 3D`, three more). Both conditions are
126
- required, so a record of such a subtype that does carry a number keeps it
127
- (`Amendment 1:2011 to SMPTE ST 430-9:2008`). The check runs in **two**
128
- places: `#parse_entry` (with a warning) and `#extract_index_entry` — the
129
- prefilter groups files by docnumber and keeps the newest of each group, so a
130
- white paper that reads as `IEEE Std 802` would otherwise join the real
131
- standard's group and push its file out. Measured on the raw corpus: 157
132
- records skipped, and no kept document has a digit-less id. The subtype match
133
- is exact, which the data supports — all 570,744 raw files carry one of three
134
- spellings (`Standard Docs` 564,611, `Whitepapers` 5,240,
135
- `Research Documents` 891), so a new spelling is the one thing to watch for.
136
- - **Bounded filenames.** `Core::DataFetcher#output_file` caps the basename at
137
- `MAX_BASENAME_BYTES` (255): when the sanitized id exceeds the OS limit it
138
- byte-truncates (`byteslice(…).scrub("")`, valid UTF-8) and appends a 12-char
139
- SHA1 of the **full** docid — bounded, unique, and **deterministic** so
140
- `save_doc`/`reconcile_staged_outputs`/`read_bib` all resolve the same name.
141
- Without this, `File.write` raised `Errno::ENAMETOOLONG`.
142
- - **Per-document commit guard.** Both the serial (`run_shard`) and parallel
143
- (`spawn_batch` fork) paths route through one `commit_entry` helper that wraps
144
- `commit_doc` in a `rescue` (logs via `Util.error`, continues). A single bad doc
145
- no longer aborts a serial crawl, and a forked worker still reaches its
146
- `File.binwrite(state_path, …)` so `merge_state_files` sees its `saved_writes`
147
- and `reconcile_staged_outputs` doesn't delete the whole batch's staged files.
148
- Reconcile also logs the straggler-cleanup count so an anomaly isn't silent.
149
- - **A title-fragment id can still slip past `#no_standard_number?`.**
150
- `#no_standard_number?` only fires for a `publicationsubtype` in
151
- `NON_STANDARD_SUBTYPES`, and a generic mid-tier `RawbibIdParser` fallback
152
- branch meant for a real catalog number (e.g. "publisher, number, part",
153
- built for `"IEEE 802.3"`) can still read a short, numbered title as one
154
- (`"WEB 3.0: The Evolution of ..."` → `WEB 3.0`) for a subtype spelling that
155
- list doesn't cover. `RawbibIdParser.fabricated_title_id?(pid, title,
156
- isbn_present)` is a second, subtype-agnostic guard on the *resolved id*
157
- itself, called from both `IdamsParser#pubid` (full parse, `title` from
158
- `@doc.btitle` — distinct from `normtitle`, which for a genuine standard
159
- holds its designation, not prose) and `DataFetcher#extract_index_entry`
160
- (cheap-parse prefilter, `title` the *last* `<title>` in the raw XML — the
161
- first is always the top-level designation, present even for a real
162
- standard). It must run in both places for the same reason
163
- `#no_standard_number?` does: the prefilter groups files by docnumber before
164
- the full parse ever runs, so a fabricated id left un-nilled there still
165
- merges a whitepaper's group with an unrelated document's and can silently
166
- drop a file from the corpus.
167
- **The check is on the id's *class*, not (only) the title prefix:** it
168
- requires `pid.is_a?(Renderer)` — i.e. the unvalidated bespoke fallback
169
- string, never a real `::Pubid::Ieee::Identifier` that pubid's own grammar
170
- actually parsed — **and** the title is a literal prefix of `pid.to_s`
171
- **and** the document carries a real ISBN. Title-prefix and ISBN alone are
172
- not enough: measured over 9,091 successfully-parsed raw-corpus records, 7
173
- genuine, correctly-resolved standards have a title that literally begins
174
- with their own real designation and also carry an ISBN (`"IEEE Std
175
- 1636.99-2013, IEEE Standard for ..."`) — an earlier version of this guard
176
- without the `Renderer` restriction nulled out their real ids. All 7 resolve
177
- to a real `Pubid::Ieee::Identifier`, never a `Renderer`, so the class check
178
- is what tells them apart from the 25 known fabricated ids in a
179
- `relaton-data-ieee` audit, all of which are `Renderer`s. A flagged record is
180
- treated like any other unparseable id: `docnumber` is nil and
181
- `DataFetcher#parse_entry` skips it with the `PubID parse error` warning.
182
-
183
- ## Testing
184
-
185
- - **Framework:** RSpec with VCR cassettes and WebMock.
186
- - **Index fixture:** `spec/ieee/fixtures/index-v2.zip` (a small structured index for
187
- the read-spec targets) is deserialized through `::Pubid::Ieee::Identifier` and
188
- pre-loaded into the `Relaton::Index` pool in `before(:suite)`
189
- (`spec/ieee/support/webmock.rb`, NIST recipe). `index_builder_spec.rb` covers
190
- `build_index` (structured rows + skip logging).
191
- When a document 404s after a cassette refresh, the fixture has gone stale
192
- against the republished index: copy the matching row **wholesale** (`:id:` *and*
193
- `:file:`) out of the live `…/relaton-data-ieee/refs/heads/v2/index-v2.zip`,
194
- rewrite the zip with the single `index-v2.yaml` entry, and **delete the cassette**
195
- so it re-records — `record: :once` will not re-record a *changed* request URI, it
196
- raises `UnhandledHTTPRequestError` instead.
197
- - **Expected-XML fixtures self-heal only when absent.** `ieee_spec.rb` and
198
- `idams_parser_spec.rb` write `fixtures/<name>.xml` with `unless File.exist?`, so
199
- reconciling a pubid rendering change means deleting the fixture, re-running, and
200
- reviewing the diff — not hand-editing it.
201
- - **VCR cassettes:** `spec/ieee/vcr_cassettes/` — `index-v2.zip` downloads are
202
- ignored by VCR (handled by the fixture).