relaton 3.1.1 → 3.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/converter/csl/from_csl.rb +195 -0
  5. data/lib/relaton/bib/converter/csl.rb +13 -1
  6. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  7. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  8. data/lib/relaton/cli/command.rb +3 -1
  9. data/lib/relaton/cli/frontend_assets.rb +7 -6
  10. data/lib/relaton/cli/subcommand_collection.rb +5 -1
  11. data/lib/relaton/core/request_error.rb +18 -0
  12. data/lib/relaton/db/config.rb +9 -1
  13. data/lib/relaton/db/workers_pool.rb +11 -1
  14. data/lib/relaton/db.rb +160 -58
  15. data/lib/relaton/ecma/bibliography.rb +2 -2
  16. data/lib/relaton/etsi/data_fetcher.rb +16 -6
  17. data/lib/relaton/iala/bibliography.rb +1 -1
  18. data/lib/relaton/iana.rb +1 -1
  19. data/lib/relaton/iec/data_parser.rb +41 -2
  20. data/lib/relaton/itu/README.adoc +1 -1
  21. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  22. data/lib/relaton/jis/README.adoc +2 -0
  23. data/lib/relaton/jis/data_fetcher.rb +74 -13
  24. data/lib/relaton/jis/hit_collection.rb +2 -1
  25. data/lib/relaton/jis/processor.rb +1 -1
  26. data/lib/relaton/jis/scraper.rb +46 -5
  27. data/lib/relaton/oasis/bibliography.rb +1 -1
  28. data/lib/relaton/ogc/hit_collection.rb +1 -1
  29. data/lib/relaton/ogc.rb +1 -1
  30. data/lib/relaton/plateau/data_fetcher.rb +10 -1
  31. data/lib/relaton/plateau/handbook_parser.rb +38 -2
  32. data/lib/relaton/plateau/hit_collection.rb +20 -5
  33. data/lib/relaton/sae.rb +1 -1
  34. data/lib/relaton/version.rb +1 -1
  35. data/lib/relaton/w3c/bibliography.rb +1 -1
  36. data/lib/relaton/xsf/bibliography.rb +1 -1
  37. data/lib/relaton/xsf.rb +1 -1
  38. metadata +35 -48
  39. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  40. data/lib/relaton/adobe/CLAUDE.md +0 -104
  41. data/lib/relaton/bib/CLAUDE.md +0 -150
  42. data/lib/relaton/bipm/CLAUDE.md +0 -187
  43. data/lib/relaton/bsi/CLAUDE.md +0 -35
  44. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  45. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  46. data/lib/relaton/cen/CLAUDE.md +0 -219
  47. data/lib/relaton/cie/CLAUDE.md +0 -90
  48. data/lib/relaton/core/CLAUDE.md +0 -140
  49. data/lib/relaton/db/CLAUDE.md +0 -385
  50. data/lib/relaton/doi/CLAUDE.md +0 -105
  51. data/lib/relaton/easc/CLAUDE.md +0 -85
  52. data/lib/relaton/ecma/CLAUDE.md +0 -217
  53. data/lib/relaton/etsi/CLAUDE.md +0 -219
  54. data/lib/relaton/evs/CLAUDE.md +0 -120
  55. data/lib/relaton/gb/CLAUDE.md +0 -47
  56. data/lib/relaton/gost/CLAUDE.md +0 -100
  57. data/lib/relaton/iala/CLAUDE.md +0 -97
  58. data/lib/relaton/iana/CLAUDE.md +0 -105
  59. data/lib/relaton/iec/CLAUDE.md +0 -13
  60. data/lib/relaton/ieee/CLAUDE.md +0 -202
  61. data/lib/relaton/ietf/CLAUDE.md +0 -382
  62. data/lib/relaton/iho/CLAUDE.md +0 -76
  63. data/lib/relaton/index/CLAUDE.md +0 -208
  64. data/lib/relaton/isbn/CLAUDE.md +0 -47
  65. data/lib/relaton/iso/CLAUDE.md +0 -38
  66. data/lib/relaton/itu/CLAUDE.md +0 -685
  67. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  68. data/lib/relaton/jis/CLAUDE.md +0 -64
  69. data/lib/relaton/logger/CLAUDE.md +0 -37
  70. data/lib/relaton/nist/CLAUDE.md +0 -194
  71. data/lib/relaton/oasis/CLAUDE.md +0 -251
  72. data/lib/relaton/ogc/CLAUDE.md +0 -242
  73. data/lib/relaton/oiml/CLAUDE.md +0 -47
  74. data/lib/relaton/omg/CLAUDE.md +0 -137
  75. data/lib/relaton/plateau/CLAUDE.md +0 -126
  76. data/lib/relaton/sae/CLAUDE.md +0 -91
  77. data/lib/relaton/sdo/CLAUDE.md +0 -81
  78. data/lib/relaton/un/CLAUDE.md +0 -91
  79. data/lib/relaton/w3c/CLAUDE.md +0 -369
  80. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,107 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with
4
- code in this repository.
5
-
6
- ## Project Overview
7
-
8
- `Relaton::Jcgm` retrieves **JCGM** (Joint Committee for Guides in Metrology)
9
- publications — the metrology **guides/GUM/VIM** and the committee **meeting**
10
- proceedings. It is **index-backed** (no scraping): a reference is parsed with
11
- `Pubid::Jcgm`, matched against a pre-built index, and the matching per-document
12
- YAML is fetched from the `relaton/relaton-data-jcgm` GitHub repo.
13
-
14
- JCGM was previously served **inside the BIPM flavor** (its records lived under
15
- `relaton-data-bipm`). It is now its own SDO/flavor: the BIPM flavor no longer
16
- matches `JCGM` in its default-prefix, no longer parses JCGM ids, and no longer
17
- harvests the `jcgm` dir.
18
-
19
- ## Architecture
20
-
21
- Namespace `Relaton::Jcgm`; require path `relaton/jcgm`. Retrieval flow:
22
-
23
- 1. **Bibliography** (`bibliography.rb`) — `get`/`search`: parses the reference
24
- with `::Pubid::Jcgm.parse`, looks it up in the index, fetches the matching
25
- YAML over HTTP. `ENDPOINT` points at `relaton-data-jcgm`. An unparseable
26
- reference **raises** `Pubid::Errors::ParseError`; the `search` rescue lists
27
- transport errors only, so it does not swallow it.
28
- 2. **Index** — `Relaton::Index.find_or_create(:jcgm, url:, file:, pubid_class:
29
- ::Pubid::Jcgm::Identifier)`. Because rows are stored as `Pubid::Jcgm`
30
- identifiers, the generated index carries the structured `_type:
31
- pubid:jcgm:{guide,gum-guide,amendment,meeting}` form (not BIPM's bespoke
32
- `{group,type,number,year}` hash). Matching is `pubid_match?`: pubid's
33
- asymmetric subset match (`query === row_id`, `Pubid::SubsetMatch`) plus an
34
- optional exact-year filter for the `year` argument. Guide editions
35
- (`200:2008` vs `200:2012`) differ only by year, so a year-less reference
36
- reaches both and the latest is chosen via `max_by(&:year)`; meetings differ
37
- by number, which the reference states. The document type is the identifier's
38
- class, and `===` requires the same class, so a guide never answers for its
39
- corrigendum.
40
- 3. **Model** (`item.rb`, `ext.rb`, `doctype.rb`, `structured_identifier.rb`,
41
- `docidentifier.rb`, …) — the records are **BIPM-shaped** (meetings are
42
- `type: proceedings` / `ext.doctype: meeting-report`; guides are `type:
43
- standard`), so the model mirrors `Relaton::Bipm`'s (a JCGM `Doctype`, a single
44
- `structuredidentifier`) rather than OIML's. `docidentifier.rb` is pubid-backed
45
- like OIML's. The bibliography/index *wiring* follows OIML; the *model* follows
46
- BIPM.
47
- 4. **Processor** (`processor.rb`) — `@prefix = "JCGM"`, `@defaultprefix =
48
- %r{^JCGM\s}`, `@pubid_flavor = :Jcgm` (prefixes sourced from
49
- `Pubid::Jcgm.prefixes`). Lazy-`require_relative`s `../jcgm` in its methods.
50
- 5. **DataFetcher** (`data_fetcher.rb`, `meetings_parser.rb`) — (re)builds the
51
- `relaton-data-jcgm` dataset. `fetch("bipm-data-outcomes")` harvests meeting
52
- proceedings from a local checkout of `metanorma/bipm-data-outcomes`'s
53
- `jcgm/meetings-{en,fr}` dirs (the JCGM-only slice of BIPM's
54
- `DataOutcomesParser`; JCGM has no sub-resolutions or parts). The curated
55
- guide/GUM/VIM records are **not** harvested — they are hand-maintained YAMLs
56
- in the data repo's top-level `static/` dir, so `relaton-data-jcgm`'s
57
- `crawler.rb` indexes them itself (like relaton-data-bipm's crawler) by looping
58
- `static/**` and calling the public, guarded `DataFetcher#add_to_index`. Rows
59
- are indexed via the `Pubid::Jcgm::Identifier` object (with `pubid_class:` on
60
- the index), so `Relaton::Index` sorts the index by id number on save and
61
- serialises each id to its `_type: pubid:jcgm:*` hash — the published index is
62
- already sorted (no "not sorted by id number" warning on consumer fetch).
63
- `add_to_index` **guards** each id: an unparseable one (the legacy `JCGM GUM` /
64
- `JCGM VIM-3` / `Corrigendum` records) is skipped with a warning rather than
65
- corrupting the whole index.
66
-
67
- ## The naive ordinal (11st / 12nd / 13rd)
68
-
69
- Meeting docnumbers print the ordinal with a **naive last-digit rule** (1→st,
70
- 2→nd, 3→rd, else th) and **no** 11/12/13 teens exception, so the real records
71
- read `JCGM 11st Meeting (2006)`, `12nd`, `13rd`. This matches the source data and
72
- `Pubid::Jcgm::Identifiers::Meeting.ordinal`, so the printed form round-trips
73
- through pubid. The DataFetcher delegates to `Meeting.ordinal` (single source of
74
- truth) rather than re-implementing it. Any validation MUST expect `11st`, not
75
- `11th`.
76
-
77
- ## Record types & the index guard
78
-
79
- The five `Pubid::Jcgm` types the index carries: `pubid:jcgm:{guide, gum-guide,
80
- amendment, corrigendum, meeting}`. The bare `GUM`/`VIM-N` records parse as
81
- `guide`, and `JCGM 200:2008 Corrigendum` parses as `corrigendum` (whose hash
82
- nests its `base` guide) — all on pubid **main** (see Dependencies). `Relaton::Index`
83
- rejects the whole index if a single row fails `from_hash`, so
84
- `DataFetcher#add_to_index` **guards** each id defensively and skips an unparseable
85
- docnumber with a warning rather than corrupting the index. Every current JCGM
86
- record parses; the guard protects against a future malformed one.
87
-
88
- ## Testing
89
-
90
- RSpec with WebMock. The offline index fixture `spec/jcgm/fixtures/index-v1.zip`
91
- is pre-loaded into the `Relaton::Index` pool in `before(:suite)`
92
- (`spec/jcgm/support/webmock.rb`); per-document requests are served from
93
- `spec/jcgm/fixtures/{data,static}/**`. `spec/jcgm/fixtures/bipm-data-outcomes/**`
94
- holds minimal source fixtures for the DataFetcher spec (the real upstream is not
95
- vendored). `spec/jcgm/relaton/jcgm/pubid_spec.rb` guards the naive ordinals and
96
- the `_type` forms.
97
-
98
- ## Dependencies
99
-
100
- `pubid` (umbrella gem, provides `Pubid::Jcgm`), `relaton-bib`, `relaton-index`,
101
- `relaton-core`.
102
-
103
- > **Dependency pin:** the JCGM support (meetings, bare `GUM`/`VIM-N` guides, the
104
- > `Corrigendum` type, and the flattened compact `to_hash`) first shipped in the
105
- > released pubid `2.0.0.pre.alpha.13`, which the gemspec pins. Its `lib/` is
106
- > identical to the pubid `main` that built the published `relaton-data-jcgm`
107
- > index — a mismatched pubid would make `Relaton::Index` reject that index.
@@ -1,64 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-jis is a Ruby gem that retrieves Japanese Industrial Standards (JIS) metadata from webdesk.jsa.or.jp and models them as Relaton bibliographic items with XML/YAML serialization and RelaxNG schema validation.
8
-
9
- ## Commands
10
-
11
- - `bundle exec rake spec` — run all tests
12
- - `bundle exec rspec spec/relaton/jis/item_spec.rb` — run a single test file
13
- - `bundle exec rspec spec/relaton/jis/item_spec.rb:15` — run a single test by line number
14
- - `bundle exec rake rubocop` — lint
15
- - `bundle exec rake` — default task (runs specs)
16
-
17
- ## Architecture
18
-
19
- The gem is currently on the `lutaml-integration` branch, a major refactoring from the old `RelatonJis::` namespace to `Relaton::Jis::` using lutaml-model for data modeling.
20
-
21
- ### Class Hierarchy
22
-
23
- All classes live under `lib/relaton/jis/`:
24
-
25
- - **`Item`** (extends `Iso::Item`) — core bibliographic item; defines the `Ext` attribute for JIS extensions; uses model `Bib::ItemData`
26
- - **`Bibitem`** (extends `Item`, includes `Bib::BibitemShared`) — XML bibitem serialization
27
- - **`Bibdata`** (extends `Item`, includes `Bib::BibdataShared`) — XML bibdata wrapper
28
- - **`Ext`** (extends `Iso::Ext`) — JIS-specific extension data (schema_version, doctype)
29
- - **`Docidentifier`** (extends `Bib::Docidentifier`) — `#pubid` parses `content` with `Pubid::Jis::Identifier` on first use (lazy, memoized, soft: nil if it does not parse). The mutators change the pubid and render it back; there is no regex. `remove_part!` clears every part level (`JIS C 0364-2-21` → `JIS C 0364`), `remove_date!` clears the year and the reaffirmation `R`, and on a supplement both act on the base — the supplement keeps its own year, because pubid renders a year-less supplement as `…/AMD 1:`. `to_all_parts!` wraps `@pubid` in pubid's `AllParts` (so `#pubid.all_parts?` reads true) but writes `content`'s own marker ` (all parts)` from the pre-wrap rendering, not pubid's `(規格群)` (which is what `#pubid.to_s`, read directly, would print). The `JIS ` publisher is rendered only if the source has it (`TR Z 0010:2008` stays without it); everything else is the pubid canonical spelling (`AMENDMENT 1` → `AMD 1`).
30
- - **`Doctype`** (extends `Bib::Doctype`) — allowed types: `japanese-industrial-standard`, `technical-report`, `technical-specification`, `amendment`
31
- - **`Scraper`** — scrapes individual JIS document pages from webdesk.jsa.or.jp; returns `Bib::ItemData`; editorial group is modeled as a contributor with `Bib::Subdivision` (not `EditorialGroup`)
32
- - **`DataFetcher`** (extends `Core::DataFetcher`) — bulk-fetches all JIS documents via threaded scraping; implements `to_yaml`/`to_xml`/`to_bibxml` for serialization dispatch; loaded on-demand via `require "relaton/jis/data_fetcher"`
33
- - **`Processor`** (extends `Core::Processor`) — Relaton processor registration; provides `get`, `fetch_data`, `from_xml`, `from_yaml`, `grammar_hash`; prefix `JIS`, defaultprefix `^(JIS|TR)\s`
34
- - **`Util`** (includes `Relaton::Bib::Util`) — logging with PROGNAME "relaton-jis"
35
-
36
- ### Key Dependencies
37
-
38
- - **relaton-iso / relaton-bib** — parent bibliographic item models this gem extends
39
- - **lutaml-model** — DSL for `attribute` definitions and serialization
40
- - **mechanize** — web scraping from JSA website
41
-
42
- ### Serialization & Validation
43
-
44
- - YAML and XML round-trip serialization via lutaml-model
45
- - RelaxNG grammars in `grammars/` validate XML output; `relaton-jis.rng` defines JIS-specific elements (DocumentType, structuredidentifier, stagename)
46
- - Tests validate XML against these grammars using ruby-jing
47
-
48
- ### Test Patterns
49
-
50
- - **Round-trip tests**: YAML → Object → YAML and XML → Object → XML (fixtures in `spec/fixtures/`)
51
- - **Schema validation**: Jing validates generated XML against RelaxNG grammars
52
- - **HTTP mocking**: VCR cassettes (`spec/vcr_cassettes/`) record external HTTP interactions; WebMock disables real network calls in tests
53
-
54
- ## Code Conventions
55
-
56
- - All files use `# frozen_string_literal: true`
57
- - Linting follows [Ribose OSS style](https://github.com/riboseinc/oss-guides); target Ruby 3.1
58
- - Ruby >= 3.1.0 required
59
-
60
- ## Testing
61
-
62
- - **Index fixture:** `spec/fixtures/index-v2.zip` (the pubid-based `index-v2`) is deserialized via `Pubid::Jis::Identifier` and pre-loaded into the `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). No rake task refreshes it: download `index-v2.zip` from the `v2` branch of relaton-data-jis into `spec/jis/fixtures/`.
63
- - **Lookup path:** `Bibliography.get`/`search` parse the reference with `pubid`. An unrecognized reference raises `Pubid::Errors::ParseError`; do not rescue it to nil. `HitCollection` takes the candidates that share the base series and number from `index-v2` (a supplement is filed under its base number). `Hit#matches?` is `Pubid::Identifier#matches?` with `ignore: %i[year reaffirmed]` (plus `:parts` for all parts), so the base document of a supplement, its parts, and the `SYMBOL` are compared. The reaffirmation mark `R` goes with the year, because `exclude(:year)` keeps it. For a dated reference, `find_by_year` picks the exact printed id first, then the same id without `R`. Two pubid gaps shape this code. `Pubid::Jis::Identifier#==` does not compare the class (`JIS TR X 0014:1999 == JIS X 0014:1999`), so `Hit#matches?` checks `instance_of?`. `SupplementIdentifier#==` ignores `symbol` and `reaffirmed`, so the exact printed id comes first. Before this rule, 301 of the 21,496 published ids resolved to a different record; `hit_collection_spec.rb` holds one example for each shape. The `DataFetcher` writes `index-v2` only. relaton-data-jis rebuilds the legacy string `index-v1` from `data/` in its own crawler, because the released `relaton-jis` gem still reads `index-v1.zip` from the `v2` branch.
64
- - **pubid dependency:** the gemspec requires `pubid ~> 2.0.0.pre.alpha.3`, the published release carrying the working lutaml-model `from_hash` (the earlier `2.0.0.pre.alpha.2` was broken). Override to a local checkout with `bundle config set local.pubid /path/to/pubid`.
@@ -1,37 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-logger is the logging infrastructure for the Relaton family. It extends Ruby's stdlib `Logger` with a **logger pool** (named loggers), **per-logger level sets** (fine-grained filtering instead of a single threshold), pluggable formatters (string and JSON), and a GitHub-issue reporting channel for bulk-fetch error summaries. Flavor gems log through it (e.g. `Relaton::Bib::Util` extends it with a `PROGNAME`).
8
-
9
- ## Development
10
-
11
- ```bash
12
- bundle exec rake # default task → rspec
13
- bundle exec rspec spec/relaton/logger/log_spec.rb # single file
14
- bundle exec rake rubocop # lint (separate task)
15
- ```
16
-
17
- ## Architecture
18
-
19
- Namespace: `Relaton::Logger`. Key classes in `lib/relaton/logger/`:
20
-
21
- - **`Log`** (`log.rb`) — subclass of `::Logger`. Replaces the single severity threshold with a `levels` Set (`add_level`/`remove_level`); `Log#add` writes only when the severity is in `levels`. Each call accepts a `progname` and a `key:` keyword used by formatters.
22
- - **`Pool`** (`pool.rb`) — a container of named loggers (`[:default]`, `[:file]`, …); logging calls fan out to every logger in the pool.
23
- - **`Config`** (`config.rb`) — `Relaton.configure { |c| ... }` / `Relaton.configuration`; the default `Configuration` seeds a `Pool` with a `:default` stderr logger at levels info/warn/error/fatal.
24
- - **Formatters** — `FormatterString` emits `[progname] SEVERITY: (key) message`; `FormatterJSON` emits a JSON line merging the call's keyword args.
25
- - **`Channels::GhIssue`** (`channels/gh_issue.rb`) — accumulates de-duplicated log lines and opens a GitHub issue via the REST API on `close` (needs `GITHUB_TOKEN`; silently skips when unset or empty). Used by `Relaton::Core::DataFetcher` to report bulk-ingest errors.
26
-
27
- ### Entry point
28
-
29
- `Relaton.logger_pool` returns the configured pool; `Relaton.logger_pool.info "msg"` broadcasts to all loggers, and `Relaton.logger_pool[:default]` accesses one by name.
30
-
31
- ## External dependencies
32
-
33
- `logger ~> 1.6`.
34
-
35
- ## Testing
36
-
37
- RSpec with SimpleCov; specs mirror the lib structure under `spec/relaton/logger/`.
@@ -1,194 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-nist is a Ruby gem that retrieves and manages NIST (National Institute of Standards and Technology) bibliographic metadata as part of the Relaton ecosystem. Data sources include the NIST Cybersecurity Resource Center (CSRC) and the NIST Library (MARC21/MODS XML).
8
-
9
- ## Development Commands
10
-
11
- ```bash
12
- # Install dependencies
13
- bundle install
14
-
15
- # Run all tests (any of these)
16
- rake
17
- rake spec
18
- bundle exec rspec
19
-
20
- # Run a single test file
21
- bundle exec rspec spec/relaton/nist/item_spec.rb
22
-
23
- # Run a specific test by line number
24
- bundle exec rspec spec/relaton/nist/item_spec.rb:10
25
-
26
- # Interactive console with gem loaded
27
- bin/console
28
-
29
- # Lint (uses Ribose OSS rubocop config)
30
- bundle exec rubocop
31
- ```
32
-
33
- ## Architecture
34
-
35
- ### Module Structure
36
-
37
- All classes live under `Relaton::Nist` (entry point: `require 'relaton/nist'`).
38
-
39
- ### Lutaml-Model Serialization
40
-
41
- All data models inherit from `Lutaml::Model::Serializable` and use declarative attribute/mapping DSL for XML and YAML serialization. The base bibliographic types come from `relaton-bib` (aliased as `Bib`).
42
-
43
- **Core model hierarchy**:
44
- - `Bib::Item` → `Relaton::Nist::Item` (`model ItemData`)
45
- - `Relaton::Nist::Bibitem` (includes `Bib::BibitemShared`)
46
- - `Relaton::Nist::Bibdata` (includes `Bib::BibdataShared`)
47
- - `Bib::ItemData` → `Relaton::Nist::ItemData`
48
-
49
- **NIST-specific extensions** (under `lib/relaton/nist/`):
50
- - `Ext` — extension block holding doctype, comment period (extends `Bib::Ext`)
51
- - `Date` — adds `abandoned` and `superseded` date types (extends `Bib::Date`)
52
- - `Relation` — adds `obsoletedBy`, `supersedes`, `supersededBy` relation types (extends `Bib::Relation`)
53
- - `CommentPeriod` — from/to/extended date ranges (extends `Lutaml::Model::Serializable`)
54
- - `Doctype` — currently only `standard` (extends `Bib::Doctype`)
55
-
56
- **Search/retrieval layer** (extends `relaton-core`):
57
- - `Bibliography` — class methods `search(text, year, opts)` and `get(code, year, opts)`
58
- - `HitCollection` < `Core::HitCollection` — searches GitHub index + CSRC JSON
59
- - `Hit` < `Core::Hit` — lazily resolves items via `Scraper`
60
- - `Processor` < `Core::Processor` — Relaton plugin interface (`get`, `fetch_data`, `from_xml`, `from_yaml`)
61
-
62
- **Data fetching/parsing**:
63
- - `DataFetcher` < `Core::DataFetcher` — fetches NIST Tech Pubs MODS XML from GitHub releases. `#source_url(source)` builds the download URL: with no tag (nil/blank/`"latest"`) it uses GitHub's `releases/latest/download/allrecords-MODS.xml` redirect, so the `relaton-data-nist` crawler auto-picks up new releases without a gem release; passing a concrete tag (e.g. `"June2026"`) pins that release. The `source` flows from `Core::DataFetcher.fetch(source)` → `#fetch(source)` → `#fetch_tech_pubs(source)` (the crawler workflow's `args` input can supply it).
64
- - **Per-record resilience (mirrors the ISO flavor).** A single bad record must never abort the crawl. `#write_file` skips (never `id.content`-crashes on) a record whose MODS entry yields **no docidentifier** (e.g. a DOI stored only in `<identifier type="doi">`, which `ModsParser#parse_doi` doesn't read), and records ids that `#pubid` can't parse (new series like `NIST RB`/`NIST CHIPS`) as "not indexed" — the YAML is still written, only the index-v2 entry is skipped. Both classes accumulate in `#failures`; the `#report_errors` override emits each at `:error` (via `#log_error` → `Util.error`), which the shared `Core::DataFetcher` gh_issue machinery turns into one "Error fetching documents" GitHub issue in CI (`GITHUB_REPOSITORY` + `GITHUB_TOKEN`). `#pubid` no longer warns per-failure; reporting is centralized in `#report_errors`.
65
- - **Draft-stage rows come from CSRC (relaton#313).** The MODS collection has no draft records, so `#fetch_csrc` adds the CSRC pubs-export feed (`PubsExport.instance.data`) after the MODS pass. It builds each row **exactly as the runtime builds a CSRC hit** — `HitCollection#hit_data` (which calls `#pubs_export_id`) → `Scraper.parse_page` — so the crawled id and the live id are one string, with the stage, revision and update from the row's uri and DOI (`NIST FIPS 197/Upd1 ipd`, `NIST SP 800-108r1 ipd`, `NIST SP 1800-12 2pd`). Only a row whose id carries a **stage** is written; finals come from MODS, so a CSRC row never shadows a base document. On the 914-row fixture feed this writes 302 staged records. Don't go back to a hand-built `"NIST #{series} #{docnumber} #{stage}"` id: it dropped the numbered drafts (`2pd`…), the `/UpdN`, and spelled revisions `Rev. 1`. A stage that CSRC does not list (e.g. `NIST FIPS 197 fpd`) still answers through the consumer's #303 family fallback. Three more rules: `#csrc_bib` clears the `fetched` date the runtime stamps, or every draft changes each day and `Core#force_write` rewrites ~300 files per incremental crawl; CSRC lists a republished draft twice (`…/ipd` and `…/ipd-(1)`, one DOI, so one id), so each id is written once and the plain uri wins; and only the build step is rescued (to a `#failures` entry) — a write error stops the crawl, as in the MODS pass. `#pubs_export_id` must not mutate its row (`sub`, not `sub!`): the crawler and the runtime share the `PubsExport` data.
66
- - **A CSRC relation names the related document (relaton#314).** `Scraper#related_id` finds the related feed row by its uri and derives its id with `HitCollection.pubs_export_id` (the class method the instance method delegates to). The entry's own `docidentifier` drops the update and the part, and the hit's code is the record's own id, so neither is the related id. `HitCollection#search_filter` ends in an `ensure` that returns `arr`: every branch must set `arr`, because an earlier `return` is overridden there.
67
- - **CSWP comes from CSRC, not the NIST Library (relaton-nist#104).** The NIST Library announced CSWP data, but its MODS release (`allrecords-MODS.xml`, and the MARC `allrecords.xml`) holds **no** CSWP record — 0 on the Sept2026 release. It publishes CSWPs only as RIS/BibTeX files under `bib/` on its `nist-pages` branch. So the index has only the CSRC draft CSWP rows (`#fetch_csrc`), and a final CSWP answers from CSRC at runtime: `HitCollection#search` asks the CSRC feed (`from_json`) and asks the index (`from_ga`) only when CSRC gives no hit. That is the chosen rule — CSRC wins, no merge — because CSRC lists every numbered CSWP with more metadata (affiliations, status, supersedes, comment period). `hit_collection_spec.rb` pins the CSRC answer.
68
- - **Legacy dated CSWP ids and CSWP translations come from a static table (relaton#406).** CSRC lists neither the ~30 **legacy dated ids** (`NIST CSWP 01142020`, the same paper as `CSWP 9`, from before NIST numbered the series) nor the translations (`NIST.CSWP.6.spa`). `cswp.yaml` holds both, built once from the 84 RIS files (header of the file says how), and `Cswp` reads it. `Bibliography.get` asks `Cswp.lookup` **first** — the parenthetical-date regex after it would strip `(spa)`. A legacy id fetches the numbered id through the normal CSRC path and adds the dated id as a non-primary docidentifier (`06222021-draft` → the CSRC `CSWP 19 ipd`); a translation, dated or numbered, answers an offline record from the table (`NIST CSWP 6(spa)`, the RIS title/authors/year/DOI, `language`/`script` from `Cswp::LANGUAGES`, a `translatedFrom` relation to the original). `normalize` reads `NIST CSWP 6 spa`, `NIST CSWP 6(spa)` and `NIST.CSWP.6.spa` as one key. `Processor#cache_pubid` returns nil (so `Db` hands `get` the String and does not cache) for **every legacy id** and for a translation that pubid does not render back. A legacy answer is the numbered record plus the dated id, and `Db` would store it under the numbered id's row, so a later `NIST CSWP 9` would carry `NIST CSWP 01142020`. pubid reads `04162018ar` as `04162018a`, `6 spa` as `6(spa)`, and fails on `04162018bg`/`02122014pl`. A translation answers only for its own RIS year. Not mapped, on purpose: `02042022-1`…`-5` have no SN and share the title "Cybersecurity White Paper: EO Response", so they still answer not-found. The CSRC `ipd` rows of `27`/`29`/`32` already answer, so the table leaves out their RIS files. A new translation means a new table row.
69
- - **`#write_file` serializes before it indexes**, so a record that fails to serialize leaves no index row naming a file that was never written. That is how a `Bib::Date` in `Scraper.fetch_dates` hid: `Nist::ItemData` takes `Nist::Date`, and YAML serialization rejects the parent class (the same strictness as the docidentifier note below).
70
- - `ModsParser` — maps MODS XML (via `loc_mods`) to `ItemData`
71
- - `Scraper` — fetches items from GitHub YAML or CSRC JSON
72
- - `PubsExport` — singleton; caches CSRC pubs-export zip with thread-safe daily updates. The cache file is `#datafile`, under `Relaton::Index.config.storage_dir` (default `Dir.home`; the `DATAFILE` constant is only that default).
73
- - **A read-only file system must not stop a search (relaton-nist#57).** When the cache write fails, `#fetch_data` keeps the downloaded zip in `@buffer`, and `#unzip` reads it with `Zip::File.open_buffer`. In rubyzip 3, `open_buffer` returns the buffer, not the block value, so collect the rows in a local. The cache write goes to a temp file and is renamed into place (`#write_datafile`), so a failed write never leaves a truncated zip that later processes read as fresh. `HitCollection#search` also rescues the local-cache errors in `LOCAL_DATA_ERRORS` from `from_json` (`#from_json_or_empty`), logs a warning, and continues with `from_ga`. Do not widen that list to `SystemCallError`: `Errno::ECONNREFUSED` is one, and a CSRC outage would then answer "not found" from the index, which `Relaton::Db` can cache. Specs assert the warning with `output(...).to_stderr_from_any_process`. A `receive(:warn)` stub on `Util` is never hit: `Kernel#warn` is private, the stub stays private, and the explicit-receiver call goes to `method_missing`.
74
-
75
- ### Pubid-backed docidentifier
76
-
77
- `Relaton::Nist::Docidentifier` (`docidentifier.rb`, `< Bib::Docidentifier`,
78
- declared flat like the rest of the NIST models) parses its `content` into a
79
- `Pubid::Nist::Identifier` kept in `@pubid`, while the lutaml `content` attribute
80
- stays a plain string for serialization. It implements the base class's abstract
81
- `remove_part!` / `remove_date!` / `to_all_parts!` (plus a NIST `remove_stage!`)
82
- by mutating the pubid graph and re-rendering via `refresh_content!`, so NIST
83
- items no longer raise `NotImplementedError` on the `Bib::ItemData#to_all_parts!`
84
- / `#remove_date!` paths. It is wired into `item.rb`
85
- (`attribute :docidentifier, Docidentifier, collection: true`) and into
86
- `item_base.rb` (`Nist::ItemBase`, the nested bibitem used inside `Relation`), so
87
- relation cross-reference ids are pubid-backed too — mirroring
88
- `iso/model/item_base.rb` / `iec/model/item_base.rb`. `ModsParser` and `Scraper`
89
- build **every** docid (primary NIST id, DOI, and relation cross-refs) as this
90
- class. Mirrors `lib/relaton/iec/model/docidentifier.rb` and
91
- `lib/relaton/ccsds/model/docidentifier.rb`.
92
-
93
- NIST-specific gotchas (why this isn't a straight copy of IEC/CCSDS):
94
-
95
- - **Adopt-on-round-trip, not a `type` gate.** A NIST DOI such as
96
- `NIST.SP.800-162` is itself a valid NIST pubid in dotted MR form whose `:human`
97
- render is `NIST SP 800-162` — so blindly parsing it would rewrite the DOI.
98
- `content=` therefore keeps the parsed pubid **only when
99
- `pubid.to_s(:human) == content`**; otherwise the raw string is preserved.
100
- A `type == "NIST"` gate looks tempting but breaks: lutaml applies setters in
101
- attribute-declaration order (`content` before `type`), so on `from_yaml` the
102
- gate sees `type == nil`, drops the pubid, and the mutators silently no-op on
103
- YAML-loaded items. The round-trip test is type/order-independent.
104
- - **Uniform collection type.** The `docidentifier` collection is typed
105
- `Nist::Docidentifier`, and lutaml's **YAML/JSON** serializer strictly rejects a
106
- parent `Bib::Docidentifier` instance in that slot (`IncorrectModelError`) —
107
- which would crash the YAML-default `DataFetcher` crawler on nearly every
108
- DOI-bearing record (XML tolerates it, which is why it hides in XML specs). So
109
- the DOI is a `Nist::Docidentifier` too (kept raw via the round-trip rule), not
110
- a `Bib::Docidentifier`.
111
- - **The date lives in the `edition` component.** NIST has no `:date` attribute;
112
- a year is carried as the edition id (`FIPS 46e1977`) or its trailing
113
- `additional_text` (`NBS CIRC 11e2.1915`), and the scalar `year`/`edition_year`
114
- fields are nil on the common parse paths. `remove_date!` clears the scalar
115
- fields *and* strips a 4-digit-year edition/`additional_text`, while preserving
116
- numbered editions/revisions (`e2`, `r5`). Dates that ride the `update`
117
- component (some NBS supplements) are left intact — `update` also encodes
118
- non-date update codes (`/Upd2`), so stripping it would corrupt identity.
119
- - **The item's `edition` comes from the DOI (relaton-nist#56).**
120
- `ModsParser#parse_edition` sets `<edition number="N">N</edition>` only for a
121
- numbered `e<N>` edition in the DOI (`NBS.CIRC.111e2` → 2, `11e2.1915` → 2):
122
- 250 of the 20,392 live MODS records. An `e<year>` edition is a date, and an
123
- `r<N>` revision is not an edition, so both stay without one. The MODS
124
- `<edition>` element is prose (`New version`, `Erratum`), never a number. The
125
- issue's "edition 0 → 1" rule came from the old Crossref feed; it is kept for
126
- a DOI `e0` (none in the live data). A DOI pubid cannot parse gives no
127
- edition, not an error.
128
- - **No `(all parts)` marker in `content`.** `content` is live-derived from
129
- `@pubid`, so `to_all_parts!` freezes the already-stripped rendering into
130
- `@raw_content` before wrapping `@pubid` in pubid's `AllParts` — matching
131
- IEC/CCSDS. `#pubid` itself, read directly, now answers `all_parts? ==
132
- true` and renders WITH pubid's generic "(all parts)" marker.
133
- - **The internal form drops `NIST `/`NBS `** (relaton#49), so
134
- `docid.to_s(form: :internal)` is `SP 800-113` for `NIST SP 800-113`. The rule
135
- is the `"NIST"` row of `Bib::Docidentifier::FORM_PUBID`: `@pubid`
136
- renders with `to_s(with_publisher: false)`. A DOI docid has type `DOI` and is
137
- unchanged. `to_s` itself is inherited from `Bib::Docidentifier`.
138
-
139
- ### Serialization Round-Trip Pattern
140
-
141
- Models support `from_xml`/`to_xml` and `from_yaml`/`to_yaml`. Tests verify round-trip fidelity by parsing a fixture, re-serializing, and comparing output to input.
142
-
143
- ### XML Schema Validation
144
-
145
- RNG (Relax NG) schemas in `grammars/` validate XML output. Tests use the `jing` gem:
146
- ```ruby
147
- schema = Jing.new("grammars/relaton-nist-compile.rng")
148
- errors = schema.validate(file)
149
- ```
150
- `relaton-nist-compile.rng` is the top-level schema that includes `relaton-nist.rng` and `basicdoc.rng`.
151
-
152
- ### HTTP Recording
153
-
154
- Tests use VCR with WebMock. Cassettes are stored in `spec/vcr_cassettes/` and
155
- `spec/support/vcr.rb` sets `record: :once` with `re_record_interval: nil`, so a
156
- cassette replays as committed and never re-records by itself (relaton#360).
157
-
158
- **When you re-record deliberately:** the NIST GitHub source data can drift (an
159
- author renamed, a `loc.gov` `identifier` dropped), so the round-tripped XML no
160
- longer matches the on-disk `spec/fixtures/*.xml` fixtures; and an upstream that
161
- throttles the recorder records a bad response. Delete the stale cassette, run
162
- the spec, and, for the integration fixtures that compare full XML (`get.xml`,
163
- `hit.xml`, `hit_bibitem.xml`), regenerate them (delete → the spec rewrites them
164
- via `File.write … unless File.exist?`) and read the diff.
165
-
166
- ### Test Data Stubbing
167
-
168
- Tests pre-load both the NIST index and CSRC pubs-export data from local fixtures in `before(:suite)` (see `spec/support/webmock.rb`), avoiding all HTTP requests for these data sources. VCR is configured to ignore both `index-v2.zip` and `pubs-export` requests (`spec/support/vcr.rb`).
169
-
170
- The gem consumes the **index-v2** index (`INDEXFILE = "index-v2"`): each row's `:id` is a `Pubid::Nist::Identifier` hash (lean `to_hash` form), so `HitCollection#from_ga` narrows candidates by number via binary search before the substring block filter, and stringifies `row[:id].to_s` at the Hit boundary. `spec/fixtures/index-v2.zip` is required for the suite to run (`rake spec:update_index` downloads it from `relaton-data-nist/v2`).
171
-
172
- - **Index**: The YAML inside `spec/fixtures/index-v2.zip` is written to a temp file and loaded through `Relaton::Index::Type.new(:nist, file:, pubid_class: ::Pubid::Nist::Identifier)`; calling `type.index` forces the offline read + `pubid_class` deserialization (and sort) before net is blocked. The instance is injected into `Relaton::Index.pool`, with `actual?` overridden to match only the remote (`url:`) lookup so the producer-side `find_or_create(:nist, file:, pubid_class:)` still gets a fresh instance. Run `rake spec:update_index` to refresh.
173
- - **PubsExport**: The `PubsExport` singleton's `@data` is set directly from `spec/fixtures/pubs-export.zip`. Run `rake spec:update_pubs_export` to refresh.
174
-
175
- To apply the index stubbing pattern to other relaton gems:
176
-
177
- 1. Add a `spec:update_index` rake task (downloads `index-v2.zip` from the gem's GitHub data repo)
178
- 2. Run `rake spec:update_index` to create `spec/fixtures/index-v2.zip`
179
- 3. In `spec/support/webmock.rb`: extract the zip's YAML, write it to a temp file, create a `Type` with the flavor's `pubid_class`, call `type.index` to deserialize + sort, override `actual?` to match the remote `url:` lookup, and inject into `Relaton::Index.pool`
180
- 4. In `spec/support/vcr.rb`: add `ignore_request` for `index-v2.zip`
181
- 5. Remove any `allow_any_instance_of(Relaton::Index::Type)` workarounds from specs
182
-
183
- ## Key Dependencies
184
-
185
- - `relaton-bib` — base bibliographic models and shared mixins
186
- - `relaton-core` — base classes for Processor, HitCollection, Hit, DataFetcher
187
- - `loc_mods` — MODS (Metadata Object Description Schema) XML parsing
188
- - `pubid` — NIST publication ID parsing
189
- - `relaton-index` — index/search utilities
190
- - `mechanize` — HTTP fetching for data sources
191
-
192
- ## Code Style
193
-
194
- RuboCop config inherits from the Ribose OSS style guide. Target Ruby version is 3.1.