relaton 3.1.1 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (67) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/frontend_assets.rb +7 -6
  7. data/lib/relaton/db.rb +31 -14
  8. data/lib/relaton/ecma/bibliography.rb +2 -2
  9. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  10. data/lib/relaton/iala/bibliography.rb +1 -1
  11. data/lib/relaton/iana.rb +1 -1
  12. data/lib/relaton/iec/data_parser.rb +41 -2
  13. data/lib/relaton/itu/README.adoc +1 -1
  14. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  15. data/lib/relaton/oasis/bibliography.rb +1 -1
  16. data/lib/relaton/ogc/hit_collection.rb +1 -1
  17. data/lib/relaton/ogc.rb +1 -1
  18. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  19. data/lib/relaton/plateau/hit_collection.rb +1 -1
  20. data/lib/relaton/sae.rb +1 -1
  21. data/lib/relaton/version.rb +1 -1
  22. data/lib/relaton/w3c/bibliography.rb +1 -1
  23. data/lib/relaton/xsf/bibliography.rb +1 -1
  24. data/lib/relaton/xsf.rb +1 -1
  25. metadata +32 -46
  26. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  27. data/lib/relaton/adobe/CLAUDE.md +0 -104
  28. data/lib/relaton/bib/CLAUDE.md +0 -150
  29. data/lib/relaton/bipm/CLAUDE.md +0 -187
  30. data/lib/relaton/bsi/CLAUDE.md +0 -35
  31. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  32. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  33. data/lib/relaton/cen/CLAUDE.md +0 -219
  34. data/lib/relaton/cie/CLAUDE.md +0 -90
  35. data/lib/relaton/core/CLAUDE.md +0 -140
  36. data/lib/relaton/db/CLAUDE.md +0 -385
  37. data/lib/relaton/doi/CLAUDE.md +0 -105
  38. data/lib/relaton/easc/CLAUDE.md +0 -85
  39. data/lib/relaton/ecma/CLAUDE.md +0 -217
  40. data/lib/relaton/etsi/CLAUDE.md +0 -219
  41. data/lib/relaton/evs/CLAUDE.md +0 -120
  42. data/lib/relaton/gb/CLAUDE.md +0 -47
  43. data/lib/relaton/gost/CLAUDE.md +0 -100
  44. data/lib/relaton/iala/CLAUDE.md +0 -97
  45. data/lib/relaton/iana/CLAUDE.md +0 -105
  46. data/lib/relaton/iec/CLAUDE.md +0 -13
  47. data/lib/relaton/ieee/CLAUDE.md +0 -202
  48. data/lib/relaton/ietf/CLAUDE.md +0 -382
  49. data/lib/relaton/iho/CLAUDE.md +0 -76
  50. data/lib/relaton/index/CLAUDE.md +0 -208
  51. data/lib/relaton/isbn/CLAUDE.md +0 -47
  52. data/lib/relaton/iso/CLAUDE.md +0 -38
  53. data/lib/relaton/itu/CLAUDE.md +0 -685
  54. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  55. data/lib/relaton/jis/CLAUDE.md +0 -64
  56. data/lib/relaton/logger/CLAUDE.md +0 -37
  57. data/lib/relaton/nist/CLAUDE.md +0 -194
  58. data/lib/relaton/oasis/CLAUDE.md +0 -251
  59. data/lib/relaton/ogc/CLAUDE.md +0 -242
  60. data/lib/relaton/oiml/CLAUDE.md +0 -47
  61. data/lib/relaton/omg/CLAUDE.md +0 -137
  62. data/lib/relaton/plateau/CLAUDE.md +0 -126
  63. data/lib/relaton/sae/CLAUDE.md +0 -91
  64. data/lib/relaton/sdo/CLAUDE.md +0 -81
  65. data/lib/relaton/un/CLAUDE.md +0 -91
  66. data/lib/relaton/w3c/CLAUDE.md +0 -369
  67. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,140 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-core is the foundation gem every flavor plugin builds on. It provides the abstract **`Relaton::Core::Processor`** base class (the registry plug-in interface), the search primitives (`HitCollection`, `Hit`), a parallel-fetch thread pool (`WorkersPool`), the crawl-politeness pair `Pacer` (shared request pacing) and `Governor` (pool-wide rate-limit back-pressure), the `DataFetcher` base for bulk ingest, and small parsing/utility mixins. It has no knowledge of any specific standards body.
8
-
9
- ## Development
10
-
11
- ```bash
12
- bundle exec rake # default task → rspec
13
- bundle exec rspec spec/core/processor_spec.rb # single file
14
- bundle exec rubocop # lint (rubocop-performance/-rake/-rspec)
15
- ```
16
-
17
- ## Architecture
18
-
19
- Namespace: `Relaton::Core`. Key classes in `lib/relaton/core/`:
20
-
21
- - **`Processor`** (`processor.rb`) — abstract base with the template methods every flavor implements: `get`, `fetch_data`, `from_xml`, `from_yaml`, `grammar_hash`, `threads`. Flavor processors subclass it and set `@short`/`@prefix`/`@defaultprefix`/`@datasets` in `initialize`.
22
- - **`HitCollection`** (`hit_collection.rb`) — holds search results, delegates to an internal array, and uses `WorkersPool` to fetch items in parallel (4 workers by default). Uses `WeakRef` to avoid a circular reference back to its hits.
23
- - **`Hit`** (`hit.rb`) — wraps one search result; its `item` is lazy-loaded on first access.
24
- - **`DataFetcher`** (`data_fetcher.rb`) — abstract base for flavor bulk fetchers; provides `gh_issue` reporting (via `Relaton::Logger::Channels::GhIssue`), `output_file`, `unique_output_file`, `write_unique`, and `serialize` helpers.
25
-
26
- **Filename collisions.** `output_file` sanitizes `.`, `,`, `/`, `:`, `(`, `)`,
27
- `-` and whitespace all to `-`, so two **distinct** docids can map to one path
28
- — the live instance is relaton-data-iana's `rpki/signed-objects` and
29
- `rpki-signed-objects`, which both give `data/rpki-signed-objects.yaml`. A
30
- writer that merely warns and writes anyway leaves one file holding the wrong
31
- document for one of two index ids: a wrong answer, not a missing one.
32
-
33
- **`unique_output_file(docid)` is the write-path entry point.** It reserves and
34
- returns `output_file(docid)` when that path is free *or already belongs to the
35
- same docid*, and only a real clash with a **different** docid gets a variant
36
- suffixed with a digest of that docid. Two consequences worth keeping:
37
-
38
- - It is a **no-op for genuine duplicates**, so each flavor's own duplicate
39
- handling (skip / merge / last-wins) still fires exactly as before. That is
40
- what makes it safe to drop into an existing writer.
41
- - The suffix is keyed on the docid, not on encounter order, so adding a record
42
- never renames an existing file. (Which member of a clashing pair keeps the
43
- plain path does follow write order, which is stable for a given corpus.)
44
-
45
- **The genuine-duplicate check must come FIRST in an adopting writer.** Once a
46
- docid has been disambiguated, `unique_output_file` returns that suffixed path
47
- for it *forever*, so `file != output_file(docid)` stays true on every repeat.
48
- Gating the flavor's duplicate handling on that comparison makes it unreachable
49
- for a repeat, which silently drops a merge (3gpp) or a skip (ecma, jis). Use
50
- `@files.include?(file)` as the duplicate test — a reserved path only ever
51
- belongs to one docid — and use the `!= output_file` comparison only to decide
52
- what to log:
53
-
54
- ```ruby
55
- file = unique_output_file docid
56
- if @files.include? file
57
- <genuine duplicate: merge / skip / warn>
58
- else
59
- Util.warn "... writing #{file} instead" if file != output_file(docid)
60
- @files << file
61
- <write + index>
62
- end
63
- ```
64
-
65
- **`write_unique(docid, content)` closes the cross-PROCESS gap.**
66
- `unique_output_file` reserves in one process's `@file_docids`, which a forked
67
- worker cannot see. `write_unique` reserves, then creates the file with
68
- `O_EXCL` (`EXCLUSIVE = File::WRONLY | File::CREAT | File::EXCL`), and returns
69
- the path it actually wrote. Four outcomes:
70
-
71
- | situation | what happens |
72
- |---|---|
73
- | we already wrote this path for this docid | plain overwrite — a genuine duplicate stays one file |
74
- | `O_EXCL` succeeds | the normal case |
75
- | `EEXIST`, single process | `unique_output_file` already proved no peer of ours holds it, so it is a leftover from an earlier crawl: overwrite |
76
- | `EEXIST`, `@cross_process` | a peer's file and a leftover are indistinguishable: take the digest path instead |
77
-
78
- `@cross_process` defaults to false, so every single-process flavor takes the
79
- third row and behaves exactly as it did before adopting this. A fetcher that
80
- forks sets it **before** the first fork, so children inherit it — see
81
- `Relaton::Ietf::DataFetcher#fetch_ieft_internet_drafts`.
82
-
83
- A forked worker only learns that a path was taken, never *which* clashing
84
- docid deserves it, so the winner would follow the race and the two filenames
85
- would swap between crawls. The parent settles that afterwards, once it can see
86
- every docid: `Relaton::Ietf::DataFetcher#reconcile_output_files`.
87
-
88
- **Deliberately not a wall-clock staleness test.** `File.mtime(file) <
89
- run_started_at` looks like the obvious way to tell a leftover from a peer, and
90
- it is a trap: crawls run into a populated `data/`, so `EEXIST` is the *common*
91
- case on a re-run and the test would decide it for every record. Its
92
- false-"stale" direction is the silent overwrite this method exists to prevent,
93
- and coarse filesystem mtime granularity makes that reachable. Gating on
94
- `@cross_process` removes the failure class instead of tuning it.
95
-
96
- **Do not drop the unconditional `delete_suffix("-")` in `output_file`.** It is
97
- the root cause of the trailing-punctuation half of these collisions, and
98
- removing it would give the clashing ids distinct names — but it renames any id
99
- ending in a character the gsub maps to `-`. Measured over this repo's index
100
- fixtures (2026-08-26):
101
-
102
- | corpus | rows | filenames that would change |
103
- |---|---|---|
104
- | `spec/itu/fixtures/index-v2.zip` | 5,361 | **808** — ids ending `:`, e.g. `ITU-R 1-3/8:` |
105
- | `spec/nist/fixtures/index-v2.zip` | 19,515 | 2 — `NBS TN 467pt1 Add.` |
106
- | `spec/ietf/fixtures/ietf-index-v2.zip` | 177,362 | 3 |
107
- | `spec/iso` + `spec/iec` v2 | 112,189 | 0 |
108
- | every v1 string-id fixture | 97,966 | 7 — IEEE `IEEE 802.3-`, OGC `20-001r2 ` |
109
-
110
- A 15% rename of the published ITU corpus does not pay for collisions that
111
- `unique_output_file` and `write_unique` already close.
112
-
113
- **Audit of every `output_file` caller** (keep this exhaustive — it is the only
114
- place the decision is recorded):
115
-
116
- | Adopted | Abstains, and why |
117
- |---|---|
118
- | `iana`, `oasis`, `calconnect`, `xsf`, `nist`, `ecma`, `jis`, `3gpp`, `iec`, `etsi`; `ietf` via `write_unique` | `ieee` — `reconcile_staged_outputs` recomputes `output_file(docnumber)` in a later pass with no reservation state, so a disambiguated path breaks the staged rename |
119
- | | `itu`, `cie` — documented positional `@seen`/`pos` dedup under a parallel crawl |
120
- | | `iso` — `File.exist?` → `rewrite_with_same_or_newer` version compare |
121
- | | `ccsds` — `merge_links` merges into the existing file by design |
122
-
123
- Note `etsi` had **no** collision handling at all before this (silent overwrite,
124
- not even a warning), and `iec` had the warn-then-overwrite-anyway shape.
125
- - **`WorkersPool`** (`workers_pool.rb`) — `SizedQueue`-backed thread pool for parallel work. **Runtime search only** — `HitCollection` uses it. No fetcher does: it carries no per-worker resources (a crawler needs one Mechanize agent per thread) and collects into an unsynchronised array. A crawler that needs a pool builds its own; see `Relaton::Itu::DataCrawlerR#in_parallel` and `Relaton::Cie::DataFetcher#process_hits`.
126
- - **`Pacer`** (`pacer.rb`) — process-wide, thread-safe request pacer: the whole pool between them starts at most one request per `gap` seconds. **Shared, not per-worker** — that is the point, and the contrast with `Relaton::Cie::DataFetcher::Pacing`, whose gap is per worker (N workers there issue N requests per gap, which is right for CIE and wrong when the budget belongs to one host). A reservation never starts before "now", so an idle stretch cannot be repaid as a burst, and the mutex is never held across the sleep. `mode: :fixed` reproduces a plain `sleep delay` loop as a rollback knob. `clock:`/`sleeper:` are injectable, so the arithmetic is specced without wall-clock waits.
127
- - **`Governor`** (`governor.rb`) — process-wide, thread-safe rate-limit back-pressure: one shared cooldown the whole pool observes, escalating per *round* (60 s → 900 s) rather than per worker, decaying on success, and latching a give-up after five barren rounds so a banned crawl fails fast instead of grinding out a CI job's remaining hours. Two round counters, deliberately: **`#throttle_rounds`** is the LIVE count since the last success (ITU logs it per retry as the current round) and `#succeeded!` resets it, while **`#give_up_rounds`** is the count snapshotted at the instant `@abandoned` latched, and is what an abort message must report — reading the live one after the latch names whatever re-accumulated since, which is how a W3C give-up at round 5 got logged as *"after 2 consecutive backoffs"*. Note also that `#wait` returning 0.0 once latched stops the pool *waiting*, not *asking*: a flavor must guard its own request sites (see `Relaton::W3c::SafeRealize#realize`) or a banned crawl keeps hammering the host until it winds down. Promoted here from `Relaton::W3c`. A flavor subclasses it and supplies two things: `THROTTLE_ERRORS` (or `.throttle?`, when the signal is a status code rather than an exception class, as for ITU) and `ENV_PREFIX`, which namespaces `<PREFIX>_THROTTLE_BASE/_MAX/_GIVEUP` so two flavors crawling in one process cannot share a ladder. Bindings: `Relaton::W3c::Governor`, `Relaton::Itu::Governor`.
128
- - **Mixins** — `DateParser` (`parse_date` for "February 2012"/"2012-02-03"/etc.), `ArrayWrapper` (`array(x)` → always an Array), `HashKeysSymbolizer` (recursive string→symbol keys).
129
-
130
- ### How flavors consume it (important)
131
-
132
- Every flavor's `Relaton::<Flavor>::Processor < Relaton::Core::Processor`. Because the umbrella registry now loads flavors lazily (only their `…/processor` file), any processor method that touches a flavor constant must `require_relative "../<flavor>"` first — see the root `CLAUDE.md` "Registry is lazy" note.
133
-
134
- ## External dependencies
135
-
136
- `nokogiri ~> 1.16`, `psych ~> 5.2.0`, `relaton-logger ~> 2.2.0.pre.alpha.1`.
137
-
138
- ## Testing
139
-
140
- RSpec with SimpleCov. Specs live under `spec/core/` and mirror the lib classes; `spec/support/` defines dummy processor/types for exercising the base classes.
@@ -1,385 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Commands
6
-
7
- ```bash
8
- # Run the umbrella (Db) suite — specs live in spec/relaton/ and run from there
9
- bundle exec rake spec:relaton
10
-
11
- # Run a single spec file (specs are CWD-relative, so cd into the suite dir)
12
- cd spec/relaton && bundle exec rspec db_spec.rb
13
-
14
- # Run a specific test by line number
15
- cd spec/relaton && bundle exec rspec db_spec.rb:234
16
-
17
- # Lint
18
- bundle exec rubocop
19
- ```
20
-
21
- ## Architecture
22
-
23
- Relaton is a Ruby gem that fetches, caches, and manages bibliographic references to technical standards from 25+ organizations (ISO, IEC, IETF, NIST, IEEE, etc.).
24
-
25
- ### Plugin Registry Pattern
26
-
27
- **Relaton::Registry** (singleton) auto-discovers and manages backend processor gems (relaton-iso, relaton-iec, relaton-ietf, etc.). Each processor implements the **Relaton::Processor** interface (`get`, `from_xml`, `from_yaml`, `grammar_hash`, `prefix`, `defaultprefix`). The registry routes reference codes to the correct processor by matching prefixes (e.g., "ISO 19115" → `:relaton_iso`).
28
-
29
- Registration is **lazy**: `register_gems` requires only each flavor's lightweight `…/processor` file, never the heavy flavor top-level, so flavor deps load on first use rather than at startup. Any processor method touching a flavor constant must `require_relative "../<flavor>"` first — see the root `CLAUDE.md` "Registry is lazy" note and `spec/relaton/lazy_loading_spec.rb`.
30
-
31
- ### Global prefix register (relaton-db#103)
32
-
33
- Separate from the reference-dispatch matching above, each processor declares the
34
- **global document-ID prefixes its SDO owns** via `@prefixes` (Array<String>) in
35
- `initialize` — e.g. NIST → `%w[NIST NBS]`, ISO → `["ISO", "ISO/IEC", "IEC/ISO",
36
- "ISO/IEC/IEEE"]`. `Core::Processor#prefixes` defaults to `[prefix]`, so the ~24
37
- single-prefix flavors need no declaration; only multi-/conflicting-prefix
38
- flavors (iso, iec, nist, ieee, bsi) set it. **Joint prefixes are listed
39
- symmetrically** by every co-publisher (both ISO and IEC list `ISO/IEC`; all three
40
- of iso/iec/ieee list `ISO/IEC/IEEE`) so the register returns every owning flavor.
41
- This is a distinct exact-match list, *not* the `@defaultprefix` regex — those
42
- require a trailing `\s` and can't match a bare prefix like `"ISO/IEC"`.
43
-
44
- Two lookups over it:
45
- - `Registry#processors_by_prefix(prefix)` → the owning **processor objects**,
46
- case-insensitive exact match, in registration (`SUPPORTED_GEMS`) order. **Lazy**
47
- — touches no flavor constant. Use this if you must stay lazy.
48
- - `Relaton.prefix_flavor(prefix)` (in `lib/relaton.rb`) → the owning **flavor
49
- module(s)** as an Array (`[]` if none). Dereferencing a module constant
50
- triggers that flavor's autoload, so this **loads** the matched flavor(s) — by
51
- design, since the caller asked for the module.
52
-
53
- Prefixes are **sourced from pubid** — the source of truth for each SDO's
54
- identifier grammar (per @ronaldtse's review on relaton-db#103). A processor sets
55
- `@pubid_flavor` to its Pubid module name (e.g. `:Iso`) in `initialize`, and
56
- `Core::Processor#prefixes` lazily reads `Pubid::<Flavor>.prefixes`
57
- (`require "pubid"` on first call, memoized). pubid returns each SDO's leading
58
- identifier tokens, including non-obvious ones (BSI `DD`, NIST `FIPS`) and joint
59
- forms listed symmetrically across co-publishers (`ISO/IEC` in both ISO's and
60
- IEC's lists). Flavors with no `@pubid_flavor` fall back to `[prefix]`. The pubid
61
- API (`Pubid.prefixes(flavor)` / `Pubid.prefix_flavors` / the `PrefixesSupport`
62
- mixin) was specified in `HANDOFFS/metanorma__pubid.md` and implemented upstream;
63
- relaton pins the pubid version that carries it.
64
-
65
- Conflicting prefixes where the *same* base identifier is co-published as distinct
66
- per-publisher documents (e.g. `IEC 80000-13` vs `ISO 80000-3` — different cover,
67
- location, even foreword) are a **separate follow-up**: the register already
68
- returns both flavors, but returning the correct publisher-specific *content*
69
- touches `Db#fetch`, not the prefix list.
70
-
71
- Scope note: only *prefixes* live here. Broader "SDO/organization metadata" (org
72
- names — relaton-db#132; logos — metanorma#346) was deliberately kept **out** of
73
- relaton proper, destined for a separate store/gem; don't add it to the processors.
74
-
75
- ### Db (lib/relaton/db.rb) — Main Public API
76
-
77
- `Relaton::Db#fetch(ref, year, opts)` is the primary entry point. It:
78
- 1. Routes the reference with `Registry#route` (see **Routing** below)
79
- 2. Handles combined references (`+` for derivedFrom, `,` for amendments) in `combine_doc`
80
- 3. Delegates to `check_bibliocache` which manages the dual-cache lookup and network fetch flow
81
-
82
- ### Routing (`Registry#route`, relaton#205)
83
-
84
- One rule for `fetch`, `fetch_db`, `fetch_async`, `fetch_std` and `docid_type`.
85
- It returns `[stdclass, pubid]`:
86
-
87
- 1. **Parse with pubid, and keep only the parsing flavor's own exact parse.**
88
- `Pubid.parse` falls back to a partial parse by any flavor (`ATN5014` → `IEC
89
- ATN5014`), and a permissive grammar reads another publisher's string exactly
90
- (`ISO REF` as IEC, `ABC 123456` as GB, a bare DOI as UN). A parse counts only
91
- when it renders the reference back (`to_s == reference`, pubid's own
92
- round-trip test) **and** the flavor claims the reference by one of its
93
- `Pubid::<Flavor>.prefixes` (`prefix_of?`: the reference ends there or goes on
94
- with a separator; a prefix that ends in punctuation, `doi:`, needs none).
95
- A flavor whose identifiers print without a publisher token opts out of the
96
- claim with `Core::Processor#bare_identifiers?`: OGC (`19-025r1`, the form
97
- its documents carry) and 3GPP (`TS 23.207:REL-18/18.0.0`, its index form).
98
- Without the opt-in both raised, and `TR 00.01U:UMTS/3.0.0` went to JIS by
99
- the `TR` regex. A URN no flavor owns (`urn:foo:bar`) makes `Pubid.parse`
100
- raise a plain `ArgumentError`, which counts as no parse.
101
- 2. **Route by class ancestry** (`processor_by_pubid`), so the registration order
102
- does not matter (a spec routes with the processors reversed).
103
- 3. **A co-published identifier: the printed form decides.** There is no
104
- canonical form of it (pubid#469), and `Pubid.parse` tries the owners of a
105
- joint prefix alphabetically, so `ISO/IEC 27001` parses as IEC. A reference
106
- that starts with a prefix several flavors own (`joint_prefixes`: `ISO/IEC`,
107
- `IEC/ISO`, `ISO/IEC/IEEE`) routes to the owner whose own prefix is the first `/` token:
108
- `ISO/IEC …` → ISO, `IEC/ISO …` → IEC. The parsed pubid is kept only when it
109
- is that flavor's. This is **relaton's routing policy, not a pubid gap**: the
110
- same string `ISO/IEC 27001:2022` is in both the ISO and the IEC catalogs, and
111
- pubid must not elect one reading of it nor route records (pubid#469). The
112
- #205 comment puts the choice here — the printed form first, then the
113
- co-publisher's flavor when the first one has no record (**Co-publisher
114
- fall-through**, below).
115
- 4. **Else the prefix regex** (`class_by_ref`): combined references, the
116
- `PREFIX(code)` wrapper, `IEV`, URNs, and spellings a flavor normalizes on
117
- render (`OGC 19-025r1`, `UN TRADE/…`, `3GPP TS …`, `IEEE 802.11-2016`).
118
- 5. **Else `Relaton::UnknownReferenceError`** (`< Relaton::Error`).
119
- `fetch_async` logs it and yields nil; `docid_type` returns `[nil, code]`;
120
- relaton-cli logs it and returns no document.
121
-
122
- **One parse per fetch (#205 PR 2).** `Db#query_and_key` turns the routed pubid
123
- into the **query pubid** once (`Core::Processor#query_pubid(ref, opts,
124
- parsed)`: the routed parse, else `#cache_pubid`, kept only when it is a
125
- `Pubid::Identifier`), keys the cache with it (`#cache_key(ref, year, opts,
126
- query)`, which folds year/all_parts on **copies**), and hands that same object
127
- to the flavor: `processor.get(query || code, year, opts)`. So
128
- `db.fetch("ISO 19115-1")` is one grammar parse, and `db_spec.rb` counts it.
129
- The year and `all_parts` still reach `get` as arguments; the pubid is unfolded.
130
- A nil query pubid (no pubid class, a policy miss such as `IEV`, a CCSDS
131
- format) sends the String and leaves the query uncached. `fetch_api` keeps the
132
- String for its URL. If `urn_to_code` rewrote the reference, the routed parse is
133
- of another string, and `query_pubid` parses the code. IEC's `urn_to_code` acts
134
- on a `urn:` only: `Relaton::Iec.urn_to_code` splits on `:`, and it used to
135
- rewrite `IEC 60034-1:1969+AMD1:1977+AMD2:1979+AMD3:1980 CSV` into
136
- `1979+AMD3 1980 CSV`.
137
-
138
- **Co-publisher fall-through (#205 PR 3).** A not-found from the lead flavor is
139
- not the last word for a co-published document. The organizations serve the
140
- joint portfolio by their own policies (IEC serves ISO-led documents, ISO serves
141
- IEC-portfolio ones; the IEC index fixture holds ~7,450 `ISO/IEC` rows), so
142
- `Db#fetch` (and `fetch_db`, `fetch_async`, and a `fetch_std` that routes) then
143
- asks the co-publishers' flavors (`Db#copublisher_fetch`):
144
-
145
- - **Order: the parsed pubid's, lead first.** `Core::Processor#copublishers(pubid)`
146
- reads the publishers after the lead from the query pubid — `copublishers` for
147
- ISO and IEC, IEEE's `publishers`/`copublisher` through its override. pubid#472
148
- made the order the printed one. `Registry#class_by_publisher` maps each to the
149
- flavor whose own prefix it is (`ASTM`, `IDF` have none, and are skipped).
150
- - **Each co-publisher uses its own flavor end to end**: its own parse of the
151
- String (`ISO/IEC 27001` reaches IEC as a `Pubid::Iec` pubid), its own cache
152
- key and bucket, its own `not_found` row. A second fetch reads ISO's
153
- `not_found` and IEC's cached document with no network call. No row crosses
154
- flavors, because `bib_retval` must read XML with the flavor that wrote it.
155
- - **A co-publisher whose flavor gives no pubid for the lead's form is
156
- skipped**, with a log line: a parse error (`IEEE/ISO 11073-10101` in ISO or
157
- IEC), or a flavor that reads the form as a miss (IEEE's `cache_pubid` gives
158
- nil for `ISO/IEC/IEEE 15288:2023/DAmd 1`). Without that rule IEEE would get
159
- the String with no cache key, so every fetch would call it again. There is
160
- no re-render into the co-publisher's arrangement: pubid#469 elects no
161
- reading, and #472's "ISO face" keeps the printed publisher order. **Only the
162
- probe parse is rescued** (`Db#copublisher_query`): a parse error from inside
163
- the co-publisher's `get` is a flavor or data bug, and it propagates.
164
- - The co-publishers get the reference **as the lead read it**
165
- (`strip_id_wrapper`: no `ISO(…)` wrapper, no en dash). Without that, a
166
- wrapped or en-dash reference raised after the lead's miss.
167
- - **Not a fall-through:** a lead hit; a `Relaton::RequestError` (a transport
168
- failure is not "not found", and it propagates); a flavor the caller names in
169
- `fetch_std`; each part of a combined reference (`combine_doc` calls
170
- `check_bibliocache` directly); a URN, which reaches the co-publisher as the
171
- URN String that its flavor cannot parse, so it is skipped.
172
- - **Known cost of the per-flavor rows:** a repeat fetch logs the lead's
173
- "not found in cache" before IEC's cached document answers, and when the lead
174
- later gains the record, the co-publisher's copy answers until the lead's
175
- `not_found` row expires (60 days) or the caller passes `no_cache`.
176
- - On the pinned pubid (`27454393`, pre-#472) an IEEE joint development lists
177
- its `publishers` in a reordered form (`IEEE/ISO/IEC 8802-3` → ISO, IEC, IEEE).
178
- It does not change an answer today: ISO and IEC cannot parse an IEEE-led form,
179
- so both are skipped.
180
-
181
- **The flavor contract for `get`** (every flavor in this gem keeps it):
182
-
183
- - **Accept a String or the flavor's pubid.** Parse only a String; a String-only
184
- rewrite (`IEV`, `upcase`, an en dash, a `BIPM ` prefix) stays in the String
185
- branch. Unwrap an `AllParts` pubid the way the String `(all parts)` is read.
186
- - **Never mutate the pubid.** It is also the cache key, and `to_all_parts`
187
- wraps the same object. Copy before a setter (`exclude`, or
188
- `pubid.class.from_hash(pubid.to_hash)`); ISO `root.date=`, IEC `date=` and
189
- JIS `year=` used to write into the caller's object.
190
- - **Use `ref.to_s` for text** — log keys (`Util.info …, key: ref.to_s`; the
191
- JSON log formatter would serialize a pubid), portal search text, messages.
192
- Routing's own parse prints back as the reference. A pubid from
193
- `#cache_pubid` (prefix-fallback routing, a named `fetch_std`, a
194
- `combine_doc` piece, a `urn_to_code` rewrite) can print a **normalized**
195
- form: `NISTIR 8200` → `NIST IR 8200`, `NIST SP 800-38A Add` → `… Add.`,
196
- `CIPM Meeting 43` → `CIPM 43rd Meeting`, `IEEE 528` → `IEEE Std 528`,
197
- `… Expert commentary` → `… Expert Commentary`, and OGC and 3GPP print with no
198
- publisher token. So a text lookup must accept the normalized print: BSI's
199
- `ExComm` rewrite is case-insensitive for this reason, and 3GPP logs with
200
- `to_s(with_publisher: true)`.
201
- - `spec/relaton/support/umbrella.rb` has the `pubid_of("ISO 19115-1")` matcher
202
- for a stubbed `get`'s first argument.
203
-
204
- `Relaton::Db#fetch_all(text, edition, year)` searches cached entries, filtering by text content (via `match_xml_text?`), edition, and/or year. Returns an array of deserialized bibliographic items from both local and global caches.
205
-
206
- The dual-cache strategy uses a **global cache** (`~/.relaton/cache`) and an optional **local cache** (project-level). `check_bibliocache` checks local first, falls back to global, and syncs between them.
207
-
208
- ### Cache (lib/relaton/db/cache.rb) — pubid-keyed index on lutaml-store
209
-
210
- relaton#189 item 2 / relaton#204. Two lutaml-store `FileSystem` stores under
211
- `<dir>/v2/`:
212
-
213
- - **`rows/`** — the index. One store key per **bucket**, `<flavor>/<root number>`
214
- (`iso/19115`), whose value is the list of the bucket's `CacheEntry` rows
215
- (`lib/relaton/db/cache_entry.rb`): `id` (the pubid's `to_hash`) or `key` (a
216
- legacy string), `status` (`doc` / `not_found`), `file`, `fetched`. A lookup
217
- reads one small bucket; a write is one atomic `update` of it. A `_versions`
218
- key holds each flavor's `grammar_hash`; a changed one drops that flavor's
219
- rows and documents on open.
220
- - **`docs/`** — the XML documents. **Several rows can point to one document.**
221
- A fetch whose answer has another identifier than the query
222
- (`ISO 19115-1` → `ISO 19115-1:2014`) writes the query row and the item row,
223
- both naming the same file. There are no redirect entries. A document is
224
- deleted with the last row that points to it; a row whose document is gone is
225
- a miss.
226
- - **The item row's key must tell editions apart (relaton#408).** `Db#item_key`
227
- parses the primary docidentifier and passes it through
228
- `Core::Processor#item_cache_key(pubid, item)`. A flavor whose printed docid
229
- omits the edition by design overrides the hook to add it from the item:
230
- IHO adds `version` from `edition` (`IHO B-12` → `IHO B-12 3.0.0`), ECMA adds
231
- `edition` and the `volume` extent (`ECMA-262` → `ECMA-262 ed16`), through
232
- `#with_components`, which fills only components the parse leaves nil, on a
233
- copy. Without it every edition got one item key and one file, and a fetch of
234
- `IHO B-12 2.0.3` overwrote the file the `IHO B-12 3.0.0` query row named. A
235
- cache with such a file is dropped by the next `Relaton::VERSION` bump
236
- (`grammar_hash`), so no separate cache-version bump was made.
237
-
238
- Matching: the **exact** row first (canonical `to_hash`, not pubid `==`, which
239
- is false after a `from_hash` round trip for some flavors, e.g. UN). Only a
240
- **dated** query then falls back to `query === row_id` in the same bucket —
241
- an undated query keeps its own row, so the 60-day expiry of undated entries
242
- still applies and it never answers with another query's year-stripped copy.
243
- **`===` alone is too wide here**: it reads an omitted `part` as "any part", so
244
- `ISO 19115:2003 === ISO 19115-1:2003` (and the same for IEC, BSI, JIS, GB).
245
- `Cache#subset_of?` therefore also requires that every component the row adds
246
- (at any depth) is allowed: only a language for the dated fallback
247
- (`LANGUAGE_COMPONENTS`), a year/date/month, an edition/version (the
248
- `item_cache_key` components; not a volume, which names another document) or a
249
- language for the date-range `candidates` (`EDITION_COMPONENTS`). A row with no `root.number` (DOI, ISBN)
250
- goes to one of 256 digest buckets (`<flavor>/~<hex>`).
251
-
252
- Expiry (`Cache#valid_row?`): an undated row, and **every** `not_found` row,
253
- is valid for 60 days (`UNDATED_TTL`) from its `fetched` date; only a `doc`
254
- row looked up with a `year` argument never expires (`fetch("ISO 9999:2030")`
255
- passes no `year`, so its rows expire too). A `not_found` for a dated query
256
- used to stay for ever (`year || age < 60`), so a document published after
257
- the first miss was never fetched (#204).
258
-
259
- Locking and atomic writes come from lutaml-store (≥ 0.3.2 — 0.3.0's
260
- FileSystem adapter was unsafe, lutaml/lutaml-store#17): an exclusive `flock`
261
- on `<root>/.lock` plus a per-root Monitor for `update`/`transaction`, and temp
262
- file + rename per write. `Cache#store` writes the document and both rows in
263
- one `rows` transaction; the lock order is always `rows`, then `docs`.
264
- Two specs check this with the real lock, in two processes (#204):
265
- `db_cache_spec.rb` ("keeps every row when two processes write") and
266
- `db_spec.rb` ("keeps one consistent cache when two processes fetch", the whole
267
- `Db#fetch` path). They start the processes with `Process.spawn`
268
- (`run_children`, `spec/relaton/support/child_processes.rb`), not `fork`, so
269
- they also run on Windows, where production has two separate commands that
270
- share `~/.relaton/cache`. A child is a fresh Ruby: RSpec stubs do not reach
271
- it, so its script replaces what it needs (the `Db` spec redefines ISO's
272
- `Bibliography.get`). A spawned child loads its code for seconds and works
273
- for less than one, so each script calls `start_barrier!` after its requires,
274
- and the children start their work together. Both specs also put every row in
275
- **one bucket** on purpose: two processes that write the same rows, or
276
- different buckets, hide a lost update (the `Cache` spec used 50 buckets
277
- before, and passed with no lock). Measured with the `flock` made a no-op
278
- (loaded into the children through `RUBYOPT`): the `Cache` spec failed 5 of 5
279
- runs and the `Db` spec 4 of 5; with a sleep added inside `update` too, both
280
- failed 3 of 3. With the real lock they passed 20 of 20.
281
-
282
- An old file-per-key cache (anything in `<dir>` beside `v2/`) is **moved** to
283
- `<dir>-v1.bak` on open, never deleted — **once**: when the `.bak` exists the
284
- old entries stay where they are, because an older relaton sharing the
285
- directory writes the old layout again, and v2 ignores it.
286
-
287
- Keys: a parsed pubid; a wrapped string (`ISO(ISO 19115-1)`), which the cache
288
- parses through the prefix's processor — `load_entry`/`save_entry` still take
289
- it; or a plain string no processor owns (bucket `_key/<string>`).
290
-
291
- Values: `Db` and `Cache` pass the document XML (`String`) or a
292
- `Relaton::Db::NotFound` (`Data` with `fetched`, in `cache_entry.rb`), never a
293
- marker string. `Db` reads with `Cache#read` and tests `is_a?(NotFound)`; keep
294
- `not_found` regexes out of `db.rb` (`db_spec.rb` guards it). The string API is
295
- kept for compatibility: `Cache#[]` and `Db#load_entry` return
296
- `"not_found <date>"` (`NotFound#to_s`), and `Cache#[]=`/`#store` still accept
297
- that string, which `Cache#coerce` turns into a `NotFound` at the entrance.
298
-
299
- ### WorkersPool (lib/relaton/db/workers_pool.rb)
300
-
301
- Thread pool for `fetch_async`. Default 10 threads per processor, overridable via `RELATON_FETCH_PARALLEL` env var.
302
-
303
- ### Cache key
304
-
305
- `Db#query_and_key` asks the processor: `Core::Processor#cache_key(ref, year,
306
- opts, query)` takes the query pubid (above; else it parses the reference **as
307
- written** with the routed flavor's pubid class, `#cache_pubid` →
308
- `pubid_class.parse`), then folds `year`
309
- (`#fold_year`) and `all_parts` (`to_all_parts`) into the pubid.
310
- `#pubid_class` reads `@pubid_identifier`, else `@pubid_flavor` —
311
- `@pubid_identifier` exists so a flavor can key its cache with pubid without
312
- also sourcing `#prefixes` from pubid.
313
-
314
- - **Canonical references only.** Relaton does not rewrite a non-canonical
315
- citation before it parses it: `I-D.draft-…`, `IETF RFC 8341`,
316
- `BIPM Metrologia …`, a W3C URL, `NIST … (IPD)` / `(January 2014)` /
317
- `NISTIR 8200:2018`, `JIS … (規格群)`, a lowercase `iec …`, an en dash — each
318
- gives no key and reaches the flavor's `get` as written, which normalizes it
319
- itself where it can (relaton#235). The canonical identifiers the data
320
- repos publish all parse as written (measured over every spec index fixture);
321
- pubid/pubid#463 and #464, which asked pubid to accept the other spellings,
322
- were closed for this.
323
- - **References with one key must get one answer from the flavor.** pubid can
324
- read several spellings as one identifier (`doi:…`, `DOI:…` and a
325
- `https://doi.org/…` URL are one `Pubid::Doi` key). If the flavor's `get`
326
- answers only some of them, a miss on another spelling is cached as
327
- `not_found` under the shared key and poisons the canonical one for 60 days.
328
- So `Doi::Crossref.get` strips every such prefix. Check this whenever a
329
- flavor's routing or its pubid grammar changes.
330
- - **Generated references must be canonical too.** `combine_doc` joins an ITU
331
- or NIST supplement with a space (`NIST SP 800-38A Add`), since pubid does not
332
- parse `NIST SP 800-38A/Add`.
333
- - **A processor overrides `cache_pubid` only for policy**, not to normalize:
334
- Adobe, IANA, IEEE and ISBN read an unparseable reference as a miss, so their
335
- parse error means "no key"; IEC answers `IEV` with the vocabulary, so `IEV`
336
- gets no key.
337
- - **A flavor parse error is a fallback, not a failure**: the query is not
338
- keyed, and the flavor's `get` receives the reference as written
339
- (`Core::Processor#cache_pubid` rescues to nil, with a log line; root
340
- `CLAUDE.md`, "A query reference the flavor's grammar cannot read falls
341
- back to the flavor as written").
342
- - **The year goes where the flavor's `get` puts it.** The default
343
- `#fold_year` sets the identifier's own year; ISO, CEN and BSI apply a year
344
- to `#root` (the base document of a supplement, the adopted document of an
345
- adoption), so their processors use `#fold_year_on_root`. A flavor with no
346
- year component drops it, which matches their `get` (it ignores the year).
347
- - **No key, no cache.** A processor with a pubid class that answers `nil` is
348
- not cached for that query: a flavor miss by the flavor's own rule (Adobe,
349
- IANA, IEEE, an incorrect ISBN, `IEV`) and a query whose answer the cache
350
- cannot hold (a CCSDS format, which filters the item's sources — CCSDS
351
- overrides `query_pubid`, so its `get` also receives the String; an OGC year,
352
- which is not the pubid's document-number year — OGC keeps that rule in
353
- `cache_key`, so its `get` still receives the pubid).
354
- - **The publication date range is never in the key.** It selects among the
355
- cached editions (`Cache#candidates` + `pub_date_in_range?`); on a miss the
356
- flavor is asked with the range and the answer is cached under **its own**
357
- identifier only, so the undated query row keeps pointing to the latest
358
- edition. Bounds may be `YYYY`, `YYYY-MM` or `YYYY-MM-DD`.
359
- - A processor with **no** pubid class (a third-party one) keeps the legacy
360
- string key from `std_id` (`ISO(ISO 19115-1:2014 (all parts))`).
361
-
362
- ## Testing
363
-
364
- - Umbrella (Db) specs live in `spec/relaton/` and run from there (`rake spec:relaton`)
365
- - RSpec with VCR cassettes in `spec/relaton/vcr_cassettes/` for recorded HTTP interactions
366
- - Tests create `testcache`/`testcache2` directories and clean them in `before(:each)`.
367
- Assert cache contents through `Relaton::Db::Cache` (`#[]`, `#rows`, `#all`),
368
- not through file names: the store names its files itself.
369
- - Cache-related tests need `<fetched>` elements in XML for `valid_entry?` to return true
370
- - Integration tests in `spec/relaton/relaton_spec.rb`; unit tests under `spec/relaton/`
371
- - **ISO lookups are stubbed, not cassette-recorded.** Flavor gems (relaton-iso/iec/nist)
372
- fetch a large live `index-v2` and deserialize every id through a pinned pubid build,
373
- so a single drifted id in the live index makes the whole index unparseable and ISO
374
- lookups return `nil`. Umbrella specs therefore stub `Relaton::Iso::Bibliography.get`
375
- (and other flavors' `.get`) to return hand-built `ItemData` — the umbrella's job is to
376
- test `Db` orchestration (`combine_doc`, caching, api fallback), not relaton-iso's index.
377
- Build stub items with the `docidentifier:` key (not `docid:`) so the id survives the
378
- cache XML round-trip. Don't reintroduce a live-index cassette for these.
379
-
380
- ## Style
381
-
382
- - RuboCop config inherits from [Ribose OSS guides](https://github.com/riboseinc/oss-guides), target Ruby 3.3
383
- - Each cache operation is atomic in lutaml-store (threads and processes). `Db`'s
384
- `@semaphore` (Mutex) only serializes its own check-then-act sequences across
385
- the two caches (validity check, clone, fetch-and-store).
@@ -1,105 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project
6
-
7
- relaton-doi is a Ruby gem that fetches bibliographic metadata via DOI identifiers from the Crossref API and converts them into Relaton bibliographic objects. It detects DOI patterns to produce flavor-specific items (NIST, IETF, BIPM, IEEE) or generic `Bib::ItemData`.
8
-
9
- ## Common Commands
10
-
11
- ```bash
12
- bundle exec rake spec # Run all tests (default rake task)
13
- bundle exec rspec spec/relaton/doi/parser_spec.rb # Run a single spec file
14
- bundle exec rspec spec/relaton/doi/parser_spec.rb:224 # Run a single example by line
15
- rubocop # Lint
16
- rubocop -a # Lint with auto-correct
17
- ```
18
-
19
- ## Architecture
20
-
21
- **Namespace:** `Relaton::Doi` (migrated from legacy `RelatonDoi`).
22
-
23
- **Core flow:** `Crossref.get(doi)` → HTTP fetch from api.crossref.org → `Parser.parse(json_hash)` → flavor-specific `ItemData`
24
-
25
- Key classes in `lib/relaton/doi/`:
26
-
27
- - **`Crossref`** — module with `get(doi)` and `get_by_id(id)`. `get` takes a
28
- String or a parsed `Pubid::Doi::Identifier`; `doi_of` reduces either to the
29
- bare `prefix/suffix` the API takes. It strips a `doi:` scheme in any case
30
- and a `doi.org` URL, because `Relaton::Db` keys all of those as one DOI — a
31
- form this method did not read would get a miss cached for the others. Uses **Mechanize** (`Mechanize.new` with a custom `USER_AGENT`). `get_by_id` retries twice, backing off by `x-rate-limit-interval * n`, then raises `Relaton::RequestError`; only a 404 returns nil.
32
- - **`Parser`** — largest file (~827 lines). Converts Crossref JSON hashes to Relaton objects. Factory method `parse(src)` delegates to `create_bibitem` which picks the right ItemData class based on DOI pattern (`/nist/` → `Nist::ItemData`, `/rfc\d+/` → `Ietf::ItemData`, etc.). Contains ~30 `parse_*` helper methods for individual bibliographic fields.
33
- - **`Processor`** — `Relaton::Processor` subclass for the Relaton registry system. Entry point for `get`, `from_xml`, `hash_to_bib`.
34
- - **`Util`** — logging utility, extends `Relaton::Bib::Util` with `PROGNAME = "relaton-doi"`.
35
-
36
- ## Test Setup
37
-
38
- - **RSpec** with `expect` syntax only (monkey patching disabled)
39
- - **VCR** cassettes in `spec/vcr_cassettes/` record Crossref HTTP responses and are never auto-rerecorded (see root `CLAUDE.md`). Because Crossref rate-limits, **re-record this suite alone** — under parallel `rake spec` it answers 429 and the empty body gets recorded as if it were data.
40
- - **XML fixtures** in `spec/fixtures/` — expected output XML files. The `read_fixture` helper auto-substitutes today's date into `<fetched>` tags.
41
- - **canon** gem for XML comparison in integration tests
42
- - Integration tests in `spec/relaton/doi_spec.rb` cover 40+ document types via VCR cassettes
43
- - Unit tests in `spec/relaton/doi/parser_spec.rb` test Parser methods directly with hash inputs
44
-
45
- ## Rate limiting
46
-
47
- Crossref advertises `X-Rate-Limit-Limit: 10` / `X-Rate-Limit-Interval: 1s` and
48
- `X-Concurrency-Limit: 3`. Both request paths honour that and **fail loud rather
49
- than degrade**:
50
-
51
- - `Parser#fetch_crossref` handles `429` *before* its generic 4xx→nil return,
52
- sleeping `retry_delay * attempt` (`Retry-After`, else `X-Rate-Limit-Interval`,
53
- else a 1s floor) with the **sleep itself** clamped to `MAX_RETRY_DELAY`, for up
54
- to `MAX_RETRIES` attempts, then raising `Relaton::RequestError`. Only the
55
- numeric form of `Retry-After` is used — RFC 9110 also permits an HTTP-date, and
56
- scanning digits out of a date would yield the day-of-month.
57
- - `Crossref.get_by_id` retries twice with `Crossref.backoff`, then raises; only a
58
- 404 returns nil.
59
-
60
- Both apply a **1-second floor**, because a throttled response may carry no
61
- rate-limit headers at all (the 429s observed here had only
62
- `Date`/`Content-Length`/`Connection`). Treating a missing header as `0` would
63
- retry with no delay — hammering the endpoint that just asked us to slow down.
64
-
65
- **A throttle must never collapse into "not found."** `fetch_crossref`'s callers
66
- (`#parent_item`, `#fetch_location`) read nil as "this record has no parent /
67
- no location", so a 429 returning nil silently produced an *incomplete* item —
68
- e.g. a book chapter with none of its editors — instead of an error. That is also
69
- how 429s ended up baked into cassettes as if they were data; see the cassette
70
- notes in the root `CLAUDE.md`.
71
-
72
- ## Key Constants in Parser
73
-
74
- - `TYPES` — maps 23 Crossref document types to Relaton types (e.g., `"book-chapter"` → `"inbook"`)
75
- - `REALATION_TYPES` — maps 37 Crossref relation types to Relaton relation types
76
- - `COUNTRIES` — `%w[USA]`, used by `parse_place` to distinguish country vs region
77
-
78
- ## Crossref markup
79
-
80
- Crossref returns abstracts and titles as JATS, and it sometimes encodes that markup as HTML
81
- entities. `Parser#normalize_markup` decodes the entities; `create_title`, `parse_abstract`,
82
- `parse_series`, `included_in_relation` and `create_org` all route through it, so an
83
- affiliation name no longer goes out double escaped (`AT&amp;T` instead of `AT&T`).
84
-
85
- **It decodes only content that holds no markup of its own** (`TAG_RE`), and it passes `nil`
86
- through. Crossref escapes an ampersand and a literal angle bracket *inside* real markup, so
87
- an unconditional decode — which is what upstream relaton-doi#26 does — turns
88
- `<jats:p>Smith &amp; Jones</jats:p>` into markup that no longer parses. The sanitizer then
89
- gives up on it and an unescapable string reaches the output: metanorma-pdfa#99 again, on the
90
- far more common input. A `nil` reaches `create_org` from a funder entry carrying only a DOI
91
- and from an affiliation carrying only a ROR id, and `CGI.unescapeHTML(nil)` raises
92
- `TypeError`, which aborted the whole fetch.
93
-
94
- The JATS also carries a `jats:` prefix on the elements and sometimes an `xlink:` prefix on an
95
- attribute, which Relaton never declares. **Removing those prefixes is not this flavor's job**
96
- — `Relaton::Bib::Sanitizer` runs on every `Bib::Title` and `Bib::Abstract` assignment and
97
- declares each undeclared prefix so the markup parses and maps to the basicdoc set. Upstream
98
- relaton-doi (relaton-doi#26) carries its own `drop_namespaces` only because that sanitizer fix
99
- is not in a released relaton-bib; here both ship in one gem, so this flavor keeps the entity
100
- decode alone. Do not add a second prefix-stripping layer. The unit specs in
101
- `spec/doi/relaton/doi/parser_spec.rb` pin the behaviour, not the layer, so they hold either way.
102
-
103
- `spec/doi/relaton/doi_spec.rb`'s shared `"fetch document"` example asserts
104
- `Nokogiri::XML(xml).errors` is empty. That is the property relaton-render requires
105
- (metanorma-pdfa#99), and it guards every integration example.