relaton 3.1.1 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (67) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/frontend_assets.rb +7 -6
  7. data/lib/relaton/db.rb +31 -14
  8. data/lib/relaton/ecma/bibliography.rb +2 -2
  9. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  10. data/lib/relaton/iala/bibliography.rb +1 -1
  11. data/lib/relaton/iana.rb +1 -1
  12. data/lib/relaton/iec/data_parser.rb +41 -2
  13. data/lib/relaton/itu/README.adoc +1 -1
  14. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  15. data/lib/relaton/oasis/bibliography.rb +1 -1
  16. data/lib/relaton/ogc/hit_collection.rb +1 -1
  17. data/lib/relaton/ogc.rb +1 -1
  18. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  19. data/lib/relaton/plateau/hit_collection.rb +1 -1
  20. data/lib/relaton/sae.rb +1 -1
  21. data/lib/relaton/version.rb +1 -1
  22. data/lib/relaton/w3c/bibliography.rb +1 -1
  23. data/lib/relaton/xsf/bibliography.rb +1 -1
  24. data/lib/relaton/xsf.rb +1 -1
  25. metadata +32 -46
  26. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  27. data/lib/relaton/adobe/CLAUDE.md +0 -104
  28. data/lib/relaton/bib/CLAUDE.md +0 -150
  29. data/lib/relaton/bipm/CLAUDE.md +0 -187
  30. data/lib/relaton/bsi/CLAUDE.md +0 -35
  31. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  32. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  33. data/lib/relaton/cen/CLAUDE.md +0 -219
  34. data/lib/relaton/cie/CLAUDE.md +0 -90
  35. data/lib/relaton/core/CLAUDE.md +0 -140
  36. data/lib/relaton/db/CLAUDE.md +0 -385
  37. data/lib/relaton/doi/CLAUDE.md +0 -105
  38. data/lib/relaton/easc/CLAUDE.md +0 -85
  39. data/lib/relaton/ecma/CLAUDE.md +0 -217
  40. data/lib/relaton/etsi/CLAUDE.md +0 -219
  41. data/lib/relaton/evs/CLAUDE.md +0 -120
  42. data/lib/relaton/gb/CLAUDE.md +0 -47
  43. data/lib/relaton/gost/CLAUDE.md +0 -100
  44. data/lib/relaton/iala/CLAUDE.md +0 -97
  45. data/lib/relaton/iana/CLAUDE.md +0 -105
  46. data/lib/relaton/iec/CLAUDE.md +0 -13
  47. data/lib/relaton/ieee/CLAUDE.md +0 -202
  48. data/lib/relaton/ietf/CLAUDE.md +0 -382
  49. data/lib/relaton/iho/CLAUDE.md +0 -76
  50. data/lib/relaton/index/CLAUDE.md +0 -208
  51. data/lib/relaton/isbn/CLAUDE.md +0 -47
  52. data/lib/relaton/iso/CLAUDE.md +0 -38
  53. data/lib/relaton/itu/CLAUDE.md +0 -685
  54. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  55. data/lib/relaton/jis/CLAUDE.md +0 -64
  56. data/lib/relaton/logger/CLAUDE.md +0 -37
  57. data/lib/relaton/nist/CLAUDE.md +0 -194
  58. data/lib/relaton/oasis/CLAUDE.md +0 -251
  59. data/lib/relaton/ogc/CLAUDE.md +0 -242
  60. data/lib/relaton/oiml/CLAUDE.md +0 -47
  61. data/lib/relaton/omg/CLAUDE.md +0 -137
  62. data/lib/relaton/plateau/CLAUDE.md +0 -126
  63. data/lib/relaton/sae/CLAUDE.md +0 -91
  64. data/lib/relaton/sdo/CLAUDE.md +0 -81
  65. data/lib/relaton/un/CLAUDE.md +0 -91
  66. data/lib/relaton/w3c/CLAUDE.md +0 -369
  67. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,177 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Development
4
-
5
- - `bundle install` — install dependencies
6
- - `bundle exec rake spec:calconnect` — run this flavor's suite
7
- - `bundle exec rubocop` — lint
8
-
9
- The root `Gemfile` git-pins pubid. Pick it up with **`bundle update pubid`**,
10
- never `bundle install` — an already-locked git source does not refloat.
11
-
12
- ## Index
13
-
14
- ### The gem builds and reads `index-v2`; the data repo owns `index-v1`
15
-
16
- `INDEXFILE` is the pubid-backed `index-v2`: rows are
17
- `Pubid::Calconnect::Identifier` hashes
18
- (`_type: pubid:calconnect:standard`, with `series`/`number`/`year`, plus
19
- `month`/`day` for the one fully dated row). The producer
20
- (`DataFetcher#index`) and the consumer (`HitCollection#index`) pass
21
- `pubid_class: ::Pubid::Calconnect::Identifier`. Omitting it on the
22
- **producer** writes v1-shaped rows under a v2 name, silently (`FileIO#save`
23
- calls `to_hash` only for instances of `pubid_class`); omitting it on the
24
- **consumer** leaves the rows raw hashes with `FileIO#sorted` false, so every
25
- lookup scans all 191 rows. `Processor#remove_index_file` passes `url: true` and
26
- `file:` only: the delete never reads the index (see
27
- `lib/relaton/index/CLAUDE.md`).
28
-
29
- **`index-v1` is not this gem's concern at all.** It exists only for released
30
- relaton **v2** clients, and `relaton-data-calconnect` owns it end to end:
31
- `build_index_v1.rb` rebuilds it there from each crawled document's own primary
32
- `docidentifier` string out of `data/` — unlike relaton-data-{ecma,w3c,iana,bipm},
33
- it does **not** derive it from the v2 rows. CalConnect can do that and they
34
- cannot, because a v1 row id is simply the document's printed id, with no edition
35
- or volume to split back out. The upshot for this side is that their v1 never
36
- depends on pubid, so nothing here can break it and there is no v1 fixture and no
37
- v1 assertion in this suite.
38
-
39
- ### Lookup
40
-
41
- `HitCollection#search_index` follows the ETSI/W3C/ECMA idiom:
42
-
43
- - **Pass the pubid, not the string.** `Type#search_candidates` narrows only when
44
- the argument is not a `String`, and a block alone never narrows — so the plain
45
- string this flavor used to pass disabled the binary search however the index
46
- was built. `pubid_class:` alone fixes nothing; both had to change together.
47
- - **Select with the subset match.** `search_index` calls
48
- `index.search(pubid)` with no block, so `Index::Type#search` selects with
49
- pubid's `pubid === row`. A date that the reference omits matches any value.
50
- pubid declares `series` strict for CalConnect (pubid#408): the series keeps
51
- `CC/CD 51016` and `CC/WD 51016` apart, and its *absence* is a value too, so
52
- `CC 36010` does not match `CC/WD 36010`. Measured over the 191-row fixture:
53
- `===` and the old `matches?(…, ignore: %i[year])` agree on all 406 pairs.
54
- - **An unparseable reference finds nothing**, with a warning, and is a miss
55
- rather than an error — it must never surface as a `Relaton::RequestError`.
56
-
57
- **This ended a silently ambiguous substring scan.** The old string search
58
- matched any row *containing* the reference: `CC/DIR 1000` answered with all five
59
- `CC/DIR 1000x` documents and `CC/A 1` with every `CC/A 1xxx`. A number now
60
- matches exactly, and a leading zero is significant (`CC/A 0001` is not
61
- `CC/A 1`).
62
-
63
- **Ordering is newest-first, and it has to be explicit.** The index is sorted by
64
- number, so rows sharing a number arrive in no meaningful order; without
65
- `#recency_key` a bare `CC/S 0601` would answer with an arbitrary one of the 2005
66
- and 2006 documents. Date segments are compared as integers and negated for
67
- descending order, with `row[:file]` breaking ties because the index sort is not
68
- stable.
69
-
70
- `Bibliography.get` no longer regex-splits the reference before searching: pubid
71
- parses `CC/DIR 10005:2019` whole, so a dated reference narrows in the index
72
- itself. The `year` **argument** still needs filtering afterwards, because an
73
- undated reference reaches every year of the document — that is why
74
- `bib_results_filter` stays, and why it now reads the year off `row[:id].date`
75
- instead of scraping a rendered string (the old `/:(\d{4})$/` would raise
76
- `TypeError` against a pubid).
77
-
78
- ### Why this flavor needs no render flags
79
-
80
- CalConnect is the simple end of the pubid migrations, and the reason is worth
81
- stating so nobody adds machinery it does not need. `Pubid::Calconnect` renders
82
- the publisher by default and models no edition and no volume, so the index key
83
- and the document's own printed id are **the same string**: `CC/DIR 10005:2019`
84
- in the row, `CC/DIR 10005:2019` in the record. There is nothing to opt in to
85
- (contrast `Relaton::ThreeGpp::Docidentifier`, which must pass
86
- `with_publisher: true`) and nothing to opt out of (contrast
87
- `Relaton::Ecma::Docidentifier`, which must suppress the edition and the volume).
88
- `DataFetcher#index_id` therefore returns the docidentifier's pubid untouched —
89
- no `dup`, no mutation.
90
-
91
- The number is a plain `String`, not a `Components::Code`, and deliberately so:
92
- it preserves leading zeros (`0001`, `0514`) and keeps a sub-number in one token
93
- (`0812-1`, `0707.1`). CalConnect models **no part** — those are numbers, not a
94
- number plus a part — which is why `Docidentifier#remove_part!` and
95
- `#to_all_parts!` are no-ops. Do not "fix" them by splitting the number.
96
-
97
- `Docidentifier` follows the IHO/W3C shape: `content=` calls `super` first, so
98
- `content` stays the source string verbatim. That is load-bearing beyond
99
- convention — `ItemData#create_id` derives the record's `id` from
100
- `content.gsub(/\W+/, "")`, so a re-render would move every published id.
101
- `#remove_date!` is the one real mutator (`CC/DIR 10005:2019` ->
102
- `CC/DIR 10005`); it writes back through `store_content`, never `content=`,
103
- because a re-parse would rebuild `@pubid` and discard the mutation.
104
-
105
- ### An unparseable docid is recorded, not just warned
106
-
107
- `Docidentifier#parse` logs at `Util.error` and leaves `#pubid` nil — it must not
108
- raise, or an already-published record would stop deserializing.
109
- `DataFetcher#add_to_index` then records
110
- `@errors[docid] = "Unparseable primary id … was not indexed (…)"`, which the
111
- inherited `Core::DataFetcher#report_errors` logs and `Logger::Channels::GhIssue`
112
- turns into a GitHub issue at the end of the crawl. (`report_errors` treats a
113
- **String** value as the message; the boolean form the parsers use means "this
114
- field failed for every record" and is the wrong shape here.)
115
-
116
- The row is skipped rather than indexed unparsed, because `Relaton::Index`
117
- rejects the **whole** index if one row fails to deserialize and its sort calls
118
- `.root.number` on every id. `#write_doc` writes the file **before** indexing, so
119
- the document is unindexed, never lost.
120
-
121
- ### Measured against the live published corpus
122
-
123
- All 191 rows of the published index (2026-09-08): every id parses; `to_s`
124
- reproduces the published string **exactly**; `from_hash(to_hash)` round-trips
125
- all 191; they render **191 distinct keys**, so nothing collapses under
126
- `add_or_update`; `root.number` is empty for none. Driving all 191 documents of
127
- `relaton-data-calconnect`'s `data/` through `write_doc` produces 191 rows with 0
128
- errors and reads back as 191 identifiers with `FileIO#sorted` true.
129
- `spec/calconnect/relaton/calconnect/pubid_contract_spec.rb` pins these
130
- properties offline against the committed `index-v2` fixture.
131
-
132
- No pubid change was needed for this migration. `Pubid::Calconnect` was written
133
- for it.
134
-
135
- ## Testing
136
-
137
- - **Framework:** RSpec with VCR cassettes and WebMock.
138
- - **Index fixture:** `spec/calconnect/fixtures/index-v2.zip` — the whole
139
- published index (191 rows), copied verbatim, so the stored shapes are exactly
140
- what the runtime deserializes. Refresh it with
141
- `bundle exec rake spec:update_index_calconnect`
142
- (`tasks/index_fixture_calconnect.rb`); `#build` refuses a source that is not a
143
- pubid index-v2 rather than writing a fixture `Relaton::Index` would reject
144
- wholesale.
145
- - It is seeded into the `Relaton::Index` pool by
146
- `spec/calconnect/support/webmock.rb`, built **with `pubid_class:`** — without
147
- which the rows stay raw hashes, `FileIO#sorted` stays false, and
148
- `Type#search` silently stops narrowing, so the suite would pass while
149
- exercising something the runtime never does. It is written to a temp file and
150
- read through `FileIO`, not stuffed into `@index` directly, because the
151
- deserialize and the sort both happen on that read path.
152
- - There is **no `index-v1` fixture**, deliberately: `index-v1` is a released
153
- relaton v2 artifact that this gem neither produces nor reads (see the index
154
- section). Don't reintroduce one.
155
- - **The `before(:each)` re-seed is not belt-and-braces.** `Index::Pool#type`
156
- replaces the pooled entry whenever `actual?` says no, and `DataFetcher#index`
157
- asks for the same `:CC` slot with a *different* file (`index-v2.yaml`). With a
158
- `before(:suite)` seed alone the fetcher evicts the fixture, and every later
159
- example goes to the network for real — which is exactly how six examples used
160
- to die with `Errno::EPERM` on `~/.relaton/cc/index-v1.yaml`. The pool key is
161
- `type.upcase.to_sym`, so the producer and the consumer share one slot.
162
- (The ECMA/OGC pattern.)
163
- - **VCR cassettes:** `spec/calconnect/vcr_cassettes/` — index downloads are
164
- ignored by VCR, matched on the constant so a version bump cannot leave the
165
- filter naming the old file and let a real download through.
166
- - `pubid_contract_spec.rb` checks the whole fixture corpus for the properties
167
- the index key depends on: every row deserializes, one distinct rendered key
168
- per row, no empty bsearch key, a `to_hash`/`from_hash` round trip, and every
169
- rendered key parsing back to the same identifier. Re-run it whenever the
170
- fixture is refreshed.
171
- - `hit_collection_spec.rb` covers the runtime search path — narrowing,
172
- ordering, and the misses. It was an empty `describe` block before this
173
- migration, so the lookup had no direct coverage at all.
174
- - The `bibdata`/`bibitem` round-trip examples and two `gets` examples validate
175
- XML with Jing, which needs a **JVM on PATH**. Without one they fail with
176
- `Jing::ExecutionError`, which is environmental, not a code defect — check
177
- `java -version` before chasing it.
@@ -1,74 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Architecture
4
-
5
- **Pubid-backed docidentifier.** `Relaton::Ccsds::Docidentifier`
6
- (`model/docidentifier.rb`, `< Bib::Docidentifier`) parses its `content` into a
7
- `Pubid::Ccsds::Identifier` kept in `@pubid`, while the lutaml `content` attribute
8
- stays a plain string for serialization. It implements the base class's abstract
9
- `remove_part!` / `remove_date!` / `to_all_parts!` by mutating the pubid graph
10
- (walking the `base` chain for supplements/corrigenda) and re-rendering
11
- via `refresh_content!`. Unparseable content falls back to the raw string. It is
12
- wired into `model/item.rb` (`attribute :docidentifier, Docidentifier`) so
13
- `from_xml`/`from_yaml` and `DataParser#parse_docidentifier` yield this class.
14
- `remove_date!` is effectively a no-op today — CCSDS ids carry no date component —
15
- but is implemented for parity with ISO/IEC. `to_all_parts!` drops the part
16
- component (e.g. `CCSDS 121.0-B-3` → `CCSDS 121-B-3`), freezes that rendering
17
- into `@raw_content` (since `content` is live-derived from `@pubid`), then
18
- wraps `@pubid` in pubid's `AllParts`. pubid-ccsds has no dedicated `AllParts`
19
- subclass — unlike pubid-iso — so the wrapper is the generic
20
- `Pubid::AllPartsIdentifier`: `content` stays the plain part-stripped id, but
21
- `#pubid` itself, read directly, now answers `all_parts? == true` and renders
22
- WITH pubid's generic "(all parts)" marker, since it's the wrapper. This
23
- mirrors `lib/relaton/iec/model/docidentifier.rb`; the shared skeleton is
24
- intentionally duplicated per flavor for now (to be hoisted into
25
- `Bib::Docidentifier` once every flavor's id is Pubid-backed).
26
-
27
- **Translation relations use pubid's subset match.**
28
- `Data::Fetcher#search_instance_translation` reads the document's **already
29
- parsed** id from `bib.docidentifier.first.pubid` (not `content`, which it would
30
- re-parse), derives the language-less reference with `pubid.exclude(:language)`,
31
- and picks the branch by `bibid == pubid` — no language means an instance
32
- (`#search_translations`), a language means a translation (`#search_relations`).
33
- This is more correct than the old `TRRGX` string test: `content` returns
34
- `@pubid.to_s`, which reorders components, so a translated corrigendum
35
- (`CCSDS 320.0-B-1-S - German Translated Cor. 1`) put the language mid-string and
36
- the `$`-anchored `TRRGX` missed it; `exclude(:language)` strips the language down
37
- the whole base chain. `TRRGX` still serves `translation_relation_types` and
38
- `DataParser#relation_type`, which match raw relation-id strings.
39
- `#search_relations` and `#search_translations` receive that pubid and select
40
- index rows with `row[:id].exclude(:language) == bibid_pid` — a
41
- language-agnostic match, so a translated row and the instance both match while
42
- another edition of the same number does not. This does **not** use `===`:
43
- pubid declares CCSDS `language` `subset_strict`, so a nil reference language
44
- means "has none" (a wildcard would be wrong for `HitCollection`, where a
45
- language-less user reference must not reach a translation). `exclude(:language)`
46
- drops the language from the row on both sides, mirroring
47
- `HitCollection#rows` (`r[:id].exclude(:edition) == pubid`).
48
-
49
- `HitCollection#rows` keeps `index.search(pubid, exact: true)` for an edition
50
- reference, although `language` is strict now. CCSDS `suffix` is still a nil
51
- wildcard in pubid, so the bare `index.search(pubid)` makes `CCSDS 101.0-B-4`
52
- also return the historical `CCSDS 101.0-B-4-S` (260 such pairs in the index
53
- fixture). The `hit_collection_spec` "historical and a translated sibling"
54
- example guards this. Drop `exact: true` only after pubid declares `suffix`
55
- `subset_strict` too.
56
-
57
- `#search_relations` excludes the document's own row with
58
- `row[:id] == bib.docidentifier.first.pubid` — pubid to pubid. It compared a
59
- `Pubid::Ccsds::Identifier` with a `String` (`.content`) before and was always
60
- false, so a re-crawl over an index that already held the document gave it a
61
- relation to itself. Covered by the `#search_relations` "excludes the document's
62
- own row" example.
63
-
64
- ## Development
65
-
66
- - `bundle install` — install dependencies
67
- - `bundle exec rake spec` — run tests
68
- - `bundle exec rubocop` — lint
69
-
70
- ## Testing
71
-
72
- - **Framework:** RSpec with VCR cassettes and WebMock
73
- - **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-ccsds.
74
- - **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).
@@ -1,219 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-cen retrieves European Committee for Standardization (CEN/CENELEC) bibliographic data using the Relaton model. It has no API or curated index — it **scrapes** the CEN/CENELEC public portal (`standards.cencenelec.eu`) with Mechanize, simulating the search form and parsing HTML tables for title, status, ICS codes, publication dates, and committee info. Uses `Relaton::Bib::ItemData` directly (no relaton-iso dependency).
8
-
9
- ## Development
10
-
11
- ```bash
12
- bundle exec rake spec:cen # the CEN suite
13
- cd spec/cen && bundle exec rspec -I . relaton/cen/docidentifier_spec.rb
14
- ```
15
-
16
- (No gem-local rubocop config; the monorepo root config applies.)
17
-
18
- ## Architecture
19
-
20
- Namespace: `Relaton::Cen`. Reference handling is **entirely `Pubid::CenCenelec`**
21
- — there is no hand-rolled identifier regex left. Because CEN publishes **no
22
- index** and has no data repo, this is the **BSI shape**, not the index-v2 shape:
23
- parse the query with pubid, select the portal hits with pubid, back the
24
- `Docidentifier` with pubid. There is no `INDEXFILE` and no `pubid_class:`.
25
- **Don't add an index.**
26
-
27
- Retrieval flow:
28
-
29
- 1. **Bibliography** (`lib/relaton/cen/bibliography.rb`) — `get(code, year, opts)`
30
- and `search` entry points. `parse` wraps
31
- `Pubid::CenCenelec::Identifier.parse` and **raises** on an unrecognized query
32
- (the root `CLAUDE.md` rule "An unrecognized query reference raises"); only the
33
- data side rescues, in `Hit#pubid`. `get` keeps an early `nil` for an **empty**
34
- reference — an empty string is no reference, not a malformed one — and
35
- otherwise lets `Pubid::Errors::ParseError` propagate. `search` keeps its
36
- `rescue Mechanize::ResponseCodeError, Net::ReadTimeout` by name; don't widen
37
- it to `StandardError`, which would swallow the parse error and turn a
38
- transport failure into a class `Relaton::Db#net_retry` does not retry.
39
- 2. **HitCollection** (`lib/relaton/cen/hit_collection.rb`, extends
40
- `Relaton::Core::HitCollection`) — the web flow: GET the portal, follow the
41
- redirect, submit the search form (`STAND_REF`) via Mechanize, parse the result
42
- table, then `sort` by a pubid-derived key.
43
- 3. **Hit** (`lib/relaton/cen/hit.rb`, extends `Relaton::Core::Hit`) — `item`
44
- lazy-loads the detail page via the scraper; `pubid` memoizes the hit code
45
- parsed with pubid and is **nil** when the grammar cannot read it, because the
46
- portal lists draft revisions such as `prEN 13306 rev` and one such row must
47
- not abort the search. The filter, the sort and the scraper all read it, which
48
- is why it is memoized (the `Relaton::Bsi::Hit#pubid` idiom).
49
- 4. **Scraper** (`lib/relaton/cen/scraper.rb`) — Mechanize/XPath HTML parser.
50
- Maps CEN date abbreviations (DOR→adapted, DAV→issued, DOA→announced,
51
- DOP→published, DOW→obsoleted) and relation headers (supersedes→obsoletes,
52
- normative reference→cites); a small `COMMITTEES` map resolves codes like
53
- TC 459→ECISS. `fetch_structuredid` takes the docnumber and partnumber from
54
- the pubid. Two regexes remain here and are **not** identifiers, so they stay:
55
- the committee regex in `fetch_contributors` and the year-of-a-date-string
56
- match in `fetch_copyright`.
57
- 5. **ItemData / Model::Item** (`item_data.rb`, `model/`) —
58
- `Relaton::Cen::ItemData` extends `Relaton::Bib::ItemData`; `Ext` adds a
59
- `StructuredIdentifier` (docnumber/partnumber, agency "CEN").
60
- 6. **Processor** (`lib/relaton/cen/processor.rb`) — registry integration. Sets
61
- `@pubid_flavor = :CenCenelec`, so `Core::Processor#prefixes` reads
62
- `Pubid::CenCenelec.prefixes` (`EN CEN CLC CWA HD ES CR ENV`) into the global
63
- prefix register; every one of the eight resolves to `[Relaton::Cen]` alone.
64
- `@defaultprefix` is deliberately **narrower** than that list — widening it to
65
- `ES` would claim a very generic token for this flavor.
66
- It lazy-`require_relative`s `../cen` inside each method, like every other
67
- flavor: the require used to sit at file load, so merely building a `Db`
68
- pulled in mechanize, isoics and pubid. `spec/relaton/lazy_loading_spec.rb`
69
- guards that.
70
-
71
- ### `#root` is the accessor that answers for every form
72
-
73
- An `AdoptedEuropeanNorm` (`CEN ISO/TS 21003-7:2019`) carries **no** number, part
74
- or year of its own — they live on the adopted ISO identifier — so
75
- `base_document.year` is nil for it while `root.year` is `"2019"`. Measured over
76
- the 67-reference probe corpus, `root.year` equals the old first-year regex on
77
- **every** parseable reference and `base_document.year` misses all three
78
- adopted-norm forms. So the year, number and part are read from `#root`; only the
79
- **family** grouping is read from `#base_document`, which keeps
80
- `EN ISO 1234` and `CEN ISO/TS 1234` apart.
81
-
82
- ### `Docidentifier` — the year rule
83
-
84
- `Relaton::Cen::Docidentifier` (`model/docidentifier.rb`, `< Bib::Docidentifier`)
85
- parses its `content` into `@pubid` while the lutaml `content` attribute stays a
86
- plain string for serialization. The parse is on the **data side**: it lazily
87
- `require "pubid"` and rescues `LoadError`/`StandardError`, so a non-CEN value (an
88
- ISBN) or a code the grammar rejects stays verbatim with `@pubid` nil, and all
89
- three mutators no-op. `Pubid#exclude` returns a **copy** (the BSI shape, not
90
- IALA's in-place setters), so each mutator replaces `@pubid` and re-renders
91
- through the aliased `store_content` — writing through `content=` would re-parse
92
- and discard the mutation.
93
-
94
- **`remove_date!` drops the LAST year, and only that one.** On a supplement the
95
- identifier names the supplement, so its own year is the last one:
96
- `EN 13250:2000/A1:2005` is amendment A1 of 2005, and the `2000` belongs to the
97
- base document it amends — excluding that would strip the base document's
98
- identity rather than a date. So the rule is `exclude(:supplement_year)` when
99
- there is a supplement year, else `exclude(:year)`:
100
-
101
- | input | `remove_date!` |
102
- |---|---|
103
- | `EN 13306:2017` | `EN 13306` |
104
- | `EN 13250:2000/A1:2005` | `EN 13250:2000/A1` |
105
- | `EN 285:2015+A1:2021` | `EN 285:2015+A1` |
106
- | `EN 61375-2-3:2015/AC:2016-11` | `EN 61375-2-3:2015/AC` |
107
- | `EN 285:2015+A1` | `EN 285+A1` |
108
-
109
- The last two rows are where this beats the old `content.sub!(/:\d{4}$/, "")`:
110
- neither string ends in a bare year, so the regex stripped **nothing** and
111
- `to_most_recent_reference` returned a dated reference. `:supplement_year` is a
112
- CEN-only key, and it resets the supplement's year and month together. Note that
113
- ISO and IEC are different — there `exclude(:year)` removes *every* year.
114
-
115
- `remove_part!` is `exclude(:part, :subpart)`, and `to_all_parts!` does both,
116
- then wraps `@pubid` in pubid's `AllParts` (a generic `Pubid::AllPartsIdentifier`
117
- — CEN has no dedicated `AllParts` subclass). `content` stays the plain
118
- part-and-date-stripped string: **`Pubid::CenCenelec::Renderer` never reads
119
- `all_parts`**, so CEN's own renderer never had a marker, and `content` is
120
- cached from the pre-wrap pubid on purpose. `#pubid` itself, read directly, now
121
- answers `all_parts? == true` and renders WITH pubid's generic "(all parts)"
122
- marker (`#to_s`), since it's the wrapper. BSI and IALA make the same
123
- content/`#pubid` trade-off, for the same reason. pubid holds a sub-part
124
- **inside** `part` (`61375-2-3` gives `"2-3"`), which is also why
125
- `fetch_structuredid` swaps the separator to
126
- reproduce the old `"2:3"` partnumber.
127
-
128
- ### Hit selection — what the query left unsaid
129
-
130
- `search_filter` sends the caller's **raw reference text** to the portal form —
131
- that is search-engine input, not identifier parsing (the BSI precedent) — and
132
- selects with pubid's subset match `query === hit.pubid`. A component that the
133
- query omits matches any value, and `===` reads the omission from the query
134
- itself, nested identifiers included (`CEN ISO/TS 21003-7` keeps its part on the
135
- adopted ISO document). That gives the rules the old comparison had:
136
-
137
- - no part → any part
138
- - no base year → any year
139
- - a supplement that carries **no year** → any supplement year:
140
- `EN 285:2015+A1` → `EN 285:2015+A1:2021`, but never → `EN 285:2015`
141
-
142
- pubid declares the CEN `type`, `stage` and `typed_stage` strict (pubid#408), so
143
- `EN 1325` matches neither the draft `prEN 1325` nor `CEN/TS 1325`. `===`
144
- requires the same class, and pubid documents that a reference never falls back
145
- to the base document of a wrapper, so **a base reference never answers with its
146
- own amendment's record**, and a supplement reference never answers with its
147
- base document. That is this flavor's rule, and it used to need a
148
- `supplement?` guard, because `matches?` enforced it only through the
149
- class-strict `==`, which was pubid's internal invariant rather than a contract.
150
- `spec/cen/relaton/cen/bibliography_spec.rb` still covers it (`never selects a
151
- supplement for a base reference`), so a pubid that relaxed `===` would fail
152
- loudly here.
153
-
154
- Measured over the 61 hit codes the cassettes hold (141 queries: each code and
155
- its forms without year, part and supplement year): `===` and the old
156
- `matches?(…, ignore: ignored_components(query))` agree on all 8,601 pairs.
157
-
158
- `absent?(id, *keys)` (`id.exclude(*keys) == id`) stays for `get`, which appends
159
- the `year` argument to a reference that names no year.
160
-
161
- Two selections got **stricter**, and both were latent bugs in the regex:
162
- `code_to_parts` required a literal `+` for its `amd` group and never compared its
163
- `ac` group at all, so a `/A1:2005` amendment was invisible to it — a query for
164
- `EN 13250:2000` matched the amendment record, and `HD 1215-2:1988` kept its
165
- corrigendum hit and was saved only by the sort order. pubid rejects both.
166
-
167
- ### Sort order
168
-
169
- `HitCollection#sort` is a memoized `sort_by!` over
170
- `[family, part_segments, -year, supplement_key]`, which reproduces the order the
171
- six `code_to_parts` fields gave: the document family ascending
172
- (`base_document.exclude(:year, :part, :subpart)`), then the part ascending with
173
- a part-less hit **first**, then the year **descending** with a year-less hit
174
- last, then the supplement, so a base document precedes its own amendments and
175
- corrigenda. Part segments are compared as **integers**, so `-10` sorts after
176
- `-2`. An unparseable hit sorts last. The order is load-bearing:
177
- `isobib_results_filter` takes the first hit that matches the year and has an
178
- item, so `EN 13306` answers with the 2017 record and `EN 1325` with the
179
- part-less one.
180
-
181
- A consolidated identifier (`EN 285:2015+A1:2021`) holds its base document and
182
- its supplements in `#identifiers` and does not answer `supplement_type`
183
- itself, so `supplements` reads `identifiers.drop(1)` for it and `[id]` for a
184
- plain amendment or corrigendum. The sort key reads the supplement's `number`
185
- and `year`, which every pubid version answers: pubid removes the
186
- `supplement_number`/`supplement_year` aliases.
187
- The `:supplement_year` exclude key is a different thing and stays.
188
-
189
- ## External dependencies
190
-
191
- `mechanize ~> 2.10`, `isoics ~> 0.1` (ICS descriptions), `pubid`,
192
- `relaton-core`, `relaton-bib`.
193
-
194
- ## Testing
195
-
196
- RSpec with WebMock + VCR (cassettes in `spec/vcr_cassettes/`, record `:once`, never auto-rerecorded). Network is blocked via WebMock. No index fixture — CEN is scraped, not indexed; `spec/fixtures/` holds XML/YAML round-trip data. The scraper is inherently brittle to portal HTML changes.
197
-
198
- - `spec/cen/relaton/cen/docidentifier_spec.rb` unit-tests the three mutators over
199
- every identifier form the portal serves, including the two rows above that the
200
- old regex left unchanged.
201
- - `spec/cen/relaton/cen/bibliography_spec.rb` unit-tests hit **selection** with
202
- **no HTTP at all**: `query === hit` is the whole of `search_filter`'s
203
- decision, and driving it directly covers pairs no cassette
204
- can. That matters here — the cassettes hold a detail page only for the
205
- document their own example fetched, so an example that selects a *different*
206
- hit from the same search (say the 2008 record in `cen_iso_ts_21003_7`) dies
207
- with `VCR::Errors::UnhandledHTTPRequestError`, not an assertion failure.
208
- - Every `docidentifier[0].content` expectation in `spec/cen/relaton/cen_spec.rb`
209
- is **unedited** by the pubid migration: what the flavor returns did not change.
210
- - `returns nil when document doesn't exist` is split in two, because
211
- `CEN NOT FOUND` is not an identifier and now raises. One example asserts
212
- `Pubid::Errors::ParseError` and makes no request; the other keeps the
213
- not-found coverage with a parseable reference.
214
- - **Trap:** `gets code` writes `spec/cen/fixtures/bibdata.xml` through
215
- `write_file`, which writes only when the file is **missing**
216
- (`spec/spec_helper.rb`). A deleted fixture is silently recreated from whatever
217
- the current cassette says, and the example then always passes. Never delete it
218
- to make a spec green; delete it only to regenerate deliberately, then read the
219
- diff.
@@ -1,90 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Index format (pubid v2 — structured `_type:` index)
4
-
5
- CIE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`):
6
- each row's `:id` is a `Pubid::Cie::Identifier` object, serialized by `relaton-index`
7
- as a `_type: pubid:cie:…` map (mirrors the NIST/ETSI flavors).
8
-
9
- - **Producer** (`data_fetcher.rb`): `#index` passes `pubid_class:
10
- ::Pubid::Cie::Identifier`; `#write_file` parses each docid via the `#pubid` helper
11
- and indexes the **object** (not the raw string). `#pubid` returns `nil` — the id is
12
- written to disk as a document but **left out of the index** — for any id pubid can't
13
- parse **or** that doesn't round-trip. The guard mirrors the read side's acceptance
14
- test (`Index::FileIO#id_supported?`, `from_hash(to_hash) == to_hash`): the loader
15
- rejects the *whole* index on the first non-round-tripping row, so a bad id must be
16
- dropped at write time rather than poison every lookup.
17
- - **Consumer** (`scrapper.rb`, `processor.rb`): both pass `pubid_class:
18
- ::Pubid::Cie::Identifier` to `Index.find_or_create`. The scrapper parses the ref to a
19
- pubid with `::Pubid::Cie.parse` — an unparseable ref **raises** `Pubid::Errors::ParseError`
20
- (partial refs such as `CIE 001` parse; the old raw-String substring fallback is gone)
21
- and passes **that** to `index.search` so index-v2 narrows candidates by number via
22
- binary search before the block runs (a String arg would force a full O(n) scan — see
23
- `Index::Type#search_candidates`); the block keeps the broad substring match, and
24
- `min_by { |r| r[:id].to_s }` picks the winner (rows are pubid objects, not
25
- `Comparable`, so compare by string form). The scrapper's index comes from the
26
- data repo's Pages machine index (`PAGES_URL`, relaton#206) — manifest + the
27
- shard for the parsed query, the whole-index zip on the same site serving only
28
- the retry on a miss; `ENDPOINT` (the `v2` branch) still serves the document
29
- files.
30
-
31
- ### pubid dependency
32
-
33
- The published `relaton-data-cie/index-v2` uses CIE **proceedings** ids
34
- (`_type: pubid:cie:proceedings` with `paper`/`page_range`) and the **flattened**
35
- `to_hash` (`number`/`year` scalar, no nested `code`). That support first shipped in
36
- the released pubid `2.0.0.pre.alpha.13`, which `relaton.gemspec` pins — an older
37
- pubid strips `paper`/`page_range` and emits the nested shape, so `Relaton::Index`
38
- would reject the whole published index.
39
-
40
- Note the CIE grammar extensions **do** parse the techstreet variant ids like
41
- `CIE x051:2025/zcunvy` (as `Conference` `@variant` ids), so — unlike the old pubid —
42
- they are indexed, not dropped.
43
-
44
- ## Crawler (`data_fetcher.rb`) — parallel detail fetch
45
-
46
- The Techstreet crawl (~1147 docs) is a headless-Chrome (Ferrum) scrape. It runs in
47
- **two phases**:
48
-
49
- - **Phase 1 — `collect_hits` (serial):** page through the search results following
50
- `//a[@class="next_page"]`, collecting every `//li[@data-product]` into one list
51
- (~12 page loads at `per_page=100`).
52
- - **Phase 2 — `process_hits` (parallel):** fan the hits out over a bounded pool.
53
- **Each worker owns its own `BrowserAgent`** (its own Chrome, built via the
54
- `build_agent` factory so the full UA/header + `navigator`/`window.chrome` stealth
55
- masking runs per worker — Cloudflare evasion must hold per worker, not for a shared
56
- singleton). Agents are built up front on the main thread (a Chrome-launch failure
57
- aborts fast instead of hanging the bounded queue) and **every** agent is quit in an
58
- `ensure`. The single memoized `#agent` is used only for Phase 1.
59
-
60
- **Concurrency knob:** `RELATON_CIE_CONCURRENCY` (default **5**, min 1), read by
61
- `.concurrency`. `relaton-data-cie`'s crawler workflow tunes it via env — no code change.
62
-
63
- **Per-worker adaptive pacing (`Pacing`)** replaces the old single global 4 s gap: each
64
- worker starts at `BASE_GAP` (~1 s) and **doubles its own gap up to `MAX_GAP`** only on
65
- trouble — a Cloudflare challenge or a Ferrum/socket error (`RETRIABLE_ERRORS`).
66
- `BrowserAgent#wait_for_challenge` raises a retriable `ChallengeError` when the challenge
67
- doesn't clear within `MAX_CHALLENGE_WAIT`, so a stuck worker backs off and retries
68
- (`#time_req` keeps the 4-try retry) instead of parsing the challenge HTML.
69
-
70
- **Thread-safety & byte-identical output:** the slow `agent.get` runs lock-free; the
71
- cheap build + `#write_file` run under one `@mutex`, so `@files`/`@errors`/`@seen` and the
72
- index mutate single-threaded. Output is order-independent by construction — `@errors`
73
- accumulates with `&&=` (AND, per-doc build is atomic under the lock), `@files` is a Set,
74
- and the structured index is sorted deterministically on save. For a duplicate output
75
- file, a `@seen[file] => catalogue-position` guard (`#superseded?`) keeps the
76
- **last-by-position** content winner the serial crawl would pick; `index.add_or_update`
77
- runs for **every** distinct id *before* that gate, so each id is still indexed. `index.save`
78
- is called **once**, after the pool drains.
79
-
80
- ## Development
81
-
82
- - `bundle install` — install dependencies
83
- - `bundle exec rake spec` — run tests
84
- - `bundle exec rubocop` — lint
85
-
86
- ## Testing
87
-
88
- - **Framework:** RSpec with VCR cassettes and WebMock
89
- - **Index fixture:** `spec/fixtures/index-v2.zip` (the published structured index) is loaded through a `Relaton::Index::Type` with `::Pubid::Cie::Identifier` and injected into the `Relaton::Index` pool in `before(:suite)` (see `spec/support/webmock.rb`, mirroring NIST/ETSI). Run `rake spec:update_index` to refresh from relaton-data-cie.
90
- - **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).