relaton 3.1.0 → 3.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (72) hide show
  1. checksums.yaml +4 -4
  2. data/README.adoc +5 -3
  3. data/lib/relaton/3gpp.rb +1 -1
  4. data/lib/relaton/bib/model/docidentifier.rb +66 -5
  5. data/lib/relaton/calconnect/hit_collection.rb +1 -1
  6. data/lib/relaton/cli/command.rb +6 -4
  7. data/lib/relaton/cli/frontend_assets.rb +7 -6
  8. data/lib/relaton/db.rb +31 -14
  9. data/lib/relaton/ecma/bibliography.rb +2 -2
  10. data/lib/relaton/etsi/data_fetcher.rb +1 -1
  11. data/lib/relaton/iala/bibliography.rb +1 -1
  12. data/lib/relaton/iana.rb +1 -1
  13. data/lib/relaton/iec/bibliography.rb +32 -20
  14. data/lib/relaton/iec/data_parser.rb +41 -2
  15. data/lib/relaton/itu/README.adoc +1 -1
  16. data/lib/relaton/itu/data_crawler_r.rb +1 -1
  17. data/lib/relaton/oasis/bibliography.rb +1 -1
  18. data/lib/relaton/ogc/hit_collection.rb +1 -1
  19. data/lib/relaton/ogc.rb +1 -1
  20. data/lib/relaton/plateau/handbook_parser.rb +31 -2
  21. data/lib/relaton/plateau/hit_collection.rb +1 -1
  22. data/lib/relaton/render.rb +17 -0
  23. data/lib/relaton/sae/data_fetcher.rb +5 -0
  24. data/lib/relaton/sae.rb +1 -1
  25. data/lib/relaton/version.rb +1 -1
  26. data/lib/relaton/w3c/bibliography.rb +1 -1
  27. data/lib/relaton/xsf/bibliography.rb +1 -1
  28. data/lib/relaton/xsf.rb +1 -1
  29. data/lib/relaton-render.rb +4 -10
  30. metadata +33 -46
  31. data/lib/relaton/3gpp/CLAUDE.md +0 -218
  32. data/lib/relaton/adobe/CLAUDE.md +0 -104
  33. data/lib/relaton/bib/CLAUDE.md +0 -150
  34. data/lib/relaton/bipm/CLAUDE.md +0 -187
  35. data/lib/relaton/bsi/CLAUDE.md +0 -35
  36. data/lib/relaton/calconnect/CLAUDE.md +0 -177
  37. data/lib/relaton/ccsds/CLAUDE.md +0 -74
  38. data/lib/relaton/cen/CLAUDE.md +0 -219
  39. data/lib/relaton/cie/CLAUDE.md +0 -90
  40. data/lib/relaton/core/CLAUDE.md +0 -140
  41. data/lib/relaton/db/CLAUDE.md +0 -385
  42. data/lib/relaton/doi/CLAUDE.md +0 -105
  43. data/lib/relaton/easc/CLAUDE.md +0 -85
  44. data/lib/relaton/ecma/CLAUDE.md +0 -217
  45. data/lib/relaton/etsi/CLAUDE.md +0 -219
  46. data/lib/relaton/evs/CLAUDE.md +0 -120
  47. data/lib/relaton/gb/CLAUDE.md +0 -47
  48. data/lib/relaton/gost/CLAUDE.md +0 -100
  49. data/lib/relaton/iala/CLAUDE.md +0 -97
  50. data/lib/relaton/iana/CLAUDE.md +0 -105
  51. data/lib/relaton/iec/CLAUDE.md +0 -13
  52. data/lib/relaton/ieee/CLAUDE.md +0 -202
  53. data/lib/relaton/ietf/CLAUDE.md +0 -382
  54. data/lib/relaton/iho/CLAUDE.md +0 -76
  55. data/lib/relaton/index/CLAUDE.md +0 -208
  56. data/lib/relaton/isbn/CLAUDE.md +0 -47
  57. data/lib/relaton/iso/CLAUDE.md +0 -38
  58. data/lib/relaton/itu/CLAUDE.md +0 -685
  59. data/lib/relaton/jcgm/CLAUDE.md +0 -107
  60. data/lib/relaton/jis/CLAUDE.md +0 -64
  61. data/lib/relaton/logger/CLAUDE.md +0 -37
  62. data/lib/relaton/nist/CLAUDE.md +0 -194
  63. data/lib/relaton/oasis/CLAUDE.md +0 -251
  64. data/lib/relaton/ogc/CLAUDE.md +0 -242
  65. data/lib/relaton/oiml/CLAUDE.md +0 -47
  66. data/lib/relaton/omg/CLAUDE.md +0 -137
  67. data/lib/relaton/plateau/CLAUDE.md +0 -126
  68. data/lib/relaton/sae/CLAUDE.md +0 -89
  69. data/lib/relaton/sdo/CLAUDE.md +0 -81
  70. data/lib/relaton/un/CLAUDE.md +0 -91
  71. data/lib/relaton/w3c/CLAUDE.md +0 -369
  72. data/lib/relaton/xsf/CLAUDE.md +0 -189
@@ -1,85 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with
4
- the EASC flavor. It is a dev doc — excluded from the packaged gem via the
5
- gemspec `files` glob.
6
-
7
- ## What this flavor is
8
-
9
- `Relaton::Easc` retrieves **EASC** (Eurasian Economic Standards Council; the
10
- Cyrillic МГС council) publications using the Relaton model. It
11
- is **index-backed** (no scraping): it searches a pre-built index via
12
- `Relaton::Index` and fetches per-document YAML from the
13
- `relaton/relaton-data-easc` GitHub repo. Reference parsing uses
14
- `Pubid::Easc::Identifier`.
15
-
16
- Two document series, both issued by EASC and catalogued on **mgscatalog.by**:
17
-
18
- - **ПМГ** (`pmg`) — Правила по межгосударственной стандартизации (interstate
19
- standardization *rules*), e.g. `ПМГ 03-2025`, plus the defense variant
20
- `ПМГ В 31-2001`.
21
- - **РМГ** (`rmg`) — Рекомендации по межгосударственной стандартизации
22
- (interstate standardization *recommendations*), e.g. `РМГ 151-2025`.
23
-
24
- ## Architecture
25
-
26
- Namespace: `Relaton::Easc`. Retrieval flow:
27
-
28
- 1. **Bibliography** (`lib/relaton/easc/bibliography.rb`) — `get(code, year)` /
29
- `search`; parses the reference with `Pubid::Easc.parse`, narrows the index by
30
- number, fetches the matching YAML over `Net::HTTP`, and returns
31
- `Item.from_yaml` (an `ItemData`), stamping `fetched`. EASC citations are
32
- always dated (`ПМГ 03-2025`), so — unlike ISO/OIML — there is no
33
- undated/most-recent handling; `get` just returns what `search` resolves.
34
- `ENDPOINT` is the `relaton-data-easc/main` raw base; `INDEXFILE` (`index-v2`)
35
- is defined in `lib/relaton/easc.rb`.
36
- 2. **Index keying** — `Relaton::Index.find_or_create(:easc, url: ...,
37
- pubid_class: ::Pubid::Easc::Identifier)`. `Pubid::Easc` (from
38
- `metanorma/pubid`) parses both the Cyrillic surface forms and the Latin
39
- transliterations (`PMG`/`RMG`/`V`) and renders canonical Cyrillic; its
40
- attributes are `series` (`"PMG"|"RMG"`, Latin-canonical), `variant`
41
- (`"V"|nil` — the «В» defense marker), `number`, `year`, and it serializes to
42
- `_type: pubid:easc:{pmg,rmg}`. `Bibliography#pubid_match?` matches on
43
- series + variant + number (nil-tolerant year), so ПМГ never matches РМГ and
44
- the «В» variant never matches the plain series.
45
- 3. **Item / ItemData / Ext** (`item.rb`, `item_data.rb`, `ext.rb`,
46
- `item_base.rb`, `bibitem.rb`, `bibdata.rb`) — `Item` extends `Bib::Item`
47
- with the typed `Ext` subclass; `Bibitem`/`Bibdata` alias it. `Ext` carries
48
- the EASC-specific fields so they round-trip natively through both XML and
49
- YAML (no per-repo merge hacks): `urn` (`urn:easc:<series>:...`), `webpage`
50
- (mgscatalog.by detail URL), `session` (adopting МГС council session, e.g.
51
- `67МГС`), `developer` (Разработчик), `joining_states` (collection of country
52
- codes, e.g. `%w[АРМ БЕИ КАЗ]`), and `assigned_to` (Закреплен за).
53
- 4. **Docidentifier** (`docidentifier.rb`) — a plain `Bib::Docidentifier`
54
- subclass; the `content` is the canonical Cyrillic citation (`ПМГ 03-2025`).
55
- Subclassed so future `#pubid` integration can hook in without changing the
56
- public interface.
57
- 5. **Doctype** (`doctype.rb`) — `TYPES = %w[pmg rmg]`.
58
- 6. **Processor** (`lib/relaton/easc/processor.rb`) — registry integration;
59
- `@prefix = "EASC"`, `@idtype = "EASC"`, and `@defaultprefix =
60
- %r{^(?:ПМГ|РМГ|PMG|RMG)\b}` routes **both** Cyrillic and Latin series
61
- prefixes here. `get` delegates to `Bibliography.get`. Per the lazy-registry
62
- invariant, each method `require_relative "../easc"` before touching a flavor
63
- constant, including `remove_index_file`.
64
-
65
- There are no scrapers — everything comes from the curated index + GitHub YAML.
66
-
67
- ## Grammar (upstream, not yet present)
68
-
69
- The RelaxNG schemas in the repo-root `grammar/` are **copies of the upstream
70
- source of truth** `metanorma/metanorma-model-iso/grammars/` (authored as `.rnc`,
71
- compiled to `.rng`). There is **no EASC grammar upstream yet**, so this flavor
72
- intentionally ships **no** `grammar/relaton-easc*.rng` and **no** Jing/XML-schema
73
- validation spec. Adding the upstream `relaton-easc.rnc` (extending
74
- `BibDataExtensionType` with the `Ext` elements above, à la the plateau pattern)
75
- is tracked as a cross-project hand-off to metanorma-model-iso; once it lands,
76
- copy the compiled pair into `grammar/` and add the round-trip Jing spec.
77
-
78
- ## Testing
79
-
80
- RSpec + WebMock. `spec/easc/support/webmock.rb` builds an offline `index-v2`
81
- in-memory from `Pubid::Easc` identifiers (so each row round-trips through
82
- `from_hash`/`to_hash` like a published index), preloads it into the
83
- `Relaton::Index` pool in `before(:suite)`, and stubs per-document requests from
84
- `spec/easc/fixtures/data/*.yaml`. No test hits the network. Run with
85
- `bundle exec rake spec:easc`.
@@ -1,217 +0,0 @@
1
- # CLAUDE.md
2
-
3
- ## Development
4
-
5
- - `bundle install` — install dependencies
6
- - `bundle exec rake spec:ecma` — run this flavor's suite (from the repo root)
7
- - The ECMA flavor needs pubid support that is not in the released
8
- `2.0.0.pre.alpha.8`, so the root `Gemfile` git-pins pubid. Pick it up with
9
- **`bundle update pubid`**, never `bundle install` — see the
10
- stale-pubid-lock trap in the root `CLAUDE.md`.
11
-
12
- ## Pubid-backed docidentifier
13
-
14
- `Relaton::Ecma::Docidentifier` parses its `content` into a
15
- `Pubid::Ecma::Identifier` kept in `@pubid`, while the lutaml `content` attribute
16
- stays a plain **string** for serialization. Parsing is **soft**: `content=`
17
- lazily requires pubid and rescues `LoadError`/`StandardError`, so a missing gem
18
- or non-ECMA content leaves `@pubid` nil rather than raising. All three mutators
19
- therefore no-op safely, and `Bib::ItemData`'s `#to_all_parts` /
20
- `#to_most_recent_reference` never raise on ECMA items.
21
-
22
- **The one ECMA-specific rule — `refresh_content!` renders**
23
- **`to_s(with_edition: false, with_volume: false)`.**
24
-
25
- `Pubid::Ecma::Identifier#to_s` renders the edition and the volume **by default**
26
- (`"ECMA-269 ed3 vol2"`). That default is deliberate on the pubid side, because
27
- the index keys on a bare `to_s` — see below. A **document's own** docidentifier
28
- is the opposite: every `ECMA-269` volume file in `relaton-data-ecma` carries
29
- `docidentifier: ECMA-269` and the same title, so the edition and the volume are
30
- index metadata, not part of the printed id. Without the opt-out, any mutation
31
- would silently promote the stored content to the index form.
32
-
33
- This is the mirror image of `Relaton::ThreeGpp::Docidentifier`, whose
34
- `refresh_content!` must pass `with_publisher: true` because *its* pubid defaults
35
- to the index rendering and *its* stored content is the fuller one. The two look
36
- contradictory only until you notice each re-renders what its own `content`
37
- already holds.
38
-
39
- Component mapping:
40
-
41
- - **`remove_date!` → clears `edition`.** ECMA has no date component; `edition`
42
- is its version discriminator. It is invisible in the rendered bare form, but
43
- the identifier really does change and a consumer reading `#pubid` sees it.
44
- - **`remove_part!` → clears `part`/`subpart`.** Real here: `ECMA-418-1` →
45
- `ECMA-418`.
46
- - **`to_all_parts!` → both, then wraps `@pubid` in pubid's `AllParts`.**
47
- `content` stays the plain stripped id — the ECMA renderer has no
48
- "(all parts)" marker, and `content` is cached from the pre-wrap pubid on
49
- purpose. `#pubid` itself, read directly, now answers `all_parts? == true`
50
- and renders WITH pubid's generic marker (`#to_s`), since it's the wrapper.
51
-
52
- ## Index
53
-
54
- ### The gem builds and reads `index-v2`; the data repo derives `index-v1`
55
-
56
- `INDEXFILE` is the pubid-backed `index-v2`: rows are `Pubid::Ecma::Identifier`
57
- hashes (`_type: pubid:ecma:{standard,technical-report,memento}` with
58
- `number`/`part`/`edition`/`volume`). The consumer (`Bibliography#index`) and
59
- the producer (`DataFetcher#index`) pass
60
- `pubid_class: ::Pubid::Ecma::Identifier`. Omitting it on the **producer** writes
61
- v1-shaped rows under a v2 name, silently (`FileIO#save` calls `to_hash` only for
62
- instances of `pubid_class`); omitting it on the **consumer** leaves the rows raw
63
- hashes with `FileIO#sorted` false, so every lookup scans all 804.
64
- `Processor#remove_index_file` passes `url: true` and `file:` only: the delete
65
- never reads the index (see `lib/relaton/index/CLAUDE.md`).
66
-
67
- This gem no longer produces the legacy `index-v1`. `relaton-data-ecma`'s
68
- `crawler.rb` derives it from the v2 rows, the IANA/BIPM/W3C shape.
69
-
70
- **Why the edition must be in the key.** `Index::Type#add_or_update` keys on a
71
- **bare** `id.to_s`, and 740 of the 804 published rows carry an edition. Before
72
- pubid rendered it, those 804 rows collapsed onto 421 keys — `ECMA-74` 22 rows to
73
- 1, `ECMA-262` 18 to 1 — so a crawl would drop **383 of 804 rows and report
74
- success**. Four rows also need `volume`: `ECMA-269` edition 3, volumes 1-4 share
75
- one docidentifier *and* one title, so the volume is the only thing that tells
76
- them apart.
77
-
78
- **`#index_id` builds the identifier from the MODEL, not from a rendered string.**
79
- It takes the docidentifier's own pubid, **duplicates** it, then sets `edition`
80
- from `bib.edition.content` and `volume` from the extent locality — the same
81
- three fields `#filename_id` reads. The `dup` is load-bearing: those two are
82
- index metadata, so setting them on the shared object would promote the
83
- document's own printed id to the index form. Filenames are unaffected either
84
- way; `output_file` is fed by `#filename_id`, giving `ecma-269-3-1.yaml`.
85
-
86
- **An unparseable docid is recorded, not just warned.** `#add_to_index` writes
87
- `@errors[docid] = "Unparseable primary id … was not indexed (…)"`, which the
88
- inherited `report_errors` logs and `Logger::Channels::GhIssue` turns into a
89
- GitHub issue at the end of the crawl. (`Core::DataFetcher#report_errors` treats
90
- a **String** value as the message; the `@errors[:key] &&= …` boolean form the
91
- parsers use means "this field failed for every record" and is the wrong shape
92
- here.) The row is skipped rather than indexed unparsed, because
93
- `Relaton::Index` rejects the **whole** index if one row fails to deserialize and
94
- its sort calls `.root.number` on every id. The data file is still written, so
95
- the document is unindexed, never lost. (The 3GPP/W3C precedent.)
96
-
97
- Verified against the live published corpus: all 804 rows build an identifier,
98
- they render **804 distinct keys** (not 421), `root.number` is empty for none,
99
- `from_hash(to_hash)` round-trips all 804, and the four `ECMA-269` volumes stay
100
- distinct.
101
-
102
- ### Lookup: `best_match`
103
-
104
- `Bibliography#search` follows the ETSI/W3C/OGC idiom:
105
-
106
- - **Pass the pubid, not the string.** `Type#search_candidates` narrows only when
107
- the argument is not a `String`, and a block alone never narrows — so the plain
108
- string this flavor used to pass disabled the binary search however the index
109
- was built. `pubid_class:` alone fixes nothing; both had to change together.
110
- Measured: `ECMA-269` now narrows to 12 of 804 rows.
111
- - **Select with the subset match.** `search` calls `index.search(pubid)` with
112
- no block, so `Index::Type#search` selects with pubid's `pubid === row`. An
113
- `edition` or `volume` that the reference omits matches any value; they are
114
- index metadata that a document's own docidentifier never carries. pubid
115
- declares `part` strict for ECMA (pubid#408), so a bare `ECMA-418` does not
116
- match `ECMA-418-1`. `===` requires the same class, which keeps `ECMA-100`
117
- and `ECMA TR/100` apart. Measured over the 804-row fixture: `===` and the
118
- old `matches?(…, ignore: %i[edition volume])` agree on all 6,492 pairs.
119
- - **An unparseable reference finds nothing**, with a warning. There is
120
- deliberately no substring-scan fallback (OGC has one): an ECMA row renders as
121
- `ECMA-262 ed17`, so a scan would answer a truncated `ECMA-26` with every
122
- ECMA-26x document, and an ambiguous answer is worse than none.
123
-
124
- **A reference must now parse WHOLE.** The old `parse_ref` regex was unanchored
125
- at the end, so trailing text after a valid prefix was ignored and still
126
- resolved; pubid rejects it. Measured:
127
-
128
- | reference | old | new |
129
- |---|---|---|
130
- | `ECMA-6 (draft)` | `ECMA-6` | none |
131
- | `ECMA-6:1991` | `ECMA-6` | none |
132
- | `ECMA-6 2nd edition` | `ECMA-6` | none |
133
- | `ECMA-6 ` (trailing space) | `ECMA-6` | `ECMA-6` |
134
- | ` ECMA-6` (leading space) | none | `ECMA-6` — `parse_ref` strips |
135
-
136
- That is the same trade-off as the missing fallback — a strict parse beats an
137
- ambiguous match — and it is the first thing to check against a report that a
138
- reference "used to resolve and now does not". `ecma-6` (lowercase) parsed under
139
- neither.
140
-
141
- #### Ordering: latest edition, then lowest volume
142
-
143
- That is the order the bespoke `compare_edition_volume` + `min` implemented, kept
144
- deliberately — but the edition is compared **segment-wise as integers**
145
- (`edition_key`), and `r[:file]` breaks the tie because the index sort is not
146
- stable. Comparing the rendered strings made `"9"` beat `"17"`, so 5 of the 421
147
- document families returned an older document than the reference asked for.
148
- Scored against each document's own published date:
149
-
150
- | family | string compare | integer compare | dates |
151
- |---|---|---|---|
152
- | ECMA-262 | ed9 | **ed17** | 2018-06 → 2026-06 |
153
- | ECMA-74 | ed9 | **ed22** | 2005-12 → 2025-12 |
154
- | ECMA-402 | ed9 | **ed13** | → 2026-06 |
155
- | ECMA-328 | ed7 | **ed10** | same direction |
156
- | ECMA-109 | ed9 | **ed11** | same direction |
157
-
158
- The integer key picks the newer document in all 5. An absent edition sorts below
159
- every present one — right for the 64 edition-less rows (mementos, a few
160
- reports), none of which shares a document with an edition-bearing row.
161
-
162
- **`ECMA-262 ed5.1` is the only dotted edition in the whole published corpus.**
163
- It is what forces the segment-wise compare (`5.1` must beat `5`), but since 262's
164
- latest is ed17 the dotted ordering never decides a bare lookup — so it is pinned
165
- by a unit assertion on `edition_key` rather than by an end-to-end example. Do not
166
- "simplify" the key to `to_i`.
167
-
168
- The old `match_ref` also matched on a **prefix** (`/^ECMA[-\s]#{id}/`), so
169
- `ECMA-43` matched `ECMA-430…434`. Four families collided this way (418, 43, 35,
170
- 13); none of the four changed its answer, so exact matching removed a latent
171
- risk rather than moving results.
172
-
173
- #### Verified against the published index
174
-
175
- `relaton-data-ecma` publishes `index-v2.zip`. Loaded through
176
- `Bibliography#index`: **804 rows**, all deserialized to identifiers, sorted,
177
- **0** keying on `""`, 804 distinct keys, and `from_hash(to_hash)` round-trips
178
- every row. Live lookups resolve every shape the flavor handles, each to a file
179
- that returns HTTP 200 — a bare `ECMA-269` → ed9, the space form `ECMA 269` →
180
- the same, `ECMA-269 ed3` → vol1, all four `ed3 vol<N>`, `ECMA-262` → ed17
181
- (2026-06), `ECMA-262 ed5.1`, `ECMA-418` vs `ECMA-418-1`, `ECMA-100` vs
182
- `ECMA TR/100`, `ECMA MEM/2021`, and `ECMA-43` narrowed to number `43` alone.
183
-
184
- ## Testing
185
-
186
- - **Framework:** RSpec with VCR cassettes and WebMock.
187
- - **Index fixture:** `spec/ecma/fixtures/index-v2.zip` — the whole published
188
- index (804 rows), copied verbatim, so the stored shapes are exactly what the
189
- runtime deserializes. Refresh it with `bundle exec rake spec:update_index_ecma`
190
- (`tasks/index_fixture_ecma.rb`); `#build` refuses a source that is not a pubid
191
- index-v2 rather than writing a fixture `Relaton::Index` would reject wholesale.
192
- - It is seeded into the `Relaton::Index` pool by `spec/ecma/support/webmock.rb`,
193
- built **with `pubid_class:`** — without it the rows stay raw hashes,
194
- `FileIO#sorted` stays false, and `Type#search` silently stops narrowing, so the
195
- suite would pass while exercising something the runtime never does. Seeded in
196
- `before(:each)`, not only `before(:suite)`, because `Index::Pool#type` replaces
197
- the pooled entry whenever `actual?` says no and `DataFetcher#index` asks for
198
- the same type with `file:` but no `url:`. With `before(:suite)` alone every
199
- example after the first data-fetcher one searched a producer index, and the
200
- consumer then built a third type and went to the network for real. The pool key
201
- is `type.upcase.to_sym`, so the fetcher's `:ecma` and the consumer's `:ECMA`
202
- share one slot. (The OGC pattern.)
203
- - **VCR cassettes:** `spec/ecma/vcr_cassettes/` — index downloads are ignored by
204
- VCR, matched on `INDEXFILE` so a version bump cannot let a real download
205
- through.
206
- - `spec/ecma/relaton/ecma/index_key_spec.rb` checks the whole fixture corpus for
207
- the properties the index depends on: every row deserializes, one distinct
208
- rendered key per row, no empty bsearch key, sorted, and a `to_hash` round-trip.
209
- - A live check of the flavor cannot use `Bibliography.get` behind an HTTP proxy:
210
- the document fetch goes through **Mechanize**, which does not read
211
- `http_proxy`/`https_proxy` the way `Net::HTTP` does, so it dies with a
212
- connection error on a host `curl` reaches. The index download (`Net::HTTP`)
213
- works, so drive `best_match` and fetch the resolved file with `Net::HTTP`
214
- instead.
215
- - The `bibdata`/`bibitem` round-trip examples validate XML with Jing, which
216
- needs a **JVM on PATH**. Without one they fail with `Jing::ExecutionError`;
217
- that is environmental, not a code defect.
@@ -1,219 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-etsi is a Ruby gem that retrieves ETSI (European Telecommunications Standards Institute) standards metadata using the Relaton bibliographic item model. It fetches data from the ETSI website and the relaton-data-etsi GitHub repository.
8
-
9
- ## Commands
10
-
11
- ```bash
12
- # Install dependencies
13
- bin/setup
14
-
15
- # Run all tests
16
- rake spec
17
-
18
- # Run a specific test file
19
- rspec spec/relaton/etsi/bibliography_spec.rb
20
-
21
- # Run a specific test by line number
22
- rspec spec/relaton/etsi/bibliography_spec.rb:15
23
-
24
- # Run linter
25
- rake rubocop
26
-
27
- # Interactive console
28
- bin/console
29
-
30
- # Install gem locally
31
- bundle exec rake install
32
- ```
33
-
34
- ## Architecture
35
-
36
- The codebase is transitioning from `RelatonEtsi` namespace to `Relaton::Etsi` namespace.
37
-
38
- ### New Architecture (lib/relaton/etsi/)
39
-
40
- Uses Lutaml for serialization:
41
- - `Item` - base bibliographic item inheriting from `Bib::Item`
42
- - `Bibitem` - for `<bibitem>` XML output (includes `Bib::BibitemShared`)
43
- - `Bibdata` - for `<bibdata>` XML output (includes `Bib::BibdataShared`)
44
- - `Ext` - ETSI-specific extension data (marker, frequency, mandate, custom_collection)
45
- - `Doctype` - ETSI document type with abbreviations (EN, ES, GS, TS, TR, etc.)
46
-
47
- ### Legacy Architecture (lib/relaton_etsi/)
48
-
49
- - `BibliographicItem` - extends `RelatonBib::BibliographicItem`
50
- - `XMLParser` - parses XML into bibliographic items
51
- - `HashConverter` - converts hashes to bibliographic items
52
- - `DocumentType` - document type with type/abbreviation mapping
53
-
54
- ### Core Components
55
-
56
- - `Bibliography` - searches and retrieves standards from relaton-data-etsi index
57
- - `DataFetcher` - fetches all documents from ETSI website CSV export
58
- - `DataParser` - parses CSV rows into bibliographic items
59
- - `Processor` - Relaton processor for integration with the relaton ecosystem
60
-
61
- ### Data Flow
62
-
63
- 1. `Bibliography.get(ref)` searches the relaton-data-etsi index
64
- 2. Fetches YAML from GitHub, converts to `Item` using `from_yaml`
65
- 3. `DataFetcher.fetch` pulls CSV from etsi.org, parses with `DataParser`, saves to output folder
66
-
67
- ### Pubid-backed index-v2
68
-
69
- The ETSI index is **pubid-structured** (`index-v2.yaml`/`.zip`): each row's `:id`
70
- is a `Pubid::Etsi::Identifier` serialized to its `_type: pubid:etsi:{etsi-standard,
71
- amendment,corrigendum}` hash (the whole published ETSI corpus round-trips on the
72
- pinned pubid — see the root `CLAUDE.md` pubid-pin note). The rows use the **flat,
73
- compact** shape (`type`/`number`/`version`/`year`/`month` as scalars under
74
- `_type`) that the published `relaton-data-etsi` index carries — the ETSI
75
- `to_hash` flattening (merged from `refactor/flatten-etsi-to-hash`) ships in the
76
- released pubid `2.0.0.pre.alpha.13` that the gemspec pins. The wiring mirrors NIST/JCGM:
77
-
78
- - **Producer** (`DataFetcher`): `index` calls `find_or_create(:etsi, file:
79
- "#{INDEXFILE}.yaml", pubid_class: ::Pubid::Etsi::Identifier)`; `#save` parses the
80
- docid via `#pubid` and stores the **pubid object** (`index.add_or_update pid,
81
- file`) so `Relaton::Index` sorts by id number and serializes each id to its
82
- `_type:` hash on save. `#pubid` returns nil (skipping the whole record) for any
83
- id it can't parse **or** `to_hash`-serialize, so one malformed record can never
84
- abort the crawl or corrupt the index. (No current ETSI record is skipped.)
85
- - **Consumer** (`Bibliography#search`): parses the reference with
86
- `::Pubid::Etsi.parse` and lets a `Pubid::Errors::ParseError` on an unrecognized ref
87
- **propagate** (ISO parity — the CLI renders a friendly message, API callers
88
- rescue it), then `#best_match` selects with pubid's subset match
89
- `pubid === row`. The `pubid` (not a String) lets `Relaton::Index` narrow
90
- candidates by number via binary search. A `version` or `date` that the ref
91
- omits matches any value, so a bare `ETSI GS ZSM 012` matches every edition
92
- and a fully-qualified ref matches only its edition.
93
- `max_by { edition_key(row[:id]) }` returns the most recent.
94
- Requires pubid `main` (partial-ref parsing, `Pubid::Errors::ParseError` on
95
- failure, `subset_strict`).
96
- - **A part-less reference strips the ROW's parts, it does not ask pubid for
97
- "all parts".** pubid declares `parts` strict for ETSI (pubid#408), so a
98
- part-less ref matches no part row on its own. `#comparable` therefore
99
- matches each row without its parts (`exclude(:part, :subpart, :parts)`, a
100
- copy — the cached index id is untouched), and `ETSI EN 300 175` reaches its
101
- 16 part rows. **Do not use `#to_all_parts` here.** Since pubid#433 "all
102
- parts" is a class, and `Pubid::AllPartsIdentifier#===` compares the document
103
- alone: it drops the version and the date as well, so
104
- `ETSI GR ZSM 011 V1.1.1 (2023-02)` answered with V2.1.1 (2024-09) — 586,487
105
- pairs over the fixture. `resolves a fully-qualified part-less reference to
106
- that exact edition` in `bibliography_spec.rb` pins it.
107
- - **A base reference does not reach its supplements.** `===` requires the
108
- same class, so `ETSI ETR 310` does not match the corrigendum
109
- `ETSI ETR 310/C1`, and a corrigendum of the base (`ETS 300 092-1/C1`) does
110
- not match a corrigendum of its amendment (`092-1/A1/C1`). The
111
- `matches?(…, ignore:)` it replaced did match them: over the 28,651-row
112
- fixture, 298 pairs changed, all of them a supplement row that the old match
113
- let in, and 218 of 94,079 references resolved to a newer amendment or
114
- corrigendum instead of the document itself (`ETSI ETR 310` →
115
- `ETR 310/C1 ed.1 (1996-10)`). `bibliography_spec.rb` pins that case.
116
- - **`#edition_key` orders on the parsed version, never on the rendered id.** ETSI
117
- versions are not zero-padded, so a String comparison of `row[:id].to_s` orders
118
- `V9.0.0` above `V19.0.0` and `ed.9` above `ed.11` — it picked the wrong edition
119
- for 983 of the 3007 multi-edition documents in the spec fixture, and a bare
120
- `ETSI TR 155 919` resolved to the 2010 edition. `Pubid::Etsi::Identifier` is
121
- **not** `Comparable` (its `<=>` returns nil), so the key is built from the
122
- components: `id.version.version` holds the bare numbers (`"19.0.0"`, or `"9"`
123
- for the `ed.N` form) and `id.date` renders as `yyyy-mm` for the tie-break.
124
- Both delegate to `base` on a `pubid:etsi:corrigendum`/`amendment` row (162 of
125
- them in the fixture), so every row shape keys the same way. The key ignores
126
- `is_edition`, which puts `ed.11` above `V1.0.0` — safe because **no** document
127
- mixes the two forms (measured: 0 of 28651 fixture rows). Do not put `.to_s`
128
- back.
129
- - **Processor** `#remove_index_file` passes `url: true` and `file:` only. It
130
- needs no `pubid_class:`: the delete never reads the index (see
131
- `lib/relaton/index/CLAUDE.md`).
132
-
133
- ### The crawl query keeps superseded editions (`version=1`)
134
-
135
- `DataFetcher::SOURCEURL` is the ETSI standards-search query, and
136
- `relaton-data-etsi/data/` is **exactly** what it returns — `crawler.rb` rebuilds
137
- `data/` on every run, so a record the query drops cannot be reinstated in the
138
- data repo. The `version` flag selects editions, not statuses: `version=0`
139
- returns only the current edition of each branch, `version=1` returns the
140
- superseded ones as well. The status flags (`withdrawn`, `historical`,
141
- `superseded`, `onApproval`, `isCurrent`) are a **separate axis** and are already
142
- all `1` — that is why the dataset carried `Historical` and `Withdrawn` documents
143
- while still missing older editions (metanorma/metanorma-pdfa#95, which wanted
144
- `ETSI EN 319 142-1 V1.2.1 (2024-01)` back). `version=0` also still returns
145
- parallel current branches, so the response's own `superseded`/`new_versions`
146
- fields, not the major version, are what `version` keys on.
147
-
148
- The flavor uses `version=1`. Measured against the live API on 2026-08-27:
149
-
150
- | | `version=0` | `version=1` |
151
- |---|---|---|
152
- | records | 28625 | 67446 (2.36x) |
153
- | pages at 50/page | 573 | 1349 |
154
- | `data/` on disk | 112 MB | ~264 MB |
155
- | `index-v2.yaml` | 5.0 MB | ~11.8 MB |
156
-
157
- Nothing else in the producer needed changing: the version is part of the ETSI
158
- docid, so `Core::DataFetcher#unique_output_file` gives each edition its own
159
- filename and `Index::Type#add_or_update` (keyed on `id.to_s`) gives each its own
160
- row. `spec/etsi/relaton/etsi/data_fetcher_spec.rb` pins the query flag, and pins
161
- that machinery too — it was already correct, but `version=1` is what makes
162
- several editions of one document a common case rather than a rare one.
163
-
164
- **Ordering rule for the data repo.** This is producer-side only; nothing changes
165
- for users until `relaton-data-etsi` re-crawls. Do **not** deploy a `version=1`
166
- crawl until the released `relaton-etsi` gem picks the newest edition. Its
167
- `Bibliography#search` uses `min_by` on the rendered id, which returns the
168
- **oldest** match, so a bare `ETSI EN 319 401` would resolve to the 2013 edition
169
- across the whole corpus. This flavor picks the newest edition (`#best_match`
170
- orders on `#edition_key`). Also check the crawl against the 6-hour GitHub
171
- Actions job cap — it roughly doubles — and confirm all three EN 319 142-1
172
- editions land in `data/` before merging the re-crawl.
173
-
174
- ### A bad page fails the crawl; it is never skipped
175
-
176
- The relaton-data-etsi crawl of 2026-09-30 (run 36761804954) died at about page
177
- 700 with `JSON::ParserError: unexpected character: 'Array'`: data.php answered
178
- once with PHP's `Array` text in place of JSON. It is transient and server side
179
- — the code had not changed since the good crawl of 2026-09-28, and a full
180
- re-crawl on 2026-10-02 gave valid JSON on all 1,353 pages (67,602 records,
181
- `RowNum` 1..67,602 with no gap). Note also that `curl` gets an HTTP 403 page from
182
- ETSI's web application firewall; Mechanize's user agent gets JSON.
183
-
184
- `DataFetcher#fetch_page` therefore parses the body **inside**
185
- `fetch_with_retry`: a body that is not a JSON array raises `BadPage`, which is
186
- retried like a network error. A page still bad after its retries is deferred
187
- and fetched again after the last page (`DEFERRED_DELAY`); if it is still bad,
188
- the crawl raises `BadPage` with the page number and the first 200 characters of
189
- the body, and `index.save` is never reached. Page 1 is not deferred (it gives
190
- `total_count`). **Do not change this to skip the page:** `crawler.rb` deletes
191
- `data/` before the crawl and commits what the crawl writes, so a skipped page
192
- unpublishes 50 documents, while a failed crawl commits nothing and the previous
193
- data stays live.
194
-
195
- The result set is sorted by deliverable number (`sort=1`), not by date, so a
196
- document that ETSI publishes during the one-hour crawl shifts the later pages
197
- by one. The page count from page 1's `total_count` would then miss the last
198
- record, so `fetch_remaining_pages` reads on past it while the pages stay full.
199
- A record read twice overwrites its own file. Two bounds keep the loop finite
200
- (data.php answers `[]` past the end today, but the loop must not depend on it):
201
- a deferred page extends the range by one page only when it lies inside page
202
- 1's `total_count`, and a crawl still reading full pages `MAX_EXTRA_PAGES` past
203
- that count raises. A deferred page that comes back `[]` inside the range also
204
- fails the crawl. **Known limit:** a document *removed* during the crawl shifts
205
- the later pages the other way, so one record can fall onto a page already read.
206
- Nothing detects that; it is rare, because every status flag is on.
207
-
208
- ## Testing
209
-
210
- - **Index fixture:** `spec/fixtures/index-v2.zip` (pubid `_type:` rows) is loaded
211
- into the `Relaton::Index` pool in `before(:suite)` (`spec/support/webmock.rb`):
212
- the YAML is written to a temp file and read through
213
- `Relaton::Index::Type.new(:etsi, file:, pubid_class: ::Pubid::Etsi::Identifier)`,
214
- and `type.index` forces the offline `pubid_class` deserialize before the net is
215
- blocked; `actual?` is overridden to match only the remote (`url:`) lookup so the
216
- producer-side `find_or_create(:etsi, file:, pubid_class:)` still gets a fresh
217
- instance. Regenerate by parsing `index-v1`'s ids through `Pubid::Etsi` into a
218
- `pubid_class` `Type` and re-zipping (`_type: pubid:etsi:…` rows).
219
- Uses RSpec with VCR for HTTP interaction recording. VCR cassettes are in `spec/vcr_cassettes/`. When tests make new HTTP requests, VCR will record them.
@@ -1,120 +0,0 @@
1
- # CLAUDE.md
2
-
3
- This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4
-
5
- ## Project Overview
6
-
7
- relaton-evs retrieves Estonian standardisation (EVS, Eesti Standardimis- ja
8
- Akrediteerimiskeskus) bibliographic data using the Relaton model. Like CEN/BSI,
9
- it publishes **no index and has no data repo**: the flavor **scrapes** the EVS
10
- e-shop (`www.evs.ee`) with Mechanize. Reference handling is entirely
11
- `Pubid::Evs` (the `:evs` flavor, registered in pubid's registry). EVS records
12
- are dominated by **adoptions**: `EVS-EN …` adopts a European document,
13
- `EVS-EN ISO …` a European adoption of an international one, `EVS-ISO …` /
14
- `EVS-IEC …` adopt international documents directly. The adoption is expressed
15
- with `adoptedFrom` relations parsed from the page's "Base Documents" group.
16
-
17
- ## Development
18
-
19
- ```bash
20
- bundle exec rake spec:evs # the EVS suite
21
- cd spec/evs && bundle exec rspec -I . relaton/evs/docidentifier_spec.rb
22
- ```
23
-
24
- (No gem-local rubocop config; the monorepo root config applies.)
25
-
26
- ## Architecture
27
-
28
- Namespace: `Relaton::Evs`. This is the **BSI/CEN shape**: parse the query with
29
- pubid, select the hit with pubid, back the `Docidentifier` with pubid. No
30
- `INDEXFILE`, no `pubid_class:`, no data repo. **Don't add an index** (unless a
31
- `relaton-data-evs` repo appears — relaton#164 tracks that possibility).
32
-
33
- Retrieval flow:
34
-
35
- 1. **Bibliography** (`lib/relaton/evs/bibliography.rb`) — `get(code, year,
36
- opts)` / `search` / `parse`. `parse` wraps `Pubid::Evs.parse` and **raises**
37
- on an unrecognized query (root CLAUDE.md law); only the data side rescues,
38
- in `Hit#pubid`. `get` keeps an early `nil` for an **empty** reference and
39
- accepts the flavor's own pubid without mutating it (`Relaton::Db` hands it
40
- the parse it routed). Transport errors are rescued **by name**
41
- (`Net::ReadTimeout`, `Net::OpenTimeout` → `Relaton::RequestError`, which
42
- `Db#net_retry` retries).
43
- 2. **HitCollection** (`hit_collection.rb`) — the search is a **direct slug
44
- probe**, not a form submit: the catalogue URL is the printed reference
45
- slugified (`EVS-EN 18216:2026` → `/en/evs-en-18216-2026`), so the product
46
- page exists iff the reference denotes a published document. A miss
47
- **redirects** to `/page-not-found` (HTTP 302, followed by Mechanize) — that
48
- redirect target is the "no hit" signal; a hard 4xx/5xx raises
49
- `Mechanize::ResponseCodeError` and is a miss (`hits` rescues to `[]`).
50
- The `et`/`et-en` language codes probe the `/et/` locale, which serves the
51
- Estonian-language product.
52
- - **The hit's `code` keeps the language code** (only the slug derivation
53
- strips it): the filter `query === hit.pubid` needs the language ON the
54
- hit for a language-carrying query to match its own record.
55
- - A reference **without a year** probes `…-<number>` and misses — the
56
- catalogue URL always carries the year. "Not found." is honest there; do
57
- not add a search-endpoint fallback without evidence the shop renders one
58
- server-side (its search is client-rendered).
59
- - Slug shapes verified against the live catalogue: plain `EVS 945:2023`,
60
- `EVS-EN 16798-1:2019`, `EVS-EN ISO 14001:2026`, the ETSI face
61
- `EVS-EN 301 406-1 V3.2.1:2026`, `EVS-EN IEC 61000-4-30:2026`.
62
- 3. **Hit** (`hit.rb`) — `item` lazy-loads the detail page via the scraper;
63
- `pubid` memoizes the hit code parsed with pubid and is **nil** when the
64
- grammar cannot read it.
65
- 4. **Scraper** (`scraper.rb`) — Mechanize/XPath over the product page:
66
- `h1` → docidentifier, `h2.product-title` → title (`Bib::Title.from_string`
67
- splits " - " into intro/main — it returns an Array, don't re-wrap), the
68
- "Valid from dd.mm.yyyy" badge → `published` date, "Base Documents" →
69
- `adoptedFrom` relations (a double adoption lists both bases in ONE text
70
- node, `ISO 14001:2026; EN ISO 14001:2026` — split on the semicolon), the
71
- scope tab (`div#tab-1`) → abstract, ICS group links → `Ext.ics` via
72
- `Isoics`. Contributor/copyright: the EVS organization (publisher); place
73
- Tallinn.
74
- 5. **ItemData / models** (`item_data.rb`, `model/`) — `Relaton::Evs::ItemData`
75
- extends `Relaton::Bib::ItemData`; `Ext` adds a `StructuredIdentifier`
76
- (docnumber/partnumber from the pubid **root**, agency "EVS"). There is no
77
- `relaton-model-evs` gem, so `Ext` does not override `get_schema_version`
78
- (inherits the nil default, like the other ten model-less flavors). The
79
- **shared ext grammar requires a `doctype`** — `fetch_doctype` always emits
80
- one (`standard`; `technical-specification`/`technical-report` for the
81
- `EVS-TS`/`EVS-TR` faces), or Jing fails on `relaton-evs-compile.rng`.
82
-
83
- ### `#root` is the accessor that answers for every form
84
-
85
- A `NationalAdoption` (`EVS-EN ISO 9001:2015/A1:2024`) holds the origin
86
- standard's number and year behind its `base` chain (base = the CEN/CENELEC
87
- document; `AdoptedEuropeanNorm#adopted` = the ISO document; `#root` recurses),
88
- so the year, number and part are read from `#root`. An `InternationalAdoption`
89
- (`EVS-ISO 9001:2015`) or `EstonianStandard` (`EVS 945:2023`) is its own root.
90
-
91
- ### `Docidentifier` — the year rule
92
-
93
- `Relaton::Evs::Docidentifier` (`model/docidentifier.rb`, `< Bib::Docidentifier`)
94
- parses its `content` into `@pubid` (DATA side: lazy `require "pubid"`, rescues
95
- `LoadError`/`StandardError`; a foreign value stays verbatim with `@pubid` nil).
96
- `Pubid#exclude` returns a **copy**, so each mutator replaces `@pubid` and
97
- re-renders through the aliased `store_content`. **`remove_date!` drops the
98
- LAST year**: on a supplement (`EVS-EN ISO 9001:2015/A1:2024`) the supplement's
99
- year goes first (`exclude(:supplement_year)` → `…/A1`), the base year only
100
- when there is none. `to_all_parts!` wraps in pubid's `AllParts` — `content`
101
- stays the part-and-date-stripped string on purpose; `#pubid` renders the
102
- "(all parts)" marker.
103
-
104
- ### Printed separator is identity
105
-
106
- `EVS EN 18216:2026` (space) and `EVS-EN 18216:2026` (hyphen) parse to the same
107
- class and root, but each renders back **as written** — pubid preserves the
108
- input's separator. Never compare EVS identifiers by `to_s`; compare by parse
109
- (`===`, `root.number`).
110
-
111
- ### Registration
112
-
113
- `Processor` (`processor.rb`) sets `@short = :relaton_evs`, `@prefix = "EVS"`,
114
- `@pubid_flavor = :Evs` (prefixes sourced from `Pubid::Evs.prefixes`) and
115
- `@defaultprefix = %r{^EVS[\s-]}`. It lazy-`require_relative`s `../evs` inside
116
- each method, like every other flavor (`spec/relaton/lazy_loading_spec.rb`
117
- guards that). `lib/relaton.rb` autoloads `:Evs`; `Db::Registry::SUPPORTED_GEMS`
118
- lists `relaton/evs`; `grammar/relaton-evs(-compile).rng` mirrors the CEN pair
119
- (the shared `biblio-standoc` ext already models doctype/flavor/ics/
120
- structuredidentifier, and its `DocRelationType` carries `adoptedFrom`).