relaton 3.1.1 → 3.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +5 -3
- data/lib/relaton/3gpp.rb +1 -1
- data/lib/relaton/bib/converter/csl/from_csl.rb +195 -0
- data/lib/relaton/bib/converter/csl.rb +13 -1
- data/lib/relaton/bib/model/docidentifier.rb +66 -5
- data/lib/relaton/calconnect/hit_collection.rb +1 -1
- data/lib/relaton/cli/command.rb +3 -1
- data/lib/relaton/cli/frontend_assets.rb +7 -6
- data/lib/relaton/cli/subcommand_collection.rb +5 -1
- data/lib/relaton/core/request_error.rb +18 -0
- data/lib/relaton/db/config.rb +9 -1
- data/lib/relaton/db/workers_pool.rb +11 -1
- data/lib/relaton/db.rb +160 -58
- data/lib/relaton/ecma/bibliography.rb +2 -2
- data/lib/relaton/etsi/data_fetcher.rb +16 -6
- data/lib/relaton/iala/bibliography.rb +1 -1
- data/lib/relaton/iana.rb +1 -1
- data/lib/relaton/iec/data_parser.rb +41 -2
- data/lib/relaton/itu/README.adoc +1 -1
- data/lib/relaton/itu/data_crawler_r.rb +1 -1
- data/lib/relaton/jis/README.adoc +2 -0
- data/lib/relaton/jis/data_fetcher.rb +74 -13
- data/lib/relaton/jis/hit_collection.rb +2 -1
- data/lib/relaton/jis/processor.rb +1 -1
- data/lib/relaton/jis/scraper.rb +46 -5
- data/lib/relaton/oasis/bibliography.rb +1 -1
- data/lib/relaton/ogc/hit_collection.rb +1 -1
- data/lib/relaton/ogc.rb +1 -1
- data/lib/relaton/plateau/data_fetcher.rb +10 -1
- data/lib/relaton/plateau/handbook_parser.rb +38 -2
- data/lib/relaton/plateau/hit_collection.rb +20 -5
- data/lib/relaton/sae.rb +1 -1
- data/lib/relaton/version.rb +1 -1
- data/lib/relaton/w3c/bibliography.rb +1 -1
- data/lib/relaton/xsf/bibliography.rb +1 -1
- data/lib/relaton/xsf.rb +1 -1
- metadata +35 -48
- data/lib/relaton/3gpp/CLAUDE.md +0 -218
- data/lib/relaton/adobe/CLAUDE.md +0 -104
- data/lib/relaton/bib/CLAUDE.md +0 -150
- data/lib/relaton/bipm/CLAUDE.md +0 -187
- data/lib/relaton/bsi/CLAUDE.md +0 -35
- data/lib/relaton/calconnect/CLAUDE.md +0 -177
- data/lib/relaton/ccsds/CLAUDE.md +0 -74
- data/lib/relaton/cen/CLAUDE.md +0 -219
- data/lib/relaton/cie/CLAUDE.md +0 -90
- data/lib/relaton/core/CLAUDE.md +0 -140
- data/lib/relaton/db/CLAUDE.md +0 -385
- data/lib/relaton/doi/CLAUDE.md +0 -105
- data/lib/relaton/easc/CLAUDE.md +0 -85
- data/lib/relaton/ecma/CLAUDE.md +0 -217
- data/lib/relaton/etsi/CLAUDE.md +0 -219
- data/lib/relaton/evs/CLAUDE.md +0 -120
- data/lib/relaton/gb/CLAUDE.md +0 -47
- data/lib/relaton/gost/CLAUDE.md +0 -100
- data/lib/relaton/iala/CLAUDE.md +0 -97
- data/lib/relaton/iana/CLAUDE.md +0 -105
- data/lib/relaton/iec/CLAUDE.md +0 -13
- data/lib/relaton/ieee/CLAUDE.md +0 -202
- data/lib/relaton/ietf/CLAUDE.md +0 -382
- data/lib/relaton/iho/CLAUDE.md +0 -76
- data/lib/relaton/index/CLAUDE.md +0 -208
- data/lib/relaton/isbn/CLAUDE.md +0 -47
- data/lib/relaton/iso/CLAUDE.md +0 -38
- data/lib/relaton/itu/CLAUDE.md +0 -685
- data/lib/relaton/jcgm/CLAUDE.md +0 -107
- data/lib/relaton/jis/CLAUDE.md +0 -64
- data/lib/relaton/logger/CLAUDE.md +0 -37
- data/lib/relaton/nist/CLAUDE.md +0 -194
- data/lib/relaton/oasis/CLAUDE.md +0 -251
- data/lib/relaton/ogc/CLAUDE.md +0 -242
- data/lib/relaton/oiml/CLAUDE.md +0 -47
- data/lib/relaton/omg/CLAUDE.md +0 -137
- data/lib/relaton/plateau/CLAUDE.md +0 -126
- data/lib/relaton/sae/CLAUDE.md +0 -91
- data/lib/relaton/sdo/CLAUDE.md +0 -81
- data/lib/relaton/un/CLAUDE.md +0 -91
- data/lib/relaton/w3c/CLAUDE.md +0 -369
- data/lib/relaton/xsf/CLAUDE.md +0 -189
|
@@ -1,177 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Development
|
|
4
|
-
|
|
5
|
-
- `bundle install` — install dependencies
|
|
6
|
-
- `bundle exec rake spec:calconnect` — run this flavor's suite
|
|
7
|
-
- `bundle exec rubocop` — lint
|
|
8
|
-
|
|
9
|
-
The root `Gemfile` git-pins pubid. Pick it up with **`bundle update pubid`**,
|
|
10
|
-
never `bundle install` — an already-locked git source does not refloat.
|
|
11
|
-
|
|
12
|
-
## Index
|
|
13
|
-
|
|
14
|
-
### The gem builds and reads `index-v2`; the data repo owns `index-v1`
|
|
15
|
-
|
|
16
|
-
`INDEXFILE` is the pubid-backed `index-v2`: rows are
|
|
17
|
-
`Pubid::Calconnect::Identifier` hashes
|
|
18
|
-
(`_type: pubid:calconnect:standard`, with `series`/`number`/`year`, plus
|
|
19
|
-
`month`/`day` for the one fully dated row). The producer
|
|
20
|
-
(`DataFetcher#index`) and the consumer (`HitCollection#index`) pass
|
|
21
|
-
`pubid_class: ::Pubid::Calconnect::Identifier`. Omitting it on the
|
|
22
|
-
**producer** writes v1-shaped rows under a v2 name, silently (`FileIO#save`
|
|
23
|
-
calls `to_hash` only for instances of `pubid_class`); omitting it on the
|
|
24
|
-
**consumer** leaves the rows raw hashes with `FileIO#sorted` false, so every
|
|
25
|
-
lookup scans all 191 rows. `Processor#remove_index_file` passes `url: true` and
|
|
26
|
-
`file:` only: the delete never reads the index (see
|
|
27
|
-
`lib/relaton/index/CLAUDE.md`).
|
|
28
|
-
|
|
29
|
-
**`index-v1` is not this gem's concern at all.** It exists only for released
|
|
30
|
-
relaton **v2** clients, and `relaton-data-calconnect` owns it end to end:
|
|
31
|
-
`build_index_v1.rb` rebuilds it there from each crawled document's own primary
|
|
32
|
-
`docidentifier` string out of `data/` — unlike relaton-data-{ecma,w3c,iana,bipm},
|
|
33
|
-
it does **not** derive it from the v2 rows. CalConnect can do that and they
|
|
34
|
-
cannot, because a v1 row id is simply the document's printed id, with no edition
|
|
35
|
-
or volume to split back out. The upshot for this side is that their v1 never
|
|
36
|
-
depends on pubid, so nothing here can break it and there is no v1 fixture and no
|
|
37
|
-
v1 assertion in this suite.
|
|
38
|
-
|
|
39
|
-
### Lookup
|
|
40
|
-
|
|
41
|
-
`HitCollection#search_index` follows the ETSI/W3C/ECMA idiom:
|
|
42
|
-
|
|
43
|
-
- **Pass the pubid, not the string.** `Type#search_candidates` narrows only when
|
|
44
|
-
the argument is not a `String`, and a block alone never narrows — so the plain
|
|
45
|
-
string this flavor used to pass disabled the binary search however the index
|
|
46
|
-
was built. `pubid_class:` alone fixes nothing; both had to change together.
|
|
47
|
-
- **Select with the subset match.** `search_index` calls
|
|
48
|
-
`index.search(pubid)` with no block, so `Index::Type#search` selects with
|
|
49
|
-
pubid's `pubid === row`. A date that the reference omits matches any value.
|
|
50
|
-
pubid declares `series` strict for CalConnect (pubid#408): the series keeps
|
|
51
|
-
`CC/CD 51016` and `CC/WD 51016` apart, and its *absence* is a value too, so
|
|
52
|
-
`CC 36010` does not match `CC/WD 36010`. Measured over the 191-row fixture:
|
|
53
|
-
`===` and the old `matches?(…, ignore: %i[year])` agree on all 406 pairs.
|
|
54
|
-
- **An unparseable reference finds nothing**, with a warning, and is a miss
|
|
55
|
-
rather than an error — it must never surface as a `Relaton::RequestError`.
|
|
56
|
-
|
|
57
|
-
**This ended a silently ambiguous substring scan.** The old string search
|
|
58
|
-
matched any row *containing* the reference: `CC/DIR 1000` answered with all five
|
|
59
|
-
`CC/DIR 1000x` documents and `CC/A 1` with every `CC/A 1xxx`. A number now
|
|
60
|
-
matches exactly, and a leading zero is significant (`CC/A 0001` is not
|
|
61
|
-
`CC/A 1`).
|
|
62
|
-
|
|
63
|
-
**Ordering is newest-first, and it has to be explicit.** The index is sorted by
|
|
64
|
-
number, so rows sharing a number arrive in no meaningful order; without
|
|
65
|
-
`#recency_key` a bare `CC/S 0601` would answer with an arbitrary one of the 2005
|
|
66
|
-
and 2006 documents. Date segments are compared as integers and negated for
|
|
67
|
-
descending order, with `row[:file]` breaking ties because the index sort is not
|
|
68
|
-
stable.
|
|
69
|
-
|
|
70
|
-
`Bibliography.get` no longer regex-splits the reference before searching: pubid
|
|
71
|
-
parses `CC/DIR 10005:2019` whole, so a dated reference narrows in the index
|
|
72
|
-
itself. The `year` **argument** still needs filtering afterwards, because an
|
|
73
|
-
undated reference reaches every year of the document — that is why
|
|
74
|
-
`bib_results_filter` stays, and why it now reads the year off `row[:id].date`
|
|
75
|
-
instead of scraping a rendered string (the old `/:(\d{4})$/` would raise
|
|
76
|
-
`TypeError` against a pubid).
|
|
77
|
-
|
|
78
|
-
### Why this flavor needs no render flags
|
|
79
|
-
|
|
80
|
-
CalConnect is the simple end of the pubid migrations, and the reason is worth
|
|
81
|
-
stating so nobody adds machinery it does not need. `Pubid::Calconnect` renders
|
|
82
|
-
the publisher by default and models no edition and no volume, so the index key
|
|
83
|
-
and the document's own printed id are **the same string**: `CC/DIR 10005:2019`
|
|
84
|
-
in the row, `CC/DIR 10005:2019` in the record. There is nothing to opt in to
|
|
85
|
-
(contrast `Relaton::ThreeGpp::Docidentifier`, which must pass
|
|
86
|
-
`with_publisher: true`) and nothing to opt out of (contrast
|
|
87
|
-
`Relaton::Ecma::Docidentifier`, which must suppress the edition and the volume).
|
|
88
|
-
`DataFetcher#index_id` therefore returns the docidentifier's pubid untouched —
|
|
89
|
-
no `dup`, no mutation.
|
|
90
|
-
|
|
91
|
-
The number is a plain `String`, not a `Components::Code`, and deliberately so:
|
|
92
|
-
it preserves leading zeros (`0001`, `0514`) and keeps a sub-number in one token
|
|
93
|
-
(`0812-1`, `0707.1`). CalConnect models **no part** — those are numbers, not a
|
|
94
|
-
number plus a part — which is why `Docidentifier#remove_part!` and
|
|
95
|
-
`#to_all_parts!` are no-ops. Do not "fix" them by splitting the number.
|
|
96
|
-
|
|
97
|
-
`Docidentifier` follows the IHO/W3C shape: `content=` calls `super` first, so
|
|
98
|
-
`content` stays the source string verbatim. That is load-bearing beyond
|
|
99
|
-
convention — `ItemData#create_id` derives the record's `id` from
|
|
100
|
-
`content.gsub(/\W+/, "")`, so a re-render would move every published id.
|
|
101
|
-
`#remove_date!` is the one real mutator (`CC/DIR 10005:2019` ->
|
|
102
|
-
`CC/DIR 10005`); it writes back through `store_content`, never `content=`,
|
|
103
|
-
because a re-parse would rebuild `@pubid` and discard the mutation.
|
|
104
|
-
|
|
105
|
-
### An unparseable docid is recorded, not just warned
|
|
106
|
-
|
|
107
|
-
`Docidentifier#parse` logs at `Util.error` and leaves `#pubid` nil — it must not
|
|
108
|
-
raise, or an already-published record would stop deserializing.
|
|
109
|
-
`DataFetcher#add_to_index` then records
|
|
110
|
-
`@errors[docid] = "Unparseable primary id … was not indexed (…)"`, which the
|
|
111
|
-
inherited `Core::DataFetcher#report_errors` logs and `Logger::Channels::GhIssue`
|
|
112
|
-
turns into a GitHub issue at the end of the crawl. (`report_errors` treats a
|
|
113
|
-
**String** value as the message; the boolean form the parsers use means "this
|
|
114
|
-
field failed for every record" and is the wrong shape here.)
|
|
115
|
-
|
|
116
|
-
The row is skipped rather than indexed unparsed, because `Relaton::Index`
|
|
117
|
-
rejects the **whole** index if one row fails to deserialize and its sort calls
|
|
118
|
-
`.root.number` on every id. `#write_doc` writes the file **before** indexing, so
|
|
119
|
-
the document is unindexed, never lost.
|
|
120
|
-
|
|
121
|
-
### Measured against the live published corpus
|
|
122
|
-
|
|
123
|
-
All 191 rows of the published index (2026-09-08): every id parses; `to_s`
|
|
124
|
-
reproduces the published string **exactly**; `from_hash(to_hash)` round-trips
|
|
125
|
-
all 191; they render **191 distinct keys**, so nothing collapses under
|
|
126
|
-
`add_or_update`; `root.number` is empty for none. Driving all 191 documents of
|
|
127
|
-
`relaton-data-calconnect`'s `data/` through `write_doc` produces 191 rows with 0
|
|
128
|
-
errors and reads back as 191 identifiers with `FileIO#sorted` true.
|
|
129
|
-
`spec/calconnect/relaton/calconnect/pubid_contract_spec.rb` pins these
|
|
130
|
-
properties offline against the committed `index-v2` fixture.
|
|
131
|
-
|
|
132
|
-
No pubid change was needed for this migration. `Pubid::Calconnect` was written
|
|
133
|
-
for it.
|
|
134
|
-
|
|
135
|
-
## Testing
|
|
136
|
-
|
|
137
|
-
- **Framework:** RSpec with VCR cassettes and WebMock.
|
|
138
|
-
- **Index fixture:** `spec/calconnect/fixtures/index-v2.zip` — the whole
|
|
139
|
-
published index (191 rows), copied verbatim, so the stored shapes are exactly
|
|
140
|
-
what the runtime deserializes. Refresh it with
|
|
141
|
-
`bundle exec rake spec:update_index_calconnect`
|
|
142
|
-
(`tasks/index_fixture_calconnect.rb`); `#build` refuses a source that is not a
|
|
143
|
-
pubid index-v2 rather than writing a fixture `Relaton::Index` would reject
|
|
144
|
-
wholesale.
|
|
145
|
-
- It is seeded into the `Relaton::Index` pool by
|
|
146
|
-
`spec/calconnect/support/webmock.rb`, built **with `pubid_class:`** — without
|
|
147
|
-
which the rows stay raw hashes, `FileIO#sorted` stays false, and
|
|
148
|
-
`Type#search` silently stops narrowing, so the suite would pass while
|
|
149
|
-
exercising something the runtime never does. It is written to a temp file and
|
|
150
|
-
read through `FileIO`, not stuffed into `@index` directly, because the
|
|
151
|
-
deserialize and the sort both happen on that read path.
|
|
152
|
-
- There is **no `index-v1` fixture**, deliberately: `index-v1` is a released
|
|
153
|
-
relaton v2 artifact that this gem neither produces nor reads (see the index
|
|
154
|
-
section). Don't reintroduce one.
|
|
155
|
-
- **The `before(:each)` re-seed is not belt-and-braces.** `Index::Pool#type`
|
|
156
|
-
replaces the pooled entry whenever `actual?` says no, and `DataFetcher#index`
|
|
157
|
-
asks for the same `:CC` slot with a *different* file (`index-v2.yaml`). With a
|
|
158
|
-
`before(:suite)` seed alone the fetcher evicts the fixture, and every later
|
|
159
|
-
example goes to the network for real — which is exactly how six examples used
|
|
160
|
-
to die with `Errno::EPERM` on `~/.relaton/cc/index-v1.yaml`. The pool key is
|
|
161
|
-
`type.upcase.to_sym`, so the producer and the consumer share one slot.
|
|
162
|
-
(The ECMA/OGC pattern.)
|
|
163
|
-
- **VCR cassettes:** `spec/calconnect/vcr_cassettes/` — index downloads are
|
|
164
|
-
ignored by VCR, matched on the constant so a version bump cannot leave the
|
|
165
|
-
filter naming the old file and let a real download through.
|
|
166
|
-
- `pubid_contract_spec.rb` checks the whole fixture corpus for the properties
|
|
167
|
-
the index key depends on: every row deserializes, one distinct rendered key
|
|
168
|
-
per row, no empty bsearch key, a `to_hash`/`from_hash` round trip, and every
|
|
169
|
-
rendered key parsing back to the same identifier. Re-run it whenever the
|
|
170
|
-
fixture is refreshed.
|
|
171
|
-
- `hit_collection_spec.rb` covers the runtime search path — narrowing,
|
|
172
|
-
ordering, and the misses. It was an empty `describe` block before this
|
|
173
|
-
migration, so the lookup had no direct coverage at all.
|
|
174
|
-
- The `bibdata`/`bibitem` round-trip examples and two `gets` examples validate
|
|
175
|
-
XML with Jing, which needs a **JVM on PATH**. Without one they fail with
|
|
176
|
-
`Jing::ExecutionError`, which is environmental, not a code defect — check
|
|
177
|
-
`java -version` before chasing it.
|
data/lib/relaton/ccsds/CLAUDE.md
DELETED
|
@@ -1,74 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Architecture
|
|
4
|
-
|
|
5
|
-
**Pubid-backed docidentifier.** `Relaton::Ccsds::Docidentifier`
|
|
6
|
-
(`model/docidentifier.rb`, `< Bib::Docidentifier`) parses its `content` into a
|
|
7
|
-
`Pubid::Ccsds::Identifier` kept in `@pubid`, while the lutaml `content` attribute
|
|
8
|
-
stays a plain string for serialization. It implements the base class's abstract
|
|
9
|
-
`remove_part!` / `remove_date!` / `to_all_parts!` by mutating the pubid graph
|
|
10
|
-
(walking the `base` chain for supplements/corrigenda) and re-rendering
|
|
11
|
-
via `refresh_content!`. Unparseable content falls back to the raw string. It is
|
|
12
|
-
wired into `model/item.rb` (`attribute :docidentifier, Docidentifier`) so
|
|
13
|
-
`from_xml`/`from_yaml` and `DataParser#parse_docidentifier` yield this class.
|
|
14
|
-
`remove_date!` is effectively a no-op today — CCSDS ids carry no date component —
|
|
15
|
-
but is implemented for parity with ISO/IEC. `to_all_parts!` drops the part
|
|
16
|
-
component (e.g. `CCSDS 121.0-B-3` → `CCSDS 121-B-3`), freezes that rendering
|
|
17
|
-
into `@raw_content` (since `content` is live-derived from `@pubid`), then
|
|
18
|
-
wraps `@pubid` in pubid's `AllParts`. pubid-ccsds has no dedicated `AllParts`
|
|
19
|
-
subclass — unlike pubid-iso — so the wrapper is the generic
|
|
20
|
-
`Pubid::AllPartsIdentifier`: `content` stays the plain part-stripped id, but
|
|
21
|
-
`#pubid` itself, read directly, now answers `all_parts? == true` and renders
|
|
22
|
-
WITH pubid's generic "(all parts)" marker, since it's the wrapper. This
|
|
23
|
-
mirrors `lib/relaton/iec/model/docidentifier.rb`; the shared skeleton is
|
|
24
|
-
intentionally duplicated per flavor for now (to be hoisted into
|
|
25
|
-
`Bib::Docidentifier` once every flavor's id is Pubid-backed).
|
|
26
|
-
|
|
27
|
-
**Translation relations use pubid's subset match.**
|
|
28
|
-
`Data::Fetcher#search_instance_translation` reads the document's **already
|
|
29
|
-
parsed** id from `bib.docidentifier.first.pubid` (not `content`, which it would
|
|
30
|
-
re-parse), derives the language-less reference with `pubid.exclude(:language)`,
|
|
31
|
-
and picks the branch by `bibid == pubid` — no language means an instance
|
|
32
|
-
(`#search_translations`), a language means a translation (`#search_relations`).
|
|
33
|
-
This is more correct than the old `TRRGX` string test: `content` returns
|
|
34
|
-
`@pubid.to_s`, which reorders components, so a translated corrigendum
|
|
35
|
-
(`CCSDS 320.0-B-1-S - German Translated Cor. 1`) put the language mid-string and
|
|
36
|
-
the `$`-anchored `TRRGX` missed it; `exclude(:language)` strips the language down
|
|
37
|
-
the whole base chain. `TRRGX` still serves `translation_relation_types` and
|
|
38
|
-
`DataParser#relation_type`, which match raw relation-id strings.
|
|
39
|
-
`#search_relations` and `#search_translations` receive that pubid and select
|
|
40
|
-
index rows with `row[:id].exclude(:language) == bibid_pid` — a
|
|
41
|
-
language-agnostic match, so a translated row and the instance both match while
|
|
42
|
-
another edition of the same number does not. This does **not** use `===`:
|
|
43
|
-
pubid declares CCSDS `language` `subset_strict`, so a nil reference language
|
|
44
|
-
means "has none" (a wildcard would be wrong for `HitCollection`, where a
|
|
45
|
-
language-less user reference must not reach a translation). `exclude(:language)`
|
|
46
|
-
drops the language from the row on both sides, mirroring
|
|
47
|
-
`HitCollection#rows` (`r[:id].exclude(:edition) == pubid`).
|
|
48
|
-
|
|
49
|
-
`HitCollection#rows` keeps `index.search(pubid, exact: true)` for an edition
|
|
50
|
-
reference, although `language` is strict now. CCSDS `suffix` is still a nil
|
|
51
|
-
wildcard in pubid, so the bare `index.search(pubid)` makes `CCSDS 101.0-B-4`
|
|
52
|
-
also return the historical `CCSDS 101.0-B-4-S` (260 such pairs in the index
|
|
53
|
-
fixture). The `hit_collection_spec` "historical and a translated sibling"
|
|
54
|
-
example guards this. Drop `exact: true` only after pubid declares `suffix`
|
|
55
|
-
`subset_strict` too.
|
|
56
|
-
|
|
57
|
-
`#search_relations` excludes the document's own row with
|
|
58
|
-
`row[:id] == bib.docidentifier.first.pubid` — pubid to pubid. It compared a
|
|
59
|
-
`Pubid::Ccsds::Identifier` with a `String` (`.content`) before and was always
|
|
60
|
-
false, so a re-crawl over an index that already held the document gave it a
|
|
61
|
-
relation to itself. Covered by the `#search_relations` "excludes the document's
|
|
62
|
-
own row" example.
|
|
63
|
-
|
|
64
|
-
## Development
|
|
65
|
-
|
|
66
|
-
- `bundle install` — install dependencies
|
|
67
|
-
- `bundle exec rake spec` — run tests
|
|
68
|
-
- `bundle exec rubocop` — lint
|
|
69
|
-
|
|
70
|
-
## Testing
|
|
71
|
-
|
|
72
|
-
- **Framework:** RSpec with VCR cassettes and WebMock
|
|
73
|
-
- **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-ccsds.
|
|
74
|
-
- **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).
|
data/lib/relaton/cen/CLAUDE.md
DELETED
|
@@ -1,219 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-cen retrieves European Committee for Standardization (CEN/CENELEC) bibliographic data using the Relaton model. It has no API or curated index — it **scrapes** the CEN/CENELEC public portal (`standards.cencenelec.eu`) with Mechanize, simulating the search form and parsing HTML tables for title, status, ICS codes, publication dates, and committee info. Uses `Relaton::Bib::ItemData` directly (no relaton-iso dependency).
|
|
8
|
-
|
|
9
|
-
## Development
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
bundle exec rake spec:cen # the CEN suite
|
|
13
|
-
cd spec/cen && bundle exec rspec -I . relaton/cen/docidentifier_spec.rb
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
(No gem-local rubocop config; the monorepo root config applies.)
|
|
17
|
-
|
|
18
|
-
## Architecture
|
|
19
|
-
|
|
20
|
-
Namespace: `Relaton::Cen`. Reference handling is **entirely `Pubid::CenCenelec`**
|
|
21
|
-
— there is no hand-rolled identifier regex left. Because CEN publishes **no
|
|
22
|
-
index** and has no data repo, this is the **BSI shape**, not the index-v2 shape:
|
|
23
|
-
parse the query with pubid, select the portal hits with pubid, back the
|
|
24
|
-
`Docidentifier` with pubid. There is no `INDEXFILE` and no `pubid_class:`.
|
|
25
|
-
**Don't add an index.**
|
|
26
|
-
|
|
27
|
-
Retrieval flow:
|
|
28
|
-
|
|
29
|
-
1. **Bibliography** (`lib/relaton/cen/bibliography.rb`) — `get(code, year, opts)`
|
|
30
|
-
and `search` entry points. `parse` wraps
|
|
31
|
-
`Pubid::CenCenelec::Identifier.parse` and **raises** on an unrecognized query
|
|
32
|
-
(the root `CLAUDE.md` rule "An unrecognized query reference raises"); only the
|
|
33
|
-
data side rescues, in `Hit#pubid`. `get` keeps an early `nil` for an **empty**
|
|
34
|
-
reference — an empty string is no reference, not a malformed one — and
|
|
35
|
-
otherwise lets `Pubid::Errors::ParseError` propagate. `search` keeps its
|
|
36
|
-
`rescue Mechanize::ResponseCodeError, Net::ReadTimeout` by name; don't widen
|
|
37
|
-
it to `StandardError`, which would swallow the parse error and turn a
|
|
38
|
-
transport failure into a class `Relaton::Db#net_retry` does not retry.
|
|
39
|
-
2. **HitCollection** (`lib/relaton/cen/hit_collection.rb`, extends
|
|
40
|
-
`Relaton::Core::HitCollection`) — the web flow: GET the portal, follow the
|
|
41
|
-
redirect, submit the search form (`STAND_REF`) via Mechanize, parse the result
|
|
42
|
-
table, then `sort` by a pubid-derived key.
|
|
43
|
-
3. **Hit** (`lib/relaton/cen/hit.rb`, extends `Relaton::Core::Hit`) — `item`
|
|
44
|
-
lazy-loads the detail page via the scraper; `pubid` memoizes the hit code
|
|
45
|
-
parsed with pubid and is **nil** when the grammar cannot read it, because the
|
|
46
|
-
portal lists draft revisions such as `prEN 13306 rev` and one such row must
|
|
47
|
-
not abort the search. The filter, the sort and the scraper all read it, which
|
|
48
|
-
is why it is memoized (the `Relaton::Bsi::Hit#pubid` idiom).
|
|
49
|
-
4. **Scraper** (`lib/relaton/cen/scraper.rb`) — Mechanize/XPath HTML parser.
|
|
50
|
-
Maps CEN date abbreviations (DOR→adapted, DAV→issued, DOA→announced,
|
|
51
|
-
DOP→published, DOW→obsoleted) and relation headers (supersedes→obsoletes,
|
|
52
|
-
normative reference→cites); a small `COMMITTEES` map resolves codes like
|
|
53
|
-
TC 459→ECISS. `fetch_structuredid` takes the docnumber and partnumber from
|
|
54
|
-
the pubid. Two regexes remain here and are **not** identifiers, so they stay:
|
|
55
|
-
the committee regex in `fetch_contributors` and the year-of-a-date-string
|
|
56
|
-
match in `fetch_copyright`.
|
|
57
|
-
5. **ItemData / Model::Item** (`item_data.rb`, `model/`) —
|
|
58
|
-
`Relaton::Cen::ItemData` extends `Relaton::Bib::ItemData`; `Ext` adds a
|
|
59
|
-
`StructuredIdentifier` (docnumber/partnumber, agency "CEN").
|
|
60
|
-
6. **Processor** (`lib/relaton/cen/processor.rb`) — registry integration. Sets
|
|
61
|
-
`@pubid_flavor = :CenCenelec`, so `Core::Processor#prefixes` reads
|
|
62
|
-
`Pubid::CenCenelec.prefixes` (`EN CEN CLC CWA HD ES CR ENV`) into the global
|
|
63
|
-
prefix register; every one of the eight resolves to `[Relaton::Cen]` alone.
|
|
64
|
-
`@defaultprefix` is deliberately **narrower** than that list — widening it to
|
|
65
|
-
`ES` would claim a very generic token for this flavor.
|
|
66
|
-
It lazy-`require_relative`s `../cen` inside each method, like every other
|
|
67
|
-
flavor: the require used to sit at file load, so merely building a `Db`
|
|
68
|
-
pulled in mechanize, isoics and pubid. `spec/relaton/lazy_loading_spec.rb`
|
|
69
|
-
guards that.
|
|
70
|
-
|
|
71
|
-
### `#root` is the accessor that answers for every form
|
|
72
|
-
|
|
73
|
-
An `AdoptedEuropeanNorm` (`CEN ISO/TS 21003-7:2019`) carries **no** number, part
|
|
74
|
-
or year of its own — they live on the adopted ISO identifier — so
|
|
75
|
-
`base_document.year` is nil for it while `root.year` is `"2019"`. Measured over
|
|
76
|
-
the 67-reference probe corpus, `root.year` equals the old first-year regex on
|
|
77
|
-
**every** parseable reference and `base_document.year` misses all three
|
|
78
|
-
adopted-norm forms. So the year, number and part are read from `#root`; only the
|
|
79
|
-
**family** grouping is read from `#base_document`, which keeps
|
|
80
|
-
`EN ISO 1234` and `CEN ISO/TS 1234` apart.
|
|
81
|
-
|
|
82
|
-
### `Docidentifier` — the year rule
|
|
83
|
-
|
|
84
|
-
`Relaton::Cen::Docidentifier` (`model/docidentifier.rb`, `< Bib::Docidentifier`)
|
|
85
|
-
parses its `content` into `@pubid` while the lutaml `content` attribute stays a
|
|
86
|
-
plain string for serialization. The parse is on the **data side**: it lazily
|
|
87
|
-
`require "pubid"` and rescues `LoadError`/`StandardError`, so a non-CEN value (an
|
|
88
|
-
ISBN) or a code the grammar rejects stays verbatim with `@pubid` nil, and all
|
|
89
|
-
three mutators no-op. `Pubid#exclude` returns a **copy** (the BSI shape, not
|
|
90
|
-
IALA's in-place setters), so each mutator replaces `@pubid` and re-renders
|
|
91
|
-
through the aliased `store_content` — writing through `content=` would re-parse
|
|
92
|
-
and discard the mutation.
|
|
93
|
-
|
|
94
|
-
**`remove_date!` drops the LAST year, and only that one.** On a supplement the
|
|
95
|
-
identifier names the supplement, so its own year is the last one:
|
|
96
|
-
`EN 13250:2000/A1:2005` is amendment A1 of 2005, and the `2000` belongs to the
|
|
97
|
-
base document it amends — excluding that would strip the base document's
|
|
98
|
-
identity rather than a date. So the rule is `exclude(:supplement_year)` when
|
|
99
|
-
there is a supplement year, else `exclude(:year)`:
|
|
100
|
-
|
|
101
|
-
| input | `remove_date!` |
|
|
102
|
-
|---|---|
|
|
103
|
-
| `EN 13306:2017` | `EN 13306` |
|
|
104
|
-
| `EN 13250:2000/A1:2005` | `EN 13250:2000/A1` |
|
|
105
|
-
| `EN 285:2015+A1:2021` | `EN 285:2015+A1` |
|
|
106
|
-
| `EN 61375-2-3:2015/AC:2016-11` | `EN 61375-2-3:2015/AC` |
|
|
107
|
-
| `EN 285:2015+A1` | `EN 285+A1` |
|
|
108
|
-
|
|
109
|
-
The last two rows are where this beats the old `content.sub!(/:\d{4}$/, "")`:
|
|
110
|
-
neither string ends in a bare year, so the regex stripped **nothing** and
|
|
111
|
-
`to_most_recent_reference` returned a dated reference. `:supplement_year` is a
|
|
112
|
-
CEN-only key, and it resets the supplement's year and month together. Note that
|
|
113
|
-
ISO and IEC are different — there `exclude(:year)` removes *every* year.
|
|
114
|
-
|
|
115
|
-
`remove_part!` is `exclude(:part, :subpart)`, and `to_all_parts!` does both,
|
|
116
|
-
then wraps `@pubid` in pubid's `AllParts` (a generic `Pubid::AllPartsIdentifier`
|
|
117
|
-
— CEN has no dedicated `AllParts` subclass). `content` stays the plain
|
|
118
|
-
part-and-date-stripped string: **`Pubid::CenCenelec::Renderer` never reads
|
|
119
|
-
`all_parts`**, so CEN's own renderer never had a marker, and `content` is
|
|
120
|
-
cached from the pre-wrap pubid on purpose. `#pubid` itself, read directly, now
|
|
121
|
-
answers `all_parts? == true` and renders WITH pubid's generic "(all parts)"
|
|
122
|
-
marker (`#to_s`), since it's the wrapper. BSI and IALA make the same
|
|
123
|
-
content/`#pubid` trade-off, for the same reason. pubid holds a sub-part
|
|
124
|
-
**inside** `part` (`61375-2-3` gives `"2-3"`), which is also why
|
|
125
|
-
`fetch_structuredid` swaps the separator to
|
|
126
|
-
reproduce the old `"2:3"` partnumber.
|
|
127
|
-
|
|
128
|
-
### Hit selection — what the query left unsaid
|
|
129
|
-
|
|
130
|
-
`search_filter` sends the caller's **raw reference text** to the portal form —
|
|
131
|
-
that is search-engine input, not identifier parsing (the BSI precedent) — and
|
|
132
|
-
selects with pubid's subset match `query === hit.pubid`. A component that the
|
|
133
|
-
query omits matches any value, and `===` reads the omission from the query
|
|
134
|
-
itself, nested identifiers included (`CEN ISO/TS 21003-7` keeps its part on the
|
|
135
|
-
adopted ISO document). That gives the rules the old comparison had:
|
|
136
|
-
|
|
137
|
-
- no part → any part
|
|
138
|
-
- no base year → any year
|
|
139
|
-
- a supplement that carries **no year** → any supplement year:
|
|
140
|
-
`EN 285:2015+A1` → `EN 285:2015+A1:2021`, but never → `EN 285:2015`
|
|
141
|
-
|
|
142
|
-
pubid declares the CEN `type`, `stage` and `typed_stage` strict (pubid#408), so
|
|
143
|
-
`EN 1325` matches neither the draft `prEN 1325` nor `CEN/TS 1325`. `===`
|
|
144
|
-
requires the same class, and pubid documents that a reference never falls back
|
|
145
|
-
to the base document of a wrapper, so **a base reference never answers with its
|
|
146
|
-
own amendment's record**, and a supplement reference never answers with its
|
|
147
|
-
base document. That is this flavor's rule, and it used to need a
|
|
148
|
-
`supplement?` guard, because `matches?` enforced it only through the
|
|
149
|
-
class-strict `==`, which was pubid's internal invariant rather than a contract.
|
|
150
|
-
`spec/cen/relaton/cen/bibliography_spec.rb` still covers it (`never selects a
|
|
151
|
-
supplement for a base reference`), so a pubid that relaxed `===` would fail
|
|
152
|
-
loudly here.
|
|
153
|
-
|
|
154
|
-
Measured over the 61 hit codes the cassettes hold (141 queries: each code and
|
|
155
|
-
its forms without year, part and supplement year): `===` and the old
|
|
156
|
-
`matches?(…, ignore: ignored_components(query))` agree on all 8,601 pairs.
|
|
157
|
-
|
|
158
|
-
`absent?(id, *keys)` (`id.exclude(*keys) == id`) stays for `get`, which appends
|
|
159
|
-
the `year` argument to a reference that names no year.
|
|
160
|
-
|
|
161
|
-
Two selections got **stricter**, and both were latent bugs in the regex:
|
|
162
|
-
`code_to_parts` required a literal `+` for its `amd` group and never compared its
|
|
163
|
-
`ac` group at all, so a `/A1:2005` amendment was invisible to it — a query for
|
|
164
|
-
`EN 13250:2000` matched the amendment record, and `HD 1215-2:1988` kept its
|
|
165
|
-
corrigendum hit and was saved only by the sort order. pubid rejects both.
|
|
166
|
-
|
|
167
|
-
### Sort order
|
|
168
|
-
|
|
169
|
-
`HitCollection#sort` is a memoized `sort_by!` over
|
|
170
|
-
`[family, part_segments, -year, supplement_key]`, which reproduces the order the
|
|
171
|
-
six `code_to_parts` fields gave: the document family ascending
|
|
172
|
-
(`base_document.exclude(:year, :part, :subpart)`), then the part ascending with
|
|
173
|
-
a part-less hit **first**, then the year **descending** with a year-less hit
|
|
174
|
-
last, then the supplement, so a base document precedes its own amendments and
|
|
175
|
-
corrigenda. Part segments are compared as **integers**, so `-10` sorts after
|
|
176
|
-
`-2`. An unparseable hit sorts last. The order is load-bearing:
|
|
177
|
-
`isobib_results_filter` takes the first hit that matches the year and has an
|
|
178
|
-
item, so `EN 13306` answers with the 2017 record and `EN 1325` with the
|
|
179
|
-
part-less one.
|
|
180
|
-
|
|
181
|
-
A consolidated identifier (`EN 285:2015+A1:2021`) holds its base document and
|
|
182
|
-
its supplements in `#identifiers` and does not answer `supplement_type`
|
|
183
|
-
itself, so `supplements` reads `identifiers.drop(1)` for it and `[id]` for a
|
|
184
|
-
plain amendment or corrigendum. The sort key reads the supplement's `number`
|
|
185
|
-
and `year`, which every pubid version answers: pubid removes the
|
|
186
|
-
`supplement_number`/`supplement_year` aliases.
|
|
187
|
-
The `:supplement_year` exclude key is a different thing and stays.
|
|
188
|
-
|
|
189
|
-
## External dependencies
|
|
190
|
-
|
|
191
|
-
`mechanize ~> 2.10`, `isoics ~> 0.1` (ICS descriptions), `pubid`,
|
|
192
|
-
`relaton-core`, `relaton-bib`.
|
|
193
|
-
|
|
194
|
-
## Testing
|
|
195
|
-
|
|
196
|
-
RSpec with WebMock + VCR (cassettes in `spec/vcr_cassettes/`, record `:once`, never auto-rerecorded). Network is blocked via WebMock. No index fixture — CEN is scraped, not indexed; `spec/fixtures/` holds XML/YAML round-trip data. The scraper is inherently brittle to portal HTML changes.
|
|
197
|
-
|
|
198
|
-
- `spec/cen/relaton/cen/docidentifier_spec.rb` unit-tests the three mutators over
|
|
199
|
-
every identifier form the portal serves, including the two rows above that the
|
|
200
|
-
old regex left unchanged.
|
|
201
|
-
- `spec/cen/relaton/cen/bibliography_spec.rb` unit-tests hit **selection** with
|
|
202
|
-
**no HTTP at all**: `query === hit` is the whole of `search_filter`'s
|
|
203
|
-
decision, and driving it directly covers pairs no cassette
|
|
204
|
-
can. That matters here — the cassettes hold a detail page only for the
|
|
205
|
-
document their own example fetched, so an example that selects a *different*
|
|
206
|
-
hit from the same search (say the 2008 record in `cen_iso_ts_21003_7`) dies
|
|
207
|
-
with `VCR::Errors::UnhandledHTTPRequestError`, not an assertion failure.
|
|
208
|
-
- Every `docidentifier[0].content` expectation in `spec/cen/relaton/cen_spec.rb`
|
|
209
|
-
is **unedited** by the pubid migration: what the flavor returns did not change.
|
|
210
|
-
- `returns nil when document doesn't exist` is split in two, because
|
|
211
|
-
`CEN NOT FOUND` is not an identifier and now raises. One example asserts
|
|
212
|
-
`Pubid::Errors::ParseError` and makes no request; the other keeps the
|
|
213
|
-
not-found coverage with a parseable reference.
|
|
214
|
-
- **Trap:** `gets code` writes `spec/cen/fixtures/bibdata.xml` through
|
|
215
|
-
`write_file`, which writes only when the file is **missing**
|
|
216
|
-
(`spec/spec_helper.rb`). A deleted fixture is silently recreated from whatever
|
|
217
|
-
the current cassette says, and the example then always passes. Never delete it
|
|
218
|
-
to make a spec green; delete it only to regenerate deliberately, then read the
|
|
219
|
-
diff.
|
data/lib/relaton/cie/CLAUDE.md
DELETED
|
@@ -1,90 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Index format (pubid v2 — structured `_type:` index)
|
|
4
|
-
|
|
5
|
-
CIE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`):
|
|
6
|
-
each row's `:id` is a `Pubid::Cie::Identifier` object, serialized by `relaton-index`
|
|
7
|
-
as a `_type: pubid:cie:…` map (mirrors the NIST/ETSI flavors).
|
|
8
|
-
|
|
9
|
-
- **Producer** (`data_fetcher.rb`): `#index` passes `pubid_class:
|
|
10
|
-
::Pubid::Cie::Identifier`; `#write_file` parses each docid via the `#pubid` helper
|
|
11
|
-
and indexes the **object** (not the raw string). `#pubid` returns `nil` — the id is
|
|
12
|
-
written to disk as a document but **left out of the index** — for any id pubid can't
|
|
13
|
-
parse **or** that doesn't round-trip. The guard mirrors the read side's acceptance
|
|
14
|
-
test (`Index::FileIO#id_supported?`, `from_hash(to_hash) == to_hash`): the loader
|
|
15
|
-
rejects the *whole* index on the first non-round-tripping row, so a bad id must be
|
|
16
|
-
dropped at write time rather than poison every lookup.
|
|
17
|
-
- **Consumer** (`scrapper.rb`, `processor.rb`): both pass `pubid_class:
|
|
18
|
-
::Pubid::Cie::Identifier` to `Index.find_or_create`. The scrapper parses the ref to a
|
|
19
|
-
pubid with `::Pubid::Cie.parse` — an unparseable ref **raises** `Pubid::Errors::ParseError`
|
|
20
|
-
(partial refs such as `CIE 001` parse; the old raw-String substring fallback is gone)
|
|
21
|
-
and passes **that** to `index.search` so index-v2 narrows candidates by number via
|
|
22
|
-
binary search before the block runs (a String arg would force a full O(n) scan — see
|
|
23
|
-
`Index::Type#search_candidates`); the block keeps the broad substring match, and
|
|
24
|
-
`min_by { |r| r[:id].to_s }` picks the winner (rows are pubid objects, not
|
|
25
|
-
`Comparable`, so compare by string form). The scrapper's index comes from the
|
|
26
|
-
data repo's Pages machine index (`PAGES_URL`, relaton#206) — manifest + the
|
|
27
|
-
shard for the parsed query, the whole-index zip on the same site serving only
|
|
28
|
-
the retry on a miss; `ENDPOINT` (the `v2` branch) still serves the document
|
|
29
|
-
files.
|
|
30
|
-
|
|
31
|
-
### pubid dependency
|
|
32
|
-
|
|
33
|
-
The published `relaton-data-cie/index-v2` uses CIE **proceedings** ids
|
|
34
|
-
(`_type: pubid:cie:proceedings` with `paper`/`page_range`) and the **flattened**
|
|
35
|
-
`to_hash` (`number`/`year` scalar, no nested `code`). That support first shipped in
|
|
36
|
-
the released pubid `2.0.0.pre.alpha.13`, which `relaton.gemspec` pins — an older
|
|
37
|
-
pubid strips `paper`/`page_range` and emits the nested shape, so `Relaton::Index`
|
|
38
|
-
would reject the whole published index.
|
|
39
|
-
|
|
40
|
-
Note the CIE grammar extensions **do** parse the techstreet variant ids like
|
|
41
|
-
`CIE x051:2025/zcunvy` (as `Conference` `@variant` ids), so — unlike the old pubid —
|
|
42
|
-
they are indexed, not dropped.
|
|
43
|
-
|
|
44
|
-
## Crawler (`data_fetcher.rb`) — parallel detail fetch
|
|
45
|
-
|
|
46
|
-
The Techstreet crawl (~1147 docs) is a headless-Chrome (Ferrum) scrape. It runs in
|
|
47
|
-
**two phases**:
|
|
48
|
-
|
|
49
|
-
- **Phase 1 — `collect_hits` (serial):** page through the search results following
|
|
50
|
-
`//a[@class="next_page"]`, collecting every `//li[@data-product]` into one list
|
|
51
|
-
(~12 page loads at `per_page=100`).
|
|
52
|
-
- **Phase 2 — `process_hits` (parallel):** fan the hits out over a bounded pool.
|
|
53
|
-
**Each worker owns its own `BrowserAgent`** (its own Chrome, built via the
|
|
54
|
-
`build_agent` factory so the full UA/header + `navigator`/`window.chrome` stealth
|
|
55
|
-
masking runs per worker — Cloudflare evasion must hold per worker, not for a shared
|
|
56
|
-
singleton). Agents are built up front on the main thread (a Chrome-launch failure
|
|
57
|
-
aborts fast instead of hanging the bounded queue) and **every** agent is quit in an
|
|
58
|
-
`ensure`. The single memoized `#agent` is used only for Phase 1.
|
|
59
|
-
|
|
60
|
-
**Concurrency knob:** `RELATON_CIE_CONCURRENCY` (default **5**, min 1), read by
|
|
61
|
-
`.concurrency`. `relaton-data-cie`'s crawler workflow tunes it via env — no code change.
|
|
62
|
-
|
|
63
|
-
**Per-worker adaptive pacing (`Pacing`)** replaces the old single global 4 s gap: each
|
|
64
|
-
worker starts at `BASE_GAP` (~1 s) and **doubles its own gap up to `MAX_GAP`** only on
|
|
65
|
-
trouble — a Cloudflare challenge or a Ferrum/socket error (`RETRIABLE_ERRORS`).
|
|
66
|
-
`BrowserAgent#wait_for_challenge` raises a retriable `ChallengeError` when the challenge
|
|
67
|
-
doesn't clear within `MAX_CHALLENGE_WAIT`, so a stuck worker backs off and retries
|
|
68
|
-
(`#time_req` keeps the 4-try retry) instead of parsing the challenge HTML.
|
|
69
|
-
|
|
70
|
-
**Thread-safety & byte-identical output:** the slow `agent.get` runs lock-free; the
|
|
71
|
-
cheap build + `#write_file` run under one `@mutex`, so `@files`/`@errors`/`@seen` and the
|
|
72
|
-
index mutate single-threaded. Output is order-independent by construction — `@errors`
|
|
73
|
-
accumulates with `&&=` (AND, per-doc build is atomic under the lock), `@files` is a Set,
|
|
74
|
-
and the structured index is sorted deterministically on save. For a duplicate output
|
|
75
|
-
file, a `@seen[file] => catalogue-position` guard (`#superseded?`) keeps the
|
|
76
|
-
**last-by-position** content winner the serial crawl would pick; `index.add_or_update`
|
|
77
|
-
runs for **every** distinct id *before* that gate, so each id is still indexed. `index.save`
|
|
78
|
-
is called **once**, after the pool drains.
|
|
79
|
-
|
|
80
|
-
## Development
|
|
81
|
-
|
|
82
|
-
- `bundle install` — install dependencies
|
|
83
|
-
- `bundle exec rake spec` — run tests
|
|
84
|
-
- `bundle exec rubocop` — lint
|
|
85
|
-
|
|
86
|
-
## Testing
|
|
87
|
-
|
|
88
|
-
- **Framework:** RSpec with VCR cassettes and WebMock
|
|
89
|
-
- **Index fixture:** `spec/fixtures/index-v2.zip` (the published structured index) is loaded through a `Relaton::Index::Type` with `::Pubid::Cie::Identifier` and injected into the `Relaton::Index` pool in `before(:suite)` (see `spec/support/webmock.rb`, mirroring NIST/ETSI). Run `rake spec:update_index` to refresh from relaton-data-cie.
|
|
90
|
-
- **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).
|