relaton 3.0.0.pre.alpha.17 → 3.0.0.pre.alpha.19

data/lib/relaton/cen/CLAUDE.md

@@ -0,0 +1,219 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+relaton-cen retrieves European Committee for Standardization (CEN/CENELEC) bibliographic data using the Relaton model. It has no API or curated index — it **scrapes** the CEN/CENELEC public portal (`standards.cencenelec.eu`) with Mechanize, simulating the search form and parsing HTML tables for title, status, ICS codes, publication dates, and committee info. Uses `Relaton::Bib::ItemData` directly (no relaton-iso dependency).
+
+## Development
+
+```bash
+bundle exec rake spec:cen                       # the CEN suite
+cd spec/cen && bundle exec rspec -I . relaton/cen/docidentifier_spec.rb
+```
+
+(No gem-local rubocop config; the monorepo root config applies.)
+
+## Architecture
+
+Namespace: `Relaton::Cen`. Reference handling is **entirely `Pubid::CenCenelec`**
+— there is no hand-rolled identifier regex left. Because CEN publishes **no
+index** and has no data repo, this is the **BSI shape**, not the index-v2 shape:
+parse the query with pubid, select the portal hits with pubid, back the
+`Docidentifier` with pubid. There is no `INDEXFILE` and no `pubid_class:`.
+**Don't add an index.**
+
+Retrieval flow:
+
+1. **Bibliography** (`lib/relaton/cen/bibliography.rb`) — `get(code, year, opts)`
+   and `search` entry points. `parse` wraps
+   `Pubid::CenCenelec::Identifier.parse` and **raises** on an unrecognized query
+   (the root `CLAUDE.md` rule "An unrecognized query reference raises"); only the
+   data side rescues, in `Hit#pubid`. `get` keeps an early `nil` for an **empty**
+   reference — an empty string is no reference, not a malformed one — and
+   otherwise lets `Pubid::Errors::ParseError` propagate. `search` keeps its
+   `rescue Mechanize::ResponseCodeError, Net::ReadTimeout` by name; don't widen
+   it to `StandardError`, which would swallow the parse error and turn a
+   transport failure into a class `Relaton::Db#net_retry` does not retry.
+2. **HitCollection** (`lib/relaton/cen/hit_collection.rb`, extends
+   `Relaton::Core::HitCollection`) — the web flow: GET the portal, follow the
+   redirect, submit the search form (`STAND_REF`) via Mechanize, parse the result
+   table, then `sort` by a pubid-derived key.
+3. **Hit** (`lib/relaton/cen/hit.rb`, extends `Relaton::Core::Hit`) — `item`
+   lazy-loads the detail page via the scraper; `pubid` memoizes the hit code
+   parsed with pubid and is **nil** when the grammar cannot read it, because the
+   portal lists draft revisions such as `prEN 13306 rev` and one such row must
+   not abort the search. The filter, the sort and the scraper all read it, which
+   is why it is memoized (the `Relaton::Bsi::Hit#pubid` idiom).
+4. **Scraper** (`lib/relaton/cen/scraper.rb`) — Mechanize/XPath HTML parser.
+   Maps CEN date abbreviations (DOR→adapted, DAV→issued, DOA→announced,
+   DOP→published, DOW→obsoleted) and relation headers (supersedes→obsoletes,
+   normative reference→cites); a small `COMMITTEES` map resolves codes like
+   TC 459→ECISS. `fetch_structuredid` takes the docnumber and partnumber from
+   the pubid. Two regexes remain here and are **not** identifiers, so they stay:
+   the committee regex in `fetch_contributors` and the year-of-a-date-string
+   match in `fetch_copyright`.
+5. **ItemData / Model::Item** (`item_data.rb`, `model/`) —
+   `Relaton::Cen::ItemData` extends `Relaton::Bib::ItemData`; `Ext` adds a
+   `StructuredIdentifier` (docnumber/partnumber, agency "CEN").
+6. **Processor** (`lib/relaton/cen/processor.rb`) — registry integration. Sets
+   `@pubid_flavor = :CenCenelec`, so `Core::Processor#prefixes` reads
+   `Pubid::CenCenelec.prefixes` (`EN CEN CLC CWA HD ES CR ENV`) into the global
+   prefix register; every one of the eight resolves to `[Relaton::Cen]` alone.
+   `@defaultprefix` is deliberately **narrower** than that list — widening it to
+   `ES` would claim a very generic token for this flavor.
+   It lazy-`require_relative`s `../cen` inside each method, like every other
+   flavor: the require used to sit at file load, so merely building a `Db`
+   pulled in mechanize, isoics and pubid. `spec/relaton/lazy_loading_spec.rb`
+   guards that.
+
+### `#root` is the accessor that answers for every form
+
+An `AdoptedEuropeanNorm` (`CEN ISO/TS 21003-7:2019`) carries **no** number, part
+or year of its own — they live on the adopted ISO identifier — so
+`base_document.year` is nil for it while `root.year` is `"2019"`. Measured over
+the 67-reference probe corpus, `root.year` equals the old first-year regex on
+**every** parseable reference and `base_document.year` misses all three
+adopted-norm forms. So the year, number and part are read from `#root`; only the
+**family** grouping is read from `#base_document`, which keeps
+`EN ISO 1234` and `CEN ISO/TS 1234` apart.
+
+### `Docidentifier` — the year rule
+
+`Relaton::Cen::Docidentifier` (`model/docidentifier.rb`, `< Bib::Docidentifier`)
+parses its `content` into `@pubid` while the lutaml `content` attribute stays a
+plain string for serialization. The parse is on the **data side**: it lazily
+`require "pubid"` and rescues `LoadError`/`StandardError`, so a non-CEN value (an
+ISBN) or a code the grammar rejects stays verbatim with `@pubid` nil, and all
+three mutators no-op. `Pubid#exclude` returns a **copy** (the BSI shape, not
+IALA's in-place setters), so each mutator replaces `@pubid` and re-renders
+through the aliased `store_content` — writing through `content=` would re-parse
+and discard the mutation.
+
+**`remove_date!` drops the LAST year, and only that one.** On a supplement the
+identifier names the supplement, so its own year is the last one:
+`EN 13250:2000/A1:2005` is amendment A1 of 2005, and the `2000` belongs to the
+base document it amends — excluding that would strip the base document's
+identity rather than a date. So the rule is `exclude(:supplement_year)` when
+there is a supplement year, else `exclude(:year)`:
+
+| input | `remove_date!` |
+|---|---|
+| `EN 13306:2017` | `EN 13306` |
+| `EN 13250:2000/A1:2005` | `EN 13250:2000/A1` |
+| `EN 285:2015+A1:2021` | `EN 285:2015+A1` |
+| `EN 61375-2-3:2015/AC:2016-11` | `EN 61375-2-3:2015/AC` |
+| `EN 285:2015+A1` | `EN 285+A1` |
+
+The last two rows are where this beats the old `content.sub!(/:\d{4}$/, "")`:
+neither string ends in a bare year, so the regex stripped **nothing** and
+`to_most_recent_reference` returned a dated reference. `:supplement_year` is a
+CEN-only key, and it resets the supplement's year and month together. Note that
+ISO and IEC are different — there `exclude(:year)` removes *every* year.
+
+`remove_part!` is `exclude(:part, :subpart)`, and `to_all_parts!` does both,
+then wraps `@pubid` in pubid's `AllParts` (a generic `Pubid::AllPartsIdentifier`
+— CEN has no dedicated `AllParts` subclass). `content` stays the plain
+part-and-date-stripped string: **`Pubid::CenCenelec::Renderer` never reads
+`all_parts`**, so CEN's own renderer never had a marker, and `content` is
+cached from the pre-wrap pubid on purpose. `#pubid` itself, read directly, now
+answers `all_parts? == true` and renders WITH pubid's generic "(all parts)"
+marker (`#to_s`), since it's the wrapper. BSI and IALA make the same
+content/`#pubid` trade-off, for the same reason. pubid holds a sub-part
+**inside** `part` (`61375-2-3` gives `"2-3"`), which is also why
+`fetch_structuredid` swaps the separator to
+reproduce the old `"2:3"` partnumber.
+
+### Hit selection — what the query left unsaid
+
+`search_filter` sends the caller's **raw reference text** to the portal form —
+that is search-engine input, not identifier parsing (the BSI precedent) — and
+selects with pubid's subset match `query === hit.pubid`. A component that the
+query omits matches any value, and `===` reads the omission from the query
+itself, nested identifiers included (`CEN ISO/TS 21003-7` keeps its part on the
+adopted ISO document). That gives the rules the old comparison had:
+
+- no part → any part
+- no base year → any year
+- a supplement that carries **no year** → any supplement year:
+  `EN 285:2015+A1` → `EN 285:2015+A1:2021`, but never → `EN 285:2015`
+
+pubid declares the CEN `type`, `stage` and `typed_stage` strict (pubid#408), so
+`EN 1325` matches neither the draft `prEN 1325` nor `CEN/TS 1325`. `===`
+requires the same class, and pubid documents that a reference never falls back
+to the base document of a wrapper, so **a base reference never answers with its
+own amendment's record**, and a supplement reference never answers with its
+base document. That is this flavor's rule, and it used to need a
+`supplement?` guard, because `matches?` enforced it only through the
+class-strict `==`, which was pubid's internal invariant rather than a contract.
+`spec/cen/relaton/cen/bibliography_spec.rb` still covers it (`never selects a
+supplement for a base reference`), so a pubid that relaxed `===` would fail
+loudly here.
+
+Measured over the 61 hit codes the cassettes hold (141 queries: each code and
+its forms without year, part and supplement year): `===` and the old
+`matches?(…, ignore: ignored_components(query))` agree on all 8,601 pairs.
+
+`absent?(id, *keys)` (`id.exclude(*keys) == id`) stays for `get`, which appends
+the `year` argument to a reference that names no year.
+
+Two selections got **stricter**, and both were latent bugs in the regex:
+`code_to_parts` required a literal `+` for its `amd` group and never compared its
+`ac` group at all, so a `/A1:2005` amendment was invisible to it — a query for
+`EN 13250:2000` matched the amendment record, and `HD 1215-2:1988` kept its
+corrigendum hit and was saved only by the sort order. pubid rejects both.
+
+### Sort order
+
+`HitCollection#sort` is a memoized `sort_by!` over
+`[family, part_segments, -year, supplement_key]`, which reproduces the order the
+six `code_to_parts` fields gave: the document family ascending
+(`base_document.exclude(:year, :part, :subpart)`), then the part ascending with
+a part-less hit **first**, then the year **descending** with a year-less hit
+last, then the supplement, so a base document precedes its own amendments and
+corrigenda. Part segments are compared as **integers**, so `-10` sorts after
+`-2`. An unparseable hit sorts last. The order is load-bearing:
+`isobib_results_filter` takes the first hit that matches the year and has an
+item, so `EN 13306` answers with the 2017 record and `EN 1325` with the
+part-less one.
+
+A consolidated identifier (`EN 285:2015+A1:2021`) holds its base document and
+its supplements in `#identifiers` and does not answer `supplement_type`
+itself, so `supplements` reads `identifiers.drop(1)` for it and `[id]` for a
+plain amendment or corrigendum. The sort key reads the supplement's `number`
+and `year`, which every pubid version answers: pubid removes the
+`supplement_number`/`supplement_year` aliases.
+The `:supplement_year` exclude key is a different thing and stays.
+
+## External dependencies
+
+`mechanize ~> 2.10`, `isoics ~> 0.1` (ICS descriptions), `pubid`,
+`relaton-core`, `relaton-bib`.
+
+## Testing
+
+RSpec with WebMock + VCR (cassettes in `spec/vcr_cassettes/`, record `:once`, 7-day re-record). Network is blocked via WebMock. No index fixture — CEN is scraped, not indexed; `spec/fixtures/` holds XML/YAML round-trip data. The scraper is inherently brittle to portal HTML changes.
+
+- `spec/cen/relaton/cen/docidentifier_spec.rb` unit-tests the three mutators over
+  every identifier form the portal serves, including the two rows above that the
+  old regex left unchanged.
+- `spec/cen/relaton/cen/bibliography_spec.rb` unit-tests hit **selection** with
+  **no HTTP at all**: `query === hit` is the whole of `search_filter`'s
+  decision, and driving it directly covers pairs no cassette
+  can. That matters here — the cassettes hold a detail page only for the
+  document their own example fetched, so an example that selects a *different*
+  hit from the same search (say the 2008 record in `cen_iso_ts_21003_7`) dies
+  with `VCR::Errors::UnhandledHTTPRequestError`, not an assertion failure.
+- Every `docidentifier[0].content` expectation in `spec/cen/relaton/cen_spec.rb`
+  is **unedited** by the pubid migration: what the flavor returns did not change.
+- `returns nil when document doesn't exist` is split in two, because
+  `CEN NOT FOUND` is not an identifier and now raises. One example asserts
+  `Pubid::Errors::ParseError` and makes no request; the other keeps the
+  not-found coverage with a parseable reference.
+- **Trap:** `gets code` writes `spec/cen/fixtures/bibdata.xml` through
+  `write_file`, which writes only when the file is **missing**
+  (`spec/spec_helper.rb`). A deleted fixture is silently recreated from whatever
+  the current cassette says, and the example then always passes. Never delete it
+  to make a spec green; delete it only to regenerate deliberately, then read the
+  diff.

data/lib/relaton/cen/README.adoc

@@ -0,0 +1,155 @@
+= Relaton::Cen retrieve CEN Standards for bibliographic use using the BibliographicItem model
+
+RelatonCen is a Ruby gem that implements the https://github.com/metanorma/metanorma-model-iso#iso-bibliographic-item[IsoBibliographicItem model].
+
+== Usage
+
+For CENELEC standards, use references with the prefix `CEN/CLC`. For example, `CEN/CLC Guide 6`.
+
+=== Search for a standard using keywords
+
+[source,ruby]
+----
+require 'relaton/cen'
+=> true
+
+hit_collection = Relaton::Cen::Bibliography.search("CEN ISO/TS 21003-7")
+=> <Relaton::Cen::HitCollection:0x00000000002560 @ref=CEN ISO/TS 21003-7 @fetched=false>
+
+item = hit_collection[0].item
+=> #<Relaton::Cen::ItemData:0x000000012a35c458
+ ...
+----
+
+=== Get a standard by its code
+
+Use `Relaton::Cen::Bibliography.get(ref, year, options)` to get a standard by its code.
+
+- `ref` is the standard code, e.g. `CEN ISO/TS 21003-7`
+- `year` is the year of the standard, e.g. `2019` (optional)
+- `options` is a hash of options:
+  - `keep_year` - keep the year in ID if true (optional)
+
+[source,ruby]
+----
+# With year in reference
+Relaton::Cen::Bibliography.get "EN 10160:1999"
+[relaton-cen] INFO: (EN 10160:1999) Fetching from standards.cencenelec.eu ...
+[relaton-cen] INFO: (EN 10160:1999) Found: `EN 10160:1999`
+=> #<Relaton::Cen::ItemData:0x00000001273d5b58
+...
+
+# With a year as a separate argument
+Relaton::Cen::Bibliography.get "EN 10160", "1999"
+[relaton-cen] INFO: (EN 10160:1999) Fetching from standards.cencenelec.eu ...
+[relaton-cen] INFO: (EN 10160:1999) Found: `EN 10160:1999`
+=> #<Relaton::Cen::ItemData:0x00000001274d58c8
+...
+
+# To get the most recent version of a standard by its code use reference without year
+Relaton::Cen::Bibliography.get "CEN/CLC Guide 6"
+[relaton-cen] INFO: (CEN/CLC Guide 6) Fetching from standards.cencenelec.eu ...
+[relaton-cen] INFO: (CEN/CLC Guide 6) Found: `CEN/CLC Guide 6:2014`
+=> #<Relaton::Cen::ItemData:0x000000012759a010
+...
+
+# To keep the year in ID use `keep_year` option
+> RelatonCen::CenBibliography.get "CEN/CLC Guide 6", nil, keep_year: true
+[relaton-cen] INFO: (CEN/CLC Guide 6) Fetching from standards.cencenelec.eu ...
+[relaton-cen] INFO: (CEN/CLC Guide 6) Found: `CEN/CLC Guide 6`
+=> #<Relaton::Cen::ItemData:0x000000012c629760
+...
+----
+
+=== XML serialization
+
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="CENISOTS2100372019" type="standard" schema-version="v1.5.6">
+      <fetched>2025-12-15</fetched>
+      <title language="en" script="Latn" type="title-main">Multilayer piping systems for hot and cold water installations inside buildings</title>
+      ...
+    </bibitem>"
+----
+
+With `bibdata: true` option XML output wrapped with `bibdata` element and `ext`
+element added.
+
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.5.6">
+      <fetched>2025-12-15</fetched>
+      <title language="en" script="Latn" type="title-main">Multilayer piping systems for hot and cold water installations inside buildings</title>
+      ...
+      <ext schema-version="v1.0.1">
+        <doctype>international-standard</doctype>
+        <flavor>cen</flavor>
+        <ics>
+          <code>23.040.20</code>
+          <text>Plastics pipes</text>
+        </ics>
+        <ics>
+          <code>91.140.60</code>
+          <text>Water supply systems</text>
+        </ics>
+        <structuredidentifier>
+          <agency>CEN</agency>
+          <docnumber>21003</docnumber>
+          <partnumber>7</partnumber>
+        </structuredidentifier>
+      </ext>
+    </bibdata>"
+----
+
+=== Typed links
+
+Each CEN document has `src` type source.
+
+[source,ruby]
+----
+item.source
+=> [#<Relaton::Bib::Uri:0x0000000126a80878
+  @content="https://standards.cencenelec.eu/ords/f?p=CEN:110:::::FSP_PROJECT,FSP_ORG_ID:68120,6137&cs=1AD0C0DDDE4500D5B9BFB0BCE76BB63F9",
+  @language=nil,
+  @locale=nil,
+  @script=nil,
+  @type="src">]
+----
+
+=== Create bibliographic item form YAML
+
+[source,ruby]
+----
+ Relaton::Cen::Item.from_yaml File.read('spec/fixtures/item.yaml')
+=> #<Relaton::Cen::ItemData:0x00000001292f71d0
+...
+----
+
+=== Create bibliographic item from XML
+
+[source,ruby]
+----
+Relaton::Cen::Item.from_xml File.read("spec/fixtures/bibdata.xml")
+=> #<Relaton::Cen::ItemData:0x0000000129353fc0
+...
+----
+
+=== Logging
+
+RelatonCen uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton-cen.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/relaton/cie/CLAUDE.md

@@ -0,0 +1,90 @@
+# CLAUDE.md
+
+## Index format (pubid v2 — structured `_type:` index)
+
+CIE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`):
+each row's `:id` is a `Pubid::Cie::Identifier` object, serialized by `relaton-index`
+as a `_type: pubid:cie:…` map (mirrors the NIST/ETSI flavors).
+
+- **Producer** (`data_fetcher.rb`): `#index` passes `pubid_class:
+  ::Pubid::Cie::Identifier`; `#write_file` parses each docid via the `#pubid` helper
+  and indexes the **object** (not the raw string). `#pubid` returns `nil` — the id is
+  written to disk as a document but **left out of the index** — for any id pubid can't
+  parse **or** that doesn't round-trip. The guard mirrors the read side's acceptance
+  test (`Index::FileIO#id_supported?`, `from_hash(to_hash) == to_hash`): the loader
+  rejects the *whole* index on the first non-round-tripping row, so a bad id must be
+  dropped at write time rather than poison every lookup.
+- **Consumer** (`scrapper.rb`, `processor.rb`): both pass `pubid_class:
+  ::Pubid::Cie::Identifier` to `Index.find_or_create`. The scrapper parses the ref to a
+  pubid with `::Pubid::Cie.parse` — an unparseable ref **raises** `Pubid::Errors::ParseError`
+  (partial refs such as `CIE 001` parse; the old raw-String substring fallback is gone)
+  and passes **that** to `index.search` so index-v2 narrows candidates by number via
+  binary search before the block runs (a String arg would force a full O(n) scan — see
+  `Index::Type#search_candidates`); the block keeps the broad substring match, and
+  `min_by { |r| r[:id].to_s }` picks the winner (rows are pubid objects, not
+  `Comparable`, so compare by string form). The scrapper's index comes from the
+  data repo's Pages machine index (`PAGES_URL`, relaton#206) — manifest + the
+  shard for the parsed query, the whole-index zip on the same site serving only
+  the retry on a miss; `ENDPOINT` (the `v2` branch) still serves the document
+  files.
+
+### pubid dependency
+
+The published `relaton-data-cie/index-v2` uses CIE **proceedings** ids
+(`_type: pubid:cie:proceedings` with `paper`/`page_range`) and the **flattened**
+`to_hash` (`number`/`year` scalar, no nested `code`). That support first shipped in
+the released pubid `2.0.0.pre.alpha.13`, which `relaton.gemspec` pins — an older
+pubid strips `paper`/`page_range` and emits the nested shape, so `Relaton::Index`
+would reject the whole published index.
+
+Note the CIE grammar extensions **do** parse the techstreet variant ids like
+`CIE x051:2025/zcunvy` (as `Conference` `@variant` ids), so — unlike the old pubid —
+they are indexed, not dropped.
+
+## Crawler (`data_fetcher.rb`) — parallel detail fetch
+
+The Techstreet crawl (~1147 docs) is a headless-Chrome (Ferrum) scrape. It runs in
+**two phases**:
+
+- **Phase 1 — `collect_hits` (serial):** page through the search results following
+  `//a[@class="next_page"]`, collecting every `//li[@data-product]` into one list
+  (~12 page loads at `per_page=100`).
+- **Phase 2 — `process_hits` (parallel):** fan the hits out over a bounded pool.
+  **Each worker owns its own `BrowserAgent`** (its own Chrome, built via the
+  `build_agent` factory so the full UA/header + `navigator`/`window.chrome` stealth
+  masking runs per worker — Cloudflare evasion must hold per worker, not for a shared
+  singleton). Agents are built up front on the main thread (a Chrome-launch failure
+  aborts fast instead of hanging the bounded queue) and **every** agent is quit in an
+  `ensure`. The single memoized `#agent` is used only for Phase 1.
+
+**Concurrency knob:** `RELATON_CIE_CONCURRENCY` (default **5**, min 1), read by
+`.concurrency`. `relaton-data-cie`'s crawler workflow tunes it via env — no code change.
+
+**Per-worker adaptive pacing (`Pacing`)** replaces the old single global 4 s gap: each
+worker starts at `BASE_GAP` (~1 s) and **doubles its own gap up to `MAX_GAP`** only on
+trouble — a Cloudflare challenge or a Ferrum/socket error (`RETRIABLE_ERRORS`).
+`BrowserAgent#wait_for_challenge` raises a retriable `ChallengeError` when the challenge
+doesn't clear within `MAX_CHALLENGE_WAIT`, so a stuck worker backs off and retries
+(`#time_req` keeps the 4-try retry) instead of parsing the challenge HTML.
+
+**Thread-safety & byte-identical output:** the slow `agent.get` runs lock-free; the
+cheap build + `#write_file` run under one `@mutex`, so `@files`/`@errors`/`@seen` and the
+index mutate single-threaded. Output is order-independent by construction — `@errors`
+accumulates with `&&=` (AND, per-doc build is atomic under the lock), `@files` is a Set,
+and the structured index is sorted deterministically on save. For a duplicate output
+file, a `@seen[file] => catalogue-position` guard (`#superseded?`) keeps the
+**last-by-position** content winner the serial crawl would pick; `index.add_or_update`
+runs for **every** distinct id *before* that gate, so each id is still indexed. `index.save`
+is called **once**, after the pool drains.
+
+## Development
+
+- `bundle install` — install dependencies
+- `bundle exec rake spec` — run tests
+- `bundle exec rubocop` — lint
+
+## Testing
+
+- **Framework:** RSpec with VCR cassettes and WebMock
+- **Index fixture:** `spec/fixtures/index-v2.zip` (the published structured index) is loaded through a `Relaton::Index::Type` with `::Pubid::Cie::Identifier` and injected into the `Relaton::Index` pool in `before(:suite)` (see `spec/support/webmock.rb`, mirroring NIST/ETSI). Run `rake spec:update_index` to refresh from relaton-data-cie.
+- **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).

data/lib/relaton/cie/README.adoc

@@ -0,0 +1,121 @@
+= Relaton::Cie
+
+RelatonCie is a Ruby gem that searches and fetches standards from the https://www.techstreet.com/cie/searches/31156444[International Commision on Illuminantion].
+
+== Usage
+
+=== Search document
+
+[source,ruby]
+----
+require 'relaton/cie'
+=> true
+
+item = Relaton::Cie::Bibliography.get 'CIE 001-1980'
+[relaton-cie] INFO: (CIE 001-1980) Fetching from Relaton repository ...
+[relaton-cie] INFO: (CIE 001-1980) Found: `CIE 001-1980`
+=> #<Relaton::Bib::ItemData:0x0000000123b05300
+...
+
+# Return nil if a document doesn't exist.
+Relaton::Cie::Bibliography.get '1111'
+[relaton-cie] INFO: (1111) Fetching from Relaton repository ...
+[relaton-cie] INFO: (1111) Not found.
+=> nil
+----
+
+=== Serialization
+
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="CIE0011980" type="standard" schema-version="v1.5.6">
+      <fetched>2025-12-23</fetched>
+      <title type="title-main">Guidelines for minimizing urban sky glow near astronomical observatories (Joint Publication IAU/CIE)</title>
+      <title type="main">Guidelines for minimizing urban sky glow near astronomical observatories (Joint Publication IAU/CIE)</title>
+      <uri type="src">https://store.accuristech.com/standards/cie-001-1980?product_id=1210107</uri>
+      ...
+    </bibitem>"
+----
+
+With `bibdata: true` option XML output is wrapped with `bibdata` element and `ext` element added.
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.5.6">
+      <fetched>2025-12-23</fetched>
+      <title type="title-main">Guidelines for minimizing urban sky glow near astronomical observatories (Joint Publication IAU/CIE)</title>
+      <title type="main">Guidelines for minimizing urban sky glow near astronomical observatories (Joint Publication IAU/CIE)</title>
+      <uri type="src">https://store.accuristech.com/standards/cie-001-1980?product_id=1210107</uri>
+      ...
+      <ext schema-version="v1.0.0">
+        <doctype>document</doctype>
+        <flavor>cie</flavor>
+      </ext>
+    </bibdata>
+----
+ === Typed links
+
+Each CIE document has `src` type source link.
+
+[source,ruby]
+----
+item.source
+=> [#<Relaton::Bib::Uri:0x0000000123cc2990
+  @content="https://store.accuristech.com/standards/cie-001-1980?product_id=1210107",
+  @language=nil,
+  @locale=nil,
+  @script=nil,
+  @type="src">]
+----
+
+=== Parse a file locally
+
+[source,ruby]
+----
+item = Relaton::Cie::Item.from_xml File.read("spec/fixtures/bibdata.xml")
+=> #<Relaton::Cie::ItemData:0x000000010d00d0d0
+...
+----
+
+=== Fetch data
+
+This gem uses the https://www.techstreet.com/cie/searches/31156444 dataset as one of the data sources.
+
+The method `Relaton::Cie::DataFetcher.fetch(output: "data", format: "yaml")` fetches all the documents from the dataset and saves them to the `./data` folder in YAML format.
+Arguments:
+
+- `output` - folder to save documents (default './data').
+- `format` - the format in which the documents are saved. Possible formats are: `yaml`, `xml`, `bibxml` (default `yaml`).
+
+[source,ruby]
+----
+require "relaton/cie/data_fetcher"
+=> true
+
+Relaton::Cie::DataFetcher.fetch
+Started at: 2021-09-08 16:37:53 +0200
+Stopped at: 2021-09-08 16:49:17 +0200
+Done in: 684 sec.
+=> nil
+----
+
+=== Logging
+
+RelatonCie uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To update the index test fixture (used by tests), run `rake spec:update_index`. This downloads the latest `index-v2.zip` from the https://github.com/relaton/relaton-data-cie[relaton-data-cie] repository.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton-cie.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).