relaton 3.0.0.pre.alpha.17 → 3.0.0.pre.alpha.18

data/lib/relaton/bib/README.adoc

@@ -0,0 +1,330 @@
+= Relaton::Bib
+
+Relaton::Bib is a Ruby gem that implements the https://github.com/metanorma/relaton-models#bibliography-uml-models[BibliographicItem model].
+
+== Usage
+
+[source,ruby]
+----
+require 'relaton/bib'
+=> true
+----
+
+=== Create bibliographic item
+
+[source,ruby]
+----
+yaml = File.read 'spec/fixtures/item.yaml'
+ => "id: ISO1231994\ntype: standard\nschema_version: 1.0.0\nfetched: \"2022-05-02\"\nformattedref: ISO 123:1994\ntitle..."
+
+item = Relaton::Bib::Item.from_yaml yaml
+ => #<Relaton::Bib::ItemData:0x000000012838bfc0 ...
+----
+
+=== Title
+
+[source,ruby]
+----
+item.title
+ => [#<Relaton::Bib::Title:0x00000001270a6810
+  @content="Geographic information -- Metadata",
+  @format=nil,
+  @language="en",
+  @locale="en-US",
+  @script="Latn",
+  @type=nil>,
+ #<Relaton::Bib::Title:0x00000001270a5cd0
+  @content="Information géographique -- Métadonnées",
+  @format=nil,
+  @language="fr",
+  @locale="fr-FR",
+  @script="Latn",
+  @type=nil>]
+
+item.title "fr"
+ => [#<Relaton::Bib::Title:0x00000001270a5cd0
+  @content="Information géographique -- Métadonnées",
+  @format=nil,
+  @language="fr",
+  @locale="fr-FR",
+  @script="Latn",
+  @type=nil>]
+----
+
+=== Abstract
+
+[source,ruby]
+----
+item.abstract
+ => [#<Relaton::Bib::LocalizedMarkedUpString:0x0000000126c05748
+  @content="This is an abstract. <em>This is emphasized</em>",
+  @language="en",
+  @locale="en-US",
+  @script="Latn">]
+
+item.abstract("en")
+ => [#<Relaton::Bib::LocalizedMarkedUpString:0x0000000126c05748
+  @content="This is an abstract. <em>This is emphasized</em>",
+  @language="en",
+  @locale="en-US",
+  @script="Latn">]
+----
+
+=== Document identifier
+
+[source,ruby]
+----
+item.docidentifier
+ => [#<Relaton::Bib::Docidentifier:0x00000001270a4b00
+  @content="ISO 123:1994",
+  @language="en",
+  @locale="en-US",
+  @primary=true,
+  @scope="global",
+  @script="Latn",
+  @type="ISO">,
+ #<Relaton::Bib::Docidentifier:0x00000001270a46a0
+  @content="10.1007/978-3-319-99155-2_1",
+  @language=nil,
+  @locale=nil,
+  @primary=nil,
+  @scope=nil,
+  @script=nil,
+  @type="DOI">]
+----
+
+=== XML serialization
+
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="ISO1231994" type="standard" schema-version="1.0.0">
+      <fetched>2022-05-02</fetched>
+      <formattedref>ISO 123:1994</formattedref>
+      <title language="en" locale="en-US" script="Latn">Geographic information -- Metadata</title>
+      <title language="fr" locale="fr-FR" script="Latn">Information géographique -- Métadonnées</title>
+      <uri type="src">https://www.iso.org/standard/22722.html</uri>
+      <uri type="doi">10.1007/978-3-319-99155-2_1</uri>
+      ...
+    </bibitem>"
+----
+
+The default root element is `bibitem`. With argument `bibdata: true` the XML wrapped with `bibdata` element.
+
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="1.0.0">
+      <fetched>2022-05-02</fetched>
+      <formattedref>ISO 123:1994</formattedref>
+      <title language="en" locale="en-US" script="Latn">Geographic information -- Metadata</title>
+      ...
+      <ext schema-version="1.0.0">
+        <doctype abbreviation="std">Standard</doctype>
+        <subdoctype>Technical Report</subdoctype>
+        ...
+      </ext>
+    </bibdata>"
+----
+
+==== Adding notes
+
+[source,ruby]
+----
+item.to_xml note: [{ content: "Note", type: "note" }]
+=> "<bibitem id="ISO1231994" type="standard" schema-version="1.0.0">
+      ...
+      <note type="note">Note</note>
+      ...
+    </bibitem>"
+----
+
+=== Create bibliographic item form YAML
+
+[source,ruby]
+----
+yaml = File.read 'spec/fixtures/item.yaml'
+ => "id: ISO1231994\ntype: standard\nschema_version: 1.0.0\nfetched: \"2022-05-02\"\nformattedref: ISO 123:1994\ntitle..."
+
+Relaton::Bib::Item.from_yaml yaml
+ => #<Relaton::Bib::ItemData:0x000000011dc9e118 ...
+----
+
+=== Create bibliographic item from RFC XML
+
+[source,ruby]
+----
+xml = File.read 'spec/fixtures/rfc.xml'
+ => "<reference anchor=\"RFC1\" target=\"10.17487/RFC0001\">\n  <front>\n    <title>Title RFC1</title>\n    <seriesInf..."
+
+Relaton::Bib::Converter::BibXml.to_item xml
+ => #<Relaton::Bib::ItemData:0x000000011dc9fc98 ...
+----
+
+=== Create bibliographic item from XML
+[source,ruby]
+----
+xml = File.read 'spec/fixtures/bibitem.xml'
+=> "<bibitem id=\"ISO1231994\" type=\"standard\" schema-version=\"1.0.0\">\n  <fetched>2022-05-02</fetched>\n  <forma..."
+
+item = Relaton::Bib::Bibitem.from_xml xml
+ => #<Relaton::Bib::ItemData:0x000000011d4b7580 ...
+
+xml = File.read 'spec/fixtures/bibdata.xml'
+item = Relaton::Bib::Bibdata.from_xml xml
+ => #<Relaton::Bib::ItemData:0x000000011d4daa08 ...
+----
+
+=== Create bibliographic item from BibTeX (@TODO: add support for BibTeX)
+
+[source,ruby]
+----
+Relaton::Bib::BibtexParser.from_bibtex File.read('spec/fixtures/techreport.bib')
+=> {"ISOTC211"=>
+  #<Relaton::Bib::Item:0x007fedee0a2ab0
+  ...
+----
+
+=== Export bibliographic item to YAML
+
+[source,ruby]
+----
+item.to_yaml
+---
+type: standard
+schema_version: 1.0.0
+fetched: '2022-05-02'
+formattedref: ISO 123:1994
+...
+----
+
+=== Export bibliographic item to Hash
+
+[source,ruby]
+----
+item.to_h
+=> {"id"=>"ISO1231994",
+ "type"=>"standard",
+ "schema_version"=>"...",
+ "title"=>[...],
+ ...}
+----
+
+=== Export bibliographic item to BibTeX
+
+[source,ruby]
+----
+item.to_bibtex
+@misc{doe1994a,
+  title = {Geographic information -- Metadata},
+  author = {Doe, John},
+  edition = {First edition},
+  publisher = {International Organization for Standardization},
+  year = {1994},
+  month = {jan},
+  address = {Geneva, Switzerland, Geneva, Geneva},
+  pages = {1--10},
+  keywords = {information},
+  timestamp = {2022-05-02},
+  url = {https://www.iso.org/standard/22722.html},
+  doi = {10.1007/978-3-319-99155-2_1},
+  month_numeric = {1}
+}
+----
+
+=== Export bibliographic item to Citeproc (@TODO: add support for Citeproc)
+
+[source,ruby]
+----
+item.to_citeproc
+=> [{"title"=>"Geographic information",
+     "edition"=>"Edition 1",
+     "author"=>[{"family"=>"Bierman", "given"=>"A."}, {"family"=>"Bierman", "given"=>"Arnold"}, {"family"=>"Bierman", "given"=>"Arnold B"}],
+     "publisher"=>"Institute of Electrical and Electronics Engineers",
+     "publisher-place"=>"bib place",
+     ...
+----
+
+=== Exporting bibliographic item to AsciiBib (@TODO: add support for AsciiBib)
+
+[source,ruby]
+----
+item.to_asciibib
+=> [%bibitem]
+   == {blank}
+   id:: ISOTC211
+   fetched:: 2022-05-02
+   title::
+   title.type:: title-main
+   title.content:: Geographic information
+   title.format:: text/plain
+   ...
+----
+
+=== Export bibliographic item to RFC XML
+
+[source,ruby]
+----
+item.to_rfcxml
+<reference anchor="ISO 123:1994" target="https://www.iso.org/standard/22722.html">
+  <front>
+    <title>Geographic information -- Metadata</title>
+    <seriesInfo name="DOI" value="10.1007/978-3-319-99155-2_1"/>
+    <seriesInfo name="ISO 123" value="1"/>
+    <author initials="J.D." surname="Doe" fullname="Prof. John Doe PhD">
+      <organization>International Organization for Standardization</organization>
+      <address>
+        <postal>
+          <street>1, rue de Varembé</street>
+          <city>Geneva</city>
+          <code>1211</code>
+          <country>Switzerland</country>
+          <postalLine>1, rue de Varembé,&lt;br/&gt;Geneva, Switzerland, 1211</postalLine>
+        </postal>
+        <phone>+41 22 749 01 11</phone>
+        <email>jdoe@email.org</email>
+        <uri>https://orcid.org/0000-0002-1825-0097</uri>
+      </address>
+    </author>
+    <author>
+      <organization abbrev="ISO">International Organization for Standardization</organization>
+      <address>
+        <postal>
+          <street>1, rue de Varembé</street>
+          <city>Geneva</city>
+          <code>1211</code>
+          <country>Switzerland</country>
+          <postalLine>1, rue de Varembé,&lt;br/&gt;Geneva, Switzerland, 1211</postalLine>
+        </postal>
+        <phone>+41 22 749 01 11</phone>
+        <email>office@iso.org</email>
+        <uri>https://www.iso.org</uri>
+      </address>
+    </author>
+    <date day="01" month="January" year="1994"></date>
+    <workgroup>Geographic information</workgroup>
+    <keyword>information</keyword>
+    <abstract>
+      <t>This is an abstract. &lt;em&gt;This is emphasized&lt;/em&gt;</t>
+    </abstract>
+  </front>
+</reference>
+----
+
+=== Logging
+
+RelatonBib uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/relaton-bib.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/relaton/bib/model/series.rb

@@ -3,7 +3,7 @@ require_relative "type/string_date"
 module Relaton
   module Bib
     class Series < Lutaml::Model::Serializable
-      attribute :type, :string, values: %w[main alt]
+      attribute :type, :string, values: %w[main alt stream]
       attribute :formattedref, Formattedref
       attribute :title, Title, collection: (1..)
       attribute :place, Place

data/lib/relaton/bipm/CLAUDE.md

@@ -0,0 +1,173 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+relaton-bipm is a Ruby gem that retrieves BIPM (Bureau International des Poids et Mesures) standards metadata for bibliographic use. It's part of the larger Relaton family of gems that handle bibliographic data for standards organizations.
+
+## Common Commands
+
+- **Install dependencies:** `bundle install`
+- **Run all tests:** `bundle exec rake spec`
+- **Run a single test file:** `bundle exec rspec spec/relaton/bipm/bibliography_spec.rb`
+- **Run a single test by line:** `bundle exec rspec spec/relaton/bipm/bibliography_spec.rb:42`
+- **Interactive console:** `bin/console`
+
+## Architecture
+
+### Namespace & Module Structure
+
+All code lives under `Relaton::Bipm` (in `lib/relaton/bipm/`). The gem name is `relaton-bipm`, the require path is `relaton/bipm`.
+
+### Pubid-backed `index-v2` (the runtime index)
+
+The flavor's runtime index is the **pubid-backed `index-v2`** (`INDEXFILE =
+"index-v2"`), whose rows are `Pubid::Bipm` identifiers serialized as
+`_type: pubid:bipm:{committee-document,meeting,metrologia-article,si-brochure}`
+— mirroring the JCGM/OIML/JIS convention, not BIPM's old bespoke
+`{group,type,number,year}` hash. The `DataFetcher` builds it (`pubid_class:
+::Pubid::Bipm::Identifier`), and the consumer reads it.
+
+The **legacy bespoke `index-v1`** (the old `{group,type,number,year}` hash form)
+is no longer produced or read by this flavor. It is still published by
+`relaton-data-bipm`'s own `crawler.rb`, which builds it with the **retained**
+`Relaton::Bipm::Id` (this flavor keeps `Id` public for exactly that). So `Id` is
+now used in **one place only** — the crawler's `index-v1` generation. It was
+also this flavor's consumer query parser until pubid's grammar widened enough to
+replace it; see **Pubid-to-pubid lookup** below.
+
+### Key Components
+
+- **`Bibliography`** (`bibliography.rb`) - Main entry point. Parses the user
+  reference with `::Pubid::Bipm.parse` (`#parse_ref`; an unparseable ref raises
+  `Pubid::Errors::ParseError`, while the machine-derived `#year_number_retry`
+  probe rescues it to nil) and matches the pubid `index-v2` rows as
+  pubid objects — see **Pubid-to-pubid lookup** below for the stem and the two
+  escapes under it. The latest edition wins via
+  `max_by { |r| [r[:id].year.to_i, r[:file]] }` (the file path only breaks a
+  tie — Metrologia and SI Brochure rows carry no year, so all 6206 of them
+  score 0 and the index sort is not stable).
+- **`Id`** (`id_parser.rb`) - The flexible bespoke regex parser. Parses BIPM
+  reference strings into `{group,type,number,year,…}` hashes; the `TYPES` hash
+  maps full EN/FR type names to abbreviations, and `normalize_hash` applies
+  `CCDS→CCTF` and leading-zero stripping. **Retained solely** as the
+  `relaton-data-bipm` crawler's `index-v1` builder — it no longer parses
+  queries, and it is **not** used to write this flavor's `index-v2`.
+  `bibliography.rb` does not require it; `lib/relaton/bipm.rb` does, which is
+  what keeps `Relaton::Bipm::Id` reachable for the crawler. JCGM is not handled
+  here.
+- **`Processor`** (`processor.rb`) - Relaton framework integration point (extends
+  `Relaton::Core::Processor`). Registers prefix `BIPM` and default prefix pattern
+  matching BIPM, CCTF, CCDS, CGPM, CIPM, JCRB (JCGM was split out into the `jcgm`
+  flavor). `remove_index_file` targets `index-v2`.
+- **`DataFetcher`** (`data_fetcher.rb`) - Bulk fetches from three data sources:
+  `bipm-data-outcomes`, `bipm-si-brochure`, `rawdata-bipm-metrologia`. Delegates
+  to specialized parsers, which route every index write through the public,
+  **guarded** `#add_to_index(docnumber, path)` — it parses `docnumber` with
+  `::Pubid::Bipm.parse` and stores the identifier object, or **warns and skips**
+  an unparseable id rather than aborting the crawl (mirrors
+  `Relaton::Jcgm::DataFetcher#add_to_index`; `Relaton::Index` rejects a whole
+  index if one row fails to deserialize). Public so `relaton-data-bipm`'s crawler
+  can index its curated `static/` docs through the same guarded path.
+- **`Item`** / **`ItemData`** (`model/item.rb`, `item_data.rb`) - The
+  bibliographic item model, extending `Relaton::Bib::Item`. Supports XML, YAML,
+  and JSON serialization.
+- **`model/`** directory - Lutaml model classes (Bibdata, Bibitem, Ext, etc.) for
+  XML/YAML serialization.
+
+### Pubid-to-pubid lookup
+
+`#get_bipm` parses the reference with `::Pubid::Bipm.parse` and matches the
+index rows as pubid objects — the same shape ISO, OIML and JCGM use. The
+bespoke `Id` grammar takes **no part in a query**. That became possible with
+pubid `a96e9f45`, which accepts the loose consumer forms this flavor once
+needed `Id` for and *normalizes* them to BIPM's canonical spelling
+(`CCDS Recommendation 2 (2009)` → `CCTF Recommendation 2 (2009)`,
+`CIPM 111e Réunion (2022)` → `CIPM 111<sup>e</sup> réunion (2022)`).
+
+**The stem.** Committee-document and meeting rows are stored language- and
+form-neutral, while a reference may name a language (`(2009, E)`) and always
+names a form (short `CCTF REC 2` vs long `CCTF Recommendation 2`). So
+`#pubid_match?` compares `exclude(:language, :form)`, adding `:year` when the
+**query** carries none — `CIPM Meeting 43` has no year, its row has 1950. That
+last clause is what `Id#==` did with
+`other_hash.delete(:year) unless hash[:year]`; without it the lookup regresses.
+
+**Two escapes sit under the narrowing**, each for a case where the query's
+bsearch key cannot equal its row's:
+
+1. **A bare `SI Brochure` names no edition**, so it keys to `""` while its row
+   keys to `"9e"`. `#search_index` rescans the whole index. A brochure query
+   with a nil edition is a *partial* reference and deliberately matches any
+   brochure row — what the old `{group: "SI", type: "Brochure"}` projection
+   meant. `Part N` names a section inside the brochure, not a document, so it
+   stays out of the key exactly as `Id#==` kept `:part` out.
+2. **`CCTF Recommendation 2009-02` parses as the literal number `2009-02`**,
+   while the row keys on `2`. A rescan cannot help — no row carries that number
+   — so `#year_number_retry` re-reads a trailing `YYYY-NN` as number plus year.
+   It runs **only after a miss**, because `NNNN-NN` is also a real BIPM number
+   (`CIPM 2005-06`), so the literal reading must win.
+
+**One rule survives from `Id#==` and must not be dropped.** It treated a
+document numbered `"1"` and a number-less one as the same document when both
+carried a year, so `CIPM Resolution 1 (1879)` reached `CIPM RES (1879)`. pubid
+has no such rule, and the six ordinal-less declarations
+(`CGPM DECL (1889)`, `CIPM RES (1879)`, …) are the only rows it applies to — but
+they are addressable *both* ways, so removing it silently broke
+`CIPM Resolution 1 (1879)` and `CGPM Declaration 1 (1889)`. `#number_collapses?`
+restores it by excluding `:number` from that one comparison. Both forms are in
+the pinned baseline. Note the corpus has **no two rows** the rule could join
+(6 number-less against 83 numbered `"1"`, no shared group/type/year), so it only
+ever widens a query, never merges records.
+
+**Cost.** `#exclude` copies the identifier, so reducing per candidate row is
+expensive — roughly **165x** a plain attribute read. Two things keep the lookup
+at or below what the `Id` grammar cost: the query is reduced **once per search**,
+not once per row, and `#pubid_match?` rejects on `CHEAP_KEYS`
+(`number group year issue article`) before reducing anything. Those are exactly
+the attributes the stem never removes, so stem equality implies all of them and
+the rejects cannot change an answer — a spec asserts that over the corpus rather
+than trusting the argument. Without both, a bare-brochure rescan built ~7,900
+copies and took **2.3 s**; with them the same lookup is 2.8 ms and a narrowed
+one 0.4–0.5 ms, against 0.5–0.7 ms under `Id`.
+
+**The regression guard.** `spec/bipm/relaton/bipm/bibliography_spec.rb` pins,
+literally, the file each of the 29 references the suite exercises resolved to
+under the `Id` grammar, captured before it was removed. Any change to matching
+that moves a lookup fails there by name. That is the evidence for the swap —
+`Id#==` was fuzzier than the stem match in ways no spec pinned (it collapsed
+number `"1"` with nil, dropped `:part`/`:append`), so equivalence had to be
+demonstrated rather than argued.
+
+`Relaton::Bipm::Id` and `id_parser.rb` **stay** — `relaton-data-bipm`'s crawler
+builds `index-v1` with them, and `id_parser_spec.rb` still covers them. Only
+`bibliography.rb` stopped requiring the file.
+
+### Data Sources
+
+The gem fetches from three external datasets:
+
+1. **bipm-data-outcomes** - CGPM/CIPM/committee resolutions, recommendations, decisions (the `jcgm` meeting dir is now harvested by the `Relaton::Jcgm::DataFetcher` instead)
+2. **bipm-si-brochure** - SI Brochure documents
+3. **rawdata-bipm-metrologia** - Metrologia journal articles (parsed from CrossRef-style data)
+
+### Testing
+
+- **Index fixture:** `spec/bipm/fixtures/index-v2.zip` (the pubid `index-v2`) is
+  deserialized via `Pubid::Bipm::Identifier` and pre-loaded into the
+  `Relaton::Index` pool in `before(:suite)` (`spec/bipm/support/webmock.rb`,
+  mirroring the JCGM/JIS setup). It was regenerated from the real
+  `relaton-data-bipm` docs' docnumbers (each parsed with `Pubid::Bipm`); the
+  handful that pubid can't parse — JCGM orphans, `CIPM 2005-06`, docs removed
+  upstream — are absent by design.
+- **Parser specs** assert the parsers call `data_fetcher.add_to_index(docnumber,
+  path)` with the right docnumber string (the pubid parsing/guarding lives on the
+  mocked `DataFetcher`, keeping the parser tests format-agnostic).
+- Tests use RSpec with VCR cassettes (`spec/bipm/vcr_cassettes/`) and WebMock.
+
+### Dependencies
+
+Key runtime dependencies: `pubid` (provides `Pubid::Bipm`), `relaton-bib` (core
+bibliographic model), `relaton-index` (index management), `relaton-core`
+(framework base), `parslet`, `mechanize` (HTTP), `faraday`, `rubyzip`.