relaton 3.0.0.pre.alpha.17 → 3.0.0.pre.alpha.18

data/lib/relaton/index/README.adoc

@@ -0,0 +1,221 @@
+= Relaton::Index
+
+This gem is a part of [Relaton](https://github.com/relaton) project. It provides a way to index Relaton documents' files and search them by references.
+
+For the data-repository layout, the index file specification (schema, `:id` v1/v2 shapes), and the GitHub Pages publishing contract, see link:../../../docs/data-repository-format.adoc[docs/data-repository-format.adoc] at the repo root.
+
+== Usage
+
+=== Creating an index object
+
+The gem provides the `Relaton::Index.find_or_create {type}, url: {url}, file: {filename}, pubid_class: {class}` method to create an index object. The first argument is the type of dataset (ISO, IEC, IHO, etc.). `url` is the URL to the zipped remote index file, `filename` is the local index file, and `pubid_class` is a class that implements `Pubid::Core::Identifier` for deserializing ID hashes into structured identifier objects. All three keywords are optional.
+
+If the URL is specified and the local file in a `/{home}/.relaton/{type}` dir doesn't exist or is outdated, the index file will be downloaded from the URL saved as a local file and an index object will be created from the file. If the file in the `/{home}/.relaton/{type}` exists and is actual, the index object will be created from the local file.
+
+If the URL isn't specified, the index object will be created from the local file in the current dir `./{filename}`. If the file doesn't exist, the empty index object will be created.
+
+If the filename isn't specified, a default `index,yaml` filename will be used.
+
+If the keys are specified, the local index file will be checked for the presence of the keys. If the keys are not found, the index file will be downloaded from the URL or an empty index object will be created if the URL isn't specified.
+
+It's possible to create many index objects for different types of datasets. The index objects are saved in the pool. The pool is a hash with the type of dataset as a key and the index object as a value. The pool is a class variable of the `Relaton::Index` class. The pool is used to find an existing index object by the type of dataset. The `Relaton::Index.find_or_create` method returns an existing index object if it was created before.
+
+If the URL or filename is specified and has a new value, different from what was used to create the index object before, the index object will be recreated.
+
+[source,ruby]
+----
+require 'relaton/index'
+
+# Create a new index object. The first argument is the type of dataset (ISO, IEC, IHO, etc.) URL and filename are optional.
+Relaton::Index.find_or_create :IHO, url: "https://raw.githubusercontent.com/relaton/relaton-data-iho/master/index.zip", file: "index-iho.yaml"
+
+# Find an existing index object (created before).
+Relaton::Index.find_or_create :IHO
+
+# If the URL or filename is specified and has a new value (different from what it was before), the index object will be recreated.
+Relaton::Index.find_or_create :IHO, url: nil, file: "index.yaml"
+
+# Remove the index from the pool.
+Relaton::Index.close :IHO
+----
+
+The aim of this gem is to be used by Relaton libraries in two ways:
+- indexing documents' files in GitHub repositories
+- searching a document by a reference when the Relaton library fetches a document
+
+=== Indexing
+
+In this case, the Relaton library creates an index object and adds documents' files to it. By default, the index object is saved to the `index.yaml` file in the root of the repository. The filename can be changed using the `filename` setting.
+
+[source,ruby]
+----
+# Create a new index object or fetch an existing one. The first argument is the type of dataset (ISO, IEC, IHO, etc.) URL should not be specified.
+index = Relaton::Index.find_or_create :IHO
+
+# Add a document to the index or update it if it already exists.
+index.add_or_update "B-4 2.19.0", "data/b-4_2_19_0.xml"
+
+# Save the index to the `index.yaml` file in the current directory.
+index.save
+----
+
+=== Searching
+
+In this case, the Relaton library should create an index object and search for a document by reference. The gem looks for the `.relaton/[TYPE]/index.yaml` file in the user's home directory. The `[TYPE]` is one of downcased ISO, IEC, IHO, etc. If the file is not found or is older than 24 hours then it will be downloaded from the URL specified in the `find_or_create` method. The URL can be specified as `true` if the index should not be fetched from the URL.
+
+[source,ruby]
+----
+# Create a new index object or fetch an existing one. URL should be specified. If the index file is not found or is older than 24 hours, it will be downloaded from the URL. By default, the index is saved as `index.yaml` file to the `/[HOME]/.relaton/iho/` folder. If the URL is specified as `true`, the index won't be fetched from the URL.
+index = Relaton::Index.find_or_create :IHO, url: "https://raw.githubusercontent.com/relaton/relaton-data-iho/master/index.zip"
+
+# Search for a document by reference
+index.search "B-4"
+# => [{ id: "B-4 2.19.0", file: "data/b-4_2_19_0.xml" }]
+
+# Search for a document by reference using a block
+index.search do |row|
+  # do something with the index row
+  row[:id] == "B-4"
+end
+# => [{ id: "B-4 2.19.0", file: "data/b-4_2_19_0.xml" }]
+----
+
+A string reference matches a row as a substring, as above. An identifier
+reference takes pubid's asymmetric subset match `reference === row[:id]`: a
+component the reference leaves out matches any value, so `OGC 12-128` answers
+with the `12-128r19` row. Pass a block for exact equality:
+
+[source,ruby]
+----
+index.search(pubid) { |row| row[:id] == pubid }
+----
+
+=== Using pubid_class for structured identifiers
+
+The `pubid_class` option allows index entries to be deserialized into structured identifier objects instead of plain strings or hashes. The class must include `Pubid::Core::Identifier` and provide a `.create(**hash)` factory method.
+
+When `pubid_class` is specified, each `:id` hash in the index is converted into a pubid object, enabling structured access to identifier components (e.g., publisher, number, part, year).
+
+[source,ruby]
+----
+require 'relaton/index'
+require 'pubid-iso'
+
+# Create an index with pubid_class to deserialize IDs into Pubid::Iso::Identifier objects
+index = Relaton::Index.find_or_create :ISO,
+  url: "https://raw.githubusercontent.com/relaton/relaton-data-iso/main/index-v2.zip",
+  pubid_class: Pubid::Iso::Identifier
+
+# Search returns entries with structured pubid objects as IDs
+results = index.search "ISO 1"
+results.first[:id]
+# => #<Pubid::Iso::Identifier: ISO 1>
+results.first[:id].publisher
+# => "ISO"
+----
+
+=== Reading the machine index on a Pages site
+
+A data repository's GitHub Pages site publishes a sharded copy of its index
+(`index/manifest.json` and `index/shard-NNNNN.json`, see
+`docs/data-repository-format.adoc`). Give its base URL as `pages_url:` in place
+of `url:`, with a `pubid_class:`:
+
+[source,ruby]
+----
+index = Relaton::Index.find_or_create :W3C,
+  pages_url: "https://relaton.github.io/relaton-data-w3c/",
+  pubid_class: Pubid::W3c::Identifier
+index.search(Pubid::W3c::Identifier.parse("W3C REC-xml-names"))
+----
+
+A parsed query reads the manifest and the one shard
+`crc32(id.root.number.to_s) % shards`. A String query and a search with only a
+block read the whole index, which the manifest names. Nothing is written to
+disk: the manifest, the shards and the whole index stay in memory for 24 hours.
+
+* An absent shard (HTTP 404) is "not found", and the search returns `[]`.
+* A network error, or an HTTP status other than 200 and 404, raises
+  `Relaton::RequestError`.
+* A missing or unsupported manifest, or a row that the pubid class cannot read,
+  raises `Relaton::Index::Error`.
+
+The manifest also carries `sha256` and `bytes` for the whole-index zip it
+names, so a dataset can be pinned and the fetched artifact verified.
+`Relaton::Index::DatasetLock.capture(pages_url)` writes that pin as a lockfile
+and verifies the artifact later; see `docs/data-repository-format.adoc`.
+
+There is no fallback to the `url:` zip.
+
+=== Family, edition and part views
+
+`Type#family(reference)` returns the rows that share the reference's root
+number — the base document, its editions, its parts, and any supplement
+layer — as a `Relaton::Index::Family`:
+
+[source,ruby]
+----
+family = index.family("ISO 1")
+family.members  # every row in the family
+family.editions # {"2015" => [...], "2020" => [...]}
+family.parts    # {"1" => [...]}
+----
+
+A String reference is parsed by the index's `pubid_class`; a parsed identifier
+is used as given. Narrowing uses the same number bucket as `search`, so a large
+index is not scanned in full.
+
+=== Remove all index records
+
+This method removes all records from the index object. The index file is not removed.
+
+[source,ruby]
+----
+index.remove_all
+----
+
+=== Remove index file
+
+This method removes the index file. The index object is cleared and can be used for indexing or loading a file from the specified URL.
+
+[source,ruby]
+----
+index.remove_file
+----
+
+=== Configuration
+
+The gem can be configured by using the `Relaton::Index.config` method. The following settings are available:
+
+- `filename` - the name of the index file. By default, it is `index.yaml`.
+- `storage` - the storage class or module. By default, it is `FileStorage` module. It can be any class or module that implements the `ctime`, `read`, and `write` class methods.
+- `storage_dir` - the directory where the `.relaton/[TYPE]` folder is created. By default, it's a user's home dir. This setting works only when the gem is used for searching. When indexing, the index is always saved in a current folder.
+
+[source,ruby]
+----
+Relaton::Index.config do |config|
+  config.filename = "index-v1.yaml"
+  config.storage = S3Storage
+  config.storage_dir = "/"
+end
+----
+
+It's also possible to redefine file name for a specific type of index:
+[source,ruby]
+----
+index = Relaton::Index.find_or_create :IHO, file: "index-v2.yaml"
+----
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton-index.
+
+== License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/relaton/index/dataset_lock.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+require "digest"
+require "json"
+require "net/http"
+require "time"
+require "uri"
+require "lutaml/model"
+
+module Relaton
+  module Index
+    # A pinned dataset: the machine-index manifest's content address written to
+    # a file, so a build can reproduce the exact index it consumed years later.
+    #
+    # The manifest (docs/data-repository-format.adoc) carries the SHA-256 and
+    # size of the published whole-index zip. Capture records them; #verify
+    # re-checks the bytes a consumer fetched.
+    #
+    #   lock = Relaton::Index::DatasetLock.capture(
+    #     "https://relaton.github.io/relaton-data-iso/",
+    #   )
+    #   File.write(".relaton-lock.yaml", lock.to_yaml)
+    #
+    #   lock = Relaton::Index::DatasetLock.from_yaml(File.read(".relaton-lock.yaml"))
+    #   lock.verify(File.binread(zip_path))
+    class DatasetLock < Lutaml::Model::Serializable
+      attribute :version, :integer
+      attribute :pages_url, :string
+      attribute :index, :string
+      attribute :count, :integer
+      attribute :shards, :integer
+      attribute :generated, :string
+      attribute :sha256, :string
+      attribute :bytes, :integer
+      attribute :captured_at, :string
+
+      SHA256_HEX = /\A\h{64}\z/
+
+      # The manifest names the artifact; these fields pin it. A manifest
+      # published before content addressing has no sha256 and cannot be
+      # locked: refusing is the point, a lock without a hash guarantees
+      # nothing.
+      def self.from_manifest(pages_url, manifest, captured_at: Time.now.utc.iso8601)
+        unless manifest.is_a?(Hash)
+          raise Error, "manifest is not a JSON object: #{manifest.class}"
+        end
+
+        sha = manifest["sha256"]
+        unless sha.is_a?(String) && SHA256_HEX.match?(sha)
+          raise Error,
+                "manifest for #{pages_url} carries no sha256 content address; " \
+                "republish the index with a generator that emits one"
+        end
+
+        index = manifest["index"]
+        unless index.is_a?(String) && !index.empty?
+          raise Error, "manifest for #{pages_url} names no index artifact"
+        end
+
+        new(
+          version: 1,
+          pages_url: pages_url,
+          index: index,
+          count: manifest["count"],
+          shards: manifest["shards"],
+          generated: manifest["generated"],
+          sha256: sha,
+          bytes: manifest["bytes"],
+          captured_at: captured_at,
+        )
+      end
+
+      # Fetch the site's manifest and return the lock for it. +fetcher+ is the
+      # injection point for tests and alternate transports; it must answer
+      # (pages_url, **timeouts) with the parsed manifest.
+      def self.capture(pages_url, fetcher: nil, open_timeout: 15, read_timeout: 30)
+        fetch = fetcher || method(:fetch_manifest)
+        from_manifest(
+          pages_url,
+          fetch.call(pages_url, open_timeout: open_timeout, read_timeout: read_timeout),
+        )
+      end
+
+      def self.fetch_manifest(pages_url, open_timeout: 15, read_timeout: 30)
+        base = pages_url.to_s
+        base = "#{base}/" unless base.end_with?("/")
+        uri = URI.join(base, "index/manifest.json")
+        res = Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https",
+                                                  open_timeout:, read_timeout:) do |http|
+          http.get(uri.request_uri)
+        end
+        raise Error, "cannot fetch #{uri}: HTTP #{res.code}" unless res.code == "200"
+
+        JSON.parse(res.body)
+      rescue JSON::ParserError => e
+        raise Error, "malformed manifest at #{uri}: #{e.message}"
+      end
+
+      # The published whole-index zip this lock addresses.
+      def zip_url
+        "#{pages_url.to_s.chomp('/')}/#{index}.zip"
+      end
+
+      # Raise unless +data+ is byte-for-byte the artifact this lock pins.
+      # Returns true so it reads as an assertion in build scripts.
+      def verify(data)
+        actual = Digest::SHA256.hexdigest(data)
+        if actual != sha256
+          raise Error,
+                "dataset hash mismatch for #{zip_url}: expected #{sha256}, got #{actual}"
+        end
+        if bytes && data.bytesize != bytes
+          raise Error,
+                "dataset size mismatch for #{zip_url}: expected #{bytes} bytes, " \
+                "got #{data.bytesize}"
+        end
+
+        true
+      end
+    end
+  end
+end

data/lib/relaton/index/family.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+module Relaton
+  module Index
+    # A document's family: every index row whose root number matches the
+    # reference's — the base document, its editions, parts, and any supplement
+    # layer (root walks a wrapper's base chain). Rows are the same
+    # `{ id:, file: }` hashes #search returns.
+    #
+    # The views are driven by the components the identifiers actually state:
+    # #editions groups rows by year/edition/version, #parts by part/parts.
+    # An identifier that states none of those appears only in #members.
+    class Family
+      attr_reader :reference, :members
+
+      def initialize(reference, members)
+        @reference = reference
+        @members = members
+      end
+
+      # @return [String] the family's narrowing key
+      def root_number
+        reference.root.number.to_s
+      end
+
+      # Rows grouped by their stated edition, ascending.
+      # @return [Hash{String => Array<Hash>}]
+      def editions
+        group_by(%w[year edition version])
+      end
+
+      # Rows grouped by their stated part, ascending.
+      # @return [Hash{String => Array<Hash>}]
+      def parts
+        group_by(%w[part parts subpart])
+      end
+
+      private
+
+      def group_by(names)
+        grouped = Hash.new { |hash, key| hash[key] = [] }
+        members.each do |row|
+          value = component(row[:id], names)
+          grouped[value] << row unless value.empty?
+        end
+        grouped.sort_by { |key, _| key }.to_h
+      end
+
+      def component(id, names)
+        hash = id.root.to_hash
+        names.each do |name|
+          value = hash[name]
+          next if value.nil?
+
+          value = value.first if value.is_a?(Array)
+          text = value.to_s
+          return text unless text.strip.empty?
+        end
+        ""
+      end
+    end
+  end
+end

data/lib/relaton/index/type.rb

@@ -130,6 +130,27 @@ module Relaton
         items.select { |i| match_item(i, id) }
       end
 
+      #
+      # The reference's family: rows sharing its root number — the base
+      # document, its editions and parts, and any supplement layer (root walks
+      # a wrapper's base chain). Narrowing uses the same number bucket as
+      # #search, so a large index is not scanned in full.
+      #
+      # @param [String, Pubid::Identifier] reference
+      # @return [Relaton::Index::Family]
+      #
+      def family(reference)
+        ref = coerce_reference(reference)
+        root = ref.root
+        members = search_candidates(ref).select do |row|
+          id = row[:id]
+          id.is_a?(::Pubid::Identifier) &&
+            id.root.class == root.class &&
+            id.root.number.to_s == root.number.to_s
+        end
+        Family.new(ref, members)
+      end
+
       #
       # Save index to storage
       #
@@ -223,6 +244,17 @@ module Relaton
         ShardSource.new(@dir, @pages_url, @pubid_class) if @pages_url
       end
 
+      # A String query is parsed by this index's flavor; a parsed identifier is
+      # used as given. A pubid-less index cannot answer a family query.
+      def coerce_reference(reference)
+        return reference unless reference.is_a?(String)
+
+        klass = @file_io.pubid_class
+        raise ArgumentError, "a String reference needs a pubid-backed index" unless klass
+
+        klass.parse(reference)
+      end
+
       def search_candidates(id)
         # A parsed query reads only its own shard. `@index`, once loaded, is
         # never refreshed, so the shards stay the fresher answer.

data/lib/relaton/index.rb

@@ -18,6 +18,9 @@ require_relative "index/type"
 
 module Relaton
   module Index
+    autoload :DatasetLock, "relaton/index/dataset_lock"
+    autoload :Family, "relaton/index/family"
+
     class Error < StandardError; end
 
     class << self

data/lib/relaton/isbn/CLAUDE.md

@@ -0,0 +1,47 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this gem does
+
+Retrieves bibliographic items from the OpenLibrary API by ISBN (10 or 13 digit) and returns `Relaton::Bib::ItemData` objects. Part of the Relaton family of gems.
+
+## Commands
+
+```sh
+bundle exec rspec                          # run all tests
+bundle exec rspec spec/relaton/isbn/parser_spec.rb  # run single spec file
+bundle exec rubocop                        # lint
+bin/console                                # IRB with gem loaded
+```
+
+## Architecture
+
+Request flow: `OpenLibrary.get(code)` → `Isbn.new(code).parse` (validates/normalizes ISBN) → `OpenLibrary.request_api(isbn)` (HTTP to openlibrary.org) → `Parser.parse(json)` (builds `ItemData`)
+
+`get` also takes a parsed `Pubid::Isbn::Identifier`: its digits (`raw`, 10 or 13) go through the same `Isbn#parse`.
+
+Key modules under `Relaton::Isbn`:
+- **OpenLibrary** — API client, main entry point via `.get`
+- **Isbn** — ISBN-10/13 validation and conversion (always normalizes to ISBN-13)
+- **Parser** — Transforms OpenLibrary JSON into `Relaton::Bib::ItemData` with titles, contributors, dates, etc.
+- **Processor** — Integration hook for relaton-core framework (`from_xml`, `from_yaml`, `get`)
+- **Util** — Logging via `Relaton::Bib::Util`
+
+## Data model (relaton-bib 2.0)
+
+Uses `Relaton::Bib::` namespace (not the old `RelatonBib::`). Key classes: `ItemData`, `Title`, `Docidentifier` (uses `.content` not `.id`), `Uri`, `Contributor`, `Person`, `Organization`, `Date` (uses `at:` not `on:`), `Place`.
+
+`Parser#parse` returns `Bib::ItemData` (not `Bibitem`) — `ItemData` is the plain data class; `Bibitem`/`Item` are lutaml-model serializers that wrap it.
+
+## Testing
+
+- **VCR** records HTTP interactions in `spec/vcr_cassettes/` (re-records every 7 days)
+- **WebMock** disables real HTTP by default
+- **SimpleCov** tracks coverage
+- Fixtures in `spec/fixtures/`
+
+## Style
+
+- RuboCop with Ribose OSS style guide, target Ruby 3.1
+- `rubocop-rails` is required but Rails cops are disabled (not a Rails app)

data/lib/relaton/isbn/README.adoc

@@ -0,0 +1,91 @@
+= Relaton::ISBN: retrieves bibliographic items using OpenLibrary API
+
+Relaton::Isbn is a Ruby gem that implements the
+https://github.com/metanorma/metanorma-model-iso#iso-bibliographic-item[IsoBibliographicItem model].
+
+You can use it to retrieve metadata of Standards from https://openlibrary.org, and
+access such metadata through the `Bibitem` object.
+
+== Usage
+
+=== Retrieving bibliographic items using OpenLibrary API
+
+To retrieve bibliographic items, use `Relaton::Isbn::OpenLibrary.get` method with ISBN-10 or ISBN-13 as an argument. Allowed prefixes are `ISBN`, `isbn:`. Prefix and hyphens are optional. The method returns `Relaton::Bib::Bibitem` object.
+
+[source,ruby]
+----
+require 'relaton/isbn'
+=> true
+
+# get document by ISBN-13
+bibitem = Relaton::Isbn::OpenLibrary.get "ISBN 978-0-12-064481-0"
+[relaton-isbn] INFO: (ISBN 978-0-12-064481-0) Fetching from OpenLibrary ...
+[relaton-isbn] INFO: (ISBN 978-0-12-064481-0) Found: `9780120644810`
+=> #<Relaton::Bib::Bibitem:0x0000000127ed25d8
+...
+
+# get document by ISBN-10
+Relaton::Isbn::OpenLibrary.get "isbn:0120644819"
+[relaton-isbn] INFO: (isbn:0120644819) Fetching from OpenLibrary ...
+[relaton-isbn] INFO: (isbn:0120644819) Found: `9780120644810`
+=> #<Relaton::Bib::Bibitem:0x0000000127f11698
+...
+----
+
+=== Serializing bibliographic items
+[source,ruby]
+----
+# serialize to XML
+bibitem.to_xml
+=> "<bibitem id="9780120644810" schema-version="v1.4.1">
+      <fetched>2026-03-04</fetched>
+      <title type="main">Graphics gems II</title>
+      <uri type="src">http://openlibrary.org/books/OL21119585M/Graphics_gems_II</uri>
+      <docidentifier type="ISBN" primary="true">9780120644810</docidentifier>
+      ...
+    </bibitem>"
+
+# serialize to bibdata XML
+bibitem.to_xml bibdata: true
+=> "<bibdata schema-version="v1.4.1">
+      <fetched>2026-03-04</fetched>
+      <title type="main">Graphics gems II</title>
+      <uri type="src">http://openlibrary.org/books/OL21119585M/Graphics_gems_II</uri>
+      <docidentifier type="ISBN" primary="true">9780120644810</docidentifier>
+      ...
+      <ext>
+        <flavor>isbn</flavor>
+      </ext>
+    </bibdata>"
+
+# serialize to hash
+puts bibitem.to_yaml
+=> "---
+    id: '9780120644810'
+    schema_version: v1.4.1
+    ..."
+----
+
+=== Logging
+
+Relaton::Isbn uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run
+`rake spec` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To
+release a new version, update the version number in `version.rb`, and then run
+`bundle exec rake release`, which will create a git tag for the version, push
+git commits and the created tag, and push the `.gem` file to
+https://rubygems.org[rubygems.org].
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/relaton/relaton-isbn.
+
+== License
+
+The gem is available as open source under the terms of the https://opensource.org/licenses/MIT[MIT License].

data/lib/relaton/iso/CLAUDE.md

@@ -0,0 +1,38 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development
+
+- `bin/setup` — install dependencies
+- `rake spec` — run tests
+- `rspec spec/relaton/iso/bibliography_spec.rb` — run a single test file
+- `rspec spec/relaton/iso/bibliography_spec.rb -e "some description"` — run a specific example
+- `rake spec:update_index` — download latest ISO index fixture (`spec/fixtures/index-v1.zip`) from relaton-data-iso
+- `bin/console` — interactive prompt with the gem loaded
+- `rubocop` — lint (Ribose OSS style guide, Ruby 3.2 target)
+
+## Architecture
+
+relaton-iso retrieves ISO standard bibliographic data. The core retrieval flow:
+
+1. **Bibliography** (`lib/relaton/iso/bibliography.rb`) — entry point via `search(pubid)` and `get(ref, year, opts)`. Handles year filtering, part matching, and type/stage validation. Both **propagate** `Pubid::Errors::ParseError` when the caller's reference can't be parsed by pubid (they do not swallow it — the CLI turns it into a friendly message; API consumers rescue it themselves). Note `look_up_with_any_types_stages` deliberately rescues `Pubid::Errors::ParseError` internally: it re-parses a *machine-derived* type/stage-stripped probe, so a failure there just means "no alternative-type/stage candidates" and must not abort a lookup whose original reference already parsed.
+2. **HitCollection** (`lib/relaton/iso/hit_collection.rb`) — searches a pre-built YAML index (`index-v1.zip` from relaton-data-iso) using `Relaton::Index`. Matches ids through the parsed pubid (`reference === row`, relaton#109). Returns sorted Hit array.
+3. **Hit** (`lib/relaton/iso/hit.rb`) — wraps an index result. The `item` attribute lazy-loads the full document from GitHub raw content (relaton-data-iso repo). `sort_weight` prioritizes published over withdrawn/deleted.
+4. **ItemData** / **Model::Item** — ISO-specific bibliographic item extending `Relaton::Bib::ItemData`.
+5. **Scraper** (`lib/relaton/iso/scraper.rb`) — parses individual ISO website pages. Used only by `Bibliography.get` as a fallback when an item is missing from the curated index; no longer drives bulk ingest.
+6. **DataFetcher** (`lib/relaton/iso/data_fetcher.rb`) — streams the ISO Open Data programme JSONL feeds (`iso_deliverables_metadata.jsonl` for documents, `iso_technical_committees.jsonl` for committee titles) and writes one YAML per primary docid into `@output`. Short-circuits on upstream `Last-Modified`; falls back to a full pass when `data/` or `index-v1.yaml` is missing. Two source modes:
+   - `iso-open-data` (default) — incremental, skip when upstream is unchanged.
+   - `iso-open-data-all` — wipe `@output` and re-emit every record.
+
+   File names come from `Core::DataFetcher#unique_output_file`, not the plain `output_file`: `output_file` is not injective (`ISO/DIS 15869.3` and `ISO/DIS 15869-3` both sanitize to `iso-dis-15869-3.yaml`), so a second distinct docid gets a digest-suffixed path. `rewrite_with_same_or_newer` (edition / substage-98) therefore only ever compares two records of the **same** docid.
+7. **DataParser** (`lib/relaton/iso/data_parser.rb`) — converts one Open Data record (`Hash`) into a `Relaton::Iso::ItemData`. Takes a `ref_index` (id → reference) for resolving `replaces`/`replacedBy` and a `tc_index` (reference → `{ "en"/"fr" => title }`) for resolving committee labels.
+
+Key dependency: `pubid-iso` gem handles ISO publication identifier parsing and comparison.
+
+## Testing
+
+- **Framework:** RSpec with VCR cassettes and WebMock
+- **Network access:** fully blocked via `WebMock.disable_net_connect!`
+- **Index fixture:** `spec/fixtures/index-v1.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iso.
+- **VCR:** cassettes in `spec/vcr_cassettes/`, record mode `:once`, re-record interval 7 days. Index download requests are ignored by VCR (handled by fixture instead).