relaton 3.0.0.pre.alpha.16 → 3.0.0.pre.alpha.18

data/lib/relaton/xsf/CLAUDE.md

@@ -0,0 +1,189 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What is relaton-xsf?
+
+A Ruby gem for bibliographic retrieval of XMPP XEP (XMPP Extension Protocol) specifications. Part of the Relaton family of gems. Fetches data from https://xmpp.org/extensions/refs/ and the relaton-data-xsf GitHub repository.
+
+## Commands
+
+- `bundle exec rake spec:xsf` — run this flavor's suite (from the repo root)
+- `bundle exec rake spec:update_index_xsf` — refresh
+  `spec/xsf/fixtures/index-v2.zip` from the published index
+- `bundle exec rubocop` — lint
+- `bundle exec rubocop -a` — lint with auto-fix
+- `bin/console` — interactive console with gem loaded
+
+## Architecture
+
+Namespace: `Relaton::Xsf` (under `lib/relaton/xsf/`). Branch `lutaml-integration` uses the new nested namespace (not the old `RelatonXsf`).
+
+Key classes and their base classes from relaton-core:
+
+| Class | Base | Role |
+|---|---|---|
+| `Processor` | `Relaton::Core::Processor` | Plugin entry point for relaton registry |
+| `Bibliography` | Module (extends self) | Search & get interface; **parses the reference into a pubid** (`parse_ref`) |
+| `HitCollection` | `Relaton::Core::HitCollection` | Collection of search results |
+| `Hit` | `Relaton::Core::Hit` | Single result; lazy-loads YAML from GitHub |
+| `DataFetcher` | `Relaton::Core::DataFetcher` | Crawls xmpp.org, parses BibXML, saves docs |
+| `Docidentifier` | `Relaton::Bib::Docidentifier` | Exposes the id as a `Pubid::Xsf::Identifier` via `#pubid` |
+| `Item` / `Bibitem` / `Bibdata` | `Relaton::Bib::Item` | Bibliographic item models (lutaml-model based) |
+
+## Index (`index-v2`, pubid-keyed)
+
+`INDEXFILE` is the pubid-backed `index-v2`: rows are `Pubid::Xsf::Identifier`
+hashes (`_type: pubid:xsf:xep` with `number`). The consumer
+(`HitCollection#index`) and the producer (`DataFetcher#index`) pass
+`pubid_class: ::Pubid::Xsf::Identifier`. Omitting it on the **producer**
+writes v1-shaped rows under a v2 name, silently (`FileIO#save` calls `to_hash`
+only for instances of `pubid_class`); omitting it on the **consumer** leaves
+the rows raw hashes with `FileIO#sorted` false, so every lookup scans all 520.
+`Processor#remove_index_file` passes `url: true` and `file:` only: the delete
+never reads the index (see `lib/relaton/index/CLAUDE.md`).
+
+An XEP identifier is a publisher and a number, nothing else — no edition, no
+date, no part, and one row per XEP (520 rows, ids unique). That makes this the
+simplest flavor in the sweep: there is no selection order to preserve, and the
+reference states every component it has, so pubid's asymmetric subset match
+(`ref === row[:id]`, `Pubid::SubsetMatch`) is exact equality after narrowing.
+
+### The index guard, and the two rows that are pages rather than XEPs
+
+`DataFetcher#add_to_index` parses the docid and, on failure, records
+`@errors[docid]` — which `report_errors` turns into a GitHub issue at the end of
+the crawl (the 3GPP/ECMA precedent) — and skips the row. The data file is still
+written, so a document that cannot be indexed is unindexed, never lost.
+
+The guard is not decoration. One unparseable row does not fail that row:
+`Relaton::Index` declares the **whole file** corrupt, deletes it, and hands back
+an **empty** index — measured, 519 good rows plus one unparseable id loads as
+**0 rows**, and the only trace is two INFO lines (`Wrong structure of file …`,
+`Considering … corrupt, removing it`). Every XSF lookup then fails silently.
+
+Two published rows are pages rather than XEPs — the XMPP repository's `README`
+and its `xep-xxxx` template, which the crawl mints records for because they sit
+in the same listing. pubid accepts them as the literal numbers `README` and
+`xxxx`, so they carry into `index-v2` like any other row and resolve normally.
+It stayed strict about everything else: `XEP banana` and a typo such as
+`XEP 00O1` are still rejected, which is what keeps an unparseable reference a
+warning rather than a silent miss.
+
+Nothing in the published corpus trips the guard today, so a recorded error means
+upstream grew a shape pubid does not know — exactly when an issue is worth
+filing.
+
+### Where the reference becomes an identifier
+
+`Bibliography.parse_ref` does it, not `HitCollection`. That split is not
+cosmetic: `Type#search_candidates` narrows only when its search argument is
+**not** a `String`, so the string has to become an identifier at the entry
+point. `HitCollection` therefore receives a `Pubid::Xsf::Identifier` (or nil,
+when the reference could not be parsed) and does nothing but the index lookup.
+`Core::HitCollection` documents `ref` as `[String, Pubid]`, so this is the
+interface working as intended.
+
+### Pubid-backed docidentifier
+
+`Relaton::Xsf::Docidentifier` parses its `content` into a
+`Pubid::Xsf::Identifier` kept in `@pubid`, while the lutaml `content` attribute
+stays a plain string. Parsing is soft — a missing gem or non-XSF content leaves
+`@pubid` nil rather than raising during deserialization. `Item` narrows the
+inherited `docidentifier` attribute to it.
+
+It is **purely additive**: `to_yaml` and `to_xml` are byte-identical before and
+after, so no published document or index needs regenerating.
+
+**It deliberately implements none of `remove_part!` / `remove_date!` /
+`to_all_parts!`.** An XEP identifier is a publisher and a number and nothing
+else, so there is genuinely nothing for any of them to strip, and
+`Bib::Docidentifier` defaults all three to no-ops for exactly that case. Empty
+overrides would assert a flavor rule that does not exist. Compare
+`Relaton::Ogc::Docidentifier`, which overrides `remove_date!` because OGC really
+does carry a revision.
+
+### Lookup
+
+`HitCollection#search` follows the ETSI/W3C/OGC idiom:
+
+- **Pass the pubid, not the string.** `Type#search_candidates` narrows only
+  when the argument is not a `String`, so the plain reference this flavor used
+  to pass disabled the binary search however the index was built.
+  `pubid_class:` alone fixes nothing; both had to change together.
+- **Matching is exact.** The old `index.search(ref)` compared a **substring**
+  of the rendered id, so a bare `001` answered with **11** documents
+  (`XEP 0001` and every `XEP 001x`) and `Bibliography#get` took `.first` —
+  a truncated reference silently resolved to whichever sorted first. It now
+  returns nothing.
+- **An unrecognized reference raises.** Like ISO, ETSI and 3GPP, the parse
+  error propagates: relaton-cli rescues `Pubid::Errors::Error` and renders
+  `"..." is not a recognized standards identifier`
+  (`gems/relaton-cli/lib/relaton/cli/command.rb:324`,
+  `subcommand_collection.rb:134`), and `Db#fetch` logs it through the
+  `StandardError` arm at `lib/relaton/db.rb:122`. `Pubid::Errors::ParseError`
+  includes the marker module `Pubid::Errors::Error`, which is what makes the
+  CLI's rescue work for every flavor.
+
+  Rescuing into a warning and an empty result — which several flavors still do
+  — collapses "this identifier is malformed" into "no such document", and a
+  caller cannot tell them apart. The parse also happens **before**
+  `HitCollection` is constructed, so it is never relabelled as the
+  `Relaton::RequestError` that collection's own rescue raises.
+- **Three reference forms are normalized** in `Bibliography#normalize_ref`, because
+  `Pubid::Xsf` accepts only the canonical `XEP 0001`:
+
+  | form | before | after |
+  |---|---|---|
+  | `XEP 0001` | resolves | resolves |
+  | `0001` (bare) | resolves (substring) | resolves — the token is added |
+  | `XEP-0001` | **no match** | resolves — the spelling xmpp.org itself uses |
+  | `xep 0001` | no match | resolves — the token match is case-insensitive |
+
+  The hyphen and case forms are new support, not preserved behaviour: the
+  substring match compared against `XEP 0001`, which has a space.
+
+### Both indexes, one crawl
+
+`relaton-data-xsf` publishes **both** from a single crawl: the `index-v2` this
+fetcher writes, and its own `index-v1` built by `build_index_v1.rb` for released
+relaton v2 consumers. Both carry the same 520 rows — building v1 from `data/`
+was originally what kept `XEP README` and `XEP xxxx` available there while they
+were unindexable in v2, and pubid accepting them has since removed that
+asymmetry.
+
+Verified against the published `index-v2`: 520 rows, all deserialize, sorted,
+**0** keying on `""`, `XEP 0001` narrows to 1 of 520, and every accepted
+spelling resolves to a file that returns HTTP 200.
+
+Data flow: `Processor#get` → `Bibliography.get` → `HitCollection.search` → `Hit#item` → fetches YAML → `Relaton::Bib::Item.from_yaml`
+
+DataFetcher flow: Crawls `https://xmpp.org/extensions/refs/`, parses each XML ref via `Relaton::Bib::Converter::BibXml.to_item`, sets `ext.flavor = "xsf"`, saves to disk.
+
+Constants: `INDEXFILE = "index-v2"`, `GHDATA_URL` points to relaton-data-xsf `v2` branch.
+
+## Testing
+
+- **Index fixture:** `spec/xsf/fixtures/index-v2.zip` is seeded into the
+  `Relaton::Index` pool by `spec/xsf/support/webmock.rb` — see **Index
+  (`index-v2`, pubid-keyed)** above for why it is re-seeded per example and
+  built with `pubid_class:`. Refresh it with `rake spec:update_index_xsf`.
+- RSpec with VCR cassettes (`spec/vcr_cassettes/`) for HTTP interactions
+- WebMock disables all external network connections
+- Fixtures in `spec/fixtures/` (item.yaml, bibdata.xml, bibitem.xml)
+- Round-trip tests verify YAML→Item→YAML and XML→Item→XML fidelity
+- `DataFetcher` is lazily required — specs that test it must `require "relaton/xsf/data_fetcher"` explicitly
+- Same for `Processor` — `require "relaton/xsf/processor"`
+
+## Key dependencies
+
+- `relaton-core` — abstract base classes (Processor, HitCollection, Hit, DataFetcher)
+- `relaton-bib` — bibliographic models, XML/YAML serialization (lutaml-model based)
+- `relaton-index` — index management for quick document lookups
+- `mechanize` — HTTP fetching and HTML parsing
+
+## Style
+
+- RuboCop with relaton shared config (inherits from riboseinc/oss-guides)
+- Target Ruby version: 3.1
+- Logging via `Relaton::Xsf::Util` (extends `Relaton::Bib::Util`, PROGNAME = "relaton-xsf")

data/lib/relaton/xsf/README.adoc

@@ -0,0 +1,127 @@
+= Relaton::XSF: bibliographic retrieval of XMPP XEP specifications
+
+== Purpose
+
+`relaton-xsf` provides bibliographic information of XMPP XEP specifications using the
+https://github.com/relaton/relaton-models#the-relaton-bibliographic-models[BibliographicItem model].
+
+== Data sources
+
+Relaton-XSF retrieves bibliographic information from https://xmpp.org/extensions/refs/[XMPP dataset].
+
+The XSF Library dataset provides documents listed in the https://github.com/relaton/relaton-data-xsf/blob/v2/index-v2.yaml[index].
+
+== Usage
+
+=== Search for a standard using keywords
+
+[source,ruby]
+----
+require 'relaton/xsf'
+=> true
+
+hit_collection = Relaton::Xsf::Bibliography.search("XEP 0001")
+=> <Relaton::Xsf::HitCollection:0x000000000013c0 @ref=XEP 0001 @fetched=false>
+
+item = hit_collection[0].item
+=> #<Relaton::Bib::ItemData:0x0000000124d510b8
+ ...
+----
+
+=== XML serialization
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="XEP0001" type="standard" schema-version="v1.4.1">
+      <fetched>2026-03-04</fetched>
+      <title language="en" script="Latn">XMPP Extension Protocols</title>
+      <uri type="src">http://xmpp.org/extensions/xep-0001.html</uri>
+      <uri type="HTML">http://xmpp.org/extensions/xep-0001.html</uri>
+      <docidentifier type="XEP" primary="true">XEP 0001</docidentifier>
+      ...
+    <bibitem>"
+----
+With argument `bibdata: true` it outputs XML wrapped by `bibdata` element and adds flavor `ext` element.
+[source,ruby]
+----
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.4.1">
+      <fetched>2026-03-04</fetched>
+      <title language="en" script="Latn">XMPP Extension Protocols</title>
+      <uri type="src">http://xmpp.org/extensions/xep-0001.html</uri>
+      <uri type="HTML">http://xmpp.org/extensions/xep-0001.html</uri>
+      <docidentifier type="XEP" primary="true">XEP 0001</docidentifier>
+      ...
+      <ext>
+        <doctype>rfc</doctype>
+        <flavor>xsf</flavor>
+      </ext>
+  </bibdata>"
+----
+
+=== Get document by reference
+[source,ruby]
+----
+item = Relaton::Xsf::Bibliography.get "XEP 0001"
+[relaton-xsf] INFO: (XEP 0001) Fetching from Relaton repository ...
+[relaton-xsf] INFO: (XEP 0001) Found: `XEP 0001`
+=> #<Relaton::Bib::ItemData:0x0000000125036f58
+...
+
+item.docidentifier.first.content
+=> "XEP 0001"
+----
+
+=== Typed source links
+
+XSF publications have `src` type source link.
+
+[source,ruby]
+----
+item.source[0].type
+=> "src"
+
+item.source[0].content
+=> "http://xmpp.org/extensions/xep-0001.html"
+----
+
+=== Fetch data
+
+This gem uses the https://xmpp.org/extensions/refs/ dataset as a data source.
+
+The method `Relaton::Xsf::DataFetcher.fetch(output: "data", format: "yaml")` fetches all the documents from the dataset and saves them to the `./data` folder in YAML format.
+Arguments:
+
+- `output` - folder to save documents (default './data').
+- `format` - the format in which the documents are saved. Possible formats are: `yaml`, `xml`, `bibxxml` (default `yaml`).
+
+[source,ruby]
+----
+require 'relaton/xsf/data_fetcher'
+
+Relaton::Xsf::DataFetcher.fetch
+Started at: 2021-09-01 18:01:01 +0200
+Stopped at: 2021-09-01 18:01:43 +0200
+Done in: 42 sec.
+=> nil
+----
+
+=== Logging
+
+Relaton::Xsf uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To update the index test fixture (used by tests), run `rake spec:update_index_xsf`. This downloads the latest published index from the https://github.com/relaton/relaton-data-xsf[relaton-data-xsf] repository.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to https://rubygems.org[rubygems.org].
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/relaton-xsf.
+
+== License
+
+The gem is available as open source under the terms of the https://opensource.org/license/bsd-2-clause/[2-Clause BSD License].

data/lib/relaton.rb

@@ -20,6 +20,7 @@ require "yaml"
 
 module Relaton
   autoload :Logger, "relaton/logger"
+  autoload :Equivalence, "relaton/equivalence"
   autoload :Core, "relaton/core"
   autoload :Index, "relaton/index"
   autoload :Bib, "relaton/bib"