relaton 2.2.0.pre.alpha.1 → 3.0.0.pre.alpha.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7836d4ee81be911e0bf0a95a3b9154895cfef800e918443a54ed73d89623b268
-  data.tar.gz: ac550984b45d4c4b558a277c23c4a3163903bdb070760eee2395f070b75b8980
+  metadata.gz: 37fab215eeedb21350f3e195a6a803d47680d35f75767d782600312cef05fa8e
+  data.tar.gz: c4542c9253678a25c5656f213ca39e12689bef778f816b70a383333c0db422d8
 SHA512:
-  metadata.gz: 7fd1668edf1c684c26e3cbf3d2177654b1b48d103290101e457a9811c49dcf6dbb85fe48750985527c545f5a3e30b627a2bff5c2d39ad87e28358528d434c891
-  data.tar.gz: fef35f22ec455e5c1094ba2016355b64f0aaa60fb9ce569dc4e4a19ea7c8c3aa0817ea94dd7ba74c039829ecf18f6d01f4541c59734c0183a63c8b0c7632d204
+  metadata.gz: bd5901f08f8b7c4359598dc38eaa65411679884c4c860350b8b129943fa366a13dcb3f0bf91048a9922d199b364caeaf4f320cdc059898f78a5f8088364b30cd
+  data.tar.gz: ccc9c3ce79b02416dd0a0c31a8dab36fe13da7556e9812ed5d403d66bcdc1c4ecb938d766f2376148315e4c0c3590e243c1ef5c5e97488df8a526df934df3af3

data/{docs/README.adoc → README.adoc}

@@ -1,12 +1,20 @@
 = Relaton
 
 image:https://img.shields.io/gem/v/relaton.svg["Gem Version", link="https://rubygems.org/gems/relaton"]
-image:https://github.com/relaton/relaton/workflows/rake/badge.svg["Build Status", link="https://github.com/relaton/relaton/actions?workflow=rake"]
-image:https://codeclimate.com/github/relaton/relaton/badges/gpa.svg["Code Climate", link="https://codeclimate.com/github/relaton/relaton"]
-image:https://img.shields.io/github/issues-pr-raw/relaton/relaton.svg["Pull Requests", link="https://github.com/relaton/relaton/pulls"]
-image:https://img.shields.io/github/commits-since/relaton/relaton/latest.svg["Commits since latest",link="https://github.com/relaton/relaton/releases"]
+image:https://github.com/relaton/relaton/actions/workflows/rake.yml/badge.svg["Build Status", link="https://github.com/relaton/relaton/actions/workflows/rake.yml"]
 
 Gem for importing and caching bibliographic references to technical standards.
+Every standards-body flavor ships inside this single gem; flavor code loads on
+demand via `autoload`, and the `Relaton::Db` API routes a reference to the right
+flavor by its prefix. The command-line interface,
+link:gems/relaton-cli/docs/README.adoc[`relaton-cli`], is a separate gem.
+
+== Installation
+
+[source,ruby]
+----
+gem "relaton"
+----
 
 == Scope
 
@@ -14,40 +22,13 @@ The Relaton gem obtains authoritative bibliographic entries for technical standa
 
 The gem also caches entries it has retrieved so that subsequent iterations do not need to go back online to retrieve the same entries. The gem uses two caches: a global cache (for all bibliographic entries retrieved by the user), and a local cache (intended to store references specific to the current document being processed.)
 
-Entries are retrieved and stored in https://github.com/relaton/relaton-models[Relaton bibliographic model], which is an expression of ISO 690. The subset of the model used and serialized for Relaton is defined in the https://github.com/relaton/relaton-bib[relaton-bib] and https://github.com/relaton/relaton-iso-bib[relaton-iso-bib] gems.
+Entries are retrieved and stored in https://github.com/relaton/relaton-models[Relaton bibliographic model], which is an expression of ISO 690. The subset of the model used and serialized for Relaton is defined by the bundled `Relaton::Bib` model.
 
 Entries are serialized to and from an internal data model, and multiple formats are intended to be supported. Currently, only https://github.com/relaton/relaton-models/blob/master/grammars/biblio.rnc[Relaton XML] is supported.
 
-Relaton imports bibliographic entries from:
-
-* 3GPP through the portal.3gpp.org website, via the https://github.com/relaton/relaton-3gpp[relaton-3gpp] gem
-* BIPM through the https://github.com/metanorma/bipm-data-outcomes[bipm-data-outcomes], https://github.com/metanorma/bipm-si-brochure[bipm-si-brochure], https://github.com/relaton/rawdata-bipm-metrologia[rawdata-bipm-metrologia] datasets, via the the https://github.com/relaton/relaton-bipm[relaton-bipm] gem
-* BSI through the knowledge.bsigroup.com/search website, via the https://github.com/relaton/relaton-bsi[relaton-bsi] gem
-* CalConnect through the standards.calconnect.org website, via the https://github.com/relaton/relaton-calconnect[relaton-calconnect] gem
-* CCSDS through the public.ccsds.org/Publications/AllPubs.aspx website, via the https://github.com/relaton/relaton-ccsds[relaton-ccsds] gem
-* CEN through the standards.cencenelec.eu website, via the https://github.com/relaton/relaton-cen[relaton-cen] gem
-* CIE through the techstreet.com website, via the https://github.com/relaton/relaton-cie[relaton-cie] gem
-* DOI through the api.crossref.org website, via the https://github.com/relaton/relaton-doi[relaton-doi] gem
-* ECMA through the ecma-international.org website, via the https://github.com/relaton/relaton-ecma[relaton-ecma] gem
-* ETSI through the etsi.org website, via the https://github.com/relaton/relaton-etsi[relaton-etsi] gem
-* GB (Chinese national standards) through the GB websites, via the https://github.com/relaton/relaton-gb[relaton-gb] gem
-* IANA through the https://github.com/ietf-tools/iana-registries[iana-registries] dataset, via the https://github.com/relaton/relaton-iana[relaton-iana] gem
-* IEC through the iec.ch website, via the https://github.com/relaton/relaton-iec[relaton-iec] gem
-* IEEE through the standards.ieee.org website, via the https://github.com/relaton/relaton-ieee[relaton-ieee] gem
-* IETF standards (Internet Drafts, RFC) through the http://xml2rfc.tools.ietf.org website, via the https://github.com/relaton/relaton-ietf[relaton-ietf] gem
-* IHO through the iho.int website, via the https://github.com/relaton/relaton-iho[relaton-iho] gem
-* ISO through the iso.org website, via the https://github.com/relaton/relaton-iso[relaton-iso] gem
-* ITU through the itu.int website, via the https://github.com/relaton/relaton-itu[relaton-itu] gem
-* ISBN through the openlibrary.org website, via the https://github.com/relaton/relaton-isbn[relaton-isbn] gem
-* JIS through the webdesk.jsa.or.jp website, via the https://github.com/relaton/relaton-jis[relaton-jis] gem
-* NIST standards through the nist.gov website, via the https://github.com/relaton/relaton-nist[relaton-nist] gem
-* OASIS through the www.oasis-open.org/standards website, via the https://github.com/relaton/relaton-oasis[relaton-oasis] gem
-* OGC through the https://github.com/opengeospatial/NamingAuthority[dataset], via the https://github.com/relaton/relaton-ogc[relaton-ogc] gem
-* OMG through the omg.org/spec website, via the https://github.com/relaton/relaton-omg[relaton-omg] gem
-* Plateau through the www.mlit.go.jp website, via the https://github.com/relaton/relaton-plateau[relaton-plateau] gem
-* UN through the documents.un.org website, via the https://github.com/relaton/relaton-un[relaton-un] gem
-* W3C through the w3c.org website, via the https://github.com/relaton/relaton-w3c[relaton-w3c] gem
-* XSF through the xmpp.org website, via the https://github.com/relaton/relaton-xsf[relaton-xsf] gem
+Relaton fetches bibliographic entries from ~29 standards bodies. Each is a
+flavor bundled in this gem (ISO, IEC, IETF, NIST, IEEE, GB, W3C, ...) -- see
+<<Flavors>> for the full list and per-flavor notes.
 
 The identifiers for which bibliographic entries are to be retrieved need to indicate which standards body they belong to. To do so, this gem adopts the convention of bracketing identifiers, and preceding them with a code that indicates the standards body:
 
@@ -55,9 +36,9 @@ The identifiers for which bibliographic entries are to be retrieved need to indi
 * Otherwise, the wrappers use the agreed abbreviation of the standards body. So `IETF(I-D.ribose-asciirfc-08)` identifies `I-D.ribose-asciirfc` as an Internet Engineering Task Force identifier.
 * Some prefixes to identifiers indicate the standards body they belong to unambiguously; e.g. `ISO` followed by a slash or space. The scope wrapper is not required for those prefixes: `ISO(ISO 639-1)` can be recognized as just `ISO 639-1`.
 
-The gem can be extended to use other standards-specific gems. Standards-specific gems like isobib register themselves against Relaton using `Relaton::Registry.instance.register`, which takes as an argument a subclass of `Relaton::Processor` defined in the gem; see isobib/lib/relaton for an example. The processor within the standards-specific gem needs to define
+Each flavor registers a `Relaton::<Flavor>::Processor` (a subclass of `Relaton::Core::Processor`) with the registry. A processor defines:
 
-* `@short`, the name of the gem
+* `@short`, the flavor name
 * `@prefix`, the regex which scopes the identifier, and constrains it to belong to a particular standards class.
 * `@defaultprefix`, the identifier prefixes which can be recognized without a scope wrapper.
 * `@idtype`, the type assigned to document identifiers for the standard class.
@@ -99,7 +80,7 @@ Relaton.configure do |conf|
 end
 ----
 
-Logger configuration is managed through the `relaton-logger` gem via `Relaton.logger_pool`:
+Logger configuration is managed via `Relaton.logger_pool`:
 
 [source,ruby]
 ----
@@ -439,3 +420,149 @@ db.save_entry("ISO(ISO 19011)", nil)
 db.load_entry("ISO(ISO 19011)")
 => nil
 ----
+
+== Organization metadata
+
+Besides bibliographic references, Relaton exposes a lightweight store of
+standards-development-organization (SDO) metadata: an organization's names (with
+translations) and its logos. This is a separate, non-flavor concern -- it does
+not go through `Relaton::Db` -- reached through the top-level
+`Relaton.organization` method.
+
+The data (names, translations and logo variants) is served as an `index.yaml`
+manifest by the https://github.com/relaton/relaton-data-sdo[`relaton-data-sdo`]
+data repository, downloaded on first use and cached under `~/.relaton/sdo`.
+
+[source,ruby]
+----
+org = Relaton.organization("ISO")   # case-insensitive; nil if unknown
+
+# Names (relaton-db#132) -- default, or a translation by language code
+# (passed positionally or as the language: keyword)
+org.name                   # => "International Organization for Standardization"
+org.name("fr")             # => "Organisation internationale de normalisation"
+org.name(language: "fr")   # => "Organisation internationale de normalisation"
+
+# Logos (metanorma#346). A single organization can have several logo variants,
+# distinguished by style (the primary discriminator), format and size.
+# All arguments are optional; omitting one matches any value.
+org.logo_query(format: "eps", size: "700x300", style: "default")
+# => [#<Relaton::Sdo::Logo style="default" format="eps" size="700x300" url=...>, ...]
+
+org.logo_query   # => every logo variant for the organization
+
+# Fetch one logo. Returns nil if nothing matches, the sole match if one does,
+# and raises Relaton::Sdo::Error if the filter is ambiguous.
+logo = org.logo(format: "eps", style: "default")
+logo.content    # raw binary content (fetched lazily, memoized)
+logo.data_uri   # "data:application/postscript;base64,..."  -- ready to embed
+logo.save                 # write to a filename derived from the URL; returns the path
+logo.save("/tmp/iso.eps") # ...or to a custom path
+----
+
+The store is configurable -- the manifest URL, the cache directory, a swappable
+storage backend (e.g. an S3-backed object), and the cache TTL:
+
+[source,ruby]
+----
+Relaton::Sdo.configure do |config|
+  config.url         = "https://example.org/my-index.yaml"
+  config.storage_dir = "/var/cache/relaton-sdo"
+end
+----
+
+== Layout
+
+----
+relaton.gemspec          # the one gemspec (union of every flavor's external deps)
+lib/
+├── relaton.rb           # entry point: autoload per flavor, then require relaton/db
+└── relaton/
+    ├── version.rb       # Relaton::VERSION — the single source of truth
+    ├── db.rb, db/       # Relaton::Db: registry, cache, workers pool
+    ├── core/ bib/ index/ logger/   # shared infrastructure
+    ├── sdo/             # Relaton.organization: SDO org/logo store (non-flavor)
+    └── iso/ iec/ ietf/ … 3gpp/     # ~29 flavor plugins
+spec/<flavor>/           # each flavor's spec suite
+grammar/                 # shared RelaxNG test schemas (test-only, not shipped)
+gems/relaton-cli/        # the separate relaton-cli gem
+----
+
+== Flavors
+
+Every flavor lives under `lib/relaton/<flavor>/` and keeps its own README with
+usage and architecture notes.
+
+Standards bodies:
+
+link:lib/relaton/3gpp/README.adoc[3gpp],
+link:lib/relaton/bipm/README.adoc[bipm],
+link:lib/relaton/bsi/README.adoc[bsi],
+link:lib/relaton/calconnect/README.adoc[calconnect],
+link:lib/relaton/ccsds/README.adoc[ccsds],
+link:lib/relaton/cen/README.adoc[cen],
+link:lib/relaton/cie/README.adoc[cie],
+link:lib/relaton/doi/README.adoc[doi],
+link:lib/relaton/ecma/README.adoc[ecma],
+link:lib/relaton/etsi/README.adoc[etsi],
+link:lib/relaton/gb/README.adoc[gb],
+link:lib/relaton/iana/README.adoc[iana],
+link:lib/relaton/iec/README.adoc[iec],
+link:lib/relaton/ieee/README.adoc[ieee],
+link:lib/relaton/ietf/README.adoc[ietf],
+link:lib/relaton/iho/README.adoc[iho],
+link:lib/relaton/isbn/README.adoc[isbn],
+link:lib/relaton/iso/README.adoc[iso],
+link:lib/relaton/itu/README.adoc[itu],
+link:lib/relaton/jis/README.adoc[jis],
+link:lib/relaton/nist/README.adoc[nist],
+link:lib/relaton/oasis/README.adoc[oasis],
+link:lib/relaton/ogc/README.adoc[ogc],
+link:lib/relaton/oiml/README.adoc[oiml],
+link:lib/relaton/omg/README.adoc[omg],
+link:lib/relaton/plateau/README.adoc[plateau],
+link:lib/relaton/un/README.adoc[un],
+link:lib/relaton/w3c/README.adoc[w3c],
+link:lib/relaton/xsf/README.adoc[xsf].
+
+Foundational libraries:
+
+link:lib/relaton/bib/README.adoc[bib],
+link:lib/relaton/core/README.adoc[core],
+link:lib/relaton/index/README.adoc[index],
+link:lib/relaton/logger/README.adoc[logger].
+
+Maintaining your own dataset? See
+link:docs/data-repository-format.adoc[Relaton data repository format] for the
+repository layout, index file specification, and GitHub Pages publishing contract.
+
+
+== Development
+
+[source,shell]
+----
+bundle install
+bundle exec rake spec              # run every flavor's spec suite
+bundle exec rake spec:iso          # run one flavor's suite
+bundle exec rake build             # build the relaton gem into pkg/
+bundle exec rake build_all         # build relaton + relaton-cli
+----
+
+Each flavor's specs live in `spec/<flavor>/` and run self-contained against the
+single gem (each in its own directory, so relative fixture/cassette/grammar
+paths resolve). See `CLAUDE.md` for the architecture notes (lazy registry,
+autoload, single version, grammar schemas).
+
+== Versioning
+
+One version, `Relaton::VERSION` in `lib/relaton/version.rb`; every flavor derives
+from it. Bump that file (and `relaton-cli`'s own version) to cut a release.
+
+== Release
+
+The `release` GitHub Actions workflow builds and pushes two gems: `relaton` and
+`relaton-cli`. Requires the `RELATON_CI_RUBYGEMS_API_KEY` secret.
+
+== License
+
+BSD-2-Clause.

data/bin/console

@@ -6,6 +6,5 @@ require "relaton"
 # You can add fixtures and/or initialization code here to make experimenting
 # with your gem easier. You can also use a different console, if you like.
 
-# (If you use this, don't forget to add pry to your Gemfile!)
 require "pry"
 Pry.start

data/lib/relaton/3gpp/bibdata.rb

@@ -0,0 +1,9 @@
+module Relaton
+  module ThreeGpp
+    # This class represents a bibliographic item as a bibdata.
+    class Bibdata < Item
+      model ItemData
+      include Bib::BibdataShared
+    end
+  end
+end

data/lib/relaton/3gpp/bibitem.rb

@@ -0,0 +1,9 @@
+module Relaton
+  module ThreeGpp
+    # This class represents a bibliographic item as a bibitem.
+    class Bibitem < Item
+      model ItemData
+      include Bib::BibitemShared
+    end
+  end
+end

data/lib/relaton/3gpp/bibliography.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+module Relaton
+  module ThreeGpp
+    # Methods for search 3GPP standards.
+    module Bibliography
+      SOURCE = "https://raw.githubusercontent.com/relaton/relaton-data-3gpp/v2/"
+
+      # @param text [String]
+      # @return [Relaton::ThreeGpp::ItemData, nil]
+      def search(text)
+        # An unrecognized reference raises; like ISO and ETSI we let it
+        # propagate — relaton-cli rescues Pubid::Errors::Error and renders
+        # "… is not a recognized standards identifier", and API callers rescue
+        # it themselves. The rescue below lists transport errors only, so it
+        # does not swallow the parse error.
+        pubid = ::Pubid::Tgpp::Identifier.parse text.to_s.strip
+        row = best_match pubid
+        return unless row
+
+        url = "#{SOURCE}#{row[:file]}"
+        resp = Net::HTTP.get_response URI(url)
+        return unless resp.code == "200"
+
+        item = Item.from_yaml(resp.body)
+        item.fetched = Date.today.to_s
+        item
+      rescue  SocketError, Timeout::Error, Errno::EINVAL, Errno::ECONNRESET,
+              EOFError, Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError,
+              Net::ProtocolError, Errno::ETIMEDOUT => e
+        raise Relaton::RequestError, e.message
+      end
+
+      # @param ref [String] the 3GPP standard Code to look up
+      # @param year [String, NilClass] not used
+      # @param opts [Hash] options
+      # @return [Relaton::ThreeGpp::ItemData, nil]
+      def get(ref, _year = nil, _opts = {})
+        Util.info "Fetching from Relaton repository ...", key: ref
+        result = search(ref)
+        unless result
+          Util.info "Not found.", key: ref
+          return
+        end
+
+        Util.info "Found: `#{result.docidentifier[0].content}`", key: ref
+        result
+      end
+
+      private
+
+      #
+      # Find the index row for a reference, newest version first.
+      #
+      # Passing the pubid to `Index::Type#search` is what enables the binary
+      # search on `id.root.number` — `Type#search_candidates` narrows only when
+      # its argument is NOT a String, so the plain text this used to pass
+      # disabled the narrowing however the index was built. With 88,464 rows in
+      # 3,767 number buckets that is the whole point of the index-v2 migration.
+      #
+      # The rows are selected with pubid's subset match `pubid === row`, the
+      # default of `Index::Type#search`. `release` and `version` are 3GPP's
+      # only optional components: when the reference omits one, it matches
+      # any value, so a bare `3GPP TS 23.207` still finds the `REL-19/19.0.0`
+      # row. pubid declares `suffix` and `parts` strict for 3GPP, because they
+      # are part of the document code (`TS 29.198-04-1`, `TR 00.01U`), not
+      # qualifiers. The document type is the identifier's class, and `===`
+      # requires the same class, so `TS 23.207` and `TR 23.207` never match.
+      #
+      # @param pubid [::Pubid::Tgpp::Identifier]
+      # @return [Hash, nil] the winning index row (`{ id:, file: }`)
+      #
+      def best_match(pubid)
+        index.search(pubid)
+             .max_by { |r| [version_key(r[:id].version), r[:id].to_s, r[:file]] }
+      end
+
+      #
+      # Order key for a 3GPP version — the highest version wins.
+      #
+      # Versions are dotted strings, so they must be compared segment by
+      # segment as integers rather than as text: `19.0.0` is newer than
+      # `4.0.0` but loses as a string. This is a deliberate behaviour change.
+      # The previous key, `min_by { r[:id] }` over raw strings, returned
+      # `REL-10/10.0.0` for `3GPP TS 23.207` — neither the newest nor the
+      # oldest of its 37 rows, just whichever sorted first as text. Scored
+      # against each document's own publication date over 870 documents in 59
+      # multi-row groups, this key picks the newest published document 51/59
+      # (86%) of the time; the old one managed 1/59 (2%).
+      #
+      # No release key is layered on top. Ranking releases numerically is wrong
+      # (the chronological order is Ph1, Ph2, REL-96 … REL-99, Release 2000,
+      # UMTS, REL-4 … REL-21, so REL-99 would outrank REL-19) and a
+      # hand-maintained 27-token table would disagree with this key on only 9
+      # of 3,982 multi-row groups — all draft TRs carried into a later release
+      # at a lower version, where the release answer is not clearly better.
+      #
+      # An absent version sorts below every present one. Ties break on the
+      # rendered id and then the file path, because the index sort is not
+      # stable.
+      #
+      # @param version [String, nil]
+      # @return [Array<Integer>]
+      #
+      def version_key(version)
+        version.to_s.split(".").map(&:to_i)
+      end
+
+      # The index is pubid-backed: `pubid_class:` is what makes Relaton::Index
+      # deserialize the rows into identifiers, sort them by `id.root.number`,
+      # and let `Type#search` bsearch. Without it the rows stay raw hashes,
+      # `FileIO#sorted` stays false, and every lookup scans all 88,464 of them.
+      def index
+        Relaton::Index.find_or_create(
+          "3GPP", url: "#{SOURCE}#{INDEXFILE}.zip", file: "#{INDEXFILE}.yaml",
+          pubid_class: ::Pubid::Tgpp::Identifier
+        )
+      end
+
+      extend Bibliography
+    end
+  end
+end