obp-access 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: '08cfedb2a758cc85495717b57a184fe2e9135e77b31449f7529470db99d82fdb'
+  data.tar.gz: 8fdb94cb0f6f7eec0f91ee0c2161a4846725f6b8aef07ecc3fd644998f1ff616
+SHA512:
+  metadata.gz: 363df11925ccc60638446288a06dd3bf58c4592eea3ac98459d69ed116693e4dd83fbff27a0533826e0c07affe64c9eaffd8abf6c3bdfbcded31e469f8d2eb98
+  data.tar.gz: c82359991c371041be7c14a11b27dcec0c255757657dfe164099c6e68809bb6c16d98f3bd319f2c809897f03b58c14d7883b12e19227f800ab8fb6aa4b4aa6e9

data/.rubocop.yml

@@ -0,0 +1,18 @@
+# Auto-generated by Cimas: Do not edit it manually!
+# See https://github.com/metanorma/cimas
+inherit_from:
+  - .rubocop_todo.yml
+  - https://raw.githubusercontent.com/riboseinc/oss-guides/master/ci/rubocop.yml
+
+# local repo-specific modifications
+# ...
+
+AllCops:
+  NewCops: enable
+  SuggestExtensions: false
+
+Metrics/MethodLength:
+  Max: 30
+
+Layout/LineLength:
+  Max: 120

data/.rubocop_todo.yml

@@ -0,0 +1,33 @@
+# This configuration was generated by
+# `rubocop --auto-gen-config`
+# on 2026-05-12 09:41:56 UTC using RuboCop version 1.86.1.
+# The point is for the user to remove these configuration records
+# one by one as the offenses are removed from the code base.
+# Note that changes in the inspected code, or installation of new
+# versions of RuboCop, may require this file to be generated again.
+
+# Offense count: 1
+Gemspec/RequiredRubyVersion:
+  Exclude:
+    - 'obp-access.gemspec'
+
+# Offense count: 1
+# This cop supports safe autocorrection (--autocorrect).
+# Configuration parameters: AllowHeredoc, AllowURI, AllowQualifiedName, URISchemes, AllowRBSInlineAnnotation, AllowCopDirectives, AllowedPatterns, SplitStrings.
+# URISchemes: http, https
+Layout/LineLength:
+  Max: 124
+
+# Offense count: 3
+# Configuration parameters: AllowedMethods, AllowedPatterns, CountRepeatedAttributes, Max.
+Metrics/AbcSize:
+  Exclude:
+    - 'lib/obp/access/elements/bibliography/bib_ref.rb'
+    - 'lib/obp/access/elements/root.rb'
+
+# Offense count: 1
+# This cop supports safe autocorrection (--autocorrect).
+# Configuration parameters: AllowHeredoc, AllowURI, AllowQualifiedName, URISchemes, AllowRBSInlineAnnotation, AllowCopDirectives, AllowedPatterns, SplitStrings.
+# URISchemes: http, https
+Layout/LineLength:
+  Max: 124

data/CLAUDE.md

@@ -0,0 +1,59 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Ruby gem (`obp-access`) that fetches informative content (introduction, scope, terms, bibliography) from the ISO Online Browsing Platform (OBP) and converts it to NISO STS XML. Published by Ribose Inc. under BSD-2-Clause license.
+
+## Commands
+
+- **Install dependencies:** `bundle install`
+- **Run tests:** `bundle exec rspec`
+- **Lint:** `bundle exec rubocop`
+- **Run a single test:** `bundle exec rspec spec/obp/access/grammar_parser_spec.rb`
+- **Console:** `bin/console` (loads the gem for interactive use)
+
+## Critical rules
+
+- **NEVER use Nokogiri for XML building.** The sts gem and lutaml-model handle all XML serialization. Nokogiri is used internally by those libraries but must not appear in our application code. The `Nokogiri::XML::Builder` usage in `elements/` is legacy tech debt being migrated to the sts gem's model-driven approach.
+- **Never use `send` to call private methods** (breaks encapsulation).
+- **Never use `respond_to?`** (poor typing — use type checking or duck typing).
+- **Never use `instance_double` in specs** — use `double` instead.
+
+## Architecture
+
+Data flows through these classes in `lib/obp/access/`:
+
+1. **`Access`** (entry point) — orchestrates the pipeline. `Obp::Access.fetch(urn)` returns an instance per language. `Access.fetch_all(urn, languages:)` returns separate instances per language.
+
+2. **`Parser`** — fetches content from the ISO OBP API (`https://www.iso.org/obp/ui`) via HTTP POST with a URN payload. Parses the JSON response to extract HTML content, titles, and images.
+
+3. **`Converter`** — wraps the HTML source, normalizes whitespace, parses it with Nokogiri, and passes DOM nodes to the Renderer.
+
+4. **`Renderer`** — recursively walks DOM nodes and dispatches them to element classes registered in `ElementRegistry`. Each element class matches against CSS classes and builds NISO STS XML.
+
+### Element system
+
+- **`ElementRegistry`** — explicit registry of element classes. Elements call `ElementRegistry.register(Class)` at load time.
+- **`Elements::Base`** — abstract base class. Subclasses implement `self.classes` (CSS class array) and `content` (XML builder).
+- **`Elements::Root`** — creates the NISO STS document skeleton (`<standard>` with `<front>`, `<body>`, `<back>`).
+- **`Elements::Terminology`** sub-elements handle term entries using the `tbx:` namespace.
+- Other elements: `section`, `introduction`, `bibliography`, `paragraph`, `list`, `array`, `figure`, `figure_group`, `table_wrap`, `title`, `copyright`, `index`, `non_normative_note`.
+
+### Supporting classes
+
+- **`GrammarParser`** — extracts part-of-speech and gender from bold term markup.
+- **`DomainExtractor`** — extracts subject-field domains from definition text.
+- **`InlineRenderer`** — renders inline HTML elements (links, xrefs, italic, bold, entailed terms) to STS XML.
+- **`Imager`** — downloads images from OBP. Uses `Parallel` for concurrent downloads.
+
+## Key dependencies
+
+- **sts** — NISO STS gem for generating STS objects from XML
+- **lutaml/model** — model serialization framework
+- **parallel** — concurrent HTTP requests
+
+## Ruby version
+
+Requires Ruby >= 3.1.

data/README.adoc

@@ -0,0 +1,97 @@
+== ISO OBP data access in Ruby
+
+=== Background
+
+The https://www.iso.org/obp/ui[ISO Online Browsing Platform (OBP)] is the ISO
+official location to obtain informative content from ISO standards.
+
+NOTE: Normative content of ISO standards is paywalled.
+
+Informative content provided on the OBP include:
+
+* Introduction (if present)
+* Scope
+* Normative references
+* Terms and definitions
+* Bibliography
+
+=== Use cases
+
+There are many situations where the informative content is already useful:
+
+* ISO project editors: obtain machine-readable content of informative clauses to
+  start a revision without waiting for the complicated STS XML file.
+
+* Terminology users: obtain term and definition details from a standard.
+
+* Bibliography users: find out what documents this standard refers to.
+
+=== Usage
+
+==== Single language
+
+[source,ruby]
+----
+obp = Obp::Access.fetch("iso:std:iso:5598:ed-3:v1:en")
+obp.to_xml(pretty: true) # => NISO STS XML string
+obp.to_sts               # => #<Sts::NisoSts::Standard>
+obp.to_xml_file          # => "/tmp/iso-std-iso-5598-.../iso-std-iso-5598-....xml"
+----
+
+==== Multilingual (specific languages)
+
+[source,ruby]
+----
+obp = Obp::Access.fetch("iso:std:iso:5598:ed-3:v1:en", languages: ["fr", "de"])
+obp.to_xml(pretty: true) # => NISO STS XML with en/fr/de langSets
+----
+
+==== Multilingual (all available languages)
+
+[source,ruby]
+----
+obp = Obp::Access.fetch("iso:std:iso:5598:ed-3:v1:en", languages: :all)
+obp.to_xml(pretty: true) # => NISO STS XML with all available language langSets
+----
+
+==== CLI
+
+Fetch a single document (English only):
+
+[source,shell]
+----
+$ obp-access fetch iso:std:iso:5598:ed-3:v1:en
+----
+
+Fetch with all available languages:
+
+[source,shell]
+----
+$ obp-access fetch -l all iso:std:iso:5598:ed-3:v1:en
+----
+
+Fetch with specific languages and save to file:
+
+[source,shell]
+----
+$ obp-access fetch -l fr,de -o output/ iso:std:iso:5598:ed-3:v1:en
+----
+
+=== Generated XML
+
+The output is NISO STS XML with TBX-Basic terminology markup:
+
+* Terms use `<tbx:termEntry>` with `<tbx:langSet>` per language
+* Grammar is encoded via `<tbx:gram>` (gender: m/f/n) and `<tbx:partOfSpeech>`
+* Domains are extracted as `<tbx:subjectField>`
+* Deprecated terms use `<tbx:normativeAuthorization value="deprecatedTerm"/>`
+
+== Credits
+
+This gem is developed, maintained and funded by
+https://www.ribose.com[Ribose Inc.]
+
+== License
+
+The gem is available as open source under the terms of the
+https://opensource.org/licenses/BSD-2-Clause[2-Clause BSD License].

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/*_test.rb"]
+end
+
+task default: :test

data/exe/obp-access

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require "obp/access"
+
+Obp::Access::CLI.start(ARGV)

data/lib/obp/access/catalog.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+
+module Obp
+  class Access
+    class Catalog
+      SOURCE_URL = "https://isopublicstorageprod.blob.core.windows.net/opendata/" \
+                   "_latest/iso_deliverables_metadata/json/iso_deliverables_metadata.jsonl"
+
+      attr_reader :deliverables
+
+      def initialize(deliverables:)
+        @deliverables = deliverables
+      end
+
+      def self.load(path: nil, url: SOURCE_URL)
+        raw = path ? read_local(path) : fetch_remote(url)
+        new(deliverables: parse_jsonl(raw).map { |data| Deliverable.new(data) })
+      end
+
+      def retrievable
+        @retrievable ||= deliverables.select(&:retrievable?)
+      end
+
+      def by_type(type)
+        deliverables.select { |d| d.deliverable_type == type }
+      end
+
+      def by_ics(code)
+        deliverables.select { |d| d.ics_codes.include?(code) }
+      end
+
+      def count
+        deliverables.size
+      end
+
+      class << self
+        private
+
+        def parse_jsonl(text)
+          text.lines.filter_map do |line|
+            line.strip!
+            next if line.empty?
+
+            JSON.parse(line)
+          end
+        end
+
+        def read_local(path)
+          File.read(path)
+        end
+
+        def fetch_remote(url)
+          uri = URI(url)
+          response = Net::HTTP.get_response(uri)
+          unless response.is_a?(Net::HTTPSuccess)
+            raise "Failed to fetch catalog: #{response.code} #{response.message}"
+          end
+
+          response.body
+        end
+      end
+    end
+  end
+end

data/lib/obp/access/cli.rb

@@ -0,0 +1,115 @@
+# frozen_string_literal: true
+
+require "thor"
+
+module Obp
+  class Access
+    class CLI < Thor
+      desc "fetch URN", "Fetch a single document from ISO OBP by URN"
+      option :output, aliases: "-o", type: :string, desc: "Output directory (default: stdout)"
+      option :languages, aliases: "-l", type: :string,
+                         desc: "Languages: 'all' or comma-separated (e.g. 'fr,de')"
+      def fetch(urn)
+        langs = parse_languages
+        if langs
+          say "Fetching #{urn} (#{langs == :all ? 'all languages' : langs.join(', ')})..."
+          Access.fetch_all(urn, languages: langs).each { |access| output(access) }
+        else
+          say "Fetching #{urn}..."
+          output(Access.fetch(urn))
+        end
+      rescue StandardError => e
+        say "Error: #{e.message}", :red
+        exit 1
+      end
+
+      desc "catalog", "Load and inspect the ISO Open Data catalog"
+      option :path, type: :string, desc: "Local JSONL file path (default: fetch remote)"
+      option :filter, type: :string, enum: %w[retrievable types], desc: "Filter mode"
+      option :type, type: :string, desc: "Filter by deliverable type (IS, TS, TR, etc.)"
+      option :ics, type: :string, desc: "Filter by ICS code"
+      def catalog
+        say "Loading catalog..."
+        cat = Access::Catalog.load(path: options[:path])
+
+        if options[:filter] == "types"
+          print_type_summary(cat)
+          return
+        end
+
+        print_deliverables(cat)
+      rescue StandardError => e
+        say "Error: #{e.message}", :red
+        exit 1
+      end
+
+      desc "retrieve", "Bulk retrieve documents from ISO OBP"
+      option :output, aliases: "-o", type: :string, required: true, desc: "Output directory"
+      option :path, type: :string, desc: "Local JSONL file path"
+      option :concurrency, aliases: "-c", type: :numeric, default: 4, desc: "Thread concurrency"
+      def retrieve
+        say "Loading catalog..."
+        cat = Access::Catalog.load(path: options[:path])
+        say "Found #{cat.retrievable.size} retrievable deliverables"
+        build_retriever(cat).run
+      rescue StandardError => e
+        say "Error: #{e.message}", :red
+        exit 1
+      end
+
+      private
+
+      def output(access)
+        if options[:output]
+          dir = File.expand_path(options[:output])
+          FileUtils.mkdir_p(dir)
+          path = File.join(dir, "#{access.urn.safe}.xml")
+          File.write(path, access.to_xml(pretty: true))
+          say "Saved to #{path}", :green
+        else
+          puts access.to_xml(pretty: true)
+        end
+      end
+
+      def print_deliverables(cat)
+        filtered = apply_filters(cat)
+        say "Total: #{filtered.size} deliverables"
+        filtered.first(20).each do |d|
+          say "  #{d.reference} [#{d.deliverable_type}] stage=#{d.current_stage} langs=#{d.languages.join(',')}"
+        end
+        say "  ... (showing first 20)" if filtered.size > 20
+      end
+
+      def apply_filters(cat)
+        return cat.by_type(options[:type]) if options[:type]
+        return cat.by_ics(options[:ics]) if options[:ics]
+        return cat.retrievable if options[:filter] == "retrievable"
+
+        cat.deliverables
+      end
+
+      def build_retriever(cat)
+        Access::Retriever.new(
+          output_dir: File.expand_path(options[:output]),
+          catalog: cat,
+          concurrency: options[:concurrency],
+        )
+      end
+
+      def parse_languages
+        case options[:languages]
+        when nil then nil
+        when "all" then :all
+        else options[:languages].split(",").map(&:strip)
+        end
+      end
+
+      def print_type_summary(cat)
+        cat.deliverables.group_by(&:deliverable_type).sort.each do |type, list|
+          published = list.count(&:published?)
+          say "  #{type || 'IS'}: #{list.size} total, #{published} published"
+        end
+      end
+    end
+  end
+end

data/lib/obp/access/converter.rb

@@ -0,0 +1,25 @@
+module Obp
+  class Access
+    class Converter
+      attr_reader :urn, :metas, :source
+
+      def initialize(urn:, metas:, source:)
+        @urn = urn
+        @metas = metas
+        @source = source
+      end
+
+      def to_xml
+        Renderer.new(urn:, metas:, nodes:).to_xml
+      end
+
+      private
+
+      def nodes
+        html = source.gsub(/[[:space:]]/, " ")
+        doc = Nokogiri::HTML(html)
+        doc.css("body > div.sts-standard").children
+      end
+    end
+  end
+end

data/lib/obp/access/deliverable.rb

@@ -0,0 +1,109 @@
+# frozen_string_literal: true
+
+module Obp
+  class Access
+    class Deliverable
+      TYPE_SEGMENTS = {
+        "IS" => nil, "TS" => "ts", "TR" => "tr", "R" => "r",
+        "PAS" => "pas", "ISP" => "isp", "GUIDE" => "guide",
+        "IWA" => "iwa", "DATA" => "data", "TTA" => "tta"
+      }.freeze
+
+      TYPE_WORDS = Set.new(TYPE_SEGMENTS.compact.keys).freeze
+
+      PUBLISHED_STAGES = [6060, 9092].freeze
+
+      attr_reader :id, :reference, :deliverable_type, :edition, :current_stage,
+                  :languages, :supplement_type, :title, :publication_date,
+                  :ics_codes, :owner_committee
+
+      def initialize(data)
+        @id = data["id"]
+        assign_metadata(data)
+      end
+
+      def published?
+        PUBLISHED_STAGES.include?(current_stage)
+      end
+
+      def base_document?
+        supplement_type.nil?
+      end
+
+      def retrievable?
+        published? && base_document? && languages.any?
+      end
+
+      def to_urn(language: "en")
+        Urn.new(build_urn(language))
+      end
+
+      def english_title
+        title["en"]
+      end
+
+      private
+
+      def assign_metadata(data)
+        @reference = data["reference"]
+        @deliverable_type = data["deliverableType"]
+        @edition = data["edition"]
+        @current_stage = data["currentStage"]
+        @supplement_type = data["supplementType"]
+        @publication_date = data["publicationDate"]
+        @owner_committee = data["ownerCommittee"]
+        assign_collections(data)
+      end
+
+      def assign_collections(data)
+        @languages = Array(data["languages"])
+        @title = data["title"] || {}
+        @ics_codes = Array(data["icsCode"])
+      end
+
+      def build_urn(language)
+        segs = ["iso", "std", org_segment]
+        type_seg = TYPE_SEGMENTS[deliverable_type]
+        segs << type_seg if type_seg
+        segs << extract_number
+        segs << "-#{extract_part}" if extract_part
+        segs << "ed-#{edition}"
+        segs << "v1"
+        segs << language
+        segs.join(":")
+      end
+
+      def org_segment
+        @org_segment ||= begin
+          tokens = parse_prefix_tokens
+          rest = tokens[1..] || []
+          org_tokens = rest.reject { |t| TYPE_WORDS.include?(t) }
+          org_tokens.empty? ? "iso" : "iso-#{org_tokens.join('-').downcase}"
+        end
+      end
+
+      def extract_number
+        @extract_number ||= begin
+          match = base_reference.match(/(\d+)(?:-\d+)?:\d{4}/)
+          match ? match[1] : "0"
+        end
+      end
+
+      def extract_part
+        @extract_part ||= begin
+          match = base_reference.match(/\d+-(\d+):\d{4}/)
+          match ? match[1] : nil
+        end
+      end
+
+      def parse_prefix_tokens
+        prefix = base_reference.match(/\A(.+?)\s+\d/)&.[](1) || "ISO"
+        prefix.split(%r{[/\s]+})
+      end
+
+      def base_reference
+        @base_reference ||= reference.sub(%r{/(?:Amd|Cor)\s+\d+(?::\d+)?$}, "")
+      end
+    end
+  end
+end

data/lib/obp/access/domain_extractor.rb

@@ -0,0 +1,63 @@
+module Obp
+  class Access
+    class DomainExtractor
+      Result = Struct.new(:domains, :clean_children, keyword_init: true)
+
+      DOMAIN_PATTERN = /\A\s*<([^>]+)>/
+      MAX_DOMAIN_LENGTH = 50
+
+      def self.extract(node)
+        state = { domains: [], clean_children: [], text_consumed: false }
+
+        node.children.each { |child| process_child(child, state) }
+
+        Result.new(domains: state[:domains], clean_children: state[:clean_children])
+      end
+
+      class << self
+        private
+
+        def process_child(child, state)
+          if !state[:text_consumed] && child.is_a?(Nokogiri::XML::Text)
+            process_leading_text(child, state)
+          else
+            state[:clean_children] << child
+            state[:text_consumed] = true
+          end
+        end
+
+        def process_leading_text(child, state)
+          extracted, remaining = extract_from_text(child.content)
+          state[:domains] = extracted
+          state[:text_consumed] = true
+          state[:clean_children] << remaining_node(remaining) unless remaining.strip.empty?
+        end
+
+        def extract_from_text(text)
+          domains = []
+          remaining = text.dup
+
+          while remaining =~ DOMAIN_PATTERN
+            candidate = $1.strip
+            break unless valid_domain?(candidate)
+
+            domains << candidate
+            remaining = remaining.sub(DOMAIN_PATTERN, "")
+          end
+
+          [domains, remaining]
+        end
+
+        def valid_domain?(text)
+          text.length <= MAX_DOMAIN_LENGTH &&
+            !text.include?("(") &&
+            !text.match?(/\d{2,}/)
+        end
+
+        def remaining_node(text)
+          Nokogiri::XML::Text.new(text, Nokogiri::HTML::Document.new)
+        end
+      end
+    end
+  end
+end

data/lib/obp/access/element_registry.rb

@@ -0,0 +1,20 @@
+module Obp
+  class Access
+    class ElementRegistry
+      class << self
+        def register(element_class)
+          elements << element_class
+          @css_classes = nil
+        end
+
+        def elements
+          @elements ||= []
+        end
+
+        def css_classes
+          @css_classes ||= elements.filter_map(&:classes).uniq
+        end
+      end
+    end
+  end
+end