relaton 3.0.0.pre.alpha.16 → 3.0.0.pre.alpha.18

data/lib/relaton/iso/README.adoc

@@ -0,0 +1,370 @@
+= Relaton::Iso: retrieve ISO Standards for bibliographic use using the BibliographicItem model
+
+Relaton::Iso is a Ruby gem that implements the https://github.com/metanorma/metanorma-model-iso#iso-bibliographic-item[IsoBibliographicItem model].
+
+You can use it to retrieve metadata of ISO Standards from https://www.iso.org, and access such metadata through the `IsoBibliographicItem` object.
+
+== Usage
+
+=== Search for standards using keywords
+
+[source,ruby]
+----
+require 'relaton/iso'
+=> true
+
+hit_collection = Relaton::Iso::Bibliography.search("ISO 19115")
+ => <Relaton::Iso::HitCollection:0x0000000003a9a8 @ref=ISO 19115 @fetched=false>
+
+hit_collection.first
+ => #<Relaton::Iso::Hit:0x00000001398ecdc8 ...>
+
+item = hit_collection[2].item
+ => #<Relaton::Iso::ItemData:0x0000000139d07070 ...>
+
+item.docidentifier.first.to_s
+ => "ISO 19115-2:2019"
+
+item.docidentifier.find { |id| id.type == "URN" }.to_s
+ => "urn:iso:std:iso:19115:-2:stage-90.93"
+----
+
+=== Fetch document by reference and year
+
+[source,ruby]
+----
+item = Relaton::Iso::Bibliography.get "ISO 19115:2003"
+[relaton-iso] INFO: (ISO 19115:2003) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115:2003) Found: `ISO 19115:2003`
+ => #<Relaton::Iso::ItemData:0x000000013c4cdac8 ...>
+
+item = Relaton::Iso::Bibliography.get "ISO 19115", "2003"
+[relaton-iso] INFO: (ISO 19115:2003) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115:2003) Found: `ISO 19115:2003`
+ => #<Relaton::Iso::ItemData:0x000000013c4c5148 ...>
+
+item.docidentifier[0].to_s
+ => "ISO 19115:2003"
+----
+
+=== Fetch non-part document
+
+[source,ruby]
+----
+item = Relaton::Iso::Bibliography.get "ISO 19115"
+[relaton-iso] INFO: (ISO 19115) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115) Found: `ISO 19115:2003`
+ => #<Relaton::Iso::ItemData:0x000000013b6a6418 ...>
+
+item.docidentifier[0].to_s
+ => "ISO 19115"
+----
+
+=== Fetch a part document
+
+[source,ruby]
+----
+item = Relaton::Iso::Bibliography.get "ISO 19115-1"
+[relaton-iso] INFO: (ISO 19115-1) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115-1) Found: `ISO 19115-1:2014`
+ => #<Relaton::Iso::ItemData:0x000000013c761410 ...>
+
+item.docidentifier[0].to_s
+ => "ISO 19115-1"
+----
+
+=== Fetch all-parts document
+
+[source,ruby]
+----
+item = Relaton::Iso::Bibliography.get "ISO 19115 (all parts)"
+[relaton-iso] INFO: (ISO 19115 (all parts)) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115 (all parts)) Found: `ISO 19115 (all parts)`
+ => #<Relaton::Iso::ItemData:0x000000013bb87950 ...>
+
+item = Relaton::Iso::Bibliography.get "ISO 19115", nil, all_parts: true
+[relaton-iso] INFO: (ISO 19115 (all parts)) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115 (all parts)) Found: `ISO 19115 (all parts)`
+ => #<Relaton::Iso::ItemData:0x000000013c855150 ...>
+
+item.docidentifier[0].to_s
+ => "ISO 19115 (all parts)"
+
+item = Relaton::Iso::Bibliography.get "ISO 19115-1 (all parts)"
+[relaton-iso] INFO: (ISO 19115-1 (all parts)) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115-1 (all parts)) Found: `ISO 19115 (all parts)`
+ => #<Relaton::Iso::ItemData:0x000000013b54c590 ...>
+
+item = Relaton::Iso::Bibliography.get "ISO 19115-1", nil, all_parts: true
+[relaton-iso] INFO: (ISO 19115-1 (all parts)) Fetching from Relaton repository ...
+[relaton-iso] INFO: (ISO 19115-1 (all parts)) Found: `ISO 19115 (all parts)`
+ => #<Relaton::Iso::ItemData:0x000000013b54dad0 ...>
+
+item.docidentifier[0].to_s
+ => "ISO 19115 (all parts)"
+----
+
+=== Non-standard ISO/IEC documents
+
+==== General
+
+ISO/IEC Directives and related documents are fetched from the `relaton-data-iso`
+repository managed by the Relaton team.
+
+The following PubID patterns are used to fetch particular documents.
+
+==== ISO/IEC Directives in Parts
+
+`ISO/IEC DIR 1 ISO SUP`::
+ISO/IEC Directives, Consolidated ISO Supplement -- Procedures specific to ISO.
+This is the undated reference.
+
+`ISO/IEC DIR 1 ISO SUP:{yyyy}`::: Available edition years from 2003 (ed1) to 2023 (ed14).
+
+`ISO/IEC DIR 1`::
+ISO/IEC Directives, Part 1. This is the undated reference.
+
+`ISO/IEC DIR 1 ISO`:: ISO-published version of "ISO/IEC Directives, Part 1".
+`ISO/IEC DIR 1 ISO:{yyyy}`::: Edition years from 1986 (ed1) to 2023 (ed19).
+
+`ISO/IEC DIR 1 IEC`:: IEC-published version of "ISO/IEC Directives, Part 1"
+`ISO/IEC DIR 1 IEC:{yyyy}`::: Edition years from 1986 (ed1) to 2023 (ed19).
+
+`ISO/IEC DIR 2`::
+ISO/IEC Directives, Part 2. This is the undated reference.
+
+`ISO/IEC DIR 2 ISO`:: ISO-published version of "ISO/IEC Directives, Part 2"
+`ISO/IEC DIR 2 ISO:{yyyy}`::: Edition years from 1986 (ed1) to 2021 (ed9).
+
+`ISO/IEC DIR 2 IEC`:: IEC-published version of "ISO/IEC Directives, Part 2"
+`ISO/IEC DIR 2 IEC:{yyyy}`::: Edition years from 1986 (ed1) to 2021 (ed9).
+
+`ISO/IEC DIR 3`::
+ISO/IEC Directives, Part 3. This is the undated reference.
+This is of historical interest, as originally ISO/IEC published the Directives
+in 3 parts:
+Part 1 "Procedures for the technical work",
+Part 2 "Methodology for the development of International Standards",
+Part 3 "Drafting and presentation of International Standards".
+In the 2004 versions, Parts 1 and 2 merged into the new Part 1, and Part 3
+was renamed as Part 2.
+
+`ISO/IEC DIR 3:{yyyy}`::: Edition years: 1986 (ed1), 1989 (ed2), 1997 (ed3).
+
+==== ISO/IEC Directives ISO Supplements related
+
+`ISO/IEC DIR 1 ISO SUP`::
+ISO/IEC Directives, Consolidated ISO Supplement -- Procedures specific to ISO.
+This is the undated reference.
+
+`ISO/IEC DIR 1 ISO SUP:{yyyy}`::: Available edition years from 2003 (ed1) to 2023 (ed14).
+
+==== ISO/IEC Directives IEC Supplements related
+
+`ISO/IEC DIR IEC SUP`::
+ISO/IEC Directives, IEC Supplement -- Procedures specific to IEC.
+This is the undated reference.
+
+`ISO/IEC DIR IEC SUP:{yyyy}`::: Available edition years from 2009 (ed4) to 2023 (ed17).
+
+`ISO/IEC DIR 1 + IEC SUP`::
+The consolidated version of "ISO/IEC Directives, Part 1 + IEC Supplement -- Procedures specific to IEC".
+This is the undated reference.
+
+`ISO/IEC DIR 1 + IEC SUP:{yyyy}`::: Available edition years from 2016 (ed12) to 2023 (ed19).
+
+==== ISO/IEC JTC 1 governance documents
+
+`ISO/IEC JTC 1 DIR`::
+ISO/IEC JTC 1 Directives. This is the undated reference. Notice that
+these are "internal directives" of ISO/IEC JTC 1. This series existed
+until 2007, after which it became "ISO/IEC Directives -- JTC 1 Supplement".
+
+`ISO/IEC JTC 1 DIR:{yyyy}`::: Available edition years are:
+2004 (5th Edition),
+2005 (5th Edition, Version 1.0),
+2006 (5th Edition, Version 2.0),
+2007 (5th Edition, Version 3.0).
+
+`ISO/IEC DIR JTC 1`::
+ISO/IEC Directives, JTC 1 Supplement -- Procedures specific to JTC 1.
+This is the undated reference.
+
+`ISO/IEC DIR JTC 1:{yyyy}`:::
+Available edition years from 2010 (ed1) to 2023 (ed12).
+
+==== ISO/TC 184/SC 4 important N-documents
+
+`ISO/TC 184/SC 4 N2412`::
+ISO/TC 184/SC 4 Supplementary directives - Rules for the structure and drafting of SC 4 standards for industrial data. 2008-12-16.
+
+`ISO/TC 184/SC 4 N2412`::
+ISO/TC 184/SC 4 Supplementary directives - Rules for the structure and drafting of SC 4 standards for industrial data. 2020-07-31.
+
+`ISO/TC 184/SC 4 N3500`::
+ISO/TC 184/SC 4 Handbook -- SC 4 Supplement to ISO directives.
+2020-08-09.
+
+=== XML serialization
+
+Possible options:
+
+- *bibdata* - If true then wrapp item with _bibdata_ element and add _ext_ element.
+- *note* - Array of hashes `{ text: "Note", type: "note" }`. These notes will be added to XML.
+
+[source,ruby]
+----
+item.to_xml
+=> "<bibitem id="ISO19115-1-2014" type="standard" schema-version="v1.5.6">
+      <title language="en" script="Latn" type="title-intro" format="text/plain">Geographic information</title>
+      <title language="en" script="Latn" type="title-main" format="text/plain">Metadata</title>
+      ...
+    </bibitem>"
+
+item.to_xml bibdata: true
+=> "<bibdata type="standard" schema-version="v1.5.6">
+      <title language="en" script="Latn" type="title-intro" format="text/plain">Geographic information</title>
+      ...
+      <ext schema-version="v1.1.2">
+        <doctype>international-standard</doctype>
+        ...
+      </ext>
+    </bibdata>"
+
+item.to_xml note: [{ content: "Note", type: "note" }]
+=> "<bibitem id="ISO19115-1-2014" type="standard" schema-version="v1.5.6">
+      ...
+      <note type="note">Note</note>
+      ...
+    </bibitem>"
+----
+
+=== Get specific language
+
+[source,ruby]
+----
+item.title 'en'
+=> [#<Relaton::Bib::Title:0x000000013e3de2f0
+  @content="Geographic information",
+  @format="text/plain",
+  @language="en",
+  @locale=nil,
+  @script="Latn",
+  @type="title-intro">,
+ #<Relaton::Bib::Title:0x000000013e3de250
+  @content="Metadata",
+  @format="text/plain",
+  @language="en",
+  @locale=nil,
+  @script="Latn",
+  @type="title-main">,
+ #<Relaton::Bib::Title:0x000000013e3de110
+  @content="Geographic information - Metadata",
+  @format="text/plain",
+  @language="en",
+  @locale=nil,
+  @script="Latn",
+  @type="main">]
+
+item.title 'fr'
+=> [#<Relaton::Bib::Title:0x000000013e3de070
+  @content="Information géographique",
+  @format="text/plain",
+  @language="fr",
+  @locale=nil,
+  @script="Latn",
+  @type="title-intro">,
+ #<Relaton::Bib::Title:0x000000013e3ddfd0
+  @content="Métadonnées",
+  @format="text/plain",
+  @language="fr",
+  @locale=nil,
+  @script="Latn",
+  @type="title-main">,
+ #<Relaton::Bib::Title:0x000000013e3dde90
+  @content="Information géographique - Métadonnées",
+  @format="text/plain",
+  @language="fr",
+  @locale=nil,
+  @script="Latn",
+  @type="main">]
+
+item = Relaton::Iso::Bibliography.get "ISO 19115:2003"
+[relaton-iso] (ISO 19115:2003) Fetching from iso.org ...
+[relaton-iso] (ISO 19115:2003) Found: `ISO 19115:2003`
+=> #<Relaton::Iso::ItemData:0x000000013eefe888 ...>
+
+item.abstract 'en'
+=> [#<Relaton::Bib::LocalizedMarkedUpString:0x000000013e4790c0
+  @content= "ISO 19115:2003 defines the schema required for describing geographic information ...",
+  @language="en",
+  @locale=nil,
+  @script="Latn">]
+----
+
+=== Typed links
+
+Each ISO document has `src` type link and optional `obp`, `rss`, and `pub` link types.
+
+[source,ruby]
+----
+item.source
+=> [#<Relaton::Bib::Uri:0x000000013ee73080
+  @content="https://www.iso.org/standard/26020.html",
+  @language=nil,
+  @locale=nil,
+  @script=nil,
+  @type="src">,
+ #<Relaton::Bib::Uri:0x000000013ee73030
+  @content="https://www.iso.org/contents/data/standard/02/60/26020.detail.rss",
+  @language=nil,
+  @locale=nil,
+  @script=nil,
+  @type="rss">]
+----
+
+[[bulk-data-ingest]]
+=== Bulk data ingest
+
+The curated dataset under https://github.com/relaton/relaton-data-iso[relaton-data-iso] is rebuilt daily by `Relaton::Iso::DataFetcher`, which streams the
+https://www.iso.org/open-data.html[ISO Open Data programme] JSONL feeds — `iso_deliverables_metadata.jsonl` for documents (~80,000 records) and `iso_technical_committees.jsonl` for committee titles — and writes one YAML per primary docid.
+
+Two source modes are exposed (also reachable via `relaton-cli`'s dataset list and the GitHub Actions workflow input):
+
+[source,ruby]
+----
+# Incremental: skip the run if upstream `Last-Modified` matches the local
+# `last_modified.txt`. Falls back to a full pass when `data/` or
+# `index-v1.yaml` is missing.
+Relaton::Iso::DataFetcher.fetch("iso-open-data", output: "data", format: "yaml")
+
+# Full refresh: wipe `output` and re-emit every record. Use when the local
+# tree is suspect or after a parser change that affects emitted YAML.
+Relaton::Iso::DataFetcher.fetch("iso-open-data-all", output: "data", format: "yaml")
+----
+
+=== Logging
+
+RelatonIso uses the relaton-logger gem for logging. By default, it logs to STDOUT. To change the log levels and add other loggers, read the https://github.com/relaton/relaton-logger#usage[relaton-logger] documentation.
+
+== Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To update the ISO index test fixture (used by WebMock in tests), run `rake spec:update_index`. This downloads the latest `index-v1.zip` from the https://github.com/relaton/relaton-data-iso[relaton-data-iso] repository.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+== Exceptional Citations
+
+Single-document lookups via `Bibliography.get` first consult the curated relaton-data-iso index (regenerated daily from the ISO Open Data programme — see <<bulk-data-ingest>>) and fall back to scraping individual pages on http://www.iso.org for items not yet indexed. The following documents are not returned as search results from the ISO website, and the gem returns manually generated references to them.
+
+* `IEV`: used in the metanorma-iso gem to reference Electropedia entries generically. Is resolved to an "all parts" reference to IEC 60050, which in turn is resolved into the specific documents cited by their top-level clause.
+
+== Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/metanorma/relaton-iso
+
+== License
+
+The gem is available as open source under the terms of the https://opensource.org/licenses/MIT[MIT license].

data/lib/relaton/iso/data_fetcher.rb

@@ -248,7 +248,7 @@ module Relaton
         docid = doc.docidentifier.detect(&:primary)
         return unless docid
 
-        file = output_file(docid.content.to_s)
+        file = docid_file(docid.content.to_s)
         if File.exist?(file)
           rewrite_with_same_or_newer(doc, docid, file)
         else
@@ -256,6 +256,21 @@ module Relaton
         end
       end
 
+      # Distinct docids can sanitize to one filename (`ISO/DIS 15869.3` and
+      # `ISO/DIS 15869-3`); take a path of our own rather than treating the
+      # other document as an older edition. A reserved path only ever belongs
+      # to one docid, and `reset_output` wiped `@output`, so an existing file
+      # is this same document again.
+      def docid_file(id)
+        file = unique_output_file(id)
+        plain = output_file(id)
+        if file != plain && !@files.include?(file)
+          Util.warn "File #{plain} exists. Docid: #{id}. " \
+                    "Writing #{file} instead."
+        end
+        file
+      end
+
       def rewrite_with_same_or_newer(doc, docid, file)
         existing = Item.from_yaml(File.read(file, encoding: "UTF-8"))
         if edition_greater?(doc, existing) || replace_substage98?(doc, existing)