relaton 3.1.1 → 3.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +5 -3
- data/lib/relaton/3gpp.rb +1 -1
- data/lib/relaton/bib/converter/csl/from_csl.rb +195 -0
- data/lib/relaton/bib/converter/csl.rb +13 -1
- data/lib/relaton/bib/model/docidentifier.rb +66 -5
- data/lib/relaton/calconnect/hit_collection.rb +1 -1
- data/lib/relaton/cli/command.rb +3 -1
- data/lib/relaton/cli/frontend_assets.rb +7 -6
- data/lib/relaton/cli/subcommand_collection.rb +5 -1
- data/lib/relaton/core/request_error.rb +18 -0
- data/lib/relaton/db/config.rb +9 -1
- data/lib/relaton/db/workers_pool.rb +11 -1
- data/lib/relaton/db.rb +160 -58
- data/lib/relaton/ecma/bibliography.rb +2 -2
- data/lib/relaton/etsi/data_fetcher.rb +16 -6
- data/lib/relaton/iala/bibliography.rb +1 -1
- data/lib/relaton/iana.rb +1 -1
- data/lib/relaton/iec/data_parser.rb +41 -2
- data/lib/relaton/itu/README.adoc +1 -1
- data/lib/relaton/itu/data_crawler_r.rb +1 -1
- data/lib/relaton/jis/README.adoc +2 -0
- data/lib/relaton/jis/data_fetcher.rb +74 -13
- data/lib/relaton/jis/hit_collection.rb +2 -1
- data/lib/relaton/jis/processor.rb +1 -1
- data/lib/relaton/jis/scraper.rb +46 -5
- data/lib/relaton/oasis/bibliography.rb +1 -1
- data/lib/relaton/ogc/hit_collection.rb +1 -1
- data/lib/relaton/ogc.rb +1 -1
- data/lib/relaton/plateau/data_fetcher.rb +10 -1
- data/lib/relaton/plateau/handbook_parser.rb +38 -2
- data/lib/relaton/plateau/hit_collection.rb +20 -5
- data/lib/relaton/sae.rb +1 -1
- data/lib/relaton/version.rb +1 -1
- data/lib/relaton/w3c/bibliography.rb +1 -1
- data/lib/relaton/xsf/bibliography.rb +1 -1
- data/lib/relaton/xsf.rb +1 -1
- metadata +35 -48
- data/lib/relaton/3gpp/CLAUDE.md +0 -218
- data/lib/relaton/adobe/CLAUDE.md +0 -104
- data/lib/relaton/bib/CLAUDE.md +0 -150
- data/lib/relaton/bipm/CLAUDE.md +0 -187
- data/lib/relaton/bsi/CLAUDE.md +0 -35
- data/lib/relaton/calconnect/CLAUDE.md +0 -177
- data/lib/relaton/ccsds/CLAUDE.md +0 -74
- data/lib/relaton/cen/CLAUDE.md +0 -219
- data/lib/relaton/cie/CLAUDE.md +0 -90
- data/lib/relaton/core/CLAUDE.md +0 -140
- data/lib/relaton/db/CLAUDE.md +0 -385
- data/lib/relaton/doi/CLAUDE.md +0 -105
- data/lib/relaton/easc/CLAUDE.md +0 -85
- data/lib/relaton/ecma/CLAUDE.md +0 -217
- data/lib/relaton/etsi/CLAUDE.md +0 -219
- data/lib/relaton/evs/CLAUDE.md +0 -120
- data/lib/relaton/gb/CLAUDE.md +0 -47
- data/lib/relaton/gost/CLAUDE.md +0 -100
- data/lib/relaton/iala/CLAUDE.md +0 -97
- data/lib/relaton/iana/CLAUDE.md +0 -105
- data/lib/relaton/iec/CLAUDE.md +0 -13
- data/lib/relaton/ieee/CLAUDE.md +0 -202
- data/lib/relaton/ietf/CLAUDE.md +0 -382
- data/lib/relaton/iho/CLAUDE.md +0 -76
- data/lib/relaton/index/CLAUDE.md +0 -208
- data/lib/relaton/isbn/CLAUDE.md +0 -47
- data/lib/relaton/iso/CLAUDE.md +0 -38
- data/lib/relaton/itu/CLAUDE.md +0 -685
- data/lib/relaton/jcgm/CLAUDE.md +0 -107
- data/lib/relaton/jis/CLAUDE.md +0 -64
- data/lib/relaton/logger/CLAUDE.md +0 -37
- data/lib/relaton/nist/CLAUDE.md +0 -194
- data/lib/relaton/oasis/CLAUDE.md +0 -251
- data/lib/relaton/ogc/CLAUDE.md +0 -242
- data/lib/relaton/oiml/CLAUDE.md +0 -47
- data/lib/relaton/omg/CLAUDE.md +0 -137
- data/lib/relaton/plateau/CLAUDE.md +0 -126
- data/lib/relaton/sae/CLAUDE.md +0 -91
- data/lib/relaton/sdo/CLAUDE.md +0 -81
- data/lib/relaton/un/CLAUDE.md +0 -91
- data/lib/relaton/w3c/CLAUDE.md +0 -369
- data/lib/relaton/xsf/CLAUDE.md +0 -189
data/lib/relaton/omg/CLAUDE.md
DELETED
|
@@ -1,137 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-omg is a Ruby gem that searches and fetches standards from The Object Management Group (OMG) at https://www.omg.org. It is part of the larger Relaton family of bibliographic data gems (v2.0.0-alpha.1).
|
|
8
|
-
|
|
9
|
-
## Commands
|
|
10
|
-
|
|
11
|
-
- **Run all tests:** `bundle exec rspec`
|
|
12
|
-
- **Run a single test:** `bundle exec rspec spec/relaton/omg/item_spec.rb`
|
|
13
|
-
- **Lint:** `bundle exec rubocop` (follows Ribose OSS style guide)
|
|
14
|
-
- **Lint with autofix:** `bundle exec rubocop -a`
|
|
15
|
-
- **Install deps:** `bundle install`
|
|
16
|
-
|
|
17
|
-
## Architecture
|
|
18
|
-
|
|
19
|
-
### Module structure
|
|
20
|
-
|
|
21
|
-
All source lives under `lib/relaton/omg/`. The main entry point is `lib/relaton/omg.rb` which loads everything under the `Relaton::Omg` module namespace.
|
|
22
|
-
|
|
23
|
-
### Class hierarchy
|
|
24
|
-
|
|
25
|
-
The gem extends `relaton-bib` (~> 2.0.0-alpha.1), the core Relaton bibliographic data library:
|
|
26
|
-
|
|
27
|
-
- `Relaton::Omg::Item` < `Bib::Item` — base item class, uses `Omg::ItemData` model (subclass of `Bib::ItemData`)
|
|
28
|
-
- `Relaton::Omg::Ext` < `Bib::Ext` — overrides `get_schema_version` to return the OMG model version
|
|
29
|
-
- `Relaton::Omg::Bibitem` < `Item` — includes `Bib::BibitemShared`, for `<bibitem>` XML
|
|
30
|
-
- `Relaton::Omg::Bibdata` < `Item` — includes `Bib::BibdataShared`, for `<bibdata>` XML
|
|
31
|
-
- `Relaton::Omg::Docidentifier` < `Bib::Docidentifier` — the document id, backed by `Pubid::Omg`; `Item` declares it, so XML and YAML deserialization produce it too
|
|
32
|
-
- `Relaton::Omg::Processor` < `Core::Processor` — relaton-core integration, delegates to `Bibliography`, `Bibitem`, `Item`
|
|
33
|
-
- `Relaton::Omg::Bibliography` — fetches standards via `Scraper`
|
|
34
|
-
- `Relaton::Omg::Scraper` — scrapes https://www.omg.org/spec for bibliographic data
|
|
35
|
-
|
|
36
|
-
### References are parsed with pubid
|
|
37
|
-
|
|
38
|
-
OMG uses `Pubid::Omg` and no hand regex. The flavor historically had no index
|
|
39
|
-
and no data repository (the BSI and CEN shape: parse the reference, and back
|
|
40
|
-
the document id with pubid). That changed: **relaton-data-omg** now crawls
|
|
41
|
-
www.omg.org/spec and serves the machine index on Pages (the relaton#206
|
|
42
|
-
contract), and `Relaton::Omg::INDEXFILE = "index-v2"` names it — the gem-side
|
|
43
|
-
consumer flip (reading the machine index in `Bibliography` the way the other
|
|
44
|
-
flavors do) is the follow-up; retrieval still scrapes live.
|
|
45
|
-
|
|
46
|
-
The grammar is `OMG <ACRONYM>[ <VERSION>][<sep><PART>]`, where `<sep>` is a
|
|
47
|
-
space, or a slash after a version. The version can carry a beta label (`2.5 beta`,
|
|
48
|
-
`2.0 beta 1`). The part is a volume or a format name (`Superstructure`, `PDF`).
|
|
49
|
-
|
|
50
|
-
- **`get` takes a String or a `Pubid::Omg::Identifier`** (the parse that
|
|
51
|
-
`Relaton::Db` routed with, relaton#205). `Scraper.scrape_page` parses only a
|
|
52
|
-
String, and the log keys print `ref.to_s`.
|
|
53
|
-
- **An unrecognized reference raises.** `Scraper.scrape_page` lets
|
|
54
|
-
`Pubid::Errors::ParseError` propagate, so
|
|
55
|
-
`OMG Model Driven Architecture Guide rev. 2.0` raises and does not show as
|
|
56
|
-
"Not found". Only the transport errors are rescued, by name.
|
|
57
|
-
- **The request URL is the acronym and the version, and nothing else.**
|
|
58
|
-
`get_doc` builds `https://www.omg.org/spec/<ACRONYM>/<VERSION>`, with each
|
|
59
|
-
space of the version turned into `/` (`1.0 beta 2` becomes `1.0/beta/2`). The
|
|
60
|
-
part stays out of the URL: `fetch_link` uses it to find the part's PDF link
|
|
61
|
-
on the version page, and `fetch_title` appends it. The cassettes replay only
|
|
62
|
-
while the URL does not change.
|
|
63
|
-
- **The docid version comes from the page, not from the query.** That is why
|
|
64
|
-
`get "OMG AMI4CCM"` answers `OMG AMI4CCM 1.1`. `fetch_docid` and the History
|
|
65
|
-
ids of `fetch_relation` are rendered by
|
|
66
|
-
`Pubid::Omg::Identifiers::Specification`, not by parsing a joined string.
|
|
67
|
-
- **pubid renders one separator.** `OMG DDS 1.4/PDF` renders back as
|
|
68
|
-
`OMG DDS 1.4 PDF`. Both spellings name the same document, and one rendering
|
|
69
|
-
keeps `==` and `matches?` true between them. If you echo a caller's
|
|
70
|
-
reference, echo the string, not `pubid.to_s`.
|
|
71
|
-
- **Known limit.** OMG's URL spelling `Beta2` (label and number joined) parses
|
|
72
|
-
as a part, not a version. It round-trips, so nothing is lost.
|
|
73
|
-
- **The acronym is the URL segment, verbatim.** 30 of the 270 acronyms in the
|
|
74
|
-
OMG catalog carry a hyphen, a slash, a plus, or start in lower case
|
|
75
|
-
(`DDS-XTypes`, `EDMC-FIBO/BE`, `VSIPL++`, `smartant`). pubid PR #376 accepts
|
|
76
|
-
all 270. A slash **before** the version belongs to the acronym
|
|
77
|
-
(`OMG EDMC-FIBO/BE 1.1` fetches `spec/EDMC-FIBO/BE/1.1`); a slash **after**
|
|
78
|
-
the version separates the part. So `OMG UML/Superstructure` is the acronym
|
|
79
|
-
`UML/Superstructure`; write the part behind a space. With pubid `ce2f75ef`
|
|
80
|
-
(PR #372) those 30 raised, and the migration waited for #376. Fix a grammar
|
|
81
|
-
gap in pubid; do not split the acronym in the flavor.
|
|
82
|
-
|
|
83
|
-
`Docidentifier` parses its `content` on the data side and rescues the parse, so
|
|
84
|
-
a value that is not an OMG identifier stays verbatim with a nil `pubid`. Its
|
|
85
|
-
mutators:
|
|
86
|
-
|
|
87
|
-
- `remove_date!` drops the **version**. OMG identifiers carry no date, and the
|
|
88
|
-
version is the discriminator, as IALA maps `remove_date!` onto its edition.
|
|
89
|
-
So `to_most_recent_reference` turns `OMG AMI4CCM 1.0` into `OMG AMI4CCM`.
|
|
90
|
-
- `remove_part!` drops the part: `OMG UML 2.1.1 Superstructure` becomes
|
|
91
|
-
`OMG UML 2.1.1`.
|
|
92
|
-
- `to_all_parts!` is the inherited no-op. An OMG part is a name, not a numbered
|
|
93
|
-
part, so there is no "all parts" form.
|
|
94
|
-
|
|
95
|
-
The processor sets `@pubid_flavor = :Omg`. That changes nothing today:
|
|
96
|
-
`Pubid::Omg.prefixes` is `["OMG"]`, the same as `@prefix`.
|
|
97
|
-
|
|
98
|
-
### Publication date comes from JSON-LD, not the visible text
|
|
99
|
-
|
|
100
|
-
`Scraper#pub_date` reads the `publicationDate` value out of the page's
|
|
101
|
-
`application/ld+json` block, not the `<dt>Publication Date:</dt>` text. Two
|
|
102
|
-
reasons, both verified against cassettes:
|
|
103
|
-
|
|
104
|
-
- **Locale.** The OMG server renders the month name in its own locale, and
|
|
105
|
-
Cloudflare caches that variant. A response can carry `Content-Language: zh-CN`
|
|
106
|
-
with an otherwise English page (`<html lang="en">`) and a `<dd>七月 2007</dd>`,
|
|
107
|
-
which made `Date.parse` raise `Date::Error`. The request already sends
|
|
108
|
-
`Accept-Language: en-us,en;q=0.5`; the CDN answers `Vary: accept-encoding`
|
|
109
|
-
only, so it does not vary on language and no request header can prevent this.
|
|
110
|
-
- **Precision.** The visible text is a lossy render of the same value:
|
|
111
|
-
`2013-03-31` prints as "March 2013". Parsing the text rounded every OMG date
|
|
112
|
-
down to day 01.
|
|
113
|
-
|
|
114
|
-
The `<dd>` text stays as a fallback for a page with no JSON-LD, and a month name
|
|
115
|
-
it cannot parse logs a warning instead of raising.
|
|
116
|
-
`spec/omg/relaton/omg/scraper_spec.rb` guards all three paths with
|
|
117
|
-
`spec/omg/fixtures/localized_date.html`. Do not "simplify" the parser back to
|
|
118
|
-
the `<dd>` text.
|
|
119
|
-
|
|
120
|
-
### Serialization formats
|
|
121
|
-
|
|
122
|
-
Items can be serialized to/from YAML and XML. Tests verify round-trip fidelity for all three classes. XML output is validated against RELAX NG schemas in `grammars/`.
|
|
123
|
-
|
|
124
|
-
### Test patterns
|
|
125
|
-
|
|
126
|
-
- RSpec with `expect` syntax (monkey patching disabled)
|
|
127
|
-
- `canon` for XML comparison
|
|
128
|
-
- `ruby-jing` for RELAX NG schema validation against `grammars/relaton-omg-compile.rng`
|
|
129
|
-
- VCR cassettes in `spec/vcr_cassettes/` for HTTP interaction recording
|
|
130
|
-
- Fixtures in `spec/fixtures/` (YAML and XML reference files)
|
|
131
|
-
- Tests follow a round-trip pattern: load fixture → parse → serialize → compare to fixture
|
|
132
|
-
|
|
133
|
-
## Style
|
|
134
|
-
|
|
135
|
-
- Ruby >= 3.1.0
|
|
136
|
-
- Rubocop inherits from Ribose OSS guide with `rubocop-rails` required but Rails cops disabled
|
|
137
|
-
- OMG document reference format: `OMG {ACRONYM}[ {VERSION}][ {PART}]` (e.g., `OMG AMI4CCM 1.0`, `OMG UML 2.1.1 Superstructure`)
|
|
@@ -1,126 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-plateau is a Ruby gem for retrieving and managing bibliographic metadata for Project PLATEAU (Japanese 3D city model standards published by MLIT). It is part of the Relaton ecosystem of bibliographic gems.
|
|
8
|
-
|
|
9
|
-
## Commands
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
# Install dependencies
|
|
13
|
-
bundle install
|
|
14
|
-
|
|
15
|
-
# Run all tests
|
|
16
|
-
bundle exec rake spec
|
|
17
|
-
|
|
18
|
-
# Run a single test file
|
|
19
|
-
bundle exec rspec spec/relaton/plateau/bibitem_spec.rb
|
|
20
|
-
|
|
21
|
-
# Run a specific test by line number
|
|
22
|
-
bundle exec rspec spec/relaton/plateau/bibitem_spec.rb:15
|
|
23
|
-
|
|
24
|
-
# Lint
|
|
25
|
-
bundle exec rake rubocop
|
|
26
|
-
|
|
27
|
-
# Default rake task (runs specs)
|
|
28
|
-
bundle exec rake
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
## Architecture
|
|
32
|
-
|
|
33
|
-
All code lives in `lib/relaton/plateau/`. The gem uses LutaML::Model::Serializable for data modeling with automatic XML/YAML serialization.
|
|
34
|
-
|
|
35
|
-
### Models
|
|
36
|
-
- **Item** (`item.rb`) — base Plateau item extending `Bib::Item`, declares `model ItemData` and adds `ext` attribute
|
|
37
|
-
- **ItemData** (`item_data.rb`) — data class extending `Bib::ItemData`, returned by `Item.from_xml`, `Item.from_yaml`, and `Bibliography.get`
|
|
38
|
-
- **Bibitem** (`bibitem.rb`) — bibliographic item variant (includes `Bib::BibitemShared`)
|
|
39
|
-
- **Bibdata** (`bibdata.rb`) — bibliographic data variant (includes `Bib::BibdataShared`)
|
|
40
|
-
- **Ext** (`ext.rb`) — extension element: doctype, subdoctype, flavor, editorialgroup, stagename, filesize, etc.
|
|
41
|
-
- **Doctype** (`doctype.rb`) — extends `Bib::Doctype`, valid values: `handbook`, `technical-report`, `annex`
|
|
42
|
-
|
|
43
|
-
### Retrieval & Data Fetching
|
|
44
|
-
- **Bibliography** (`bibliography.rb`) — module with `get(code)` for index-based document retrieval, returns `ItemData`
|
|
45
|
-
- **HitCollection** / **Hit** — search the relaton-data-plateau index and fetch YAML documents
|
|
46
|
-
- **DataFetcher** (`data_fetcher.rb`) — extends `Core::DataFetcher`, scrapes MLIT JSON APIs for handbooks and technical reports
|
|
47
|
-
- **Parser** / **HandbookParser** / **TechnicalReportParser** — parse JSON into `ItemData` objects
|
|
48
|
-
- **Processor** (`processor.rb`) — standard Relaton processor plugin (`get`, `from_xml`, `from_yaml`, `fetch_data`, `grammar_hash`)
|
|
49
|
-
|
|
50
|
-
### Serialization Methods
|
|
51
|
-
- `bib.to_xml` — XML (bibitem format)
|
|
52
|
-
- `Item.to_yaml(bib)` — YAML (class method)
|
|
53
|
-
- `bib.to_rfcxml` — BibXML/RFC XML
|
|
54
|
-
- `Item.from_xml(xml)` — parse XML into `ItemData`
|
|
55
|
-
- `Item.from_yaml(yaml)` — parse YAML into `ItemData`
|
|
56
|
-
|
|
57
|
-
### Data Sources
|
|
58
|
-
- Handbooks: `https://www.mlit.go.jp/plateau/_next/data/1.3.0/libraries/handbooks.json`
|
|
59
|
-
- Technical Reports: `https://www.mlit.go.jp/plateau/_next/data/1.3.0/libraries/technical-reports.json`
|
|
60
|
-
- Pre-fetched index: `https://raw.githubusercontent.com/relaton/relaton-data-plateau/v2/` (`HitCollection::ENDPOINT`)
|
|
61
|
-
|
|
62
|
-
### Pubid-backed index (index-v2)
|
|
63
|
-
|
|
64
|
-
The index is **pubid-structured** (`_type: pubid:plateau:*` rows), mirroring
|
|
65
|
-
NIST/CIE. The producer and the consumer `Relaton::Index.find_or_create` call
|
|
66
|
-
sites (`data_fetcher.rb`, `hit_collection.rb`) pass
|
|
67
|
-
`pubid_class: ::Pubid::Plateau::Identifier`, and `INDEXFILE = "index-v2"`.
|
|
68
|
-
`processor.rb#remove_index_file` passes `url: true` and `file:` only: the delete
|
|
69
|
-
never reads the index (see `lib/relaton/index/CLAUDE.md`).
|
|
70
|
-
|
|
71
|
-
**Ids are pubid-canonical.** The parsers emit the form `Pubid::Plateau.parse`
|
|
72
|
-
accepts, which also matches the MLIT source:
|
|
73
|
-
- Handbook: `PLATEAU Handbook #NN 第X.Y版` (Japanese edition label — the parser
|
|
74
|
-
keeps the source `第X.Y版`; the Latinized `#edition` is retained only for the
|
|
75
|
-
structured `Bib::Edition`/`ext` fields).
|
|
76
|
-
- Technical Report: `PLATEAU Technical Report #NN` (**no edition** — pubid TRs
|
|
77
|
-
carry none; the document still has a `1.0` `Bib::Edition`).
|
|
78
|
-
- Sub-numbers use a hyphen (`#46-1`), not the source underscore (`#46_1`).
|
|
79
|
-
|
|
80
|
-
`DataFetcher#save_document` parses the canonical id via `#pubid` (a round-trip
|
|
81
|
-
guard: `from_hash(to_hash) == to_hash`) and stores the `Pubid::Plateau::Identifier`
|
|
82
|
-
object; unparseable ids are warned and skipped (defensive — the parser already
|
|
83
|
-
emits canonical, so nothing is expected to skip).
|
|
84
|
-
|
|
85
|
-
**Search accepts both forms.** `HitCollection#find` parses the query via `#ref`
|
|
86
|
-
(`Pubid::Plateau.parse`, memoized; an unparseable query **raises**
|
|
87
|
-
`Pubid::Errors::ParseError`; `Bibliography.get` rescues transport errors only,
|
|
88
|
-
as `Relaton::RequestError`, and `Hit#item` raises that same error on a
|
|
89
|
-
non-200) and matches on **pubid objects**, mirroring the
|
|
90
|
-
other pubid flavors: an exact edition uses `index.search(ref, exact: true)`
|
|
91
|
-
(pubid `==` compares type + number + annex + edition); a family query
|
|
92
|
-
(`ref.edition.nil?` — an edition-less handbook, or any Technical Report) uses
|
|
93
|
-
the default subset match `ref === row`, where the omitted edition matches any
|
|
94
|
-
value. pubid declares `annex` strict for PLATEAU (pubid#408), so
|
|
95
|
-
`PLATEAU Handbook #03` does not reach the annex `#03-1`;
|
|
96
|
-
`hit_collection_spec.rb` pins that against the index fixture. This keeps the
|
|
97
|
-
family-vs-exact decision and the match on the *same* parsed identifier (no raw
|
|
98
|
-
`@ref` regex). Canonical (`第1.0版`), edition-less (`#00`), and **legacy Latin**
|
|
99
|
-
(`#00 1.0`) references all resolve — `Pubid::Plateau` normalizes Latin input to
|
|
100
|
-
the canonical id (metanorma/pubid #269; needs the pubid `main` pin). No
|
|
101
|
-
relaton-side Latin shim exists by design.
|
|
102
|
-
|
|
103
|
-
`EDITION_SUFFIX` is **not** used for matching; it lives only in `#to_all_editions`,
|
|
104
|
-
which strips the edition off the fetched document's `Bib::Docidentifier`/docnumber
|
|
105
|
-
**strings** (relaton-bib content, not pubid) to build the family record — it
|
|
106
|
-
matches both Japanese and Latin suffixes so it works against the current (Latin)
|
|
107
|
-
VCR-cassette documents and future canonical ones.
|
|
108
|
-
|
|
109
|
-
### Schema Validation
|
|
110
|
-
RNG grammar files in `grammars/` define the XML schema. Tests validate fixtures against `relaton-plateau-compile.rng` using the `ruby-jing` gem.
|
|
111
|
-
|
|
112
|
-
## Testing
|
|
113
|
-
|
|
114
|
-
- **Index fixture:** `spec/fixtures/index-v2.zip` (pubid-backed, `_type: pubid:plateau:*` rows) is pre-loaded into the `Relaton::Index` pool in `before(:suite)` via a `Relaton::Index::Type` built with `::Pubid::Plateau::Identifier` (configured in `spec/support/webmock.rb`, mirroring NIST/CIE). Run `rake spec:update_index` to refresh from relaton-data-plateau.
|
|
115
|
-
- RSpec with `expect` syntax (no monkey-patching)
|
|
116
|
-
- VCR cassettes record HTTP interactions (in `spec/fixtures/vcr_cassettes/`)
|
|
117
|
-
- `canon` for XML comparison assertions
|
|
118
|
-
- `jing` matcher for RNG schema validation
|
|
119
|
-
- SimpleCov for code coverage
|
|
120
|
-
|
|
121
|
-
## Key Conventions
|
|
122
|
-
|
|
123
|
-
- Document identifiers are pubid-canonical: `PLATEAU Handbook #XX 第Y.Z版` (Japanese edition) or `PLATEAU Technical Report #XX` (no edition); sub-numbers use a hyphen (`#XX-N`). See "Pubid-backed index" above.
|
|
124
|
-
- The gem supports multi-format serialization: XML, YAML, BibXML (RFC XML)
|
|
125
|
-
- RuboCop follows Ribose OSS style guide; target Ruby version is 3.0
|
|
126
|
-
- The `ext` element in XML/YAML carries PLATEAU-specific metadata (doctype, flavor, editorialgroup, etc.)
|
data/lib/relaton/sae/CLAUDE.md
DELETED
|
@@ -1,91 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md — SAE flavor
|
|
2
|
-
|
|
3
|
-
`Relaton::Sae` fetches SAE International standards (relaton/relaton-sae#1).
|
|
4
|
-
|
|
5
|
-
## Development
|
|
6
|
-
|
|
7
|
-
- `bundle exec rake spec:sae` — this flavor's suite (from the repo root).
|
|
8
|
-
- No VCR: the specs stub HTTP with WebMock. `spec/sae/fixtures/crossref/`
|
|
9
|
-
holds three Crossref cursor pages built from real records (11 works:
|
|
10
|
-
six J3016/J3194 editions, three AMS4121 revisions, a part and a
|
|
11
|
-
letter-pair number). `spec/sae/fixtures/index-v2.yaml` and `fixtures/data/` are the
|
|
12
|
-
**output of `DataFetcher` over those pages**: regenerate both together by
|
|
13
|
-
running the crawler over the page fixtures, never by hand.
|
|
14
|
-
|
|
15
|
-
## Where the data comes from
|
|
16
|
-
|
|
17
|
-
**sae.org answers every script request with a CAPTCHA** (product pages and
|
|
18
|
-
search alike), so no scraper is possible. SAE deposits every standard edition
|
|
19
|
-
with Crossref instead: prefix `10.4271`, type `standard`, ~47,700 records. The
|
|
20
|
-
DOI suffix is SAE's own edition code in lowercase:
|
|
21
|
-
|
|
22
|
-
| DOI | SAE code | scheme |
|
|
23
|
-
|---|---|---|
|
|
24
|
-
| `10.4271/j3016_202104` | `SAE J3016_202104` | J-series: `_YYYYMM` edition date |
|
|
25
|
-
| `10.4271/ams4121j` | `SAE AMS4121J` | aerospace: revision letter, no date |
|
|
26
|
-
| `10.4271/j2337/2_201702` | `SAE J2337/2_201702` | a part |
|
|
27
|
-
|
|
28
|
-
Crossref's text search does **not** find a standard by its id (it matches
|
|
29
|
-
titles), so an undated reference cannot be answered live. The flavor is
|
|
30
|
-
therefore index-based, the ECMA/IALA shape:
|
|
31
|
-
|
|
32
|
-
- **Producer** — `DataFetcher` (`relaton fetch-data sae-crossref`) walks
|
|
33
|
-
`api.crossref.org/prefixes/10.4271/works?filter=type:standard` with the
|
|
34
|
-
deep-paging cursor (1000 a request, one request a second through
|
|
35
|
-
`Core::Pacer`; 429/5xx and transport errors retried with back-off, five
|
|
36
|
-
attempts, then `RequestError`, because
|
|
37
|
-
a partial crawl would publish a partial index). Each work goes through
|
|
38
|
-
`Relaton::Doi::Parser`, then gets SAE's code as its **primary**
|
|
39
|
-
docidentifier (type `SAE`), the DOI beside it, and `ext` doctype
|
|
40
|
-
`standard`, flavor `sae`. `fetched` is cleared: a data file carries no
|
|
41
|
-
fetch date, or every daily crawl would rewrite and commit all ~47,700
|
|
42
|
-
files. `Bibliography#fetch_doc` sets it when it reads a document.
|
|
43
|
-
- **Consumer** — `Bibliography.get` reads the `relaton-data-sae` Pages index
|
|
44
|
-
(`PAGES_URL`) and the document from `ENDPOINT` (raw GitHub).
|
|
45
|
-
|
|
46
|
-
## Three identifier forms
|
|
47
|
-
|
|
48
|
-
`DoiCode` gives each record three strings:
|
|
49
|
-
|
|
50
|
-
- `#native` — the code in the DOI's spelling, `SAE J3016_202104`,
|
|
51
|
-
`SAE AMSQQA250/4B`. The output file is named after it
|
|
52
|
-
(`data/sae-amsqqa250-4b.yaml`), so a file name never depends on pubid.
|
|
53
|
-
- `#printed` — the code as SAE prints it (`SAE AMS-QQ-A-250/4B`): pubid's
|
|
54
|
-
render of the native code, or the native code when pubid cannot read it.
|
|
55
|
-
It is the primary docidentifier.
|
|
56
|
-
- `#reference` — what the index row is parsed from: the native code when the
|
|
57
|
-
DOI has a date, else the code with the year of the record's `approved` date
|
|
58
|
-
(`SAE AMS4121J:2017`), so a dated reference also reaches an aerospace
|
|
59
|
-
revision.
|
|
60
|
-
|
|
61
|
-
`Pubid::Sae` (pubid >= 2.0.9, pubid#665) reads all three, the month
|
|
62
|
-
included, so the `Db` keys the query and the fetched document with the
|
|
63
|
-
default `cache_pubid`. A reference pubid cannot read (`SAE J3016_2021`,
|
|
64
|
-
`SAE CPGM2016`) answers nil with a log line (relaton#235); it does not raise.
|
|
65
|
-
|
|
66
|
-
## Selection
|
|
67
|
-
|
|
68
|
-
`index.search(pubid)` with no block, so the subset match `query === row`:
|
|
69
|
-
a revision, a year or a month the query omits matches any value; the type
|
|
70
|
-
(`J`, `AMS`, …) is compared, and a part and an amendment are `subset_strict`
|
|
71
|
-
(`SAE J2337` does not answer with `SAE J2337/2`). Of the matches the highest
|
|
72
|
-
revision wins (`[length, letters]`, so `AA` follows `Z`), then the latest
|
|
73
|
-
year and month, then the file name (the index sort is not stable). A year
|
|
74
|
-
argument goes on a **copy** of a pubid handed in by `Db`.
|
|
75
|
-
|
|
76
|
-
The index key carries the month, so two editions of one year are two rows.
|
|
77
|
-
Two DOIs that spell one code (`ams-4121j`, `ams4121j`) share a key, and
|
|
78
|
-
`DataFetcher#add_to_index` keeps the later one (DOI date, then approval
|
|
79
|
-
date).
|
|
80
|
-
|
|
81
|
-
## What pubid still cannot read
|
|
82
|
-
|
|
83
|
-
About 960 of the ~47,700 deposits: 791 `CP*` collected-paper products (not
|
|
84
|
-
standards) and ~170 one-off shapes (`SUP1` supplements, language tails such
|
|
85
|
-
as `_PTBR`, `_4Q2012`, `_MMYYYY` dates). See pubid's `lib/pubid/sae/CLAUDE.md`.
|
|
86
|
-
Such a record is written, reported in `@errors` (the crawl's GitHub issue),
|
|
87
|
-
and not indexed: one row that fails to deserialize makes `Relaton::Index`
|
|
88
|
-
reject the whole index.
|
|
89
|
-
|
|
90
|
-
A multi-letter tail stays in `number` (`J1939DA`, and also `AMS4121AA`), so
|
|
91
|
-
a two-letter revision is its own bucket: `SAE AMS4121` does not reach it.
|
data/lib/relaton/sdo/CLAUDE.md
DELETED
|
@@ -1,81 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
Dev notes for `Relaton::Sdo` — the SDO organization & logo store.
|
|
4
|
-
|
|
5
|
-
## What this is
|
|
6
|
-
|
|
7
|
-
A **non-flavor** component that answers metanorma#346 (central logo store) and
|
|
8
|
-
its sister relaton-db#132 (map abbreviation → organisation name/translation).
|
|
9
|
-
It is **not** a bibliographic flavor: it has **no processor**, **no
|
|
10
|
-
`Db::Registry` entry**, and is not reachable from `Db#fetch`. It is a standalone
|
|
11
|
-
top-level store, exposed through `Relaton.organization`.
|
|
12
|
-
|
|
13
|
-
This deliberately honors the scope decision in `lib/relaton/db/CLAUDE.md`
|
|
14
|
-
("org metadata … kept out of the processors"): nothing here touches a processor.
|
|
15
|
-
The user's choice was to still surface it from the `relaton` gem itself (one
|
|
16
|
-
`require`, one entry point) as a **lazily-autoloaded** component — referencing
|
|
17
|
-
`Relaton::Sdo` or calling `Relaton.organization` is what loads it.
|
|
18
|
-
|
|
19
|
-
## Public API
|
|
20
|
-
|
|
21
|
-
```ruby
|
|
22
|
-
org = Relaton.organization("ISO") # case-insensitive; nil if unknown
|
|
23
|
-
org.name # default (untranslated) name
|
|
24
|
-
org.name("fr") # translation, or nil
|
|
25
|
-
org.name(language: "fr") # same, via the language: keyword
|
|
26
|
-
org.logo_query(format:, size:, style:) # all args optional → matching Logo list
|
|
27
|
-
org.logo(format:, size:, style:) # one Logo: nil if 0, the match if 1,
|
|
28
|
-
# raises Sdo::Error if the filter is ambiguous
|
|
29
|
-
logo.content # raw binary (lazy HTTP GET of logo.url, memoized)
|
|
30
|
-
logo.data_uri # "data:<mime>;base64,…"
|
|
31
|
-
logo.save # write to a filename derived from the URL; returns path
|
|
32
|
-
logo.save("/path.eps") # write to a custom path
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
## Data model & retrieval flow
|
|
36
|
-
|
|
37
|
-
`Store` (singleton) lazily builds an index from a single **`index.yaml`**
|
|
38
|
-
manifest served by the `relaton-data-sdo` data repo, and memoizes it (`reset!`
|
|
39
|
-
drops the memo — used by specs). Shape:
|
|
40
|
-
|
|
41
|
-
```yaml
|
|
42
|
-
organizations:
|
|
43
|
-
ISO:
|
|
44
|
-
name:
|
|
45
|
-
- content: International Organization for Standardization # default
|
|
46
|
-
- { language: fr, content: Organisation internationale de normalisation }
|
|
47
|
-
logo:
|
|
48
|
-
- { style: default, format: eps, size: 700x300, url: "https://…/iso.eps" }
|
|
49
|
-
- { style: red, format: svg, url: "https://…/red.svg",
|
|
50
|
-
applicability: "stage>=60" } # opaque text; the store never interprets it
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
- **`style`** is the primary discriminator for multiple logos per org (e.g.
|
|
54
|
-
`default`, `red`, `grey`, `iso_1972`, `white_paper_2022`, BIPM `si_aspect_full`).
|
|
55
|
-
- **`applicability`** (year/stage/doctype/cover-color rules) is carried through
|
|
56
|
-
verbatim and exposed; **selection is the consumer's job** (metanorma), not the
|
|
57
|
-
store's.
|
|
58
|
-
|
|
59
|
-
`Config` (`Relaton::Sdo.configuration`) holds the index `url`, the cache
|
|
60
|
-
`storage_dir` (default `~/.relaton/sdo`), the `storage` backend (defaults to
|
|
61
|
-
`Relaton::Index::FileStorage`; swappable, e.g. S3), and the `ttl` (24 h).
|
|
62
|
-
`Fetcher.fetch_index` serves a fresh on-disk copy within the TTL, re-downloads
|
|
63
|
-
otherwise, and falls back to a stale cache if the network fails. A URL with no
|
|
64
|
-
scheme (or `file://`) is read straight off disk — handy for fixtures/specs. Logo
|
|
65
|
-
binaries are fetched on demand and memoized on the `Logo` (no disk cache yet).
|
|
66
|
-
|
|
67
|
-
## Tests
|
|
68
|
-
|
|
69
|
-
`spec/sdo/` is a self-contained suite (auto-discovered by `rake spec`). It uses
|
|
70
|
-
WebMock to stub the index and logo URLs, points `storage_dir` at a fresh
|
|
71
|
-
`Dir.mktmpdir` per example, and resets the store between examples. It
|
|
72
|
-
`require "relaton"` because the public entry point (`Relaton.organization`)
|
|
73
|
-
lives in the root entry file, which the shared spec_helper does not load.
|
|
74
|
-
|
|
75
|
-
## Scope / not yet done
|
|
76
|
-
|
|
77
|
-
- The real `relaton-data-sdo` data repo (assets + build GHA) — see
|
|
78
|
-
`HANDOFFS/relaton__relaton-data-sdo.md`. Until it publishes, `Config#url`
|
|
79
|
-
points at its (future) raw `index.yaml`.
|
|
80
|
-
- No structured interpretation of `applicability`; no on-disk cache of logo
|
|
81
|
-
binaries; no writeback/authoring.
|
data/lib/relaton/un/CLAUDE.md
DELETED
|
@@ -1,91 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-un is a Ruby gem that provides bibliographic data models for UN documents, part of the Relaton family of gems. It extends `relaton-bib` with UN-specific metadata using the LutaML-Model serialization framework. Currently on the `lutaml-integration` branch undergoing a major migration from the old `RelatonUn` namespace to `Relaton::Un`.
|
|
8
|
-
|
|
9
|
-
## Commands
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
# Install dependencies
|
|
13
|
-
bundle install
|
|
14
|
-
|
|
15
|
-
# Run all tests (also the default rake task)
|
|
16
|
-
bundle exec rake spec
|
|
17
|
-
|
|
18
|
-
# Run a specific test file
|
|
19
|
-
bundle exec rspec spec/relaton/un/bibitem_spec.rb
|
|
20
|
-
|
|
21
|
-
# Run a specific test by line number
|
|
22
|
-
bundle exec rspec spec/relaton/un/bibitem_spec.rb:7
|
|
23
|
-
|
|
24
|
-
# Lint
|
|
25
|
-
bundle exec rubocop
|
|
26
|
-
|
|
27
|
-
# Interactive console with gem loaded
|
|
28
|
-
bin/console
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
## Architecture
|
|
32
|
-
|
|
33
|
-
### Flavor Model Pattern
|
|
34
|
-
|
|
35
|
-
The gem follows the Relaton flavor pattern: base classes from `relaton-bib` (`Bib::Item`) are extended with a flavor-specific `Ext` class containing UN-specific metadata.
|
|
36
|
-
|
|
37
|
-
```
|
|
38
|
-
Bib::ItemData (from relaton-bib)
|
|
39
|
-
└── Relaton::Un::ItemData # UN flavor identity, enables NamespaceHelper#namespace → Relaton::Un
|
|
40
|
-
|
|
41
|
-
Bib::Item (from relaton-bib)
|
|
42
|
-
└── Relaton::Un::Item # model ItemData; adds Ext attribute
|
|
43
|
-
├── Relaton::Un::Bibitem # includes Bib::BibitemShared (for <bibitem> XML)
|
|
44
|
-
└── Relaton::Un::Bibdata # includes Bib::BibdataShared (for <bibdata> XML)
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
### Lookup
|
|
48
|
-
|
|
49
|
-
`Bibliography.get` takes a String (`UN TRADE/CEFACT/2004/32`, the `UN ` token
|
|
50
|
-
optional) or a parsed `Pubid::Un::Identifier`, which renders the document
|
|
51
|
-
symbol without the token (`TRADE/CEFACT/2004/32`) — the form the UN data
|
|
52
|
-
itself writes. Either becomes the symbol the search matches (`document_symbol`).
|
|
53
|
-
`Relaton::Db` routes UN by the `UN ` prefix (`Registry#route` falls back to
|
|
54
|
-
`class_by_ref`): `Pubid::Un` renders the symbol without the token, so a
|
|
55
|
-
`UN …` reference is not an exact parse. A symbol without the token
|
|
56
|
-
(`TRADE/CEFACT/2004/32`) is not routed at all — no flavor claims it — and
|
|
57
|
-
raises `Relaton::UnknownReferenceError`. `Pubid::Un` also parses a bare DOI
|
|
58
|
-
exactly (`10.6028/NIST.IR.8245`), but UN does not claim it (`10.` is a DOI
|
|
59
|
-
prefix), so it does not route to UN either.
|
|
60
|
-
|
|
61
|
-
### Key Classes (all in `lib/relaton/un/`)
|
|
62
|
-
|
|
63
|
-
- **ItemData** — extends `Bib::ItemData`; the UN-specific data container used by `Item` via `model ItemData`
|
|
64
|
-
- **Item** — extends `Bib::Item`, adds `ext` attribute of type `Ext`
|
|
65
|
-
- **Bibitem / Bibdata** — extend Item with shared mixins for two XML serialization formats
|
|
66
|
-
- **Ext** — UN-specific metadata: `doctype`, `distribution`, `session`, `submissionlanguage`, `job_number`; extends `Bib::Ext`
|
|
67
|
-
- **Session** — session metadata (number, date, item numbers/names, collaborator, agenda_id)
|
|
68
|
-
- **Doctype** — restricts document types to: recommendation, plenary, addendum, communication, corrigendum, reissue, agenda, budgetary, sec-gen-notes, expert-report, resolution
|
|
69
|
-
|
|
70
|
-
### LutaML-Model Serialization
|
|
71
|
-
|
|
72
|
-
All model classes use `Lutaml::Model::Serializable`. XML mappings are defined in `xml do` blocks:
|
|
73
|
-
- `map_attribute` for XML attributes (e.g., `schema-version` → `schema_version`)
|
|
74
|
-
- `map_element` for XML elements (e.g., `session-date` → `session_date`)
|
|
75
|
-
- `collection: true` for array attributes
|
|
76
|
-
- `values: [...]` for restricted string values
|
|
77
|
-
|
|
78
|
-
### Schema Validation
|
|
79
|
-
|
|
80
|
-
RelaxNG grammars in `grammars/` validate XML output. The compiled schema is `grammars/relaton-un-compile.rng`. Tests use `ruby-jing` for validation.
|
|
81
|
-
|
|
82
|
-
### Testing Approach
|
|
83
|
-
|
|
84
|
-
Tests are round-trip serialization tests: load a fixture, serialize to XML/YAML, compare against the original fixture. Bibitem and Bibdata specs also validate XML against the RelaxNG schema. Fixtures live in `spec/fixtures/`.
|
|
85
|
-
|
|
86
|
-
## Dependencies
|
|
87
|
-
|
|
88
|
-
- **relaton-bib** (~> 2.0.0-alpha.1) — base bibliographic model with LutaML-Model integration
|
|
89
|
-
- **faraday** — HTTP client
|
|
90
|
-
- **ruby-jing** (dev) — RelaxNG schema validation in tests
|
|
91
|
-
- **canon** (dev) — XML comparison in tests
|