relaton 3.1.1 → 3.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +5 -3
- data/lib/relaton/3gpp.rb +1 -1
- data/lib/relaton/bib/converter/csl/from_csl.rb +195 -0
- data/lib/relaton/bib/converter/csl.rb +13 -1
- data/lib/relaton/bib/model/docidentifier.rb +66 -5
- data/lib/relaton/calconnect/hit_collection.rb +1 -1
- data/lib/relaton/cli/command.rb +3 -1
- data/lib/relaton/cli/frontend_assets.rb +7 -6
- data/lib/relaton/cli/subcommand_collection.rb +5 -1
- data/lib/relaton/core/request_error.rb +18 -0
- data/lib/relaton/db/config.rb +9 -1
- data/lib/relaton/db/workers_pool.rb +11 -1
- data/lib/relaton/db.rb +160 -58
- data/lib/relaton/ecma/bibliography.rb +2 -2
- data/lib/relaton/etsi/data_fetcher.rb +16 -6
- data/lib/relaton/iala/bibliography.rb +1 -1
- data/lib/relaton/iana.rb +1 -1
- data/lib/relaton/iec/data_parser.rb +41 -2
- data/lib/relaton/itu/README.adoc +1 -1
- data/lib/relaton/itu/data_crawler_r.rb +1 -1
- data/lib/relaton/jis/README.adoc +2 -0
- data/lib/relaton/jis/data_fetcher.rb +74 -13
- data/lib/relaton/jis/hit_collection.rb +2 -1
- data/lib/relaton/jis/processor.rb +1 -1
- data/lib/relaton/jis/scraper.rb +46 -5
- data/lib/relaton/oasis/bibliography.rb +1 -1
- data/lib/relaton/ogc/hit_collection.rb +1 -1
- data/lib/relaton/ogc.rb +1 -1
- data/lib/relaton/plateau/data_fetcher.rb +10 -1
- data/lib/relaton/plateau/handbook_parser.rb +38 -2
- data/lib/relaton/plateau/hit_collection.rb +20 -5
- data/lib/relaton/sae.rb +1 -1
- data/lib/relaton/version.rb +1 -1
- data/lib/relaton/w3c/bibliography.rb +1 -1
- data/lib/relaton/xsf/bibliography.rb +1 -1
- data/lib/relaton/xsf.rb +1 -1
- metadata +35 -48
- data/lib/relaton/3gpp/CLAUDE.md +0 -218
- data/lib/relaton/adobe/CLAUDE.md +0 -104
- data/lib/relaton/bib/CLAUDE.md +0 -150
- data/lib/relaton/bipm/CLAUDE.md +0 -187
- data/lib/relaton/bsi/CLAUDE.md +0 -35
- data/lib/relaton/calconnect/CLAUDE.md +0 -177
- data/lib/relaton/ccsds/CLAUDE.md +0 -74
- data/lib/relaton/cen/CLAUDE.md +0 -219
- data/lib/relaton/cie/CLAUDE.md +0 -90
- data/lib/relaton/core/CLAUDE.md +0 -140
- data/lib/relaton/db/CLAUDE.md +0 -385
- data/lib/relaton/doi/CLAUDE.md +0 -105
- data/lib/relaton/easc/CLAUDE.md +0 -85
- data/lib/relaton/ecma/CLAUDE.md +0 -217
- data/lib/relaton/etsi/CLAUDE.md +0 -219
- data/lib/relaton/evs/CLAUDE.md +0 -120
- data/lib/relaton/gb/CLAUDE.md +0 -47
- data/lib/relaton/gost/CLAUDE.md +0 -100
- data/lib/relaton/iala/CLAUDE.md +0 -97
- data/lib/relaton/iana/CLAUDE.md +0 -105
- data/lib/relaton/iec/CLAUDE.md +0 -13
- data/lib/relaton/ieee/CLAUDE.md +0 -202
- data/lib/relaton/ietf/CLAUDE.md +0 -382
- data/lib/relaton/iho/CLAUDE.md +0 -76
- data/lib/relaton/index/CLAUDE.md +0 -208
- data/lib/relaton/isbn/CLAUDE.md +0 -47
- data/lib/relaton/iso/CLAUDE.md +0 -38
- data/lib/relaton/itu/CLAUDE.md +0 -685
- data/lib/relaton/jcgm/CLAUDE.md +0 -107
- data/lib/relaton/jis/CLAUDE.md +0 -64
- data/lib/relaton/logger/CLAUDE.md +0 -37
- data/lib/relaton/nist/CLAUDE.md +0 -194
- data/lib/relaton/oasis/CLAUDE.md +0 -251
- data/lib/relaton/ogc/CLAUDE.md +0 -242
- data/lib/relaton/oiml/CLAUDE.md +0 -47
- data/lib/relaton/omg/CLAUDE.md +0 -137
- data/lib/relaton/plateau/CLAUDE.md +0 -126
- data/lib/relaton/sae/CLAUDE.md +0 -91
- data/lib/relaton/sdo/CLAUDE.md +0 -81
- data/lib/relaton/un/CLAUDE.md +0 -91
- data/lib/relaton/w3c/CLAUDE.md +0 -369
- data/lib/relaton/xsf/CLAUDE.md +0 -189
data/lib/relaton/gb/CLAUDE.md
DELETED
|
@@ -1,47 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
4
|
-
|
|
5
|
-
## Project Overview
|
|
6
|
-
|
|
7
|
-
relaton-gb retrieves Chinese National Standards (Guobiao — `GB`, `GB/T`, `GB/Z`, plus social-org and sector standards) using the Relaton model. It scrapes several Chinese government portals (`openstd.samr.gov.cn`, `ttbz.org.cn`, `std.gov.cn`) with Mechanize. Depends on relaton-iso for ISO-based item/extension models, and on Chinese-specific classifiers (`cnccs`, `gb-agencies`).
|
|
8
|
-
|
|
9
|
-
## Development
|
|
10
|
-
|
|
11
|
-
```bash
|
|
12
|
-
bundle exec rake # default task → rspec
|
|
13
|
-
bundle exec rspec spec/relaton/gb/bibliography_spec.rb # single file
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
(RuboCop is configured via the shared `.hound.yml`; the monorepo root config applies otherwise.)
|
|
17
|
-
|
|
18
|
-
## Architecture
|
|
19
|
-
|
|
20
|
-
Namespace: `Relaton::Gb`. Note: GB places its model classes at the top of `lib/relaton/gb/` (`item.rb`, `bibitem.rb`, `bibdata.rb`, `ext.rb`, …) rather than under a `model/` subdir.
|
|
21
|
-
|
|
22
|
-
Retrieval flow:
|
|
23
|
-
|
|
24
|
-
1. **Bibliography** (`lib/relaton/gb/bibliography.rb`) — `get`/`search` dispatch by prefix: `GB*` → `GbScraper` (openstd.samr.gov.cn), `T/` → `TScraper` (ttbz.org.cn), others → `SecScraper` (std.gov.cn).
|
|
25
|
-
2. **Scrapers** (`scraper.rb` + `gb_scraper.rb` / `t_scraper.rb` / `sec_scraper.rb`) — scrape the relevant portal for hits, then scrape the detail page into an `ItemData`.
|
|
26
|
-
3. **HitCollection / Hit** (`hit_collection.rb`, `hit.rb`, extend the `Relaton::Core` bases) — `Hit#item` lazy-loads the document; a hit holds `pid`, `docref`, `release_date`, `status`.
|
|
27
|
-
4. **Item / ItemData / Ext** (`item.rb`, `item_data.rb`, `ext.rb`) — `Item` extends `Bib::Item`; `Ext` carries GB-specific fields (`gbtype`, `ccs`, `stagename`, `plannumber`). Supporting models: `gb_type.rb`, `ccs.rb` (Chinese classification), `stage_name.rb`, `committee.rb`, `structured_identifier.rb`.
|
|
28
|
-
5. **Processor** (`lib/relaton/gb/processor.rb`) — registry integration; `@prefix = "CN"`, `@defaultprefix = %r{^(GB|GB/T|GB/Z) }`, `@pubid_flavor = :Gb`. Lazy-`require_relative`s `../gb` in its methods.
|
|
29
|
-
|
|
30
|
-
## Identifiers (pubid)
|
|
31
|
-
|
|
32
|
-
GB parses identifiers with `Pubid::Gb::Identifier`. It publishes no index, so it has no `INDEXFILE` and no `pubid_class:` (the BSI/CEN shape, see the root `CLAUDE.md`). The attributes it reads are `publisher`, `mandate`, `number`, `part`, `year` and `all_parts`.
|
|
33
|
-
|
|
34
|
-
- **Query side raises.** `Bibliography.get` parses the caller's code without a rescue, so a reference that is not a GB identifier raises `Pubid::Errors::ParseError`. `get` also takes the `Pubid::Gb::Identifier` that `Relaton::Db` parsed (relaton#205) and never mutates it: an `AllParts` pubid (or a String `(all parts)`) sets `all_parts` and unwraps its identifier, and every change after that is on the `exclude(:year)` copy. The year comes from the argument or from `pubid.year`; `all_parts` sets `part = "1"` on the copy to find part 1. `search` still routes on the raw string (`GB|GJ|GS` → `GbScraper`, `T/…` → `TScraper`, else `SecScraper`), so the portal for each reference and the cassettes are unchanged.
|
|
35
|
-
- **Hit filter.** `search_filter` parses each `hit.docref` (rescued: a docref is data; an unparseable one drops the hit) and keeps the hits that `query.matches?`. It ignores `:year` only when the query has no year. It **never** ignores `:part`: `GB/T 20223` and `GB/T 20223.1` are two documents, and `gb_spec.rb` asserts that `GB/T 20223-2006` does not return `GB/T 20223.1-2006`. This is a deliberate departure from the migration hand-off, which proposed to ignore a part the query omits. It also stops `GB/T 1.1` from matching `GB/T 1.10`, which the old substring compare allowed.
|
|
36
|
-
- **`Docidentifier`** (`docidentifier.rb`) follows the CEN/OMG shape: `content=` parses into `@pubid` (rescued, data side), and the mutators replace the pubid with an `exclude` copy and write back through `store_content`. `to_all_parts!` excludes part and year, sets `all_parts`, and pubid renders the ` (all parts)` suffix itself (and parses it back). A nil pubid makes every mutator a no-op.
|
|
37
|
-
- **Scrapers** (`scraper.rb`) read the docref through a rescued `docref_pubid`. `get_prefix` keys `yaml/prefixes.yaml` with the first segment of `publisher` (a `Pubid::Components::Publisher`, so read it as `publisher.to_s`; `T/GZAEPI` → `T`). An unknown key gives nil, and `get_gbtype` keeps a nil prefix rather than raising — a portal docref is data. `get_mandate` maps `mandate` `T` → recommended, `Z` → guidelines, nil → mandatory. `get_contributors` passes `publisher.to_s` to `gb_agencies`, which names the same organization for `GB` and `GB/T`. `parse_docref` returns `[id without part and year, part, year]`; the first value is `plannumber`.
|
|
38
|
-
- **Pubid traps.** `Pubid::Gb` accepts any upper-case publisher code (`ISO 123` parses), so use `foo` or `GB/T` for a parse-error example. `year` is a reader over `date` (no `year=`); set it with `date = Pubid::Components::Date.new(year:)`. Local (`DB11/T`) forms do not parse. pubid PR #388 (`dcea7380`) renamed `publisher_code` to the `publisher` component, added the year to `to_hash`, and made `GBn` parse; a pubid older than that breaks this flavor. pubid renders an em dash (`T/GZAEPI 001—2018`) as a hyphen; the fixture `tgzaepi_001_2018.xml` still records the em dash, and its spec is pending until pubid decides.
|
|
39
|
-
- **Prefix register.** `@pubid_flavor = :Gb` sources `Relaton.prefix_flavor` from `Pubid::Gb.prefixes`, so `GB`, `GB/T`, `JB/T`, `T/` … resolve to `Relaton::Gb`. `CN` is not a pubid prefix, so `prefix_flavor("CN")` is `[]`; no caller in this repo used it, and `Db` routing still reads `@prefix`. `HB` resolves to `[Gb, Nist, Bsi]`, pinned in `spec/relaton/prefix_flavor_spec.rb`.
|
|
40
|
-
|
|
41
|
-
## External dependencies
|
|
42
|
-
|
|
43
|
-
`cnccs ~> 0.1.1`, `gb-agencies ~> 0.0.1`, `mechanize ~> 2.10`, `csv ~> 3.0`, `relaton-core`, `relaton-iso`.
|
|
44
|
-
|
|
45
|
-
## Testing
|
|
46
|
-
|
|
47
|
-
RSpec with WebMock + VCR (cassettes in `spec/vcr_cassettes/`). Network is blocked via WebMock. SimpleCov and canon are loaded via `spec/support/`. No index fixture — GB is scraped, not indexed; `spec/fixtures/` holds XML/YAML round-trip data.
|
data/lib/relaton/gost/CLAUDE.md
DELETED
|
@@ -1,100 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
This file provides guidance to Claude Code (claude.ai/code) when working with the
|
|
4
|
-
`Relaton::Gost` flavor.
|
|
5
|
-
|
|
6
|
-
## Project Overview
|
|
7
|
-
|
|
8
|
-
`Relaton::Gost` retrieves **GOST** standards — the interstate/national standards of the
|
|
9
|
-
former-Soviet space (Государственный стандарт), published by Rosstandart and the
|
|
10
|
-
Euro-Asian Council for Standardization (EASC/МГС). It is **index-backed** (no scraping):
|
|
11
|
-
it searches a pre-built **pubid-structured** index via `Relaton::Index` and fetches
|
|
12
|
-
per-document YAML from the `relaton/relaton-data-gost` GitHub repo. The model layer mirrors
|
|
13
|
-
the `Relaton::Easc` sibling; the pubid-backed retrieval mirrors `Relaton::Oiml`.
|
|
14
|
-
|
|
15
|
-
## Architecture
|
|
16
|
-
|
|
17
|
-
Namespace: `Relaton::Gost`. Retrieval flow:
|
|
18
|
-
|
|
19
|
-
1. **Bibliography** (`bibliography.rb`) — `get(code, year, opts)` parses the citation with
|
|
20
|
-
`Pubid::Gost.parse`, looks it up in the **pubid-structured** index, fetches the matching
|
|
21
|
-
`row[:file]` YAML over `Net::HTTP` (`Item.from_yaml`, raising `Relaton::RequestError` on
|
|
22
|
-
non-200), and stamps `fetched`. `search` picks the **latest edition** (`max_by { year }`)
|
|
23
|
-
among number-matched candidates; `get` then strips the year for an **undated** citation via
|
|
24
|
-
`to_most_recent_reference` (so `GOST 1.0` renders undated but resolves the newest edition),
|
|
25
|
-
while a dated citation (`GOST 1.0-92`) or explicit `year` pins that edition. Mirrors
|
|
26
|
-
`lib/relaton/oiml/bibliography.rb`.
|
|
27
|
-
2. **Index** — `Relaton::Index.find_or_create(:gost, url: "#{ENDPOINT}index-v2.zip",
|
|
28
|
-
file: "index-v2.yaml", pubid_class: ::Pubid::Gost::Identifier)`. Each row's `:id` is a
|
|
29
|
-
`_type: pubid:gost:{interstate,national}-standard` hash that `Relaton::Index` rebuilds into
|
|
30
|
-
a `Pubid::Gost` identifier via `from_hash`; passing the parsed pubid to `#search` lets it
|
|
31
|
-
binary-search candidates by number (see `Bibliography#rows`). `INDEXFILE =
|
|
32
|
-
"index-v2"` is in `lib/relaton/gost.rb`.
|
|
33
|
-
3. **Item / ItemData / Ext** (`item.rb`, `item_data.rb`, `ext.rb`) — `Item` extends
|
|
34
|
-
`Bib::Item`, re-declares `ext` to the typed `Gost::Ext` and `docidentifier` to the pubid
|
|
35
|
-
`Gost::Docidentifier`. `Ext`'s GOST fields (`urn`, `webpage`, `ics_code`, `developer`,
|
|
36
|
-
`keywords` [collection], `pages`, `designation_original`) round-trip natively through YAML
|
|
37
|
-
**and** XML (mapped in both `xml` and `key_value`). (The live dataset uses `urn`/`webpage`/
|
|
38
|
-
`ics_code`/`pages`/`designation_original`; `developer`/`keywords` are declared but unused.)
|
|
39
|
-
4. **Doctype** (`doctype.rb`) — `TYPES = %w[interstate national preliminary methodological]`
|
|
40
|
-
(GOST, GOST R, PST, OD categories). Carried in the inherited `content` attribute.
|
|
41
|
-
5. **Docidentifier** (`docidentifier.rb`) — wraps a `Pubid::Gost` identifier (`attr_reader
|
|
42
|
-
:pubid`); `content=` re-parses to keep `@pubid` in sync, and `remove_date!` nil-s the pubid
|
|
43
|
-
`year` and re-renders `content` (GOST carries the edition as `year`, not `date`) so
|
|
44
|
-
`to_most_recent_reference` yields `GOST R 34.12`. Mirrors `Oiml::Docidentifier`.
|
|
45
|
-
6. **Processor** (`processor.rb`) — registry integration; `@short = :relaton_gost`,
|
|
46
|
-
`@prefix = "GOST"`, `@idtype = "GOST"`, and `@defaultprefix = %r{^(?:GOST|ГОСТ)\b}` so **both
|
|
47
|
-
the Latin `GOST` and Cyrillic `ГОСТ` surface forms route here** (the `\b` stops it swallowing
|
|
48
|
-
longer tokens). `remove_index_file` passes `url: true` and `file:` only — the delete never
|
|
49
|
-
reads the index, so it needs no `pubid_class` (see `lib/relaton/index/CLAUDE.md`). Every method touching a
|
|
50
|
-
flavor constant lazy-`require_relative`s `../gost` first (the lazy-registry invariant;
|
|
51
|
-
`spec/relaton/lazy_loading_spec.rb` guards it).
|
|
52
|
-
|
|
53
|
-
There are no scrapers — everything comes from the curated index + GitHub YAML.
|
|
54
|
-
|
|
55
|
-
## Pubid
|
|
56
|
-
|
|
57
|
-
The flavor is **pubid-backed** via `Pubid::Gost` (metanorma/pubid#108), on the `pubid` `main`
|
|
58
|
-
branch the root `Gemfile` already pins (for JCGM). `Bibliography#rows` selects a dated
|
|
59
|
-
citation with `index.search(query, exact: true)` (pubid `==`) and an undated one with the
|
|
60
|
-
default subset match `query === row`: the omitted year matches any value, so it matches every
|
|
61
|
-
edition sharing subtype + number. `===` requires the same class, so interstate and national
|
|
62
|
-
stay distinct, and `1.1` does not match `1.10`. pubid declares `copublisher` strict for GOST
|
|
63
|
-
(pubid#408), so an omitted copublisher means "none". Measured over the full
|
|
64
|
-
`relaton-data-gost` index (37,508 rows): `===` and the old `matches?(…, ignore: [:year])` agree
|
|
65
|
-
on all 94,279 pairs. The
|
|
66
|
-
`relaton-data-gost` `index-v2.yaml` was built with this **same** pubid, so the rows deserialize
|
|
67
|
-
(a mismatched pubid would make `Relaton::Index` reject the whole index — cf. the root
|
|
68
|
-
`CLAUDE.md` JCGM note). The pinned release `2.0.0.pre.alpha.13` carries `Pubid::Gost` and
|
|
69
|
-
`subset_strict`; any lower pin must too.
|
|
70
|
-
|
|
71
|
-
## Dataset
|
|
72
|
-
|
|
73
|
-
`relaton/relaton-data-gost` is the live data repo behind `ENDPOINT` (a real pubid-structured
|
|
74
|
-
`index-v2.yaml` + `data/*.yaml`). The specs run fully offline against a small committed subset
|
|
75
|
-
(see Testing).
|
|
76
|
-
|
|
77
|
-
## Testing
|
|
78
|
-
|
|
79
|
-
`spec/gost/` — self-contained, run via `bundle exec rake spec:gost` (or `cd spec/gost &&
|
|
80
|
-
bundle exec rspec -I . .`). Coverage:
|
|
81
|
-
|
|
82
|
-
- `ext_spec.rb`, `item_spec.rb` — YAML round-trip of the typed ext fields.
|
|
83
|
-
- `bibliography_spec.rb` — retrieval against real records copied from `relaton-data-gost`:
|
|
84
|
-
the pubid-structured index fixture `fixtures/index-v2.zip` is pre-loaded into the
|
|
85
|
-
`Relaton::Index` pool in `before(:suite)` (with `pubid_class`), and `fixtures/data/*.yaml`
|
|
86
|
-
are served by WebMock (`support/webmock.rb`) — **no live network**. Covers dated/undated/
|
|
87
|
-
Cyrillic lookups, undated→latest-edition, year-pinning, not-found (`nil`), and non-200
|
|
88
|
-
(`Relaton::RequestError`). Fixture docs: interstate `GOST 1.0` (editions 2015 + 92) and
|
|
89
|
-
national `GOST R 34.12-2015`.
|
|
90
|
-
- `processor_spec.rb` — processor shape + Latin/Cyrillic prefix routing.
|
|
91
|
-
|
|
92
|
-
To refresh the fixture, copy the wanted rows from `relaton-data-gost/index-v2.yaml` into
|
|
93
|
-
`fixtures/index-v2.yaml`, re-zip it to `fixtures/index-v2.zip`, and copy the matching
|
|
94
|
-
`data/*.yaml` files.
|
|
95
|
-
|
|
96
|
-
There is **no RelaxNG grammar** for GOST and no XML-schema validation spec: the metanorma
|
|
97
|
-
document model (`metanorma-model-iso/grammars/`) defines no GOST grammar, so there is nothing
|
|
98
|
-
to copy into `grammar/` and nothing to validate `to_xml` against. This matches the sibling
|
|
99
|
-
`easc` flavor (also grammar-less). If an official GOST grammar is ever added upstream, copy the
|
|
100
|
-
`<flavor>{,-compile}.rng` pair into `grammar/` and add a Jing-validation spec then.
|
data/lib/relaton/iala/CLAUDE.md
DELETED
|
@@ -1,97 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Architecture
|
|
4
|
-
|
|
5
|
-
**Pubid-backed docidentifier.** `Relaton::Iala::Docidentifier`
|
|
6
|
-
(`docidentifier.rb`, `< Bib::Docidentifier`) parses its `content` into a
|
|
7
|
-
`Pubid::Iala::Identifier` kept in `@pubid`, while the lutaml `content` attribute
|
|
8
|
-
stays a plain string for serialization. Parsing is **soft**: `content=` lazily
|
|
9
|
-
`require "pubid"` + `require "pubid/iala"` (pubid core must load first for
|
|
10
|
-
`Pubid::PrefixesSupport`) and rescues `LoadError`/`StandardError`, so a missing
|
|
11
|
-
pubid gem or non-IALA/unparseable content leaves `@pubid` nil rather than raising.
|
|
12
|
-
|
|
13
|
-
It implements the base class's abstract `remove_part!` / `remove_date!` /
|
|
14
|
-
`to_all_parts!` by mutating the pubid and re-rendering via `refresh_content!`
|
|
15
|
-
(which writes through the aliased `store_content` — the inherited string setter —
|
|
16
|
-
so in-place mutations aren't clobbered by a re-parse). IALA's identifier models
|
|
17
|
-
only `publisher`/`number`/`edition`/`language`, so the mapping is IALA-specific:
|
|
18
|
-
|
|
19
|
-
- **`remove_date!` → clears `edition`.** IALA carries no date component; `edition`
|
|
20
|
-
is its version discriminator, so this yields the version-agnostic ("most
|
|
21
|
-
recent") reference, e.g. `IALA S1070 Ed 2.0` → `IALA S1070`.
|
|
22
|
-
- **`remove_part!` → clears the (unused) `part`/`subpart` attributes.** A no-op
|
|
23
|
-
for the rendered string today: IALA folds any subpart inline into `number`
|
|
24
|
-
(e.g. `0103-1`), so the part is not a separable component. Implemented anyway
|
|
25
|
-
so it never raises, and it starts working automatically if pubid-iala ever
|
|
26
|
-
models `part` separately.
|
|
27
|
-
- **`to_all_parts!` → `remove_part!` + `remove_date!`, then wraps `@pubid` in
|
|
28
|
-
pubid's `AllParts`.** `content` stays the edition-stripped string — IALA's
|
|
29
|
-
own renderer has no "(all parts)" marker, and `content` is cached from the
|
|
30
|
-
pre-wrap pubid on purpose. `#pubid` itself, read directly, now answers
|
|
31
|
-
`all_parts? == true` and renders WITH pubid's generic marker (`#to_s`),
|
|
32
|
-
since it's the wrapper. Mirrors `lib/relaton/iec/model/docidentifier.rb`.
|
|
33
|
-
|
|
34
|
-
All three no-op safely when `@pubid` is nil, so `Bib::ItemData#to_all_parts` /
|
|
35
|
-
`#to_most_recent_reference` never raise on IALA items.
|
|
36
|
-
|
|
37
|
-
**Index (`index-v2`, pubid-keyed).** `INDEXFILE` is the pubid-backed
|
|
38
|
-
`index-v2` published by `relaton-data-iala`: 701 rows of
|
|
39
|
-
`_type: pubid:iala:*` with `number`/`edition`/`language`. The consumer
|
|
40
|
-
(`Bibliography#index`) passes
|
|
41
|
-
`pubid_class: ::Pubid::Iala::Identifier` — that is what makes `Relaton::Index`
|
|
42
|
-
deserialize the rows into identifiers, sort them by `id.root.number`, and let
|
|
43
|
-
`Type#search` bsearch. Omitting it leaves the rows as raw hashes and
|
|
44
|
-
`FileIO#sorted` false, so every lookup scans all 701 rows, silently. That was
|
|
45
|
-
the state before this flavor adopted the W3C pattern; the data repo already
|
|
46
|
-
published a pubid-shaped `index-v2`, only the consumer was unwired.
|
|
47
|
-
`Processor#remove_index_file` passes `url: true` and `file:` only: the delete
|
|
48
|
-
never reads the index (see `lib/relaton/index/CLAUDE.md`).
|
|
49
|
-
|
|
50
|
-
`Bibliography#best_match` follows the ETSI/W3C idiom:
|
|
51
|
-
|
|
52
|
-
- **Pass the pubid, not the string.** `Type#search_candidates` narrows only when
|
|
53
|
-
the argument is not a `String`, so passing the reference text — which this
|
|
54
|
-
flavor used to do — disables the binary search however the index was built.
|
|
55
|
-
- **Selection is pubid's asymmetric subset match** (`Pubid::SubsetMatch`):
|
|
56
|
-
`pubid === r[:id]`, the reference on the left. A component the reference omits
|
|
57
|
-
matches any value, so a bare `IALA M0001` finds the `Ed 9.0 (E)` row, while
|
|
58
|
-
`IALA R0106 Ed 2.1 (F)` reaches only its own. `edition` and `language` are
|
|
59
|
-
IALA's only optional components.
|
|
60
|
-
- **The document type is never a wildcard.** It is the identifier's class, and
|
|
61
|
-
`===` requires the same class. 17 of the 309 index numbers are shared by two
|
|
62
|
-
types (`R1001` and `C1001` are different documents), so they stay apart. An
|
|
63
|
-
`Annex` row carries its base identifier, which `===` compares with its own
|
|
64
|
-
subset match.
|
|
65
|
-
- **Newest edition wins, deterministically.** Editions are dotted versions and
|
|
66
|
-
must be compared segment-by-segment as integers: `"10.0"` is newer than
|
|
67
|
-
`"9.0"` but loses as a string. No published edition has a segment above 9
|
|
68
|
-
today, so this is latent — the same bug already fixed for ETSI. Ties break on
|
|
69
|
-
the edition text and then the file path, because the index sort is not stable.
|
|
70
|
-
Among rows of one edition the language-neutral record wins: it is the base
|
|
71
|
-
document, the others are translations.
|
|
72
|
-
- **An unparseable reference raises.** `parse_ref` lets
|
|
73
|
-
`Pubid::Errors::ParseError` propagate, and the full-scan substring fallback is
|
|
74
|
-
gone with it: a substring match answered an ambiguous reference with whichever
|
|
75
|
-
row sorted first, instead of telling the caller the reference is not an
|
|
76
|
-
identifier.
|
|
77
|
-
|
|
78
|
-
`require "pubid"` sits in `lib/relaton/iala.rb` (the IANA/IHO form), so pubid
|
|
79
|
-
loads with the flavor. `Processor#remove_index_file` names no pubid class, so the
|
|
80
|
-
cold `Db#clear` path that `spec/relaton/lazy_loading_spec.rb` guards does not
|
|
81
|
-
depend on it. `Docidentifier` keeps its own
|
|
82
|
-
lazy, `LoadError`-rescuing require: it is reached by deserialization, which must
|
|
83
|
-
not depend on the flavor entry file having been loaded.
|
|
84
|
-
|
|
85
|
-
## Testing
|
|
86
|
-
|
|
87
|
-
- `bundle exec rake spec:iala` — run the IALA suite.
|
|
88
|
-
- Specs live in `spec/iala/relaton/iala/` and run self-contained (see the root
|
|
89
|
-
`CLAUDE.md` Testing section).
|
|
90
|
-
- `spec/iala/fixtures/index-v2.zip` is a 19-row curated subset of the published
|
|
91
|
-
index, copied row-for-row so the stored shapes stay verbatim. It covers the
|
|
92
|
-
matching rules above: a multi-edition document (`R0101`, editions `2` and
|
|
93
|
-
`3.0`), a four-language document (`R0106`), a number shared by two types
|
|
94
|
-
(`R1001` / `C1001`), a sub-part folded into the number (`C0103-1`), and the
|
|
95
|
-
language-neutral/translation split. `spec/iala/support/webmock.rb` seeds it
|
|
96
|
-
into the `Relaton::Index` pool **with `pubid_class:`** — without that the
|
|
97
|
-
suite would pass while exercising something the runtime never does.
|
data/lib/relaton/iana/CLAUDE.md
DELETED
|
@@ -1,105 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Development
|
|
4
|
-
|
|
5
|
-
- `bundle install` — install dependencies
|
|
6
|
-
- `bundle exec rake spec:iana` — run this flavor's tests
|
|
7
|
-
(rubocop is configured via the root `.rubocop.yml` but is not in the bundle)
|
|
8
|
-
|
|
9
|
-
## What this flavor is
|
|
10
|
-
|
|
11
|
-
IANA entries are protocol **registries**, not numbered standards. An identifier
|
|
12
|
-
is a hierarchical registry slug capped at one "/":
|
|
13
|
-
|
|
14
|
-
```
|
|
15
|
-
IANA _6lowpan-parameters
|
|
16
|
-
IANA _6lowpan-parameters/lowpan_nhc
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
`Parser#docnumber` builds the bare slug (`registry[/sub-registry]`) by joining a
|
|
20
|
-
nested registry's XML `id` to its parent's; `Core::DataFetcher#output_file`
|
|
21
|
-
turns that into the document filename (`data/_6lowpan-parameters-lowpan_nhc.yaml`).
|
|
22
|
-
|
|
23
|
-
`output_file` collapses `/` and `-` to the same character, so two distinct
|
|
24
|
-
registries can want one file — `rpki/signed-objects` and `rpki-signed-objects`
|
|
25
|
-
do, the only such pair in the 3405 published rows. `save_doc` therefore writes
|
|
26
|
-
through `Core::DataFetcher#unique_output_file`, which hands the second one a
|
|
27
|
-
digest-suffixed path instead of overwriting the first; the warning stays so the
|
|
28
|
-
clash is visible in the crawl log. Before that fix the corpus had 3405 index
|
|
29
|
-
rows over 3404 files, and one id resolved to the wrong document.
|
|
30
|
-
|
|
31
|
-
## Index (`index-v2`, pubid-keyed)
|
|
32
|
-
|
|
33
|
-
`INDEXFILE = "index-v2"` (`lib/relaton/iana.rb`). Rows carry
|
|
34
|
-
`Pubid::Iana::Identifiers::Registry` identifiers, serialized as:
|
|
35
|
-
|
|
36
|
-
```yaml
|
|
37
|
-
- :id:
|
|
38
|
-
_type: pubid:iana:registry
|
|
39
|
-
sub_registry: lowpan_nhc # absent for a top-level registry
|
|
40
|
-
number: _6lowpan-parameters
|
|
41
|
-
:file: data/_6lowpan-parameters-lowpan_nhc.yaml
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
`number` holds the **top-level** slug — not a document number, because an IANA
|
|
45
|
-
identifier has none (digits only ever appear glued inside a slug token:
|
|
46
|
-
`sip-parameters/sip-parameters-13`, `idna-tables-11.0.0`). It is the index key:
|
|
47
|
-
`Relaton::Index::Type#candidates_by_number` bsearches on `id.root.number.to_s`,
|
|
48
|
-
so a registry and all of its sub-registries share one bucket (655 buckets over
|
|
49
|
-
the 3405 published rows; median 3, max 73 — `pcep`) and `Bibliography#pubid_match?`
|
|
50
|
-
separates them. `registry` is a **derived reader** over `number`, so it is not a
|
|
51
|
-
serialized key.
|
|
52
|
-
|
|
53
|
-
The pair that must stay in step, both passing
|
|
54
|
-
`pubid_class: ::Pubid::Iana::Identifier`:
|
|
55
|
-
|
|
56
|
-
| role | file |
|
|
57
|
-
|---|---|
|
|
58
|
-
| producer | `data_fetcher.rb` — `#index` + the guarded `#add_to_index` |
|
|
59
|
-
| consumer | `bibliography.rb` — `#index`, `#parse_ref`, `#pubid_match?` |
|
|
60
|
-
|
|
61
|
-
`processor.rb#remove_index_file` passes `url: true` and `file:` only: the delete
|
|
62
|
-
never reads the index (see `lib/relaton/index/CLAUDE.md`).
|
|
63
|
-
|
|
64
|
-
`#add_to_index` rescues and warns rather than raising: `Relaton::Index` rejects
|
|
65
|
-
the **whole** index if one row fails to deserialize, so an unparseable slug has
|
|
66
|
-
to be skipped at write time instead of aborting the crawl.
|
|
67
|
-
|
|
68
|
-
### Why a stale index is loud, not silent
|
|
69
|
-
|
|
70
|
-
A row serialized before `number` existed (`{_type, registry, sub_registry}`)
|
|
71
|
-
deserializes with `number` **nil and no error at all**: lutaml ignores unknown
|
|
72
|
-
keys, and `Index::FileIO#id_supported?` skips its `to_hash`/`from_hash`
|
|
73
|
-
validation for a concrete subclass — which every IANA id is. pubid therefore
|
|
74
|
-
fails at *render* time instead: `Identifier#require_number!` raises
|
|
75
|
-
`ArgumentError` ("… regenerate the index") from `to_s`, `to_urn` and
|
|
76
|
-
`to_mr_string`. **If you see that error the index is stale — rebuild it, don't
|
|
77
|
-
work around it.** `spec/iana/relaton/iana_spec.rb`'s "the pubid index-v2
|
|
78
|
-
fixture" block is the tripwire for the fixture.
|
|
79
|
-
|
|
80
|
-
### Migration note (v1 → v2)
|
|
81
|
-
|
|
82
|
-
`index-v1` was keyed on the bare slug string, so `Index::Type#search` matched by
|
|
83
|
-
**substring** and `min_by { |i| i[:id] }` picked the shortest hit — `IANA rpki`
|
|
84
|
-
also matched `rpki/signed-objects`. v2 matches the identifier exactly, so a
|
|
85
|
-
partial reference no longer resolves.
|
|
86
|
-
|
|
87
|
-
This flavor no longer produces `index-v1`. `relaton-data-iana`'s own
|
|
88
|
-
`crawler.rb` derives it from `index-v2` (a plain YAML transform:
|
|
89
|
-
`sub_registry ? "#{number}/#{sub_registry}" : number`) so released gem lines
|
|
90
|
-
keep resolving.
|
|
91
|
-
|
|
92
|
-
## Testing
|
|
93
|
-
|
|
94
|
-
- **Framework:** RSpec with VCR cassettes and WebMock.
|
|
95
|
-
- **Index fixture:** `spec/iana/fixtures/index-v2.zip` (all 3405 published rows)
|
|
96
|
-
is deserialized through `pubid_class` once per process and **re-seeded into
|
|
97
|
-
the `Relaton::Index` pool before every example** — see
|
|
98
|
-
`spec/iana/support/webmock.rb` for why (the producer-side `find_or_create`
|
|
99
|
-
evicts it otherwise). Its `actual?` override claims only the consumer
|
|
100
|
-
(`url:`) lookup.
|
|
101
|
-
- **VCR cassettes:** `spec/iana/vcr_cassettes/` — index downloads are ignored by
|
|
102
|
-
VCR (`support/vcr.rb`), since the fixture answers them.
|
|
103
|
-
- The XML round-trip specs validate against `grammar/relaton-iana-compile.rng`
|
|
104
|
-
with Jing, which needs a **Java runtime**; without one they fail with
|
|
105
|
-
`Jing::ExecutionError`, unrelated to any code change.
|
data/lib/relaton/iec/CLAUDE.md
DELETED
|
@@ -1,13 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Development
|
|
4
|
-
|
|
5
|
-
- `bundle install` — install dependencies
|
|
6
|
-
- `bundle exec rake spec` — run tests
|
|
7
|
-
- `bundle exec rubocop` — lint
|
|
8
|
-
|
|
9
|
-
## Testing
|
|
10
|
-
|
|
11
|
-
- **Framework:** RSpec with VCR cassettes and WebMock
|
|
12
|
-
- **Index fixture:** `spec/fixtures/index-v2.zip` is pre-loaded into `Relaton::Index` pool in `before(:suite)` (configured in `spec/support/webmock.rb`). Run `rake spec:update_index` to refresh from relaton-data-iec. The fixture is the lean pubid-v2 `to_hash` index (`_type`-tagged rows); rows are deserialized via `Pubid::Iec::Identifier.from_hash` and validated by round-trip in relaton-index (no `id_keys` allowlist).
|
|
13
|
-
- **VCR cassettes:** `spec/vcr_cassettes/` — index download requests are ignored by VCR (handled by fixture).
|
data/lib/relaton/ieee/CLAUDE.md
DELETED
|
@@ -1,202 +0,0 @@
|
|
|
1
|
-
# CLAUDE.md
|
|
2
|
-
|
|
3
|
-
## Development
|
|
4
|
-
|
|
5
|
-
- `bundle install` — install dependencies
|
|
6
|
-
- `bundle exec rake spec` — run tests
|
|
7
|
-
- `bundle exec rubocop` — lint
|
|
8
|
-
|
|
9
|
-
## Index (pubid-structured `index-v2`)
|
|
10
|
-
|
|
11
|
-
IEEE consumes and produces the **structured `index-v2`** (`INDEXFILE = "index-v2"`
|
|
12
|
-
in `lib/relaton/ieee.rb`): each row's `:id` is a `Pubid::Ieee::Identifier`
|
|
13
|
-
serialized to the `_type: pubid:ieee:*` (lutaml) form, keyed on the index via
|
|
14
|
-
`pubid_class: ::Pubid::Ieee::Identifier`.
|
|
15
|
-
|
|
16
|
-
- **Write side.** Unlike the doc-fetching pipeline, the index is built by a
|
|
17
|
-
**separate step**: `Relaton::Ieee::DataFetcher.build_index(dir:, index_file:)`
|
|
18
|
-
(`data_fetcher.rb`). It globs the per-document `data/*.yaml` files, extracts each
|
|
19
|
-
primary non-trademark IEEE docidentifier, parses it via `.pubid` (a guarded
|
|
20
|
-
`::Pubid::Ieee::Identifier.parse` + `from_hash(to_hash) == to_hash` round-trip
|
|
21
|
-
check), `add_or_update`s the object, and `save`s. Ids pubid can't parse/round-trip
|
|
22
|
-
are skipped; a per-file skip warning **and** a final
|
|
23
|
-
`IEEE index-v2.yaml: N/M indexed, K skipped (P% coverage)` line are logged so the
|
|
24
|
-
loss is never silent. `DataFetcher.fetch` (doc generation) is unchanged — the
|
|
25
|
-
data repo (`relaton-data-ieee`) calls `build_index` after obtaining `data/`.
|
|
26
|
-
- **Read side.** `bibliography.rb#search` passes `pubid_class:` to `find_or_create`,
|
|
27
|
-
parses the query with `parse_pubid` (falling back to the raw String on failure)
|
|
28
|
-
and passes the pubid to `index.search` for number-narrowing. A parsed query
|
|
29
|
-
selects its rows by **pubid equality with the year aside**
|
|
30
|
-
(`id.exclude(:year) == pubid.exclude(:year)`), so a redline, a draft, a
|
|
31
|
-
corrigendum and another number (`1003.10` for `1003.1`) never match. A dated
|
|
32
|
-
query (its own year, or the `year` argument) takes that year; an **undated**
|
|
33
|
-
query takes the **latest** edition, and `get` returns it as
|
|
34
|
-
`to_most_recent_reference` (undated IEEE docids, no date, an `instanceOf`
|
|
35
|
-
relation to the edition; `opts[:keep_year]` keeps it dated) — relaton-ieee#29.
|
|
36
|
-
Before this, a substring match + `min_by(to_s)` answered `IEEE 528` with the
|
|
37
|
-
oldest edition, `528-1994`. When equality finds nothing (an unparseable query,
|
|
38
|
-
or one of the ~4% of forms whose year pubid keeps elsewhere, e.g.
|
|
39
|
-
`IEEE Std 1003.1-1988/INT-1992`, the corrigendum's legacy base year), the old
|
|
40
|
-
substring match is the fallback. `processor.rb#remove_index_file` passes
|
|
41
|
-
`url: true` and `file:` only: the delete never reads the index (see
|
|
42
|
-
`lib/relaton/index/CLAUDE.md`).
|
|
43
|
-
- **Why not `pubid === row`.** pubid declares no `subset_strict` attribute for
|
|
44
|
-
IEEE, so a nil `parts`, `prefix`, `draft` or `draft_status` in the query is a
|
|
45
|
-
wildcard: over the published index (12,616 rows) `===` let `IEEE 802.3` match
|
|
46
|
-
`802.3.1-2011` and `IEEE 754` match `IEEE Unapproved P754/D9.0`. Switch to
|
|
47
|
-
`index.search(pubid)` once pubid makes those strict, and re-measure `===`
|
|
48
|
-
against the equality over the full index first.
|
|
49
|
-
- **`Relaton::Ieee::Docidentifier`.** `Item` holds this class (the producer
|
|
50
|
-
creates it too: `IdamsParser#docidentifier`, `DataFetcher`), because
|
|
51
|
-
`Bib::Docidentifier#remove_date!` is a no-op. Its `remove_date!` re-renders an
|
|
52
|
-
`IEEE` docid through pubid without the year (`to_s(trademark: true)` for the
|
|
53
|
-
`scope: trademark` one) and leaves an id **with no year** untouched: a
|
|
54
|
-
re-render would change a draft's spelling (`P802.5t™/D-2.5` →
|
|
55
|
-
`P802.5t®/D2.5`). `ItemData#to_most_recent_reference` drops the ISBN and
|
|
56
|
-
DOI, which name one edition.
|
|
57
|
-
|
|
58
|
-
## Identifier resolution (`RawbibIdParser`, pubid-first)
|
|
59
|
-
|
|
60
|
-
`RawbibIdParser.parse(normtitle, stdnumber)` turns a raw IEEE catalog
|
|
61
|
-
`normtitle` into a **`::Pubid::Ieee::Identifier`** (`idams_parser.rb`/prefilter
|
|
62
|
-
consume `.to_s` / `.to_s(trademark: true)` / docnumber). It is **pubid-first**:
|
|
63
|
-
pubid parses ~93.5% of normtitles directly; the bespoke regex case statement
|
|
64
|
-
(`#parse_fallback`, ~430 lines) only normalizes the messy remainder into a
|
|
65
|
-
canonical string pubid then parses (→ 98.7% pubid objects, 100% coverage).
|
|
66
|
-
|
|
67
|
-
- **Faithfulness guard.** `#parse` always renders the bespoke reference
|
|
68
|
-
(`parse_fallback(...).to_id`) and only accepts a pubid parse when it kept every
|
|
69
|
-
non-date digit (`#faithful?`, a date-stripped **digit** multiset — digit-level
|
|
70
|
-
so a benign re-split like `P1476/D4` vs `P14764` counts as faithful while a real
|
|
71
|
-
loss like `/D3.0`→`/D.0` does not). If no pubid parse is faithful it returns the
|
|
72
|
-
private fallback `Renderer` (the old bespoke id string) — so there is **zero
|
|
73
|
-
silent digit loss**. ~1.3% of docs keep the `Renderer`.
|
|
74
|
-
- **An id must have a digit in its number.** The last two fallback branches
|
|
75
|
-
("publisher, number" and the `else` that uses `stdnumber`) accept a number
|
|
76
|
-
only when it has a digit. Without that rule the first two words of any title
|
|
77
|
-
became an id (`A Call to Action …` → `A Call`), and a category `stdnumber`
|
|
78
|
-
became one too (`White Paper` → `IEEE White Paper`). Since the id is the
|
|
79
|
-
output filename, 176 records shared 42 files and 134 were overwritten. Now
|
|
80
|
-
`parse` returns nil for them, and `DataFetcher#parse_entry` skips them with
|
|
81
|
-
the `PubID parse error` warning. A real standard with a title-like
|
|
82
|
-
`normtitle` falls through to `stdnumber` and keeps its id
|
|
83
|
-
(`IEEE Standard for Ethernet …` / `802.3bm-2015` → `IEEE Std 802.3bm-2015`);
|
|
84
|
-
16 such standards were hidden by the old branch. **The digit rule alone is
|
|
85
|
-
not enough** — see `#no_standard_number?` under **Fetch robustness**.
|
|
86
|
-
- **No public `PubId` class.** The old `Relaton::Ieee::PubId` is gone; its
|
|
87
|
-
rendering survives only as the private `RawbibIdParser::Renderer` used by the
|
|
88
|
-
fallback. pubid (`::Pubid::Ieee::Identifier`) is the identifier abstraction.
|
|
89
|
-
- **Coordination with pubid.** IEEE ids now render in pubid's canonical form
|
|
90
|
-
(`/D-5`→`/D5`, `/Amd 5-2012` with a space, `802.16-2012 - Redline`, trademark
|
|
91
|
-
`™`/`®` attached to the document **number**, full corrigendum expansion). This
|
|
92
|
-
required a batch of pubid fixes (draft-verbatim, redline, revision incl.
|
|
93
|
-
numbered `RevN`, edition, trademark, historical formats, update_codes one-offs).
|
|
94
|
-
`Core::DataFetcher#output_file` also sanitizes commas so pubid's `", Mar 2011"`
|
|
95
|
-
dates don't leak into filenames. The fallback `Renderer`'s suffix strings are
|
|
96
|
-
**not** the canonical spellings and shouldn't be "fixed" to match: they are the
|
|
97
|
-
input to `to_id`, which pubid re-parses, so `/D-N`, `/R-N`, `/E-N` are
|
|
98
|
-
deliberately the *parseable* forms pubid canonicalizes to `/DN` etc. The
|
|
99
|
-
**trademark** rendering is the exception that must track pubid: `IdamsParser`
|
|
100
|
-
emits the `scope: trademark` docidentifier with `to_s(trademark: true)` on
|
|
101
|
-
whatever `parse` returned, so the `Renderer`'s ®/™ rule
|
|
102
|
-
(`Renderer::Id::REGISTERED_SERIES`) mirrors pubid#322 — series read off the
|
|
103
|
-
*number* with a leading project `P` stripped, `8802` ungated, everything else
|
|
104
|
-
gated on IEEE being a (co)publisher. Otherwise a `P802.x` draft would print ™
|
|
105
|
-
down the fallback path and ® down the pubid path. The one
|
|
106
|
-
exception is `amd_to_s`, which emits the **spaced** `/Amd N`: pubid parses both
|
|
107
|
-
spellings, so the spaced one costs nothing and keeps the rendering canonical
|
|
108
|
-
for the ~1.3% of ids that miss the faithfulness guard and keep the `Renderer`
|
|
109
|
-
string. Changing any of these also moves the `extract_index_entry` dedup key in
|
|
110
|
-
`data_fetcher.rb`.
|
|
111
|
-
|
|
112
|
-
## Fetch robustness (no silent corpus loss)
|
|
113
|
-
|
|
114
|
-
Amendment docnumbers can be pathologically long — pubid's `to_s` embeds the full
|
|
115
|
-
"(Amendment to … as amended by …)" clause (300+ chars). Two guards keep one such
|
|
116
|
-
id from silently halving the crawl (see the `data_fetcher.rb` history):
|
|
117
|
-
|
|
118
|
-
- **A record with no standard number is skipped.** `#no_standard_number?` is
|
|
119
|
-
true when `publicationsubtype` is in `NON_STANDARD_SUBTYPES` (`Whitepapers`,
|
|
120
|
-
`Research Documents`) **and** `stdnumber` has no digit. IEEE gives such a
|
|
121
|
-
document no designation and cites it by title, so every id built from it is
|
|
122
|
-
a title fragment. The digit rule in `RawbibIdParser` cannot catch all of
|
|
123
|
-
them, because pubid parses a digit out of the title itself
|
|
124
|
-
(`IEEE 802 Nendica Report: …` → `IEEE Std 802`, three documents; `IEEE 3D
|
|
125
|
-
Body Processing …` → `IEEE Std 3D`, three more). Both conditions are
|
|
126
|
-
required, so a record of such a subtype that does carry a number keeps it
|
|
127
|
-
(`Amendment 1:2011 to SMPTE ST 430-9:2008`). The check runs in **two**
|
|
128
|
-
places: `#parse_entry` (with a warning) and `#extract_index_entry` — the
|
|
129
|
-
prefilter groups files by docnumber and keeps the newest of each group, so a
|
|
130
|
-
white paper that reads as `IEEE Std 802` would otherwise join the real
|
|
131
|
-
standard's group and push its file out. Measured on the raw corpus: 157
|
|
132
|
-
records skipped, and no kept document has a digit-less id. The subtype match
|
|
133
|
-
is exact, which the data supports — all 570,744 raw files carry one of three
|
|
134
|
-
spellings (`Standard Docs` 564,611, `Whitepapers` 5,240,
|
|
135
|
-
`Research Documents` 891), so a new spelling is the one thing to watch for.
|
|
136
|
-
- **Bounded filenames.** `Core::DataFetcher#output_file` caps the basename at
|
|
137
|
-
`MAX_BASENAME_BYTES` (255): when the sanitized id exceeds the OS limit it
|
|
138
|
-
byte-truncates (`byteslice(…).scrub("")`, valid UTF-8) and appends a 12-char
|
|
139
|
-
SHA1 of the **full** docid — bounded, unique, and **deterministic** so
|
|
140
|
-
`save_doc`/`reconcile_staged_outputs`/`read_bib` all resolve the same name.
|
|
141
|
-
Without this, `File.write` raised `Errno::ENAMETOOLONG`.
|
|
142
|
-
- **Per-document commit guard.** Both the serial (`run_shard`) and parallel
|
|
143
|
-
(`spawn_batch` fork) paths route through one `commit_entry` helper that wraps
|
|
144
|
-
`commit_doc` in a `rescue` (logs via `Util.error`, continues). A single bad doc
|
|
145
|
-
no longer aborts a serial crawl, and a forked worker still reaches its
|
|
146
|
-
`File.binwrite(state_path, …)` so `merge_state_files` sees its `saved_writes`
|
|
147
|
-
and `reconcile_staged_outputs` doesn't delete the whole batch's staged files.
|
|
148
|
-
Reconcile also logs the straggler-cleanup count so an anomaly isn't silent.
|
|
149
|
-
- **A title-fragment id can still slip past `#no_standard_number?`.**
|
|
150
|
-
`#no_standard_number?` only fires for a `publicationsubtype` in
|
|
151
|
-
`NON_STANDARD_SUBTYPES`, and a generic mid-tier `RawbibIdParser` fallback
|
|
152
|
-
branch meant for a real catalog number (e.g. "publisher, number, part",
|
|
153
|
-
built for `"IEEE 802.3"`) can still read a short, numbered title as one
|
|
154
|
-
(`"WEB 3.0: The Evolution of ..."` → `WEB 3.0`) for a subtype spelling that
|
|
155
|
-
list doesn't cover. `RawbibIdParser.fabricated_title_id?(pid, title,
|
|
156
|
-
isbn_present)` is a second, subtype-agnostic guard on the *resolved id*
|
|
157
|
-
itself, called from both `IdamsParser#pubid` (full parse, `title` from
|
|
158
|
-
`@doc.btitle` — distinct from `normtitle`, which for a genuine standard
|
|
159
|
-
holds its designation, not prose) and `DataFetcher#extract_index_entry`
|
|
160
|
-
(cheap-parse prefilter, `title` the *last* `<title>` in the raw XML — the
|
|
161
|
-
first is always the top-level designation, present even for a real
|
|
162
|
-
standard). It must run in both places for the same reason
|
|
163
|
-
`#no_standard_number?` does: the prefilter groups files by docnumber before
|
|
164
|
-
the full parse ever runs, so a fabricated id left un-nilled there still
|
|
165
|
-
merges a whitepaper's group with an unrelated document's and can silently
|
|
166
|
-
drop a file from the corpus.
|
|
167
|
-
**The check is on the id's *class*, not (only) the title prefix:** it
|
|
168
|
-
requires `pid.is_a?(Renderer)` — i.e. the unvalidated bespoke fallback
|
|
169
|
-
string, never a real `::Pubid::Ieee::Identifier` that pubid's own grammar
|
|
170
|
-
actually parsed — **and** the title is a literal prefix of `pid.to_s`
|
|
171
|
-
**and** the document carries a real ISBN. Title-prefix and ISBN alone are
|
|
172
|
-
not enough: measured over 9,091 successfully-parsed raw-corpus records, 7
|
|
173
|
-
genuine, correctly-resolved standards have a title that literally begins
|
|
174
|
-
with their own real designation and also carry an ISBN (`"IEEE Std
|
|
175
|
-
1636.99-2013, IEEE Standard for ..."`) — an earlier version of this guard
|
|
176
|
-
without the `Renderer` restriction nulled out their real ids. All 7 resolve
|
|
177
|
-
to a real `Pubid::Ieee::Identifier`, never a `Renderer`, so the class check
|
|
178
|
-
is what tells them apart from the 25 known fabricated ids in a
|
|
179
|
-
`relaton-data-ieee` audit, all of which are `Renderer`s. A flagged record is
|
|
180
|
-
treated like any other unparseable id: `docnumber` is nil and
|
|
181
|
-
`DataFetcher#parse_entry` skips it with the `PubID parse error` warning.
|
|
182
|
-
|
|
183
|
-
## Testing
|
|
184
|
-
|
|
185
|
-
- **Framework:** RSpec with VCR cassettes and WebMock.
|
|
186
|
-
- **Index fixture:** `spec/ieee/fixtures/index-v2.zip` (a small structured index for
|
|
187
|
-
the read-spec targets) is deserialized through `::Pubid::Ieee::Identifier` and
|
|
188
|
-
pre-loaded into the `Relaton::Index` pool in `before(:suite)`
|
|
189
|
-
(`spec/ieee/support/webmock.rb`, NIST recipe). `index_builder_spec.rb` covers
|
|
190
|
-
`build_index` (structured rows + skip logging).
|
|
191
|
-
When a document 404s after a cassette refresh, the fixture has gone stale
|
|
192
|
-
against the republished index: copy the matching row **wholesale** (`:id:` *and*
|
|
193
|
-
`:file:`) out of the live `…/relaton-data-ieee/refs/heads/v2/index-v2.zip`,
|
|
194
|
-
rewrite the zip with the single `index-v2.yaml` entry, and **delete the cassette**
|
|
195
|
-
so it re-records — `record: :once` will not re-record a *changed* request URI, it
|
|
196
|
-
raises `UnhandledHTTPRequestError` instead.
|
|
197
|
-
- **Expected-XML fixtures self-heal only when absent.** `ieee_spec.rb` and
|
|
198
|
-
`idams_parser_spec.rb` write `fixtures/<name>.xml` with `unless File.exist?`, so
|
|
199
|
-
reconciling a pubid rendering change means deleting the fixture, re-running, and
|
|
200
|
-
reviewing the diff — not hand-editing it.
|
|
201
|
-
- **VCR cassettes:** `spec/ieee/vcr_cassettes/` — `index-v2.zip` downloads are
|
|
202
|
-
ignored by VCR (handled by the fixture).
|