iev 0.4.7 → 0.4.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a965bb1653227f1b4ba8ebc27e9f3a67950d359cec2f809b46d4e0edebec18e9
-  data.tar.gz: 5414e80f45caf460970d75052dc7bd7ec931d697f267de0ea2df6abd6a2209ba
+  metadata.gz: a016c3e0df9de5b20b48d9354f3a61ebbc604ac94e07b116a07ce2023b244eea
+  data.tar.gz: badb0152246599074ab74131274b70972f8ab8fa3dd1e4b49b06bd38dde81a33
 SHA512:
-  metadata.gz: 805143019d38cf0ed15e6526b8b727e681e8e3361b25d3c58ea71f6b2b4cf66eb3dda9882bcb8f038317bc4f1bd84554f6abdd14e4575774c2572a08b45c8865
-  data.tar.gz: 204a790f8cc565859a3abb9f64b137b235a559bbb411f52fee15989b467c5860ff2fe3073c65495f5c28e1e01a251375c533139cac07eaefb7341b5016669d1a
+  metadata.gz: 6a1527631c2f47363372dca4f12f9c3b4026a3d936ecfabf8336dc061c3a188fb79d1e4797aa028c11d13936ef6164d09c2e396b50315cc532e01cda65f30d08
+  data.tar.gz: 7d1c71fcb8b1f87bbc7787688b8937780cb775fc43a6e7058d59f41a269cc2f4bdcd725ce303bc160ae5317505465ba88bd0c453d4cf311063f3207ed8e99bd3

data/.claude/scheduled_tasks.lock

@@ -0,0 +1 @@
+{"sessionId":"7e802925-11d5-47e6-9ff2-7e151d3b2dfb","pid":23315,"procStart":"Wed Jul  8 10:33:56 2026","acquiredAt":1783506837511}

data/.rubocop_todo.yml

@@ -1,6 +1,6 @@
 # This configuration was generated by
 # `rubocop --auto-gen-config`
-# on 2026-06-18 12:23:48 UTC using RuboCop version 1.86.1.
+# on 2026-06-22 10:01:24 UTC using RuboCop version 1.86.1.
 # The point is for the user to remove these configuration records
 # one by one as the offenses are removed from the code base.
 # Note that changes in the inspected code, or installation of new
@@ -11,7 +11,7 @@ Gemspec/RequiredRubyVersion:
   Exclude:
     - 'iev.gemspec'
 
-# Offense count: 64
+# Offense count: 72
 # This cop supports safe autocorrection (--autocorrect).
 # Configuration parameters: Max, AllowHeredoc, AllowURI, AllowQualifiedName, URISchemes, AllowRBSInlineAnnotation, AllowCopDirectives, AllowedPatterns, SplitStrings.
 # URISchemes: http, https
@@ -35,7 +35,7 @@ Lint/ShadowedException:
   Exclude:
     - 'lib/iev/source_parser.rb'
 
-# Offense count: 27
+# Offense count: 29
 # Configuration parameters: AllowedMethods, AllowedPatterns, CountRepeatedAttributes, Max.
 Metrics/AbcSize:
   Exclude:
@@ -64,12 +64,12 @@ Metrics/CyclomaticComplexity:
     - 'lib/iev/term_builder.rb'
     - 'lib/iev/utilities.rb'
 
-# Offense count: 36
+# Offense count: 38
 # Configuration parameters: CountComments, CountAsOne, AllowedMethods, AllowedPatterns.
 Metrics/MethodLength:
   Max: 73
 
-# Offense count: 1
+# Offense count: 2
 # Configuration parameters: CountKeywordArgs, MaxOptionalParameters.
 Metrics/ParameterLists:
   Max: 7

data/Gemfile

@@ -14,6 +14,7 @@ end
 gem "benchmark"
 gem "canon"
 gem "openssl"
+gem "playwright-ruby-client", require: false
 gem "rake"
 gem "rspec"
 gem "rubocop"

data/TODO.reconcile/01-design.md

@@ -0,0 +1,57 @@
+# 01 — Architecture & autoloads
+
+## Audit of current approach (scripts/reconcile.rb)
+
+Problems:
+1. **Monolithic** — 400-line script, module of static methods, no classes
+2. **Not model-driven** — builds raw Hashes instead of Glossarist model objects
+3. **No field-level diff** — knows THAT content changed, not WHAT changed
+4. **No before/after values** — loses the historical content
+5. **No machine-readable change log** — no way to audit changes
+6. **No specs**
+7. **Not MECE** — merge, diff, serialize all mixed together
+8. **Not autoloaded** — everything in a script file
+
+## New architecture
+
+```
+lib/iev/reconciler.rb                      module + autoloads
+lib/iev/reconciler/change.rb               field-level change (field, lang, old, new)
+lib/iev/reconciler/change_set.rb           collection of Change per concept
+lib/iev/reconciler/status_mapper.rb        termbase status -> V3 ConceptStatus
+lib/iev/reconciler/termbase_loader.rb      termbase.yaml -> {code => ManagedConcept}
+lib/iev/reconciler/live_loader.rb          HTML pages -> {code => ManagedConcept}
+lib/iev/reconciler/content_differ.rb       diff two ManagedConcepts -> ChangeSet
+lib/iev/reconciler/concept_merger.rb       merge termbase + live -> ReconciledConcept
+lib/iev/reconciler/reconciled_concept.rb   ManagedConcept + ChangeSet + source
+lib/iev/reconciler/report.rb               dataset-level change reporting
+lib/iev/reconciler/pipeline.rb             orchestration
+```
+
+Each class has ONE responsibility (MECE). The Pipeline composes them.
+
+## History reporting
+
+Three layers:
+
+1. **Model-level** — `ManagedConcept#dates` (accepted/amended/retired),
+   `ManagedConcept#status` (valid/draft/superseded/retired),
+   `ManagedConcept#related` (supersedes/superseded_by).
+
+2. **Concept-level annotations** — Each amended concept gets a
+   `ConceptData#annotations` entry describing what field changed, with
+   before/after text.
+
+3. **Dataset-level report** — Summary statistics + machine-readable
+   change log written to `report/`:
+   - `summary.yaml` — aggregate counts by change type, section, language
+   - `changes.csv` — one row per field-level change
+   - `retired.yaml` — concepts retired since termbase snapshot
+   - `new_concepts.yaml` — concepts added since termbase snapshot
+
+## Checklist
+
+- [x] Audit current approach
+- [x] Design new architecture
+- [x] Create lib/iev/reconciler.rb with autoloads
+- [x] Register in lib/iev.rb

data/TODO.reconcile/02-change-models.md

@@ -0,0 +1,44 @@
+# 02 — Change + ChangeSet models
+
+## Goal
+
+The core of history reporting. A `Change` captures a single field-level
+difference between the termbase snapshot and the live snapshot. A
+`ChangeSet` groups all changes for one concept.
+
+## Change
+
+```ruby
+Change = Struct.new(:code, :field, :language, :old_value, :new_value,
+                    :detected_at, keyword_init: true)
+```
+
+Fields:
+- `code`: IEV code (e.g. "102-01-01")
+- `field`: `:designation` | `:definition` | `:notes` | `:examples` | `:status` | `:language_added` | `:language_removed`
+- `language`: ISO 639-2 code ("eng", "fra", nil for concept-level)
+- `old_value`: the termbase value (nil for additions)
+- `new_value`: the live value (nil for removals)
+- `detected_at`: date the change was observed (mirror date)
+
+## ChangeSet
+
+```ruby
+class ChangeSet
+  include Enumerable
+  def initialize(code)
+  def add(change)
+  def each
+  def empty?
+  def by_field(field)
+  def by_language(lang)
+  def summary  # hash of field => count
+end
+```
+
+## Checklist
+
+- [x] Implement Change (Struct)
+- [x] Implement ChangeSet
+- [x] Spec: ChangeSet query methods
+- [x] Spec: ChangeSet#summary

data/TODO.reconcile/02-termbase-loader.md

@@ -0,0 +1,48 @@
+# 02 — Termbase loader
+
+## Goal
+
+Load `iev-data/termbase.yaml` into a uniform per-code hash keyed by IEV
+code. The termbase is the authoritative source for lifecycle dates and
+historical content.
+
+## What termbase.yaml contains
+
+22,228 concepts in the old ISO/IEV termbase format:
+
+```yaml
+102-01-01:
+  termid: 102-01-01
+  term: equality
+  eng:
+    id: 102-01-01
+    terms: [...]
+    definition: "..."
+    language_code: eng
+    notes: [...]
+    examples: [...]
+    entry_status: Standard
+    date_accepted: '2008-08-01T00:00:00+00:00'
+    date_amended: '2008-08-01T00:00:00+00:00'
+    review_date: '2008-08-01T00:00:00+00:00'
+    review_decision_date: '2008-08-01T00:00:00+00:00'
+    review_decision_event: published
+  fra:
+    ...
+```
+
+## Deliverable
+
+Class: `Iev::Reconciler::TermbaseLoader`
+
+- Loads termbase.yaml (22,228 concepts).
+- Returns a Hash keyed by IEV code string → concept hash.
+- Each concept hash normalizes per-language data into a consistent
+  structure so downstream stages don't need to handle the old format
+  ad-hoc.
+
+## Checklist
+
+- [x] Define TermbaseLoader interface
+- [x] Implement
+- [x] Test: loads all 22,228 codes; sample code has expected fields

data/TODO.reconcile/03-live-loader.md

@@ -0,0 +1,28 @@
+# 03 — Live HTML loader
+
+## Goal
+
+Load the mirrored HTML pages from `iev-data-latest/pages/` using the
+existing `Iev::Scraper::PageParser` to extract structured concept data.
+
+## Source
+
+17,473 HTML pages at `../iev-data-latest/pages/*.html`. Each page is
+the current Electropedia display page for one IEV code.
+
+## Deliverable
+
+Class: `Iev::Reconciler::LiveLoader`
+
+- Walks `Iev::Fetcher::PageStore#each_concept`.
+- For each page, uses `Iev::Scraper::PageParser` to parse into the raw
+  hash that `build_concept_from_raw` expects.
+- Returns a Hash keyed by IEV code → parsed raw hash.
+- Extracts the publication date from the HTML (e.g. "Publication date:
+  2008-08") for concepts not in termbase.
+
+## Checklist
+
+- [x] Define LiveLoader interface
+- [x] Implement
+- [x] Test: sample page parses with expected localized_concepts

data/TODO.reconcile/03-status-mapper.md

@@ -0,0 +1,24 @@
+# 03 — StatusMapper
+
+## Goal
+
+Map termbase `entry_status` strings to Glossarist V3 `ConceptStatus`
+enum values. Stateless, single responsibility.
+
+## Mapping
+
+| Termbase | V3 |
+|----------|-----|
+| Standard / Published / Effective | valid |
+| Draft | draft |
+| Not Valid | notValid |
+| Superseded | superseded |
+| Retired | retired |
+
+Unknown values default to `valid`.
+
+## Checklist
+
+- [x] Implement StatusMapper
+- [x] Spec: all mapped values
+- [x] Spec: unknown defaults to valid

data/TODO.reconcile/04-concept-merger.md

@@ -0,0 +1,45 @@
+# 04 — Concept merger
+
+## Goal
+
+For each code in the union of termbase + live sources, merge the two
+into a single V3 `ManagedConcept` with full lifecycle data.
+
+## Three populations
+
+1. **In both** (~16,643 codes): termbase provides dates/status,
+   live provides current content. If content differs, record an
+   amendment.
+2. **Termbase only** (~5,585 codes): not on live Electropedia → mark
+   `status: retired`, add `ConceptDate(type: retired)`.
+3. **Live only** (~830 codes): not in 2020 termbase → new concept,
+   `ConceptDate(type: accepted)` from HTML publication date.
+
+## Merge rules
+
+| Field | Source |
+|-------|--------|
+| `identifier` | either (same code) |
+| `status` | termbase `entry_status` → mapped to `ConceptStatus` enum |
+| `dates` | termbase `date_accepted`/`date_amended`/`review_date`/`review_decision_date` → `ConceptDate` array |
+| `localized_concepts.designations` | **live** if available, else termbase |
+| `localized_concepts.definition` | **live** if available, else termbase |
+| `localized_concepts.notes/examples` | **live** if available, else termbase |
+| `sources` | live parsed sources (richer) |
+| `related` | termbase supersession data + any extracted from live |
+
+## Status mapping
+
+| Termbase `entry_status` | V3 `ConceptStatus` |
+|--------------------------|---------------------|
+| `Standard` / `Published` | `valid` |
+| `Draft` | `draft` |
+| `Not Valid` | `notValid` |
+| `Superseded` | `superseded` |
+| `Retired` / absent from live | `retired` |
+
+## Checklist
+
+- [x] Define merge rules
+- [x] Implement ConceptMerger
+- [x] Test: in-both, termbase-only, live-only cases

data/TODO.reconcile/04-termbase-loader.md

@@ -0,0 +1,28 @@
+# 04 — TermbaseLoader
+
+## Goal
+
+Load `termbase.yaml` and build `Glossarist::ManagedConcept` objects
+with full lifecycle dates from the old termbase format.
+
+## Input
+
+`iev-data/termbase.yaml` — 22,228 concepts in old ISO/IEV format.
+
+## Output
+
+`Hash<String, Glossarist::ManagedConcept>` keyed by IEV code.
+
+Each ManagedConcept has:
+- `id` = code
+- `status` from StatusMapper
+- `dates` from `date_accepted` + `date_amended` (ConceptDate objects)
+- `localized_concepts` hash with one LocalizedConcept per language
+- Each LocalizedConcept has ConceptData with terms, definition, notes, examples
+
+## Checklist
+
+- [x] Implement TermbaseLoader
+- [x] Spec: loads sample termbase entry correctly
+- [x] Spec: dates extracted properly
+- [x] Spec: multi-language concepts

data/TODO.reconcile/05-content-diff.md

@@ -0,0 +1,45 @@
+# 05 — Content diff detector
+
+## Goal
+
+Detect content changes between the termbase snapshot and the live HTML
+for concepts that exist in both sources. Each detected change becomes
+an amendment record.
+
+## What to diff
+
+Per localized concept (per language):
+
+1. **Designations** (terms): added, removed, or changed text
+2. **Definition**: text content changed
+3. **Notes**: added, removed, or changed
+4. **Examples**: added, removed, or changed
+
+## Output
+
+For each changed field, record:
+
+```ruby
+{
+  field: :definition,
+  language: :eng,
+  old_value: "...",      # from termbase
+  new_value: "...",      # from live
+  changed_at: MIRROR_DATE # the date we captured the live page
+}
+```
+
+These feed into the merger to append `ConceptDate(type: amended)` and
+optionally an annotation describing the change.
+
+## Approach
+
+String normalization before comparison: strip whitespace, normalize
+unicode, ignore case-only differences in designation. Real semantic
+diffs (definition rewrite) vs cosmetic diffs (encoding fix).
+
+## Checklist
+
+- [x] Define diff strategy
+- [x] implement ContentDiff
+- [x] Test: identical → no change; modified definition → one diff

data/TODO.reconcile/05-live-loader.md

@@ -0,0 +1,20 @@
+# 05 — LiveLoader
+
+## Goal
+
+Load live HTML pages and build `Glossarist::ManagedConcept` objects
+using the existing `Iev::Scraper::PageParser`.
+
+## Input
+
+`iev-data-latest/pages/*.html` — 17,473 current Electropedia pages.
+
+## Output
+
+`Hash<String, Glossarist::ManagedConcept>` keyed by IEV code.
+
+## Checklist
+
+- [x] Implement LiveLoader
+- [x] Spec: parses sample HTML page
+- [x] Spec: skips placeholder pages (no localized_concepts)

data/TODO.reconcile/06-content-differ.md

@@ -0,0 +1,32 @@
+# 06 — ContentDiffer
+
+## Goal
+
+Compare two `ManagedConcept` objects and produce a `ChangeSet` with
+field-level differences. This is the core of change reporting.
+
+## What to diff
+
+Per localized concept (per language):
+- `designation` (term text)
+- `definition` (text content)
+- `notes` (added/removed/changed)
+- `examples` (added/removed/changed)
+
+At concept level:
+- `status` (e.g. valid → retired)
+- language added (in live but not termbase)
+- language removed (in termbase but not live)
+
+## Normalization
+
+Before comparison: strip HTML tags, normalize whitespace, normalize
+unicode quotes. Avoid false positives from encoding differences.
+
+## Checklist
+
+- [x] Implement ContentDiffer
+- [x] Spec: identical concepts → empty ChangeSet
+- [x] Spec: definition changed → one Change
+- [x] Spec: language added → one Change
+- [x] Spec: status changed → one Change

data/TODO.reconcile/06-v3-serializer.md

@@ -0,0 +1,42 @@
+# 06 — V3 serializer
+
+## Goal
+
+Serialize the merged `ManagedConcept` objects into Glossarist V3 YAML
+files using the existing `Glossarist::ManagedConceptCollection` API.
+
+## Output
+
+```
+output/
+  concepts/
+    concept-102-01-01.yaml
+    concept-102-01-02.yaml
+    ...
+  register.yaml          # DatasetRegister
+```
+
+Each concept file uses `schema_version: "3"` and the V3 field names
+from the concept model:
+
+```yaml
+---
+schema_version: "3"
+identifier: "102-01-01"
+status: valid
+dates:
+  - type: accepted
+    date: "2008-08-01"
+  - type: amended
+    date: "2008-08-01"
+localized_concepts:
+  eng: ...
+  fra: ...
+related: ...
+```
+
+## Checklist
+
+- [x] Define output layout
+- [x] Implement V3 serializer (reuse ManagedConceptCollection#save)
+- [x] Test: round-trip — load output back, verify fields

data/TODO.reconcile/07-cli-and-run.md

@@ -0,0 +1,27 @@
+# 07 — CLI command + end-to-end run
+
+## Goal
+
+Wire the reconciliation pipeline into a runnable command and execute
+it to produce the final dataset.
+
+## Deliverable
+
+Either:
+- A new Thor CLI command: `iev reconcile --termbase <path> --output <dir>`
+- Or a standalone script: `scripts/reconcile.rb`
+
+The command should:
+1. Load termbase.yaml
+2. Load live HTML from PageStore
+3. Merge + diff
+4. Serialize V3 YAML
+5. Write register.yaml
+6. Report: total concepts, in-both, termbase-only, live-only, amendments detected
+
+## Checklist
+
+- [x] Define command interface
+- [x] Implement
+- [x] Run end-to-end and verify output
+- [x] Spot-check 5 concepts manually (in-both, termbase-only, live-only, with changes)

data/TODO.reconcile/07-concept-merger.md

@@ -0,0 +1,32 @@
+# 07 — ConceptMerger + ReconciledConcept
+
+## Goal
+
+Merge termbase and live concepts into a single `ManagedConcept` with
+full lifecycle data, and package it with its `ChangeSet` in a
+`ReconciledConcept` value object.
+
+## ReconciledConcept
+
+```ruby
+ReconciledConcept = Struct.new(:managed_concept, :change_set, :source,
+                               keyword_init: true)
+# source: :both | :termbase_only | :live_only
+```
+
+## Merge rules
+
+| Source | Behavior |
+|--------|----------|
+| both | live content + termbase dates; if changes detected, add ConceptDate(amended) + annotations |
+| termbase only | mark retired, add ConceptDate(retired), preserve full content |
+| live only | new concept, add ConceptDate(accepted) from publication date |
+
+## Checklist
+
+- [x] Implement ReconciledConcept (Struct)
+- [x] Implement ConceptMerger
+- [x] Spec: merge-both produces live content + termbase dates
+- [x] Spec: termbase-only produces retired status
+- [x] Spec: live-only produces accepted date
+- [x] Spec: changes produce annotation entries

data/TODO.reconcile/08-report.md

@@ -0,0 +1,53 @@
+# 08 — Report
+
+## Goal
+
+Generate dataset-level change reports from the reconciled concepts.
+This answers "what changed across the entire IEV?"
+
+## Output files
+
+```
+report/
+  summary.yaml       # aggregate statistics
+  changes.csv        # one row per field-level change
+  retired.yaml       # concepts retired since termbase
+  new_concepts.yaml  # concepts added since termbase
+```
+
+## summary.yaml
+
+```yaml
+total_concepts: 23058
+sources:
+  in_both: 16643
+  termbase_only: 5585
+  live_only: 830
+changes:
+  total: 5499
+  by_field:
+    definition: 4200
+    designation: 350
+    notes: 580
+    examples: 120
+    status: 249
+  by_language:
+    eng: 5000
+    fra: 450
+    deu: 49
+  by_section:
+    "113-01": 45
+    "102-01": 12
+    ...
+```
+
+## changes.csv
+
+Columns: `code, section, field, language, detected_at, old_value, new_value`
+
+## Checklist
+
+- [x] Implement Report
+- [x] Spec: summary counts correct
+- [x] Spec: CSV format correct
+- [x] Spec: retired/new lists correct

data/TODO.reconcile/09-pipeline.md

@@ -0,0 +1,23 @@
+# 09 — Pipeline
+
+## Goal
+
+Orchestrate the full reconciliation: load → merge → serialize → report.
+
+## Flow
+
+```
+Pipeline#run
+  ├── TermbaseLoader.load(path) → Hash<code, ManagedConcept>
+  ├── LiveLoader.load(dir)      → Hash<code, ManagedConcept>
+  ├── for each code in union:
+  │     ├── ConceptMerger.merge(code, tb, live) → ReconciledConcept
+  │     └── collect ReconciledConcept
+  ├── save concepts to output/concepts/*.yaml
+  └── Report.new(reconciled).write_to(output/report/)
+```
+
+## Checklist
+
+- [x] Implement Pipeline
+- [x] Spec: pipeline produces expected output with test fixtures

data/TODO.reconcile/10-script-and-run.md

@@ -0,0 +1,15 @@
+# 10 — Script wrapper + end-to-end run
+
+## Goal
+
+Rewrite `scripts/reconcile.rb` as a thin wrapper around the library
+code. Then run the full pipeline and verify output.
+
+## Checklist
+
+- [x] Rewrite scripts/reconcile.rb as thin wrapper
+- [x] Run end-to-end pipeline
+- [x] Verify concepts/*.yaml output
+- [x] Verify report/*.yaml output
+- [x] Spot-check 5 concepts (in-both, retired, new, with changes)
+- [x] Spot-check report accuracy