arrolio 0.1.0 → 0.1.2

data/TODO.complete/77-text-formatting-parity.md

@@ -0,0 +1,55 @@
+---
+priority: P2
+impact: low
+depends_on: [70, 79]
+layer: adapter
+status: in_progress
+est: 1d
+---
+
+## Problem
+
+Section number formatting and several inline-formatting details
+differ from the reference.
+
+## Status (2026-08-05)
+
+- [x] **Section number "2.1" extracts as "2.1" not "21"** —
+      delimiter included via `autonum-delim` + `caption-label`
+      class match in `extract_from_heading`.
+- [x] **`<sub>` produces baseline_shift=:sub, scale=0.7** —
+      `baseline_for_style` in `collect_inline_runs` checks the
+      inline_styles mapping; elements mapping to `:subscript` or
+      `:superscript` set the baseline state carried through the
+      walker.
+- [x] **`<sup>` produces baseline_shift=:sup, scale=0.7** — same
+      path as `<sub>`.
+- [x] **Note labels have em-dash suffix** —
+      `formatted_note_label(label)` appends the configured suffix
+      (default "—") from `flow_rules.note.label_suffix`.
+
+## Still pending
+
+- [ ] **List bullet indent**: match the XSL's
+      `provisional-distance-between-starts` for list labels.
+- [ ] **Hyperlink underline**: extend `Style::Definition` with an
+      `underline` boolean; renderer emits `/UL true` before
+      drawing underlined runs.
+- [ ] **`<strong>` weight via Bold variant**: the `:strong` style
+      currently inherits the body font. Renderer should pick the
+      Bold variant automatically based on style name (similar to
+      how table headers do it in TODO 72).
+
+## Done-When
+
+- [x] Section number "2.1" extracts as "2.1" not "21"
+- [x] Note labels have the em-dash suffix
+- [x] `<sub>` / `<sup>` get baseline_shift + scale
+- [ ] List bullet indent matches reference
+- [ ] Hyperlinks render underlined
+- [ ] `<strong>` uses the Bold font variant
+
+## Expected improvement
+
+Estimated 2-3% overall similarity once the remaining items land
+(primarily from list indent matching and strong/bold rendering).

data/TODO.complete/78-bibliography-formatting-parity.md

@@ -0,0 +1,62 @@
+---
+priority: P2
+impact: low
+depends_on: [70]
+layer: adapter
+status: done
+est: 1d
+---
+
+## Problem
+
+Bibliography page renders content but with wrong formatting. The
+reference uses numbered references `[1]`, `[2]` with hanging indent
+so wrap lines align under the citation text, not under the tag.
+
+A separate bug in `extract_bibliography` compounded this: the
+`each_child` loop pushed Arrays into items without flattening,
+hiding all bibliography items from the flow builder.
+
+## Status (2026-08-06)
+
+- [x] **`extract_bibliography` now flattens** — uses `items.concat`
+      instead of `items <<`.
+- [x] **`bibliography_item_flowable` emits a TextFlowable with
+      hanging_indent** equal to the marker width. First line: tag
+      flush left + citation starts after marker. Wrap lines:
+      indented by marker_w so they align under the citation text.
+- [x] **TextFlowable supports hanging_indent** — new attribute
+      reduces effective width for layout and signals the renderer
+      to offset wrap lines.
+
+## Verification
+
+Page 28 bibliography page now renders entries with proper hanging
+indent:
+```
+[1] OIML V 1:2013, International Vocabulary of Terms in Legal
+    Metrology (VIML)
+[2] OIML V 2-200:2012, International Vocabulary of Metrology —
+    Basic and General Concepts and Associated Terms (VIM)
+```
+
+## Still pending
+
+- **4pt gap between entries** — current `:bibitem` style has
+  `margin_top: 4`, `margin_bottom: 4` but it's not visible in the
+  extracted output.
+- **Bibliography section heading style** — heading renders but
+  may not use the exact level-1 style.
+
+## Done-When
+
+- [x] Bibliography entries render as single paragraphs
+- [x] All entries appear in output (was 0 due to concat bug)
+- [x] Hanging indent so wrap lines align with text start
+- [ ] 4pt gap between entries (visual only — text diff unaffected)
+- [ ] Page 28 similarity > 85% (currently ~50% due to body drift)
+
+## Measurement
+
+`bundle exec rake parity:check` — page 28 bibliography page now
+renders all 12 entries with proper hanging indent.

data/TODO.complete/79-line-breaking-parity.md

@@ -0,0 +1,80 @@
+---
+priority: P2
+impact: high
+depends_on: [70]
+layer: text
+status: in_progress
+est: 3d
+---
+
+## Problem
+
+FOP uses Knuth-Plass optimal fitting. The original Arroolio KP
+implementation had multiple bugs producing per-word lines and
+overfills. As of 2026-08-06, the major issues are fixed:
+
+## Status (2026-08-06)
+
+Fixed bugs:
+- [x] **Badness formula only counted negative ratios.** Now uses
+      `100 * |ratio|^3` clamped to 10_000 for both directions.
+- [x] **TOLERANCE was 100.0.** Now 2.5, rejecting pathological
+      overfills.
+- [x] **`build_placed_runs` skipped Glue items entirely** — every
+      space between words was dropped. Now emits `InlineRun(' ')`
+      per Glue.
+- [x] **`total_stretch`/`total_shrink` summed the entire document.**
+      Renamed to `line_stretch(start, stop)` / `line_shrink(start, stop)`,
+      scoped to the current line.
+- [x] **`Arroolio::Style::Definition` typo** (double-o). Fixed.
+- [x] **Emergency stretch glue at the end was rendering as trailing
+      space.** build_placed_runs now skips Glue items with infinite
+      stretch.
+- [x] **Each Glue became its own Tj block**, confusing pdftotext
+      into dropping spaces between words ("durabilityerror"). Now
+      Box+Glue items merge into a single placed_run per group.
+
+Remaining gaps:
+- **Hyphenation** — long words at line boundaries get pushed to
+  the next line, leaving the previous line underfull. Needs TeX
+  hyphenation patterns (external dependency).
+- **Slight overfill on dense paragraphs** — KP picks overfull lines
+  when the alternative has higher demerits. Acceptable but not
+  pixel-perfect.
+- **Consecutive-flag penalty** — flagged breaks (hyphens) next to
+  each other aren't penalised correctly.
+
+## Approach
+
+1. **Tune demerits to match FOP.** FOP uses Knuth-Plass with
+   specific constants. Compare our line break choices against the
+   reference PDF on >20 body paragraphs.
+
+2. **Add hyphenation.** Use `text-hyphen` or `ruby-hyphen` gem
+   (or a TeX pattern data file) to compute break opportunities
+   within words. ItemBuilder should emit `Box + Penalty(flagged)`
+   at each valid hyphenation point. This matches FOP/TeX behavior.
+
+3. **Handle underfull last line.** Last line of a justified
+   paragraph is left-aligned (already done in renderer).
+
+## Done-When
+
+- [x] Knuth-Plass is the default line breaker for body text
+- [x] Badness formula is correct for positive and negative ratios
+- [x] KP produces multi-word lines (no per-word regression)
+- [x] Spaces between words are preserved in extracted text
+- [x] Last line of a justified paragraph is left-aligned
+- [x] Emergency stretch glue doesn't render as trailing space
+- [ ] Line break positions match FOP on >80% of body paragraphs
+- [ ] Hyphenation is supported for long words at line boundaries
+- [ ] Overall similarity > 60% on OIML r060/1 fixture
+
+## Measurement
+
+`bundle exec rake parity:check` — current baseline 47.88%
+overall, 30 pages vs 28 reference. Body pages 5-15 mostly OK
+(55-87%); table pages 18-25 LOW (5-35%) due to TODO 72 work
+still in progress.
+
+Last measured: 2026-08-06.

data/TODO.complete/80-parity-diff-harness-in-ci.md

@@ -0,0 +1,68 @@
+---
+priority: P0
+impact: critical
+depends_on: [70]
+layer: harness
+status: done
+est: 1d
+completion_date: 2026-08-05
+---
+
+## Problem
+
+Throughout development, "28 pages, matches the reference" was claimed
+without ever running the PDF diff. The diff harness existed
+(`Harness::PdfDiff`) but was not integrated into the development loop
+or CI, and the script that called it had a unit bug that disguised
+the actual parity number.
+
+## Approach (delivered)
+
+1. **`rake parity:check`** renders the OIML r060/1 fixture through
+   the generic pipeline and diffs it against the reference PDF,
+   printing per-page similarity and an overall score.
+
+2. **Unit-correct reporting.** `PdfDiff#similarity` is a Float in
+   [0, 1]; the script now multiplies by 100 for display and
+   compares against fractional thresholds (0.5 / 0.1) rather than
+   the previous buggy > 50 / > 10 integer checks.
+
+3. **`scripts/parity_check.rb` invokes `ParityCheck.run`** at the
+   bottom (previously the module was defined but never invoked —
+   the script exited silently with no output).
+
+4. **`ConfigDrivenPipeline` resolves fonts via FontScanner** so
+   the rendered PDF embeds the actual Jost/Times fonts the
+   reference uses, making the diff meaningful instead of comparing
+   against system fallbacks.
+
+## Done-When
+
+- [x] `rake parity:check` renders + diffs + prints per-page report
+- [x] Reporting shows percentages in [0, 100] not [0, 1]
+- [x] Script invokes the diff (no silent exit)
+- [x] Fonts are embedded in the rendered PDF, so the diff is
+      meaningful
+
+## Verification
+
+```
+$ bundle exec rake parity:check
+Overall similarity: 24.58%
+Our pages: 30, Reference pages: 28
+
+  Page 1: 48.0% [PARTIAL]
+  Page 2: 100.0% [OK]
+  ...
+```
+
+## Not Yet Done
+
+- `spec/arrolio/parity_spec.rb` regression spec — the next parity
+  PR should add this with a floor threshold (start at 20%, raise
+  as TODOs land). The spec belongs in the gem'sRSpec suite so CI
+  fails on regressions.
+
+- GitHub Actions wiring — the spec runs locally via `bundle exec
+  rspec`; CI invocation is added in the workflow file (`.github/
+  workflows/*.yml`) by the release engineer.

data/TODO.complete/81-conditional-style-refinements.md

@@ -0,0 +1,70 @@
+---
+priority: P1
+impact: high
+depends_on: [70]
+layer: adapter
+status: in_progress
+est: 2d
+---
+
+## Problem
+
+The XSL has many conditional style refinements (`refine_*` templates)
+that apply margin/space properties only in specific contexts (annex
+headings, list-item paragraphs, term definitions). The converter
+captures these as unconditional properties in `layout_spec.yml`,
+causing spacing to be wrong for most content.
+
+## Bugs found (2026-08-08)
+
+- [x] **heading_1 margin_bottom: 30pt applied unconditionally.** The
+      XSL `refine_title-style` only sets `margin-bottom: 30pt` when
+      `ancestor::mn:annex`. Our YAML applied it to ALL level-1
+      headings, adding 30pt × ~8 headings = 240pt of dead space.
+      Fixed: removed from heading_1; needs separate `heading_1_annex`
+      style.
+- [x] **edition_label always included year.** XSL `get_edition` uses
+      `version/revision-date` (child element), which is empty when
+      `<version>` has text content. Our profile mapped
+      `revision_date: version` (text), always extracting the year.
+      Fixed: profile now uses `version/revision-date`, and
+      `derive_metadata_fields` falls back to label without year.
+- [x] **Doc title block missing.** XSL renders zzSTDTitle1 paragraphs
+      as "Part N - title-part" from bibdata. Our adapter extracted the
+      paragraph text instead of constructing from metadata. Fixed:
+      `extract_title_block` now constructs from `part_number` and
+      `title_part` metadata.
+- [ ] **Term entry spacing is 3× too tall.** Each term entry component
+      (number, preferred, definition) inherits `term` style with
+      margin_bottom: 12pt. The XSL applies it once per entry (on the
+      outer block). Fix needs either a container flowable or
+      per-component style override.
+- [ ] **heading_2 space-after: 12pt may be unconditional.** XSL
+      `refine_title-style` level 2 adds `space-after: 12pt`
+      unconditionally. Verify against reference.
+- [ ] **clause-style space-before/space-after.** XSL
+      `refine_clause_style` adds space properties conditionally.
+
+## Approach
+
+1. **Audit all `refine_*` templates** in the XSL for conditional
+   attributes. Map each condition to a style selector.
+
+2. **Extend the converter** to emit context-specific styles (e.g.,
+   `heading_1_annex`, `p_in_list`, `term_definition`) when the
+   XSL condition applies.
+
+3. **Or:** model FO space-resolution rules (space-after of previous
+   vs space-before of next → take the max, not the sum).
+
+## Done-When
+
+- [ ] All conditional margins match the reference PDF on pages 5-15
+- [ ] Term entries use 12pt spacing per entry, not per component
+- [ ] Annex headings have 30pt margin; body headings don't
+- [ ] Overall parity > 55% on OIML r060/1 fixture
+
+## Measurement
+
+`bundle exec rake parity:check` — current baseline 53.53%.
+Last measured: 2026-08-08.

data/TODO.complete/82-semantic-presentation-dedup.md

@@ -0,0 +1,65 @@
+---
+priority: P1
+impact: high
+depends_on: [66]
+layer: adapter
+status: done
+est: 1d
+---
+
+## Problem
+
+The standoc presentation XML contains paired semantic + format
+elements for cross-references, terms, and identifiers:
+
+```xml
+<xref target="X" id="id123">clause=5.1.1</xref>
+<semx element="xref" source="id123">
+  <fmt-xref target="X">clause=5.1.1</fmt-xref>
+</semx>
+```
+
+Both the semantic `<xref>` and presentation `<fmt-xref>` carry the
+same text. Our inline walker was collecting text from BOTH, producing
+doubled output: `clause=5.1.1clause=5.1.1`.
+
+## Fix (2026-08-08)
+
+Added semantic elements to `skip_metadata_elements` in the standoc
+profile:
+
+```yaml
+skip_metadata_elements:
+  ...
+  - xref
+  - eref
+  - identifier
+```
+
+The presentation equivalents (`fmt-xref`, `fmt-identifier`,
+`fmt-preferred`) remain in `inline_styles` and provide the display
+text.
+
+## Remaining gaps
+
+- [ ] **`clause=5.1.1` should render as "clause 5.1.1" or "5.1.1".**
+      The `=` in the raw text is a locality separator. The XSL has
+      i18n templates for locality rendering. Needs locality parsing
+      in the inline walker.
+- [ ] **Some fmt-* elements may also be doubled** when their semantic
+      counterpart has no `id` attribute (can't match via `source`).
+      Audit all element pairs.
+- [ ] **`fmt-name` in skip_elements vs inline.** The block-level
+      `convert_children` skips `fmt-name`, but the inline
+      `collect_inline_runs` doesn't. Verify consistency.
+
+## Done-When
+
+- [x] No doubled xref/eref/identifier text in output
+- [ ] Locality references render as "clause X.Y.Z" not "clause=X.Y.Z"
+- [ ] All semantic/presentation pairs deduplicated
+
+## Measurement
+
+Parity improved from 50.52% → 53.53% after deduplication.
+Last measured: 2026-08-08.

data/TODO.complete/83-nested-list-support.md

@@ -0,0 +1,74 @@
+---
+priority: P1
+impact: high
+depends_on: [66]
+layer: adapter
+status: done
+est: 0.5d
+---
+
+## Problem
+
+The adapter's `convert_list` only collected `<p>` paragraph children
+of `<li>` elements, silently dropping nested `<ol>`/`<ul>` lists.
+Similarly, `list_flowable` in the flow builder only converted
+`Content::Paragraph` items, skipping `Content::List`.
+
+This caused entire sub-lists to disappear from the output. For
+example, item b) in section 6.1:
+
+```
+b) For legally relevant software... shall be applied.
+   1) The exception described in clause 5.1.1...
+   2) The level of conformity...
+   3) Updating the legally relevant software...
+   4) The software documentation shall include...
+```
+
+Items 1-4 were completely absent from our output.
+
+## Fix (2026-08-08)
+
+### Adapter (`convert_list`)
+
+```ruby
+content = []
+each_element(li) do |child|
+  next if child.parent && child.parent != li  # direct children only
+
+  if child.name == para_name
+    content << convert_paragraph(child)
+  elsif list_mapping.key?(child.name)
+    nested_kind = list_mapping[child.name]['kind']&.to_sym || :bullet
+    content << convert_list(child, kind: nested_kind)
+  end
+end
+```
+
+### Flow builder (`list_flowable`)
+
+```ruby
+def flowable_for_list_content(content)
+  case content
+  when Content::Paragraph then [paragraph_flowable(content)]
+  when Content::List then [list_flowable(content)]
+  else []
+  end
+end
+```
+
+The `ListFlowable` already handles arbitrary body flowables via
+`f.emit`, so nested `ListFlowable` renders correctly at increased
+indent.
+
+## Done-When
+
+- [x] Nested `<ol>`/`<ul>` inside `<li>` are collected by adapter
+- [x] `Content::List` inside list item content is converted by flow builder
+- [x] Nested list markers render at increased indent
+- [x] Item b) sub-items 1-4 visible in output
+
+## Measurement
+
+Verified on OIML r060/1 page 26: items 1-4 now visible.
+Last measured: 2026-08-08.

data/TODO.complete/84-figure-svg-rendering.md

@@ -0,0 +1,70 @@
+---
+priority: P1
+impact: critical
+depends_on: []
+layer: render
+status: done
+est: 1d
+---
+
+## Problem
+
+Figures in the standoc presentation XML contain inline `<svg>` elements
+inside `<image>` tags. The adapter was only looking for external image
+file references (`images/figure-1.svg`) which don't exist, causing all
+4 figures to be missing from the output — the biggest source of
+pagination drift.
+
+## Fix (2026-08-09)
+
+### Adapter (`extract_figure_image`)
+
+When a `<figure>` has an `<image>` element:
+1. Search recursively for `<svg>` inside the image element
+2. If found, serialize the SVG XML and extract viewBox dimensions
+3. Store as `Content::Image` with `inline-svg:` prefix
+
+When no `<image>` element exists, search recursively in the figure for
+standalone `<svg>`.
+
+### Flow builder (`image_flowable`)
+
+Recognize the `inline-svg:` prefix:
+1. Extract SVG XML from the prefixed source string
+2. Write to a temp `.svg` file
+3. Pass to `ImageFlowable` which delegates to the renderer
+
+### Renderer (existing)
+
+The renderer already rasterizes `.svg` files via `rsvg-convert` to PNG
+and caches the result. No renderer changes needed.
+
+### Config (`flow_rules.yml`)
+
+Set `max_display_width: 450` (was default 106) to allow full-width
+figure rendering.
+
+## Impact
+
+Parity: 53.53% → **56.98%** (+3.45%)
+Page count: 26 → **27** (reference 28)
+
+| Page | Before | After |
+|------|--------|-------|
+| 5 | 94.1% | 99.6% |
+| 9 | 35.7% | 87.4% |
+| 12 | 37.4% | 64.0% |
+| 14 | 45.8% | 79.5% |
+
+## Done-When
+
+- [x] Inline SVG extracted from `<image>` elements
+- [x] viewBox dimensions used for aspect ratio
+- [x] rsvg-convert rasterizes SVG to PNG
+- [x] Figures appear in output at correct size
+- [x] Parity improved by 3.45%
+
+## Measurement
+
+`bundle exec rake parity:check` — 56.98%.
+Last measured: 2026-08-09.

data/TODO.complete/85-locality-reference-formatting.md

@@ -0,0 +1,48 @@
+---
+priority: P2
+impact: medium
+depends_on: [84]
+layer: adapter
+status: done
+est: 0.5d
+---
+
+## Problem
+
+Cross-references to clauses in the document rendered as raw locality
+strings: `clause=5.1.1` instead of `clause 5.1.1`. The `=` is a
+locality separator used in standoc presentation XML that the XSL i18n
+templates replace with a space.
+
+## Fix (2026-08-09)
+
+Added `format_locality_text` method in the inline walker. When text is
+collected from inside a `<fmt-xref>` element, the `=` between a word
+and a value is replaced with a space:
+
+```ruby
+def format_locality_text(text)
+  text.gsub(/([a-zA-Z]+)=/, '\\1 ')
+end
+```
+
+Examples:
+- `clause=5.1.1` → `clause 5.1.1`
+- `table=2` → `table 2`
+- `figure=4` → `figure 4`
+
+The walker tracks `in_xref` context to only apply this transformation
+inside cross-references, not globally.
+
+## Done-When
+
+- [x] `clause=X` renders as `clause X`
+- [x] Only applied inside fmt-xref elements
+- [x] Non-locality `=` preserved (e.g., in URLs or formulas)
+- [x] Specs cover locality formatting
+
+## Measurement
+
+Improves readability of ~15 cross-references. Parity impact is small
+(text content change, not spacing).
+Last measured: 2026-08-09.

data/TODO.complete/86-list-item-text-overflow.md

@@ -0,0 +1,80 @@
+---
+priority: P1
+impact: high
+depends_on: [79]
+layer: text
+status: done
+est: 2d
+---
+
+## Problem
+
+List item body paragraphs sometimes lose text after cross-references.
+The placed box correctly contains 2 lines of text (verified via debug),
+but the renderer only outputs line 1 — line 2 is silently dropped.
+
+## Example
+
+Input paragraph:
+```
+For legally relevant software of digital load cells the following
+statements according to [OIML_D_31_2008] shall be applied.
+```
+
+Rendered PDF (before fix): the line ran to x=597 on a 595pt page —
+"be applied." was drawn off-page and clipped by readers.
+
+## Root cause (found 2026-08-17)
+
+NOT a font-metrics mismatch — TTF advance widths match the embedded
+render exactly. The Knuth-Plass breaker rejected every intermediate
+break of the paragraph (best candidate ratio 2.68 > TOLERANCE 2.5),
+so the only surviving path was the FORCED final break, which bypasses
+the ratio check: the whole paragraph rendered as one 552.6pt line on
+a 426.7pt measure.
+
+## Fix
+
+- `KnuthPlass::Breaker`: TeX-style emergency-stretch pass. When the
+  first pass yields no feasible solution, re-solve with
+  EMERGENCY_STRETCH added to every line's stretchability — the
+  paragraph breaks properly instead of overflowing.
+- `KnuthPlass::Breaker`: removed the `prune?` cap that disabled
+  active nodes beyond `line_widths.length + 10` lines — any
+  paragraph or table cell longer than 11 lines got its tail crammed
+  into one overfull line (visible as overlapping table-cell text).
+- `KnuthPlass::Breaker#build_placed_runs`: style-preserving run
+  grouping. A line is merged into one text operation per rendering
+  signature; bold/italic/sub/superscript runs keep their formatting
+  (previously the whole line took the first run's style). Trailing
+  glue at the break is dropped so justify counts real word gaps.
+- `TextFlowable#collect_runs`: re-inserts the word separator that a
+  line break consumed when a split paragraph is rebuilt — otherwise
+  re-layout glues boundary words ("…increasing anddecreasing").
+  Hyphen breaks gain no space ("analogue-active" stays joined).
+- `Renderer::Pdf#render_line_runs`: justified lines are drawn as one
+  positioned text operation per word (FOP's model) with the stretch
+  accumulated in each word's x. PDF word spacing (Tw) does not apply
+  to two-byte CID-encoded embedded fonts, so justification cannot
+  rely on it; the old boundary-shift approach concentrated the whole
+  line's stretch at run boundaries (34pt holes around math runs).
+- `GenericFlowBuilder`: `require 'tmpdir'` (was a transitive-require
+  accident).
+- Prefix sums for item widths/stretch/shrink make width queries O(1).
+
+## Done-When
+
+- [x] "be applied." renders after "shall" in list item b)
+- [x] Text never overflows body width in list items
+- [x] GlyphMeasurer widths match rendered widths (verified: measured
+      501.6pt vs rendered 501.2pt for the failing line)
+- [x] Specs cover list item with long inline tokens
+
+## Measurement
+
+Overall similarity 62.7% → 59.9% (2026-08-17). Text pages unchanged;
+the drop is pages 23–27, where previously-overlapping table cells now
+occupy their true height. Our table rows are taller than FOP's
+(column-width debt — TODO 72/92); the old render accidentally matched
+page boundaries while its cell text overlapped. Recovering the metric
+requires table layout parity, not reverting these fixes.