arrolio 0.1.0

data/TODO.implementation/63-layout-rules-profile.md

@@ -0,0 +1,110 @@
+---
+priority: P1
+phase: 19
+depends_on: [62]
+layer: conformance
+est: 2d
+status: pending
+---
+
+## Problem
+
+A baseline ruleset for "did Arrolio itself produce a clean layout?"
+This is the **render-time self-check**: catch Arrolio's own
+regressions, not the source document's. Includes rules every
+production publisher expects:
+
+- No orphan headings (heading at very bottom of page).
+- No widow/orphan lines (single line of paragraph alone on a page).
+- Every page has the running header.
+- Tables don't break mid-row.
+- List items don't break mid-item.
+- Footnote and reference on the same page.
+- TOC entries resolve (no broken citations).
+- Page numbering is contiguous.
+
+These rules complement semantic-diff (TODO 55): diff catches
+"output changed", layout-rules catch "output is wrong even if it
+matches yesterday's output".
+
+## Approach
+
+File: `lib/arrolio/conformance/profiles/layout_rules.rb`.
+
+```ruby
+module Arrolio::Conformance::Profiles
+  module LayoutRules
+    def self.profile
+      Arrolio::Conformance::Profile.new(
+        name: "layout-baseline",
+        version: "1.0.0",
+        spec_reference: "Arrolio baseline layout expectations",
+        rules: [
+          # Headings
+          Rule.new(id: "no-orphan-heading",
+                   applies_to: :flowable,
+                   severity: :warning,
+                   test: ->(t, ctx) { !(t.heading? && t.last_on_page?) }),
+          Rule.new(id: "heading-keep-with-next",
+                   applies_to: :flowable,
+                   severity: :error,
+                   test: ->(t, ctx) { !t.heading? || t.kept_with_next? }),
+
+          # Paragraphs
+          Rule.new(id: "no-widow",
+                   applies_to: :paragraph,
+                   severity: :warning,
+                   test: ->(t, ctx) { t.line_count > 1 || t.first_on_page? }),
+
+          # Tables
+          Rule.new(id: "table-no-mid-row-break",
+                   applies_to: :table_row,
+                   severity: :error,
+                   test: ->(t, ctx) { !t.split_mid_row? }),
+
+          # Pagination
+          Rule.new(id: "page-numbering-contiguous",
+                   applies_to: :document,
+                   severity: :error,
+                   test: ->(t, ctx) { t.page_numbers_contiguous? }),
+          Rule.new(id: "every-page-has-footer",
+                   applies_to: :page,
+                   severity: :warning,
+                   test: ->(t, ctx) { t.region(:after).placed_boxes.any? }),
+
+          # Cross-references
+          Rule.new(id: "citations-resolved",
+                   applies_to: :field_run,
+                   severity: :error,
+                   test: ->(t, ctx) { !t.citation? || ctx.citation_for(t.ref_id) }),
+        ]
+      )
+    end
+  end
+end
+```
+
+Plus an OIML-specific extension profile that adds rules particular
+to OIML house style (e.g. "every clause has a number", "running
+header shows current top-level section title").
+
+### Engine integration
+
+Engine emits an `Output::Page[]` with enough metadata for the rules
+to evaluate: `heading?`, `last_on_page?`, `kept_with_next?`,
+`line_count`, `first_on_page?`, `split_mid_row?`,
+`page_numbers_contiguous?`, etc.
+
+Run the layout-rules profile automatically after layout (configurable).
+Failed `:error` rules raise `LayoutError`; `:warning` rules log.
+
+## Done-When
+
+- [ ] LayoutRules.profile has at least 8 rules covering the
+      categories above.
+- [ ] A document with an orphan heading fails `no-orphan-heading`.
+- [ ] A document with a mid-row table break fails
+      `table-no-mid-row-break`.
+- [ ] A document with broken citations fails `citations-resolved`.
+- [ ] Engine integration: rules run automatically unless disabled.
+- [ ] OIML extension profile applies on top of baseline.

data/TODO.implementation/64-pdfa-and-pdfua-conformance.md

@@ -0,0 +1,74 @@
+---
+priority: P2
+phase: 19
+depends_on: [62, 48]
+layer: conformance
+est: 5d
+status: pending
+---
+
+## Problem
+
+veraPDF ships rule profiles for ISO 19005 (PDF/A) and ISO 14289
+(PDF/UA). Arrolio should be able to validate its own output against
+these standards — without requiring veraPDF itself.
+
+This is a pragmatic subset, not full veraPDF parity. Goal: catch
+the common 80% of conformance failures before users hit them, then
+let veraPDF do the authoritative check.
+
+## Approach
+
+Files under `lib/arrolio/conformance/profiles/`:
+
+- `pdfa_1b.rb` — PDF/A-1b baseline rules:
+  - `/MarkInfo` not required (Level B).
+  - All fonts embedded (no standard-14 references).
+  - No unembedded JPG2000 in /JBIG2Decode without /DecodeParms.
+  - No JavaScript actions.
+  - No encrypted streams.
+  - `/ColorSpace` does not reference /DeviceN or /DevRGB without ICC.
+
+- `pdfa_2b.rb` — PDF/A-2b additions:
+  - JPEG2000 allowed (with restrictions).
+  - Transparency allowed (with restrictions).
+  - Object-level XMP metadata.
+
+- `pdfua_1.rb` — PDF/UA-1 rules:
+  - `/MarkInfo /true` on Catalog.
+  - `/StructTreeRoot` present.
+  - Every page has `/Tabs /S` (structured reading order).
+  - Every image has `/Alt` text.
+  - Every heading maps to the right `<Hn>` element.
+  - No nested marked-content sequences beyond depth limit.
+  - Font has `/ToUnicode` CMap (text extractable).
+
+- `pdfua_2.rb` — PDF/UA-2 (when standard is final).
+
+Each profile is a Ruby module with `.profile` returning a
+`Conformance::Profile`. Rules are written in the same DSL as TODO 63.
+
+### veraPDF cross-validation
+
+- A spec under `spec/conformance/veraPDF_cross_check_spec.rb` runs
+  both Arrolio's profiles and veraPDF (via shell) on the same
+  fixture PDFs; expected disagreements are catalogued.
+- Rake task `rake conformance:cross_check` runs the suite.
+
+## Approach for partial-coverage
+
+Be honest in the docs: "Arrolio catches N% of PDF/A-1b rules.
+Run veraPDF for authoritative validation." List the rules NOT
+implemented so users know what's missing.
+
+## Done-When
+
+- [ ] `pdfa_1b` profile covers ≥ 30 of the most common rules.
+- [ ] `pdfua_1` profile covers ≥ 20 of the most common rules.
+- [ ] An Arrolio-rendered PDF (without fonts embedded) fails
+      `pdfa_1b` with a useful error.
+- [ ] An Arrolio-rendered PDF (without StructTree) fails
+      `pdfua_1`.
+- [ ] veraPDF cross-check spec runs (when veraPDF is installed) and
+      documents the disagreements.
+- [ ] README documents the partial-coverage scope clearly.

data/TODO.implementation/65-feature-extractor.md

@@ -0,0 +1,64 @@
+---
+priority: P2
+phase: 19
+depends_on: [22]
+layer: conformance
+est: 2d
+status: pending
+---
+
+## Problem
+
+Modeled on `veraPDF/features/`, Arrolio needs a feature extractor
+that walks a PDF (ours or a third-party's) and produces a structured
+feature report — fonts used, images embedded, colour spaces, page
+sizes, annotations, signatures, metadata. Useful for:
+
+- Debugging: "why does this PDF render wrong on viewer X?"
+- Conformance pre-checks before running full validation.
+- Diff inputs (compare features between two PDFs as a quick scan).
+- Documentation: "what does this PDF contain?"
+
+## Approach
+
+Files under `lib/arrolio/conformance/features/`:
+
+- `feature_extractor.rb` — top-level orchestrator; walks a
+  `Pdfrb::Document`, dispatches per object type, accumulates
+  `FeatureReport`.
+
+- `feature_report.rb` — frozen value object: pages, fonts, images,
+  colour_spaces, annotations, signatures, metadata, embedded_files,
+  outlines, acroform_fields.
+
+- Per-category extractors:
+  - `font_features.rb` — base_font, subtype, embedded?, encoding,
+    to_unicode?, num_glyphs.
+  - `image_features.rb` — width, height, bits_per_component, color_space,
+    filter, embedded?, smask?.
+  - `color_space_features.rb` — name, type (DeviceRGB/CMYK/ICC/etc.),
+    icc_profile?.
+  - `annotation_features.rb` — subtype, rect, action.
+  - `page_features.rb` — mediabox, rotate, contents_size,
+    annotation_count.
+  - `metadata_features.rb` — info dict fields, xmp_packet?.
+
+- `feature_serializer.rb` — `to_yaml`, `to_json`, `to_html_table`.
+
+### Diff integration
+
+A `feature_diff` mode that compares two FeatureReports and emits a
+summary ("font Helvetica added", "image /Im3 missing in v2"). This
+is a fast pre-check before the full semantic comparator (TODO 55).
+
+## Done-When
+
+- [ ] FeatureExtractor produces a complete FeatureReport on a
+      fixture PDF.
+- [ ] All category extractors run without error on FOP-produced PDFs.
+- [ ] YAML/JSON/HTML serialisation works.
+- [ ] A simple "feature_diff" highlights added/removed/changed
+      features between two PDFs.
+- [ ] Round-trip: extract from rendered PDF shows expected font
+      list, image count, page count.
+- [ ] Spec coverage for each category extractor.

data/TODO.implementation/66-auto-fixer.md

@@ -0,0 +1,75 @@
+---
+priority: P2
+phase: 19
+depends_on: [62, 64]
+layer: conformance
+est: 3d
+status: pending
+---
+
+## Problem
+
+Modeled on `veraPDF/metadata/fixer/`, Arrolio needs an auto-fixer
+that takes the failed assertions from a conformance profile and
+applies known-safe fixes:
+
+- Missing `/CreationDate` → fill in `Time.now`.
+- Missing `/Producer` → `"Arrolio #{Arrolio::VERSION} + Pdfrb"`.
+- Missing `/MarkInfo /true` → add (when StructTree exists).
+- Missing `/Tabs /S` on pages → add.
+- Missing font `/ToUnicode` → generate from font encoding (best effort).
+- XMP packet missing → emit one from `/Info` fields.
+
+The fixer is **opt-in**: it never modifies without being asked. It
+also reports what it changed.
+
+## Approach
+
+Files under `lib/arrolio/conformance/fixer/`:
+
+- `base_fixer.rb` — abstract; takes a `Pdfrb::Document` + a
+  `ValidationResult`, returns a `FixResult`.
+
+- `fix_result.rb` — value object: applied_fixes (`Array<AppliedFix>`),
+  skipped_fixes (`Array<SkippedFix>`), unchanged.
+
+- `applied_fix.rb` — value object: rule_id, fix_kind, before, after,
+  location.
+
+- Per-rule fixers (one file each, loaded only when the rule failed):
+  - `creation_date_fixer.rb`
+  - `producer_fixer.rb`
+  - `mark_info_fixer.rb`
+  - `tabs_fixer.rb`
+  - `to_unicode_fixer.rb`
+  - `xmp_packet_fixer.rb`
+
+Each fixer is **conservative**: if it can't be sure the fix is
+correct, it skips (records the reason) rather than guessing.
+
+### Pipeline integration
+
+```ruby
+# After render:
+doc = render_to_pdfrb_doc(content, layout_spec)
+report = Arrolio::Conformance::Validator.(
+  Arrolio::Conformance::Profiles::Pdfa1b.profile,
+  doc
+)
+fixed_doc, fix_result = Arrolio::Conformance::Fixer.(doc, report)
+fixed_doc.write(io: output_io)
+```
+
+`Fixer` is opt-in via a render option `fix_conformance: true`.
+
+## Done-When
+
+- [ ] A PDF missing `/Producer` gets it added.
+- [ ] A PDF missing `/CreationDate` gets it added.
+- [ ] A PDF missing `/MarkInfo /true` (but with StructTree) gets it
+      added.
+- [ ] Each applied fix is recorded with before/after evidence.
+- [ ] Fixer skips fixes it's not sure about (no guessing).
+- [ ] Round-trip: fixed PDF re-validated passes the rules it failed
+      before.
+- [ ] `render(fix_conformance: true)` flag wires it in.

data/TODO.implementation/README.md

@@ -0,0 +1,169 @@
+# TODO.implementation — Arrolio Work Breakdown
+
+Arrolio is the **Ruby-only alternative to Apache FOP / XSL-FO**. It takes
+a typed content tree plus a typed layout spec, lays out pages, and renders
+to PDF via the sibling `pdfrb` gem. No XML input. No FO intermediate.
+The Ruby model is the source of truth.
+
+This directory tracks all implementation work as one file per TODO,
+named `{NN}-{slug}.md`.
+
+## Architectural layering (recap)
+
+```
+Content Tree (Ruby, typed, medium-independent)
+       ↓
+LayoutSpec (Ruby, typed: PageTemplate, Region, Style, Flow)
+       ↓
+Engine::Paged (Knuth-Plass breaking, frame threading)
+       ↓
+Output::Page[] (laid-out pages with placed boxes)
+       ↓
+Renderer::Pdf (walks Output tree, emits bytes via Pdfrb)
+       ↓
+PDF bytes
+```
+
+Each layer is independently testable. Output::Page[] is the contract
+between Engine and Renderer.
+
+## Primary end-to-end correctness: three subsystems
+
+Arrolio is validated by three complementary subsystems, all
+exercised against real OIML documents:
+
+1. **OIML E2E pipeline** (TODOs 52–54) — parse real OIML XML via
+   `metanorma-document`, apply an OIML LayoutSpec, render to PDF via
+   Pdfrb. Proves the full pipeline works end-to-end.
+
+2. **PDF diff subsystem** (TODOs 55–59), modeled on
+   `~/src/lutaml/canon/` — semantic comparator with dimensions
+   (text, font, position, colour, image, vector, structure),
+   normative/informative classifier, multiple formatters, RSpec
+   matchers, optional pixel diff. Catches "output changed in a way
+   that matters".
+
+3. **Conformance subsystem** (TODOs 62–66), modeled on
+   `~/src/external/veraPDF-library/` — rule-based validator with
+   profiles for layout baseline (TODO 63), PDF/A (TODO 64), and
+   PDF/UA (TODO 64); feature extractor for debugging (TODO 65);
+   auto-fixer for known-safe repairs (TODO 66). Catches "output is
+   wrong even when it matches yesterday's output".
+
+## Critical path (P0)
+
+To reach **minimum viable PDF** (a one-page document with text), in order:
+
+```
+01 → 02 → 06 → 07 → 09 → 10 → 11 → 13
+                                      ↓
+21 → 22 → 23 (output)  ←── 20 (engine) ←── 16-19 (flowables)
+```
+
+After that, page templates (Phase 7) and tables (Phase 8) deliver
+FOP-equivalent capability. Then the OIML E2E pipeline (Phase 17)
+proves the engine on real-world documents, and the diff subsystem
+makes correctness verifiable per-commit.
+
+## Phase index
+
+| Phase | Range | Layer | Outcome |
+|---|---|---|---|
+| 1 | 01–05 | foundation | Skeleton, content contract, layout spec, style system, visitors |
+| 2 | 06–09 | metrics | AFM + TrueType glyph metrics + Registry + GlyphMeasurer |
+| 3 | 10–13 | text | Break opportunities, Greedy + Knuth-Plass, Line + alignment |
+| 4 | 14–15 | knuth | Universal Knuth elements + Breaker (refactor of Phase 3) |
+| 5 | 16–20 | flowable | Flowable hierarchy + Frame + Engine::Paged driver |
+| 6 | 21–23 | output | Output::Page[] + Renderer::Pdf + end-to-end smoke |
+| 7 | 24–26 | template | PageTemplate, PageSequenceMaster, two-pass citations |
+| 8 | 27–30 | table | Table model + Fixed/Auto layouts + TableFlowable |
+| 9 | 31–32 | list | List model + ListFlowable (proper label/body columns) |
+| 10 | 33–35 | media | Image loading via Pdfrb + SVG renderer + Form XObjects |
+| 11 | 36–38 | inline | InlineBuilder + Hyperlink + Sub/superscript |
+| 12 | 39–42 | xref | Leader + FieldRuns + Bookmark/outline + Destinations + TOC |
+| 13 | 43–45 | page | Footnotes + Multi-column + Page numbering + Section tracking |
+| 14 | 46–47 | color | Color management + XMP/Info metadata write |
+| 15 | 48–49 | a11y | PDF/UA StructTree + marked content + alt text |
+| 16 | 50–51 | composer | High-level Composer facade + style presets |
+| 17 | **52–59** | **harness/diff** | **OIML E2E pipeline + canon-style diff subsystem** |
+| 18 | 60–61 | polish | YARD docs + performance/stress tests |
+| 19 | **62–66** | **conformance** | **veraPDF-style validator + profiles + feature extractor + fixer** |
+
+**Total: 66 TODOs.**
+
+## Phase 17 in detail — OIML E2E + diff subsystem
+
+| TODO | What it builds |
+|---|---|
+| 52 | OIML adapter (metanorma-document → Arrolio content) + fixture corpus |
+| 53 | OIML LayoutSpec — Ruby encoding of `oiml.xsl`'s visual intent |
+| 54 | Pipeline (`Arrolio::Oiml::Pipeline.(xml, io:)`) + CLI + sanity spec |
+| 55 | **PDF comparator** — semantic tree diff with dimensions, normative/informative classifier, profiles |
+| 56 | **Diff formatters** — summary, pretty, by-page, by-element, JSON, HTML, JUnit |
+| 57 | **Path locator + enrichment** — canonical paths, char-level diffs, source positions |
+| 58 | **RSpec matchers + Harness API** — `match_pdf`, `be_equivalent_to_pdf`, `Arrolio::Harness.diff` |
+| 59 | **Pixel diff mode** — optional raster-based diff for visual regressions |
+
+## Phase 19 in detail — Conformance & validation (veraPDF-inspired)
+
+| TODO | What it builds |
+|---|---|
+| 62 | Validation framework — Rule, Profile, TestAssertion, ValidationResult, Validator, Location |
+| 63 | Layout baseline rules — no orphan headings, no widow/orphans, contiguous page numbering |
+| 64 | PDF/A + PDF/UA conformance profiles — pragmatic rule subset; cross-check vs veraPDF |
+| 65 | Feature extractor — fonts/images/colour/annotations dump for debugging |
+| 66 | Auto-fixer — applies known-safe fixes (CreationDate, Producer, MarkInfo, XMP) |
+
+After TODO 66 lands, Arrolio can self-validate its own output against
+PDF/A and PDF/UA rules — a veraPDF in Ruby for documents Arrolio
+produces.
+
+## Front-matter shape
+
+```yaml
+---
+priority: P0     # P0 blocks MVP; P1 = FOP parity; P2 = production; P3 = polish
+phase: 1
+depends_on: []   # TODO numbers that must be done first
+layer: foundation
+est: 2d          # 1d = single focused session; 5d = multi-week module
+status: pending  # pending | in_progress | done
+---
+```
+
+## What Arrolio deliberately does NOT do
+
+- Parse XML input (FO or otherwise). **Loom's** job.
+- Author DSLs (YAML, custom syntax). **Loom's** job.
+- Compile to multiple targets (CSS, FO, IDML). **Loom's** job.
+- Speak PDF byte format directly. **Pdfrb's** job.
+- Own the semantic content model. **metanorma-document's** job —
+  Arrolio defines a content contract, metanorma-document fills it.
+- Replace veraPDF authoritatively for PDF/A. **veraPDF's** job.
+  Arrolio's conformance profiles (Phase 19) are a pragmatic
+  pre-check; veraPDF remains authoritative.
+
+If a TODO crosses these boundaries, it belongs in another gem.
+
+## Verification
+
+Every TODO carries Done-When criteria. The global verification is:
+
+- `bundle exec rake` — runs rspec + rubocop; must be green after every TODO.
+- After TODO 23 (end-to-end smoke): a single-paragraph PDF renders correctly.
+- After TODO 30 (TableFlowable): an OIML-style multi-page doc with tables
+  renders.
+- After TODO 54 (OIML pipeline): real OIML XML renders to PDF.
+- After TODO 58 (RSpec matchers): every commit validates against FOP
+  references with `be_equivalent_to_pdf`.
+- After TODO 64 (PDF/A + PDF/UA): Arrolio's output passes the
+  common-subset conformance rules.
+
+## Status legend
+
+- `pending` — not started.
+- `in_progress` — actively being worked.
+- `done` — complete; specs pass; ready for review.
+
+(Initial seed: all 66 TODOs are `pending`. TODO 01 is partially done —
+the gem skeleton already exists.)