canon 0.1.23 → 0.2.1

data/docs/features/diff-formatting/index.adoc

@@ -72,6 +72,19 @@ Make invisible characters visible in diff output:
 * Non-ASCII character detection
 * CJK-safe Unicode symbols
 
+Controlled via `Canon::Config`:
+
+* `true` — (default) apply the full visualization map; spaces → `░`, tabs → `⇥`, NBSP → `␣`
+* `false` — disable all substitution; useful for copying output or CI tooling that cannot render Unicode symbols
+* `:content_only` — reserved; currently behaves as `true`
+
+[source,ruby]
+----
+Canon::Config.configure do |cfg|
+  cfg.xml.diff.character_visualization = false
+end
+----
+
 See link:character-visualization.adoc[Character visualization] for details.
 
 === Context and grouping
@@ -83,6 +96,40 @@ Control how much surrounding context to show:
 
 See link:context-and-grouping.adoc[Context and grouping] for details.
 
+=== Display preprocessing
+
+Normalize both documents through the same formatter before the line diff, so
+that formatting differences between expected and actual do not confuse the LCS
+algorithm:
+
+* `:none` — use documents as-is (default)
+* `:pretty_print` — run through `Canon::PrettyPrinter::Xml` (one tag per line)
+* `:c14n` — run through XML C14N normalization
+
+See link:display-preprocessing.adoc[Display preprocessing] for details,
+including the character-visualization constraint for future extensibility.
+
+=== Pretty-diff mode
+
+`:pretty_diff` is a text-LCS diff mode that bypasses DiffNodeMapper entirely.
+It applies `display_preprocessing` to both sides and runs `Diff::LCS.sdiff`
+on the resulting plain-text lines, guaranteeing that every line-level change
+is visible — even when DiffNodeMapper's DOM-address correlation is unreliable.
+
+Use it as a reliable fallback when `:by_line` shows no output for changes you
+know are present (issue #85).
+
+[source,ruby]
+----
+formatter = Canon::DiffFormatter.new(
+  mode: :pretty_diff,
+  display_preprocessing: :pretty_print,
+  context_lines: 3,
+)
+----
+
+See link:pretty-diff-mode.adoc[Pretty-diff mode] for details and limitations.
+
 === Algorithm-specific output
 
 Different diff algorithms produce different output styles:

data/docs/features/diff-formatting/pretty-diff-mode.adoc

@@ -0,0 +1,154 @@
+---
+title: Pretty-diff mode
+parent: Diff Formatting
+grand_parent: Features
+nav_order: 9
+---
+= Pretty-diff mode
+:toc:
+:toclevels: 2
+
+== Purpose
+
+`:pretty_diff` is a text-LCS diff mode that bypasses DiffNodeMapper entirely.
+It applies `display_preprocessing` to both sides, then runs
+`Diff::LCS.sdiff` on the resulting plain-text lines.
+
+This guarantees that every line-level change is visible, regardless of whether
+DiffNodeMapper's DOM-address correlation is accurate.
+
+== When to use
+
+Use `:pretty_diff` when:
+
+* `:by_line` mode shows no diff output for changes you *know* are present
+  (the issue #85 class of bug where DiffNodeMapper maps DiffNodes to the
+  wrong post-preprocessing line numbers)
+* You want a quick human-readable text diff and do not need inline
+  character-level highlighting
+
+Use `:by_line` when:
+
+* Inline character highlighting within a line matters
+* `show_diffs: :normative` / `:informative` filtering is required (these
+  filters rely on DiffNodes that `:pretty_diff` does not use)
+
+== Usage
+
+=== Basic usage
+
+[source,ruby]
+----
+formatter = Canon::DiffFormatter.new(
+  mode: :pretty_diff,
+  display_preprocessing: :normalize_pretty_print,  # recommended for mixed-content XML
+  context_lines: 3,
+  use_color: true,
+)
+output = formatter.format(differences, :xml, doc1: expected, doc2: actual)
+----
+
+Combining with `display_preprocessing: :normalize_pretty_print` is strongly
+recommended for XML with inline markup (Metanorma, DocBook, DITA).  It breaks
+every mixed-content element onto separate lines and visualizes inter-element
+whitespace, so the LCS diff can pinpoint changes inside `<p>`, `<formattedref>`,
+and similar elements.
+
+`display_preprocessing: :pretty_print` is sufficient for element-only XML
+(no mixed content).
+
+=== Controlling per-element whitespace visualization
+
+By default, `:normalize_pretty_print` treats all inter-element whitespace as
+structural formatting noise and drops it silently — both compact XML and
+indented XML serialize to identical lines.  This is the correct default for
+purely structural elements such as `<formattedref>` or `<bibdata>`.
+
+For mixed-content elements where whitespace *presence* is meaningful (e.g. the
+word-boundary space in `<p>See <xref/>`), opt in with
+`collapse_whitespace_elements`:
+
+[source,ruby]
+----
+formatter = Canon::DiffFormatter.new(
+  mode: :pretty_diff,
+  display_preprocessing: :normalize_pretty_print,
+  collapse_whitespace_elements: %w[p li td th],
+  context_lines: 3,
+  use_color: true,
+)
+----
+
+Or via `Canon::Config` (recommended in `spec_helper.rb`):
+
+[source,ruby]
+----
+Canon::Config.configure do |cfg|
+  cfg.xml.diff.mode = :pretty_diff
+  cfg.xml.diff.display_preprocessing = :normalize_pretty_print
+  cfg.xml.diff.collapse_whitespace_elements = %w[p li td th]
+end
+----
+
+For preformatted elements where every whitespace character is significant, use
+`preserve_whitespace_elements` instead (e.g. `%w[pre code]`).
+
+See
+link:display-preprocessing.adoc#_controlling_whitespace_visualization_with_element_classification[display-preprocessing:
+element classification] for the full three-way classification system and HTML
+format defaults.
+
+== Output format
+
+The output uses the same header/context/separator conventions as `:by_line`:
+
+[source]
+----
+Pretty diff (XML mode):
+  <section>
+    <title>Introduction</title>
+- <para>This describes the <strong>old</strong> behaviour.</para>
++ <para>This describes the <strong>new</strong> behaviour.</para>
+    <note>See also: Appendix A.</note>
+  </section>
+----
+
+* Lines prefixed `- ` are removals (present in `doc1`, absent in `doc2`)
+* Lines prefixed `+ ` are additions (absent in `doc1`, present in `doc2`)
+* Lines prefixed `  ` are context (unchanged)
+* `--- ---` separates non-adjacent change blocks
+
+== Context windowing
+
+`context_lines` (default 3) controls how many unchanged lines are shown around
+each change.  Nearby windows whose expanded ranges overlap or are adjacent are
+merged into a single block.  A `--- ---` separator is emitted between blocks
+that are not adjacent.
+
+Setting `context_lines: 0` shows only the changed lines with no surrounding
+context.
+
+== Limitations
+
+`:pretty_diff` trades DiffNode integration for reliability:
+
+* `show_diffs: :normative` / `:informative` filtering is **not supported**.
+  All line differences are shown regardless of their classification.
+* Inline character-level highlighting (e.g. highlighting the changed word
+  within a changed line) is **not available**; granularity is whole-line only.
+* The `differences` array passed to `DiffFormatter#format` is **ignored**;
+  the diff is computed purely from `doc1` and `doc2` via LCS.
+
+== Implementation note
+
+`Diff::LCS.sdiff` is invoked with `::Diff::LCS` (absolute constant path)
+to avoid resolving against `Canon::Diff` which is an unrelated internal module.
+
+== See also
+
+* link:display-preprocessing.adoc[Display preprocessing] — normalising input
+  before the diff
+* link:context-and-grouping.adoc[Context and grouping] — context_lines and
+  diff grouping
+* link:display-filtering.adoc[Display filtering] — normative/informative
+  filtering (`:by_line` only)

data/docs/features/environment-configuration/override-system.adoc

@@ -16,7 +16,7 @@ This page explains how the priority chain works, when overrides apply, and how t
 
 == Priority chain
 
-Configuration values are resolved using a strict three-level priority:
+Configuration values are resolved using a strict four-level priority:
 
 [source]
 ----
@@ -26,18 +26,25 @@ Configuration values are resolved using a strict three-level priority:
 └──────────────┬──────────────────┘
                ↓ overrides
 ┌─────────────────────────────────┐
-│ 2. Programmatic Configuration   │ ← Medium Priority
+│ 2. Programmatic Configuration   │
 │    (config.xml.diff.algorithm)  │
 └──────────────┬──────────────────┘
                ↓ overrides
 ┌─────────────────────────────────┐
-│ 3. Default Values               │ ← Lowest Priority
+│ 3. Profile Values               │
+│    (from YAML config profile)   │
+└──────────────┬──────────────────┘
+               ↓ overrides
+┌─────────────────────────────────┐
+│ 4. Default Values               │ ← Lowest Priority
 │    (defined in Canon::Config)   │
 └─────────────────────────────────┘
 ----
 
 **Rule**: Higher priority always wins, regardless of when values are set.
 
+See link:../configuration-profiles.adoc[Configuration Profiles] for details on the profile layer.
+
 == How overrides work
 
 === Environment variables override programmatic settings

data/docs/features/index.adoc

@@ -77,6 +77,15 @@ Customize how differences are displayed.
 * link:diff-formatting/context-and-grouping[Context and Grouping] - Surrounding lines
 * link:diff-formatting/character-visualization[Character Visualization] - Whitespace visibility
 
+=== Configuration Profiles
+
+link:configuration-profiles.adoc[**Configuration Profiles**]::
+Bundle all settings into named presets defined in YAML.
++
+* Built-in profiles (metanorma, metanorma_debug)
+* Custom profiles from local YAML files
+* Profile inheritance
+
 === System Configuration
 
 link:environment-configuration/[**Environment Configuration**]::

data/docs/features/match-options/html-policies.adoc

@@ -226,6 +226,9 @@ Canon::Comparison.equivalent?(html1, html2, preprocessing: :rendered)
 Pretty-prints before comparison:
 - Consistent indentation
 - One element per line
+- Whitespace-only text nodes in strip-context elements (e.g. `<div>`) are
+  removed after formatting, so differences in structural whitespace between
+  elements do not produce false normative differences
 - Good for visual diffs
 
 [source,ruby]

data/docs/features/match-options/index.adoc

@@ -30,6 +30,7 @@ Match options control which aspects of documents are compared and how strictly t
 * link:profiles.adoc[Match Profiles] - Predefined configurations
 * link:algorithm-specific-behavior.adoc[Algorithm-Specific Behavior] - How DOM and Semantic algorithms interpret options differently
 * link:html-policies.adoc[HTML-Specific Policies] - HTML format-specific comparison policies
+* link:pretty-printed-fixtures.adoc[Pretty-Printed Fixture Support] - Comparing compact generated XML against hand-indented fixture heredocs without spurious whitespace differences
 
 == Match dimensions overview
 
@@ -188,34 +189,25 @@ Canon::Comparison.equivalent?(xml1, xml2,
 ----
 ====
 
-==== Whitelist and blacklist options
+==== Element classification options
 
-You can explicitly specify which elements are whitespace-sensitive using either short or long option names:
+You can explicitly classify elements for whitespace handling:
 
 [source,ruby]
 ----
-# Short names (preferred)
 Canon::Comparison.equivalent?(xml1, xml2,
   match: {
     structural_whitespace: :strict,
-    sensitive_elements: ["pre", "code", "sample"],
-    insensitive_elements: ["div", "span"]
-  }
-)
-
-# Long names (backward-compatible)
-Canon::Comparison.equivalent?(xml1, xml2,
-  match: {
-    structural_whitespace: :strict,
-    whitespace_sensitive_elements: ["pre", "code", "sample"],
-    whitespace_insensitive_elements: ["div", "span"]
+    preserve_whitespace_elements: ["pre", "code", "sample"],
+    collapse_whitespace_elements: ["p", "li", "td"],
+    strip_whitespace_elements: ["div", "span"]
   }
 )
 ----
 
 **Element names are strings** (not symbols) for consistency with XML/HTML conventions.
 
-**Blacklist takes precedence over whitelist** — if an element appears in both lists, whitespace is stripped.
+**Priority order**: strip > preserve > collapse > format defaults.
 
 ==== respect_xml_space option
 
@@ -248,21 +240,19 @@ When determining if an element is whitespace-sensitive, Canon uses this priority
 ----
 1. respect_xml_space: false → User config only (ignore xml:space)
    ↓
-2. User whitelist → Use whitelist (user explicitly declared)
+2. Ancestor walk (strip wins; then preserve; then collapse)
    ↓
-3. Format defaults → HTML: [:pre, :textarea, :script, :style], XML: []
+3. xml:space="preserve" → Element classified as :preserve
    ↓
-4. User blacklist → Remove from defaults/whitelist
+4. xml:space="default" → Use configured behaviour
    ↓
-5. xml:space="preserve" → Element is sensitive
-   ↓
-6. xml:space="default" → Use steps 1-4
-----
+5. Format defaults (HTML: preserve + collapse; XML: strip)
 
 ==== Format-specific defaults
 
-**HTML**:: `["pre", "textarea", "script", "style"]` - These elements preserve whitespace by HTML specification
-**XML**:: `[]` - No default whitespace-sensitive elements, purely user-controlled
+**HTML preserve**:: `["pre", "textarea", "script", "style"]` - Every whitespace character significant
+**HTML collapse**:: `["p", "li", "td", "th", "dt", "dd", "h1"-"h6", ...]` - Whitespace collapsed
+**XML**:: No defaults — all elements are :strip unless explicitly configured
 
 ==== Two types of whitespace sensitivity
 
@@ -272,28 +262,28 @@ Canon handles two distinct whitespace concerns:
 
 **2. Text content comparison** — how non-whitespace text content is compared. Controlled by `structural_whitespace` and `text_content` dimension behaviors (`:strict`, `:normalize`, `:ignore`).
 
-The `sensitive_elements` / `insensitive_elements` options control both concerns:
+The `preserve_whitespace_elements`, `collapse_whitespace_elements`, and `strip_whitespace_elements` options control both concerns:
 
 [source,ruby]
 ----
 # For XML: structural whitespace is stripped by default
-# Use sensitive_elements to preserve whitespace in specific elements
+# Use preserve_whitespace_elements to preserve whitespace in specific elements
 xml1 = "<root><item>Test</item></root>"
 xml2 = "<root>\n  <item>Test</item>\n</root>"
 
-# With sensitive_elements, whitespace inside <item> is preserved
+# With preserve_whitespace_elements, whitespace inside <item> is preserved
 Canon::Comparison.equivalent?(xml1, xml2,
   match: {
     structural_whitespace: :strict,
-    sensitive_elements: ["item"]
+    preserve_whitespace_elements: ["item"]
   }
 )
 # => true
 ----
 
-**Precedence**: blacklist (`insensitive_elements`) > whitelist (`sensitive_elements`) > format defaults
+**Precedence**: `strip_whitespace_elements` > `preserve_whitespace_elements` > `collapse_whitespace_elements` > format defaults
 
-**No inheritance**: Only the immediate parent element's name is checked — not ancestor elements.
+**Ancestor-based**: The closest matching ancestor determines classification.
 
 ==== Examples
 
@@ -321,17 +311,17 @@ Canon::Comparison.equivalent?(xml1, xml2,
 # => false
 ----
 
-.Using sensitive_elements whitelist
+.Using preserve_whitespace_elements
 [source,ruby]
 ----
-# Make <sample> elements whitespace-sensitive (strings, not symbols)
+# Make <sample> elements preserve whitespace exactly (strings, not symbols)
 xml1 = "<sample>\n  content\n</sample>"
 xml2 = "<sample>content</sample>"
 
 Canon::Comparison.equivalent?(xml1, xml2,
   match: {
     structural_whitespace: :strict,
-    sensitive_elements: ["sample"]
+    preserve_whitespace_elements: ["sample"]
   }
 )
 # => false (structural whitespace differs in <sample>)
@@ -340,30 +330,30 @@ Canon::Comparison.equivalent?(xml1, xml2,
 .Overriding HTML defaults
 [source,ruby]
 ----
-# Make <script> NOT whitespace-sensitive (override HTML default)
+# Make <script> NOT whitespace-preserved (override HTML default)
 Canon::Comparison.equivalent?(html1, html2,
   format: :html,
   match: {
     structural_whitespace: :strict,
-    insensitive_elements: ["script"]
+    strip_whitespace_elements: ["script"]
   }
 )
 ----
 
-.Using text_content: :normalize with whitespace_insensitive_elements
+.Using text_content: :normalize with strip_whitespace_elements
 [source,ruby]
 ----
-# HTML defaults: [:pre, :code, :textarea, :script, :style]
-# Excluding :code means it's no longer whitespace-sensitive
+# HTML defaults: pre, code, textarea, script, style are :preserve
+# Adding code to strip list removes it from preserve
 html1 = '<root><pre>  indented  </pre><code>  code  </code></root>'
 html2 = '<root><pre>  indented  </pre><code>code</code></root>'
 
-# With :code blacklisted, whitespace in <code> is normalized (formatting-only)
+# With :code in strip list, whitespace in <code> is normalized (formatting-only)
 # HTML uses text_content: :normalize by default
 Canon::Comparison.equivalent?(html1, html2,
   format: :html,
   match: {
-    whitespace_insensitive_elements: [:code],
+    strip_whitespace_elements: [:code],
   }
 )
 # => true (whitespace differences in <code> are formatting-only)
@@ -737,7 +727,7 @@ expect(actual).to be_xml_equivalent_to(expected,
   match: { structural_whitespace: :strict }
 )
   .with_options(
-    sensitive_elements: ["pre", "code", "sample"],
+    preserve_whitespace_elements: ["pre", "code", "sample"],
     respect_xml_space: true
   )
 
@@ -746,7 +736,7 @@ expect(html).to be_html_equivalent_to(expected,
   match: { structural_whitespace: :strict }
 )
   .with_options(
-    insensitive_elements: ["script", "style"]
+    strip_whitespace_elements: ["script", "style"]
   )
 ====