canon 0.1.21 → 0.1.22

data/docs/features/diff-formatting/themes.adoc

@@ -0,0 +1,353 @@
+---
+layout: default
+title: Diff Display Themes
+parent: Diff Formatting
+grand_parent: Features
+nav_order: 6
+---
+
+:toc:
+:toclevels: 3
+
+== Purpose
+
+Canon provides a theme system for diff display that controls colors, backgrounds, text effects, and visual markers. Themes make diffs readable on different terminal backgrounds and allow personalizing the visual style.
+
+== Available themes
+
+Canon ships with four predefined themes:
+
+[cols="1,2,3"]
+|===
+| Theme | Best For | Key Characteristics
+
+| `:light`
+| Light terminal backgrounds
+| Light marker backgrounds (`light_red`, `light_green`), dark text, `:black` structure elements
+
+| `:dark`
+| Dark terminal backgrounds (default)
+| Saturated foreground colors, no backgrounds, `:cyan` informative, `:bright_blue` formatting
+
+| `:retro`
+| Amber CRT / low blue light / accessibility
+| Monochromatic amber/yellow, inverse video for removed, `:bright_yellow` text on `:yellow` background
+
+| `:claude`
+| Maximum visual contrast (Claude Code style)
+| `:white` on `:red` background for removed, `:black` on `:green` background for added
+|===
+
+== Theme structure
+
+Each theme is a nested hash with these top-level sections:
+
+[cols="1,3"]
+|===
+| Section | Controls
+
+| `diff`
+| Colors for removed, added, changed, unchanged, formatting, and informative diffs
+
+| `xml`
+| Colors for XML syntax elements: tags, attribute names, attribute values, text, comments, CDATA
+
+| `html`
+| Same as `xml` but for HTML output
+
+| `structure`
+| Colors for line numbers, pipe separators, context lines
+
+| `visualization`
+| Unicode characters for invisible whitespace (space, tab, newline, nbsp)
+
+| `display_mode`
+| How changed lines display: `:separate`, `:inline`, or `:mixed`
+|===
+
+=== Styling properties
+
+Each styled element supports these properties:
+
+[cols="1,1,3"]
+|===
+| Property | Type | Description
+
+| `color`
+| Symbol
+| ANSI foreground color
+
+| `bg`
+| Symbol or nil
+| ANSI background color (nil = no background)
+
+| `bold`
+| Boolean
+| Bold text
+
+| `underline`
+| Boolean
+| Underlined text
+
+| `strikethrough`
+| Boolean
+| Strikethrough text
+
+| `italic`
+| Boolean
+| Italic text (terminal support varies)
+|===
+
+=== Valid color values
+
+.Standard ANSI colors (universally supported)
+[cols="2,1"]
+|===
+| Color | ANSI Code
+
+| `:black` | 30
+| `:red` | 31
+| `:green` | 32
+| `:yellow` | 33
+| `:blue` | 34
+| `:magenta` | 35
+| `:cyan` | 36
+| `:white` | 37
+|===
+
+.Bright variants (supported by most terminals)
+[cols="2,1"]
+|===
+| Color | ANSI Code
+
+| `:bright_red` | 91
+| `:bright_green` | 92
+| `:bright_yellow` | 93
+| `:bright_blue` | 94
+| `:bright_magenta` | 95
+| `:bright_cyan` | 96
+| `:default` | Default terminal color
+|===
+
+NOTE: The Rainbow gem (Canon's terminal color library) doesn't support `:bright_black` or `:bright_white` directly. Themes use `:white` or `:black` as substitutes.
+
+== Diff color sections
+
+Each theme defines colors for six diff categories:
+
+[cols="1,2,2"]
+|===
+| Category | Markers | Description
+
+| `removed`
+| `-` (normative), `[` (formatting), `<` (informative)
+| Lines or content present in the old document but not the new
+
+| `added`
+| `+` (normative), `]` (formatting), `>` (informative)
+| Lines or content present in the new document but not the old
+
+| `changed`
+| `*` (mixed)
+| Lines with both old and new content; has `content_old` and `content_new` sub-sections
+
+| `unchanged`
+| ` ` (space)
+| Context lines with no differences
+
+| `formatting`
+| `[` / `]`
+| Pure whitespace/formatting differences (no semantic change)
+
+| `informative`
+| `<` / `>`
+| Tracked differences that don't affect equivalence
+|===
+
+== Theme specifications
+
+=== Light theme (`:light`)
+
+Designed for light terminal backgrounds (white/light gray).
+
+* Removed: `:red` text with `:light_red` background on markers, `:strikethrough` on content
+* Added: `:green` text with `:light_green` background on markers
+* Changed: `:bright_red` old / `:bright_green` new, bold
+* Formatting: `:bright_blue` (visible on light backgrounds)
+* Informative: `:bright_magenta`
+* Structure: `:black` line numbers and pipes
+* Comments: `:magenta` italic
+
+=== Dark theme (`:dark`, default)
+
+Designed for dark terminal backgrounds.
+
+* Removed: `:red` text, `:strikethrough` on content
+* Added: `:green` text
+* Changed: `:yellow` marker, `:bright_red` old / `:bright_green` new
+* Formatting: `:bright_blue` (subtle but visible on dark backgrounds)
+* Informative: `:cyan` (distinct from formatting)
+* Structure: `:white` line numbers and pipes
+* Comments: `:cyan` italic
+
+=== Retro theme (`:retro`)
+
+Amber CRT phosphor look. Monochromatic amber/yellow, good for low blue light and accessibility.
+
+* Removed: Inverse video - `:bright_yellow` on `:yellow` background, bold
+* Added: `:bright_white` (less emphasis than removed)
+* Changed: `:bright_yellow` old with strikethrough / `:bright_white` new with underline
+* Formatting: `:yellow` (subtle)
+* Informative: `:bright_yellow` bold (stands out)
+* Structure: `:yellow` throughout
+* Comments: `:yellow` italic
+
+=== Claude theme (`:claude`)
+
+Claude Code diff style. Maximum visual contrast with colored backgrounds.
+
+* Removed: `:white` text on `:red` background
+* Added: `:black` text on `:green` background
+* Changed: `:white` on `:magenta` background for marker
+* Formatting: `:yellow`
+* Informative: `:bright_cyan`
+* Structure: `:yellow` line numbers and pipes
+* Comments: `:cyan` italic
+
+== Configuration
+
+=== Setting theme by name
+
+.Ruby API
+[source,ruby]
+----
+Canon::Config.configure do |config|
+  config.xml.diff.theme = :claude
+end
+----
+
+.ENV variable
+[source,bash]
+----
+export CANON_DIFF_THEME=claude
+----
+
+Valid values: `light`, `dark`, `retro`, `claude`
+
+=== Resolution priority
+
+Theme is resolved in this order (highest to lowest):
+
+. `CANON_DIFF_THEME` environment variable
+. `config.xml.diff.theme_inheritance` (base theme + overrides)
+. `config.xml.diff.custom_theme` (complete custom theme hash)
+. `config.xml.diff.theme` (theme name)
+. `:dark` (default)
+
+=== Theme inheritance
+
+Create a custom theme by inheriting from a base theme and overriding specific properties:
+
+[source,ruby]
+----
+Canon::Config.configure do |config|
+  config.xml.diff.theme_inheritance = {
+    base: :dark,
+    overrides: {
+      diff: {
+        removed: { content: { bg: :light_red } }
+      }
+    }
+  }
+end
+----
+
+Only the overridden properties change; everything else inherits from the base theme.
+
+.Programmatic theme building
+[source,ruby]
+----
+custom_theme = Canon::DiffFormatter::Theme.inherit_from(:dark)
+  .merge(diff: { removed: { content: { bg: :light_red } } })
+  .build
+
+Canon::Config.configure do |config|
+  config.xml.diff.custom_theme = custom_theme
+end
+----
+
+=== Full custom theme
+
+Provide a complete theme hash with all required keys:
+
+[source,ruby]
+----
+Canon::Config.configure do |config|
+  config.xml.diff.custom_theme = Canon::DiffFormatter::Theme.inherit_from(:light)
+    .merge(diff: { removed: { content: { color: :magenta } } })
+    .build
+end
+----
+
+== Example script
+
+Canon includes an example script that demonstrates all themes with a sample XML diff:
+
+[source,bash]
+----
+bundle exec ruby examples/show_themes.rb
+----
+
+This shows each theme's color output side-by-side, plus plain text output and usage examples.
+
+== Theme validation
+
+All themes (built-in and custom) are validated for:
+
+* **Completeness**: All required sections and keys present
+* **Valid values**: Colors are valid ANSI, booleans are true/false
+* **MECE**: No missing or extra keys
+
+[source,ruby]
+----
+result = Canon::DiffFormatter::Theme.validate(my_theme)
+result.valid?          # => true/false
+result.missing_keys    # => ["diff.removed.content.color", ...]
+result.invalid_values  # => ["diff.removed.content.color must be one of ..."]
+----
+
+== Rainbow gem compatibility
+
+Canon uses the https://github.com/kucaahbe/rainbow[Rainbow] gem for terminal colors. Some color names that Canon's theme system accepts are translated internally:
+
+[cols="2,2"]
+|===
+| Theme Color | Rainbow Method Chain
+
+| `:bright_red`
+| `.red.bright`
+
+| `:bright_blue`
+| `.blue.bright`
+
+| `:bright_green`
+| `.green.bright`
+
+| `:bright_cyan`
+| `.cyan.bright`
+
+| `:bright_magenta`
+| `.magenta.bright`
+
+| `:bright_yellow`
+| `.yellow.bright`
+|===
+
+Colors `:bright_black` and `:bright_white` are not supported by Rainbow in 16-color mode. Themes use `:white`, `:black`, or `:cyan` as substitutes.
+
+== See also
+
+* link:colors-and-symbols.adoc[Colors and symbols] - Diff markers and classification
+* link:display-filtering.adoc[Display filtering] - Control which diff types appear
+* link:../environment-configuration/index.html[Environment configuration] - ENV variable setup
+* link:../../reference/environment-variables.html[Environment variables reference] - Complete variable listing

data/docs/features/environment-configuration/index.adoc

@@ -99,6 +99,29 @@ export CANON_HTML_DIFF_USE_COLOR=true
 
 Valid values: `true`, `false`, `1`, `0`, `yes`, `no`
 
+=== Diff display theme
+
+Choose a color theme for diff output. Themes control foreground/background colors, text effects, and visual markers.
+
+[source,bash]
+----
+# Set theme for dark terminal
+export CANON_DIFF_THEME=dark
+
+# Set theme for light terminal
+export CANON_DIFF_THEME=light
+
+# Amber CRT retro look
+export CANON_DIFF_THEME=retro
+
+# Claude Code style (red/green backgrounds)
+export CANON_DIFF_THEME=claude
+----
+
+Valid values: `light`, `dark`, `retro`, `claude`
+
+See link:../diff-formatting/themes.adoc[Diff display themes] for complete theme documentation.
+
 === Context and grouping
 
 [source,bash]

data/docs/internals/diff-char-range-pipeline.adoc

@@ -0,0 +1,249 @@
+---
+title: DiffCharRange Pipeline
+parent: Internals
+nav_order: 2
+---
+= DiffCharRange Pipeline
+
+== Purpose
+
+This document explains the two-phase pipeline that enriches semantic DiffNodes with character-level positions and assembles them into display-ready DiffLines. This pipeline replaced the previous DiffNodeMapper approach that used Diff::LCS on text lines, which caused structural mismatches for XML.
+
+== The Problem
+
+The previous approach (`DiffNodeMapper`) ran `Diff::LCS.sdiff(lines1, lines2)` on text lines. This was fundamentally wrong because:
+
+1. LCS operates on text lines, not XML structure
+2. It caused spurious deletes/inserts for closing tags that merely moved to a different line
+3. Character-level highlighting was independently re-tokenized in the formatter
+4. A string-concatenation hack (`merge_adjacent_removals!`) was needed to paper over structural mismatches
+
+== The Solution
+
+The new pipeline follows a strict two-phase separation:
+
+* **Phase 1 (Enrichment)**: DiffNodes are enriched with character positions and line counts. LCS on serialized content is acceptable here -- it operates on the change itself (short strings like "Hello World" vs "Hello Universe"), NOT on document lines.
+* **Phase 2 (Rendering)**: Formatter renders pre-computed DiffCharRanges. NO LCS, NO tokenization, NO computation. Just read char ranges and apply colors.
+
+== Architecture
+
+[source]
+----
+PHASE 1: DIFFNODE GENERATION + ENRICHMENT
+==========================================
+
+  Comparator (DOM/semantic) produces DiffNode[]
+    |
+    | serialized_before, serialized_after, dimension, etc.
+    |
+    v
+  DiffNodeEnricher (NEW)
+    |
+    | For each DiffNode:
+    |   1. SourceLocator: find serialized content in source text
+    |      -> char_offset, line_number, col in text1 and text2
+    |   2. TextDecomposer: decompose serialized_before vs serialized_after
+    |      -> common_prefix, changed_old, changed_new, common_suffix
+    |   3. Create DiffCharRange objects for each part
+    |      -> attached to DiffNode
+    |   4. Compute line_range_before, line_range_after
+    |
+    v
+  DiffNode[] (enriched with char_ranges, line counts)
+
+
+PHASE 2: RENDERING (NO LCS, NO COMPUTATION)
+============================================
+
+  DiffNode[] -> DiffLineBuilder (maps DiffCharRanges to DiffLines)
+           -> DiffLines (each carrying DiffCharRange[])
+           -> DiffBlockBuilder -> DiffContextBuilder -> DiffReport  [UNCHANGED]
+           -> Formatter (reads DiffCharRanges, applies colors)
+----
+
+== New Classes
+
+=== DiffCharRange
+
+*Location*: `lib/canon/diff/diff_char_range.rb`
+
+Value object: a character range within a source line linked to a DiffNode.
+
+[source,ruby]
+----
+class DiffCharRange
+  attr_reader :line_number, :start_col, :end_col, :side, :status, :role, :diff_node
+
+  # side: :old (text1) or :new (text2)
+  # status: :unchanged / :removed / :added / :changed_old / :changed_new
+  # role: :before / :changed / :after
+end
+----
+
+Status meanings:
+
+[cols="2,4"]
+|===
+| Status | Meaning
+
+| `:unchanged` | Same in both documents (before-text / after-text portions)
+| `:changed_old` | Old text being replaced (changed-text, text1 side)
+| `:changed_new` | New replacement text (changed-text, text2 side)
+| `:removed` | Entire range only in text1 (whole-line deletion)
+| `:added` | Entire range only in text2 (whole-line addition)
+|===
+
+=== TextDecomposer
+
+*Location*: `lib/canon/diff/text_decomposer.rb`
+
+Pure function: decomposes two strings into common prefix / changed / common suffix.
+
+[source,ruby]
+----
+TextDecomposer.decompose("Hello World", "Hello Universe")
+# => {
+#   common_prefix: "Hello ",
+#   changed_old: "World",
+#   changed_new: "Universe",
+#   common_suffix: ""
+# }
+----
+
+Algorithm: character-by-character prefix scan + reverse suffix scan. O(n).
+
+=== SourceLocator
+
+*Location*: `lib/canon/diff/source_locator.rb`
+
+Locates serialized content within source text, returns char offset + line/col.
+
+[source,ruby]
+----
+SourceLocator.locate("Hello World", text1, line_map)
+# => { char_offset: 17, line_number: 1, col: 9 }
+----
+
+Uses `String#index` on the full text, then maps char offset to line/col via a pre-built line offset map.
+
+=== DiffNodeEnricher
+
+*Location*: `lib/canon/diff/diff_node_enricher.rb`
+
+Enriches DiffNodes with character position data. This is Phase 1 of the pipeline.
+
+**Input**: `DiffNode[]`, `text1`, `text2` (preprocessed strings)
+**Output**: Same `DiffNode[]` enriched with `char_ranges`, `line_range_before`, `line_range_after`
+
+**Algorithm**:
+----
+Step 1: Build line offset maps for text1 and text2
+Step 2: For each DiffNode:
+  a. SourceLocator.locate(serialized_before, text1) -> position in text1
+  b. SourceLocator.locate(serialized_after, text2) -> position in text2
+  c. TextDecomposer.decompose(serialized_before, serialized_after) -> 3 parts
+  d. Map each part to DiffCharRange objects at correct line/col positions
+  e. Store char_ranges on DiffNode
+  f. Compute line_range_before, line_range_after from char_ranges
+----
+
+=== DiffLineBuilder
+
+*Location*: `lib/canon/diff/diff_line_builder.rb`
+
+Assembles DiffLines from enriched DiffNodes. This is Phase 2 -- NO LCS, NO computation.
+
+**Input**: `DiffNode[]` (enriched), `text1`, `text2`
+**Output**: `DiffLine[]` (each carrying `DiffCharRange[]`)
+
+**Algorithm**:
+----
+Step 1: Walk DiffNodes in document order (sorted by line_range_before)
+Step 2: For each DiffNode, create DiffLines from its char_ranges:
+  - char_ranges with side: :old -> DiffLine.char_ranges
+  - char_ranges with side: :new -> DiffLine.new_char_ranges
+  - Determine DiffLine type from the DiffNode's dimension and line ranges
+Step 3: Fill in unchanged lines between DiffNodes
+  - Use cumulative offset from line_range differences
+  - Detect reflow (lines that moved but content unchanged)
+Step 4: Return DiffLine[] in document order
+----
+
+**Reflow detection**: Lines that exist in text1 but whose content appears within an adjacent changed line in text2 (e.g., closing tag moved to previous line) are detected by checking if the stripped content is a substring of the adjacent line. These become formatting-only DiffLines (`[`/`]` markers), NOT deletions.
+
+== Per-Dimension Strategies
+
+[cols="2,4"]
+|===
+| Dimension | Strategy
+
+| `:text_content` | Locate serialized_before/after, decompose into prefix/changed/suffix
+| `:attribute_values` | Find `key="old_value"` in old line, `key="new_value"` in new line
+| `:attribute_presence` | Find `key="value"` for added/removed attributes
+| `:attribute_order` | Highlight entire attribute section as formatting
+| `:comments` | Locate `<!--...-->`, decompose if changed
+| `:structural_whitespace` | Mark affected lines as formatting-only
+| `:element_structure` | Full-element deletion/insertion (all lines)
+|===
+
+== Modified Classes
+
+=== DiffNode
+
+Added enrichment attributes:
+
+* `attr_accessor :char_ranges` -- `Array<DiffCharRange>`
+* `attr_accessor :line_range_before` -- `[start_line, end_line]` in text1
+* `attr_accessor :line_range_after` -- `[start_line, end_line]` in text2
+
+=== DiffLine
+
+Added:
+
+* `attr_reader :char_ranges` (text1 side DiffCharRange[])
+* `attr_reader :new_char_ranges` (text2 side DiffCharRange[])
+* `attr_reader :new_content` (text2 line text, for :changed lines)
+* `add_char_range(cr)`, `add_new_char_range(cr)`, `has_char_ranges?`
+
+== The 3-Part Decomposition
+
+When a text node changes (e.g., "Hello World" -> "Hello Universe"), the change is decomposed into 3 logical parts:
+
+[source]
+----
+"Hello World"  ->  "Hello Universe"
+^^^^^^^^         ^^^^^^^^ ^^^^^^^^
+  :before          :before  :changed
+                            (the actual diff)
+                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
+                            ^^^^^^^^
+                             :after (empty in this case)
+
+Each part produces DiffCharRange objects:
+  :before  -> status: :unchanged   (white/default)
+  :changed -> status: :changed_old (red)  / :changed_new (green)
+  :after   -> status: :unchanged   (white/default)
+----
+
+Each part can map to **multiple DiffCharRanges** across **multiple source lines** when the content spans line boundaries.
+
+== Mixed Change Detection
+
+A "mixed" change (`*` marker) occurs when a line has **multiple separate changed regions**. The detection counts contiguous changed regions, not total ranges:
+
+* Simple replacement "John Doe" -> "Jane Doe": 3 ranges but 1 changed region -> `-`/`+`
+* True mixed "foo bar baz" -> "foo qux baz quux": 2+ separate changed regions -> `*`
+
+The `count_changed_regions` method in `XmlFormatter` walks the ranges array counting transitions into `:changed_old`/`:changed_new` status.
+
+== Deleted Files
+
+The following file was removed and replaced by this pipeline:
+
+* `lib/canon/diff/diff_node_mapper.rb` -- replaced by `DiffNodeEnricher` + `DiffLineBuilder`
+
+== See Also
+
+* link:diffnode-enrichment[DiffNode Enrichment] -- How metadata is attached to DiffNodes
+* link:../advanced/diff-pipeline[Diff Pipeline Architecture] -- The 6-layer pipeline overview
+* link:../features/diff-formatting/colors-and-symbols[Colors and Symbols] -- How char ranges are rendered with color

data/docs/internals/diffnode-enrichment.adoc

@@ -609,3 +609,4 @@ The old API still works for backwards compatibility, but enriched properties pro
 * link:../understanding/architecture.adoc[Architecture] - 4-layer architecture overview
 * link:../understanding/algorithms/[Algorithms] - DOM and Semantic algorithm details
 * link:../features/diff-formatting/[Diff Formatting] - Layer 4 rendering options
+* link:diff-char-range-pipeline[DiffCharRange Pipeline] - How enriched DiffNodes are processed into character-level display positions