canon 0.1.23 → 0.2.1

data/docs/features/diff-formatting/character-visualization.adoc

@@ -1,528 +1,227 @@
 ---
+layout: default
 title: Character Visualization
 parent: Diff Formatting
 grand_parent: Features
-nav_order: 4
+nav_order: 3
 ---
-= Character visualization
+
 :toc:
 :toclevels: 3
 
-== Purpose
-
-Canon's character visualization system makes invisible characters (spaces, tabs, zero-width characters) visible in diff output, helping you quickly identify whitespace differences that cause test failures.
-
-Visualization is **CJK-safe**, using Unicode symbols that don't conflict with Chinese, Japanese, or Korean text.
-
-== When visualization is applied
-
-Character visualization is applied **only to diff lines** (additions, deletions, and changes), not to context lines (unchanged lines). This ensures:
-
-* Context lines display content in original form
-* Only actual changes show visualization
-* Differences are easier to spot
-
-Within changed lines showing token-level diffs, unchanged tokens are displayed in the terminal's default color (not red/green) to distinguish them from actual changes.
-
-== Default character map
-
-Canon provides a comprehensive CJK-safe character mapping.
-
-=== Common whitespace
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|Regular space
-|U+0020
-|`░`
-|Light Shade (U+2591)
-
-|Tab
-|U+0009
-|`⇥`
-|Rightwards Arrow to Bar (U+21E5)
-
-|Non-breaking space
-|U+00A0
-|`␣`
-|Open Box (U+2423)
-|===
-
-=== Line endings
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|Line feed (LF)
-|U+000A
-|`↵`
-|Downwards Arrow with Corner Leftwards (U+21B5)
-
-|Carriage return (CR)
-|U+000D
-|`⏎`
-|Return Symbol (U+23CE)
-
-|Windows line ending (CRLF)
-|U+000D U+000A
-|`↵`
-|Downwards Arrow with Corner Leftwards (U+21B5)
-
-|Next line (NEL)
-|U+0085
-|`⏎`
-|Return Symbol (U+23CE)
-
-|Line separator
-|U+2028
-|`⤓`
-|Downwards Arrow to Bar (U+2913)
-
-|Paragraph separator
-|U+2029
-|`⤓`
-|Downwards Arrow to Bar (U+2913)
-|===
-
-=== Unicode spaces
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|En space
-|U+2002
-|`▭`
-|White Rectangle (U+25AD)
-
-|Em space
-|U+2003
-|`▬`
-|Black Rectangle (U+25AC)
+== Overview
 
-|Four-per-em space
-|U+2005
-|`⏓`
-|Metrical Short Over Long (U+23D3)
+Canon replaces invisible characters (spaces, tabs, non-breaking spaces, etc.)
+with visible Unicode symbols before rendering diff output. This makes it easy
+to spot whitespace differences that would otherwise be indistinguishable from
+surrounding content.
 
-|Six-per-em space
-|U+2006
-|`⏕`
-|Metrical Two Shorts Over Long (U+23D5)
-
-|Thin space
-|U+2009
-|`▯`
-|White Vertical Rectangle (U+25AF)
-
-|Hair space
-|U+200A
-|`▮`
-|Black Vertical Rectangle (U+25AE)
-
-|Figure space
-|U+2007
-|`□`
-|White Square (U+25A1)
-
-|Narrow no-break space
-|U+202F
-|`▫`
-|White Small Square (U+25AB)
-
-|Medium mathematical space
-|U+205F
-|`▭`
-|White Rectangle (U+25AD)
-
-|Ideographic space
-|U+3000
-|`⎵`
-|Bottom Square Bracket (U+23B5)
-
-|Ideographic half space
-|U+303F
-|`⏑`
-|Metrical Breve (U+23D1)
-|===
-
-=== Zero-width characters
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|Zero-width space
-|U+200B
-|`→`
-|Rightwards Arrow (U+2192)
-
-|Zero-width non-joiner
-|U+200C
-|`↛`
-|Rightwards Arrow with Stroke (U+219B)
-
-|Zero-width joiner
-|U+200D
-|`⇢`
-|Rightwards Dashed Arrow (U+21E2)
-
-|Zero-width no-break space (BOM)
-|U+FEFF
-|`⇨`
-|Rightwards White Arrow (U+21E8)
-|===
-
-=== Bidirectional/RTL markers
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|Left-to-right mark
-|U+200E
-|`⟹`
-|Long Rightwards Double Arrow (U+27F9)
-
-|Right-to-left mark
-|U+200F
-|`⟸`
-|Long Leftwards Double Arrow (U+27F8)
-
-|LTR embedding
-|U+202A
-|`⇒`
-|Rightwards Double Arrow (U+21D2)
-
-|RTL embedding
-|U+202B
-|`⇐`
-|Leftwards Double Arrow (U+21D0)
-
-|Pop directional formatting
-|U+202C
-|`↔`
-|Left Right Arrow (U+2194)
-
-|LTR override
-|U+202D
-|`⇉`
-|Rightwards Paired Arrows (U+21C9)
-
-|RTL override
-|U+202E
-|`⇇`
-|Leftwards Paired Arrows (U+21C7)
-|===
-
-=== Control characters
-
-[cols="1,1,1,2"]
-|===
-|Character |Unicode |Symbol |Description
-
-|Null
-|U+0000
-|`␀`
-|Symbol for Null (U+2400)
-
-|Soft hyphen
-|U+00AD
-|`­‐`
-|Hyphen (U+2010)
-
-|Backspace
-|U+0008
-|`␈`
-|Symbol for Backspace (U+2408)
-
-|Delete
-|U+007F
-|`␡`
-|Symbol for Delete (U+2421)
-|===
-
-== CJK safety
-
-The visualization characters are specifically chosen to avoid conflicts with CJK text:
-
-**Avoided characters**:
-
-* **No middle dots** (`·`) - commonly used as separators in CJK
-* **No bullets** (`∙`) - used in CJK lists
-* **No circles** (`◌◍◎`) - look similar to CJK characters like ○ ●
-* **No small dots** (`⋅`) - conflict with CJK punctuation
-
-**Used instead**:
-
-* Box characters (`□▭▬▯▮▫`) for various space types
-* Arrow symbols (`→↛⇢⇨⟹⟸⇒⇐`) for zero-width and directional characters
-* Control Pictures block symbols (`␀␈␡`) for control characters
-
-== Examples in use
-
-=== Space added
-
-.Regular space added
-[example]
-====
-[source]
-----
-  10|     -| <tag>Value</tag>           # No space
-    |  10+| <tag>░Value</tag>           # Space added (green light shade)
-----
-
-The `░` symbol clearly shows a regular space was added between `<tag>` and `Value`.
-====
-
-=== Tab vs spaces
-
-.Tab replaced with spaces
+.Example output with character visualization enabled (default)
 [example]
 ====
-[source]
 ----
-  15|     -| <tag>⇥Value</tag>          # Tab (red arrow-to-bar)
-    |  15+| <tag>░░Value</tag>          # Two spaces (green light shades)
+    |   3- | ░░<item>Alpha</item>
+    |   3+ | ░░<item>Beta</item>
 ----
-
-The difference between a tab (`⇥`) and two spaces (`░░`) is immediately visible.
-====
-
-=== Non-breaking space
-
-.Non-breaking space from web copy-paste
-[example]
+The `░` characters represent U+0020 spaces used for indentation.
 ====
-Without visualization, these look identical:
-
-[source,xml]
-----
-<foreword id="fwd">
-<foreword id="fwd">
-----
 
-With visualization:
+== Configuration
 
-[source]
-----
-   4|     -| <foreword░id="fwd">         # Regular space (U+0020)
-    |   4+| <foreword␣id="fwd">          # Non-breaking space (U+00A0)
-----
+`character_visualization` is a `DiffConfig` option accessible via
+`Canon::Config`, environment variables, or the `DiffFormatter` constructor.
 
-The different symbols (`░` vs `␣`) clearly show that one uses a regular space while the other uses a non-breaking space, likely from copying from a web page.
-====
+=== Values
 
-=== Zero-width space
+[cols="1,4"]
+|===
+|Value |Behaviour
+
+|`true`
+|(default) The full default visualization map is applied.  Spaces appear as
+`░`, tabs as `⇥`, non-breaking spaces as `␣`, and so on.
+
+|`false`
+|All visualization is disabled.  Characters appear exactly as they are stored
+in the document.  Useful when copying failure-message output into a text
+editor, or when downstream tooling cannot handle Unicode symbols.
+
+|`:content_only`
+|**Reserved for future use** — currently behaves identically to `true`.
+Future intent: apply visualization only to text-node content, leaving
+structural indentation whitespace plain so that pretty-printed diffs remain
+visually uncluttered.  See the <<future-work>> section below.
+|===
 
-.Zero-width space (completely invisible)
-[example]
-====
-Zero-width characters are invisible but affect comparison:
+=== Via `Canon::Config` (RSpec `spec_helper.rb`)
 
-[source,xml]
-----
-<item>Widget</item>
-<item>Widget</item>  <!-- Contains U+200B zero-width space after "Widget" -->
+[source,ruby]
 ----
+Canon::Config.configure do |cfg|
+  # Disable for all XML diffs in this test suite
+  cfg.xml.diff.character_visualization = false
 
-The diff shows:
-
-[source]
-----
-   5|     -| <item>Widget</item>
-    |   5+| <item>Widget→</item>         # Zero-width space visualized as →
+  # Or disable for HTML as well
+  cfg.html.diff.character_visualization = false
+end
 ----
 
-The rightwards arrow (`→`) reveals the presence of a zero-width space.
-====
-
-== Real-world scenarios
+The setting is format-specific: `cfg.xml`, `cfg.html`, `cfg.json`,
+`cfg.yaml`, and `cfg.string` each have their own `diff.character_visualization`.
 
-=== Web copy-paste
+=== Via Environment Variable
 
-**Problem**: Text copied from web pages often contains non-breaking spaces (U+00A0) instead of regular spaces.
-
-.Detection example
-[example]
-====
-[source]
-----
-   4|     -| <p>Hello░world</p>          # U+0020 (regular space)
-    |   4+| <p>Hello␣world</p>           # U+00A0 (non-breaking space)
+[source,bash]
 ----
+# Disable globally
+export CANON_CHARACTER_VISUALIZATION=false
 
-The `␣` symbol immediately identifies the non-breaking space.
-====
-
-=== Smart quotes
-
-**Problem**: Text editors may automatically convert straight quotes to curly quotes.
+# Disable for XML only
+export CANON_XML_DIFF_CHARACTER_VISUALIZATION=false
 
-.Detection example
-[example]
-====
-[source]
-----
-  10|     -| <title>John's Book</title>  # Straight apostrophe
-    |  10+| <title>John's Book</title>  # Curly apostrophe (U+2019)
+# Set to :content_only for XML only
+export CANON_XML_DIFF_CHARACTER_VISUALIZATION=content_only
 ----
 
-Non-ASCII warning will alert you to the smart quote.
-====
-
-=== Template generation
-
-**Problem**: Generated output has invisible character differences.
+=== Via `DiffFormatter` directly
 
-.Detection example
-[example]
-====
-[source]
+[source,ruby]
 ----
-  20|     -| <item>Value→</item>         # Zero-width space present
-    |  20+| <item>Value</item>           # No zero-width space
+formatter = Canon::DiffFormatter.new(
+  use_color: false,
+  mode: :by_line,
+  character_visualization: false,
+)
 ----
 
-The `→` symbol reveals the zero-width space in generated content.
-====
-
-== Customizing character visualization
+== Default Visualization Map
 
-You can customize the visualization map for specific needs.
+The following characters are visualized by default:
 
-=== Custom map
+[cols="2,1,1,3"]
+|===
+|Name |Unicode |Visualization |Notes
 
-[source,ruby]
-----
-require 'canon/diff_formatter'
+|Space
+|U+0020
+|`░`
+|Most common; appears as indentation and between tokens
 
-# Create custom visualization map
-custom_map = Canon::DiffFormatter.merge_visualization_map({
-  ' '  => '·',      # Use middle dot for spaces (if not using CJK)
-  "\t" => '→',      # Use simple arrow for tabs
-  "\u200B" => '⚠'   # Warning symbol for zero-width space
-})
+|Tab
+|U+0009
+|`⇥`
+|Structural indentation in tab-indented files
 
-# Use custom map with formatter
-formatter = Canon::DiffFormatter.new(
-  use_color: true,
-  visualization_map: custom_map
-)
-----
+|No-Break Space
+|U+00A0
+|`␣`
+|Common in HTML (`&nbsp;`) and word-processor output
 
-=== When to customize
+|Thin Space
+|U+2009
+|`·`
+|Typographic use
 
-**Use custom visualization when**:
+|En Space
+|U+2002
+|`◦`
+|
 
-* Working with non-CJK text exclusively
-* Prefer simpler symbols
-* Need specific character highlighting
-* Integrating with existing tools
+|Em Space
+|U+2003
+|`▬`
+|
 
-**Keep defaults when**:
+|Line Feed
+|U+000A
+|`↵`
+|End-of-line (shown in certain modes)
 
-* Working with CJK text
-* Maximum compatibility needed
-* Standard behavior preferred
+|Carriage Return
+|U+000D
+|`←`
+|Windows line endings
 
-== Configuration
+|Zero-Width Space
+|U+200B
+|`⁰`
+|Invisible in output; often a bug
+|===
 
-Character visualization is automatically enabled when `use_color: true` and applies across all Canon interfaces.
+The complete map is loaded from
+`lib/canon/diff_formatter/character_map.yml` at startup.
 
-=== Enabling/disabling
+== Interaction with `display_preprocessing`
 
-Visualization is tied to color output:
+Character visualization is applied **after** display preprocessing.
 
-[source,ruby]
-----
-# Enable (visualization active)
-diff: { use_color: true }
+When `display_preprocessing: :pretty_print` is active, Canon runs both
+documents through `Canon::PrettyPrinter::Xml` before the line diff.  The
+pretty-printer produces only:
 
-# Disable (no visualization)
-diff: { use_color: false }
-----
+* ASCII U+0020 spaces for indentation
+* ASCII U+000A newlines between elements
 
-=== Interface configuration
+Because the default visualization map visualizes U+0020 spaces as `░`,
+structural indentation introduced by the pretty-printer *will* appear as `░`
+characters in context lines.  This is intentional: the diff output stays
+consistent regardless of how indentation was introduced.
 
-.Ruby API
+.Example (display_preprocessing: :pretty_print, character_visualization: true)
 [example]
 ====
-[source,ruby]
 ----
-# Visualization enabled by default
-Canon::Comparison.equivalent?(doc1, doc2,
-  verbose: true,
-  diff: { use_color: true }  # Visualization active
-)
-
-# Disable for plain text
-Canon::Comparison.equivalent?(doc1, doc2,
-  verbose: true,
-  diff: { use_color: false }  # No visualization
-)
+    |   1  | <root>
+    |   2- | ░░<item>Alpha</item>
+    |   2+ | ░░<item>Beta</item>
+    |   3  | </root>
 ----
+The `░░` represents the 2-space indentation added by the pretty-printer.
 ====
 
-.CLI
-[example]
-====
-[source,bash]
-----
-# Enable (default)
-canon diff file1.xml file2.xml --verbose
+If you want structure-free visualization (e.g. indentation stays as plain
+spaces but content whitespace is still visualized), set
+`character_visualization: :content_only` once that feature is implemented.
+See <<future-work>>.
 
-# Disable
-canon diff file1.xml file2.xml --no-color --verbose
-----
-====
+== Combining with `context_lines`
+
+Context lines (unchanged lines shown around a diff hunk) are also subject to
+character visualization.  Reducing `context_lines` limits the number of
+visualized lines displayed:
 
-.RSpec
-[example]
-====
 [source,ruby]
 ----
-Canon::RSpecMatchers.configure do |config|
-  # Enable for local development
-  config.xml.diff.use_color = !ENV['CI']
+Canon::Config.configure do |cfg|
+  cfg.xml.diff.context_lines = 1       # show only 1 context line
+  cfg.xml.diff.character_visualization = true
 end
 ----
-====
-
-== Troubleshooting
-
-=== Visualization not showing
-
-**Problem**: Invisible characters not visualized.
-
-**Solutions**:
-
-* Ensure `use_color: true`
-* Check terminal supports Unicode
-* Verify the characters are in diff lines (not context lines)
-
-=== Wrong symbols displayed
 
-**Problem**: Symbols appear garbled or as boxes.
+== [[future-work]]Future Work: `:content_only`
 
-**Solutions**:
+The `:content_only` value is reserved for a planned DOM-level pre-serialization
+pass that would:
 
-* Use terminal with Unicode support
-* Install Unicode-compatible font
-* Check terminal encoding (should be UTF-8)
+1. Walk the parsed DOM tree before serialization.
+2. Apply character visualization *only* to text node content.
+3. Leave structural whitespace (indentation, element-separator newlines)
+   plain.
 
-=== CJK text affected
+This would allow pretty-printed output to have clean indentation while
+still making content whitespace visible.
 
-**Problem**: Visualization conflicts with CJK text.
+The constraint that blocks this today: visualization is currently applied
+as a post-serialization string substitution by the by-line formatters.
+Implementing `:content_only` requires moving the substitution step into a
+DOM walk before `Nokogiri::XML::Node#to_xml` is called.
 
-**Solution**: Canon's defaults are CJK-safe. If using custom map, avoid the characters listed in "CJK safety" section.
+When implemented, `:content_only` will become the recommended value for
+suites that use `display_preprocessing: :pretty_print`.
 
-== See also
+== See Also
 
-* link:index.adoc[Diff Formatting] - Overview of formatting options
-* link:colors-and-symbols.adoc[Colors and Symbols] - Color scheme details
-* link:../../interfaces/cli/index.adoc[CLI Interface] - Command-line usage
-* link:../../interfaces/ruby-api/index.adoc[Ruby API] - Programmatic usage
+* link:display-preprocessing.html[Display Preprocessing]
+* link:index.html[Diff Formatting overview]
+* link:../../reference/options-across-interfaces.html[Options Across Interfaces]
+* link:../../reference/environment-variables.html[Environment Variables]