canon 0.1.3 → 0.1.5

data/docs/CHARACTER_VISUALIZATION.adoc

@@ -0,0 +1,567 @@
+---
+layout: default
+title: Character Visualization
+nav_order: 34
+parent: Customizing Behavior
+---
+= Canon character visualization
+:toc:
+:toclevels: 3
+
+== Scope
+
+This document describes Canon's whitespace and special character visualization
+system, which makes invisible characters visible in diff output.
+
+For diff formatting options, see link:DIFF_FORMATTING[Diff formatting].
+
+== General
+
+When comparing documents, invisible characters like spaces, tabs, and
+zero-width characters can cause mysterious test failures. Canon's character
+visualization makes these characters visible in diff output, helping you
+quickly identify the exact difference.
+
+Visualization is **CJK-safe**, using Unicode symbols that don't conflict with
+Chinese, Japanese, or Korean text.
+
+== When visualization is applied
+
+Character visualization is applied **only to diff lines** (additions,
+deletions, and changes), not to context lines (unchanged lines). This ensures:
+
+* Context lines display content in original form
+* Only actual changes show visualization
+* Differences are easier to spot
+
+Within changed lines showing token-level diffs, unchanged tokens are displayed
+in the terminal's default color (not red/green) to distinguish them from
+actual changes.
+
+== Default character map
+
+Canon provides a comprehensive CJK-safe character mapping.
+
+=== Common whitespace
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|Regular space
+|U+0020
+|`░`
+|Light Shade (U+2591)
+
+|Tab
+|U+0009
+|`⇥`
+|Rightwards Arrow to Bar (U+21E5)
+
+|Non-breaking space
+|U+00A0
+|`␣`
+|Open Box (U+2423)
+|===
+
+=== Line endings
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|Line feed (LF)
+|U+000A
+|`↵`
+|Downwards Arrow with Corner Leftwards (U+21B5)
+
+|Carriage return (CR)
+|U+000D
+|`⏎`
+|Return Symbol (U+23CE)
+
+|Windows line ending (CRLF)
+|U+000D U+000A
+|`↵`
+|Downwards Arrow with Corner Leftwards (U+21B5)
+
+|Next line (NEL)
+|U+0085
+|`⏎`
+|Return Symbol (U+23CE)
+
+|Line separator
+|U+2028
+|`⤓`
+|Downwards Arrow to Bar (U+2913)
+
+|Paragraph separator
+|U+2029
+|`⤓`
+|Downwards Arrow to Bar (U+2913)
+|===
+
+=== Unicode spaces
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|En space
+|U+2002
+|`▭`
+|White Rectangle (U+25AD)
+
+|Em space
+|U+2003
+|`▬`
+|Black Rectangle (U+25AC)
+
+|Four-per-em space
+|U+2005
+|`⏓`
+|Metrical Short Over Long (U+23D3)
+
+|Six-per-em space
+|U+2006
+|`⏕`
+|Metrical Two Shorts Over Long (U+23D5)
+
+|Thin space
+|U+2009
+|`▯`
+|White Vertical Rectangle (U+25AF)
+
+|Hair space
+|U+200A
+|`▮`
+|Black Vertical Rectangle (U+25AE)
+
+|Figure space
+|U+2007
+|`□`
+|White Square (U+25A1)
+
+|Narrow no-break space
+|U+202F
+|`▫`
+|White Small Square (U+25AB)
+
+|Medium mathematical space
+|U+205F
+|`▭`
+|White Rectangle (U+25AD)
+
+|Ideographic space
+|U+3000
+|`⎵`
+|Bottom Square Bracket (U+23B5)
+
+|Ideographic half space
+|U+303F
+|`⏑`
+|Metrical Breve (U+23D1)
+|===
+
+=== Zero-width characters
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|Zero-width space
+|U+200B
+|`→`
+|Rightwards Arrow (U+2192)
+
+|Zero-width non-joiner
+|U+200C
+|`↛`
+|Rightwards Arrow with Stroke (U+219B)
+
+|Zero-width joiner
+|U+200D
+|`⇢`
+|Rightwards Dashed Arrow (U+21E2)
+
+|Zero-width no-break space (BOM)
+|U+FEFF
+|`⇨`
+|Rightwards White Arrow (U+21E8)
+|===
+
+=== Bidirectional/RTL markers
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|Left-to-right mark
+|U+200E
+|`⟹`
+|Long Rightwards Double Arrow (U+27F9)
+
+|Right-to-left mark
+|U+200F
+|`⟸`
+|Long Leftwards Double Arrow (U+27F8)
+
+|LTR embedding
+|U+202A
+|`⇒`
+|Rightwards Double Arrow (U+21D2)
+
+|RTL embedding
+|U+202B
+|`⇐`
+|Leftwards Double Arrow (U+21D0)
+
+|Pop directional formatting
+|U+202C
+|`↔`
+|Left Right Arrow (U+2194)
+
+|LTR override
+|U+202D
+|`⇉`
+|Rightwards Paired Arrows (U+21C9)
+
+|RTL override
+|U+202E
+|`⇇`
+|Leftwards Paired Arrows (U+21C7)
+|===
+
+=== Control characters
+
+[cols="1,1,1,2"]
+|===
+|Character |Unicode |Symbol |Description
+
+|Null
+|U+0000
+|`␀`
+|Symbol for Null (U+2400)
+
+|Soft hyphen
+|U+00AD
+|`­‐`
+|Hyphen (U+2010)
+
+|Backspace
+|U+0008
+|`␈`
+|Symbol for Backspace (U+2408)
+
+|Delete
+|U+007F
+|`␡`
+|Symbol for Delete (U+2421)
+|===
+
+== CJK safety
+
+The visualization characters are specifically chosen to avoid conflicts with
+CJK text:
+
+**Avoided characters**:
+
+* **No middle dots** (`·`) - commonly used as separators in CJK
+* **No bullets** (`∙`) - used in CJK lists
+* **No circles** (`◌◍◎`) - look similar to CJK characters like ○ ●
+* **No small dots** (`⋅`) - conflict with CJK punctuation
+
+**Used instead**:
+
+* Box characters (`□▭▬▯▮▫`) for various space types
+* Arrow symbols (`→↛⇢⇨⟹⟸⇒⇐`) for zero-width and directional characters
+* Control Pictures block symbols (`␀␈␡`) for control characters
+
+== Examples in use
+
+=== Space added
+
+.Regular space added
+[example]
+====
+[source]
+----
+  10|     -| <tag>Value</tag>           # No space
+    |  10+| <tag>░Value</tag>           # Space added (green light shade)
+----
+
+The `░` symbol clearly shows a regular space was added between `<tag>` and
+`Value`.
+====
+
+=== Tab vs spaces
+
+.Tab replaced with spaces
+[example]
+====
+[source]
+----
+  15|     -| <tag>⇥Value</tag>          # Tab (red arrow-to-bar)
+    |  15+| <tag>░░Value</tag>          # Two spaces (green light shades)
+----
+
+The difference between a tab (`⇥`) and two spaces (`░░`) is immediately
+visible.
+====
+
+=== Non-breaking space
+
+.Non-breaking space from web copy-paste
+[example]
+====
+Without visualization, these look identical:
+
+[source,xml]
+----
+<foreword id="fwd">
+<foreword id="fwd">
+----
+
+With visualization:
+
+[source]
+----
+   4|     -| <foreword░id="fwd">         # Regular space (U+0020)
+    |   4+| <foreword␣id="fwd">          # Non-breaking space (U+00A0)
+----
+
+The different symbols (`░` vs `␣`) clearly show that one uses a regular space
+while the other uses a non-breaking space, likely from copying from a web page.
+====
+
+=== Zero-width space
+
+.Zero-width space (completely invisible)
+[example]
+====
+Zero-width characters are invisible but affect comparison:
+
+[source,xml]
+----
+<item>Widget</item>
+<item>Widget</item>  <!-- Contains U+200B zero-width space after "Widget" -->
+----
+
+The diff shows:
+
+[source]
+----
+   5|     -| <item>Widget</item>
+    |   5+| <item>Widget→</item>         # Zero-width space visualized as →
+----
+
+The rightwards arrow (`→`) reveals the presence of a zero-width space.
+====
+
+=== Mixed invisible characters
+
+.Multiple whitespace types
+[example]
+====
+[source]
+----
+  30|     -| <p>Text▬more</p>           # Em space (red black rectangle)
+    |  30+| <p>Text░more</p>            # Regular space (green light shade)
+----
+
+Different space types shown with different symbols.
+====
+
+== Real-world scenarios
+
+=== Web copy-paste
+
+**Problem**: Text copied from web pages often contains non-breaking spaces
+(U+00A0) instead of regular spaces.
+
+.Detection example
+[example]
+====
+[source]
+----
+   4|     -| <p>Hello░world</p>          # U+0020 (regular space)
+    |   4+| <p>Hello␣world</p>           # U+00A0 (non-breaking space)
+----
+
+The `␣` symbol immediately identifies the non-breaking space.
+====
+
+=== Smart quotes
+
+**Problem**: Text editors may automatically convert straight quotes to curly
+quotes.
+
+.Detection example
+[example]
+====
+[source]
+----
+  10|     -| <title>John's Book</title>  # Straight apostrophe
+    |  10+| <title>John's Book</title>  # Curly apostrophe (U+2019)
+----
+
+Non-ASCII warning will alert you to the smart quote.
+====
+
+=== Template generation
+
+**Problem**: Generated output has invisible character differences.
+
+.Detection example
+[example]
+====
+[source]
+----
+  20|     -| <item>Value→</item>         # Zero-width space present
+    |  20+| <item>Value</item>           # No zero-width space
+----
+
+The `→` symbol reveals the zero-width space in generated content.
+====
+
+== Customizing character visualization
+
+You can customize the visualization map for specific needs.
+
+=== Custom map
+
+[source,ruby]
+----
+require 'canon/diff_formatter'
+
+# Create custom visualization map
+custom_map = Canon::DiffFormatter.merge_visualization_map({
+  ' '  => '·',      # Use middle dot for spaces (if not using CJK)
+  "\t" => '→',      # Use simple arrow for tabs
+  "\u200B" => '⚠'   # Warning symbol for zero-width space
+})
+
+# Use custom map with formatter
+formatter = Canon::DiffFormatter.new(
+  use_color: true,
+  visualization_map: custom_map
+)
+
+# The custom map merges with defaults, so unspecified
+# characters still use the default visualization
+----
+
+=== When to customize
+
+**Use custom visualization when**:
+
+* Working with non-CJK text exclusively
+* Prefer simpler symbols
+* Need specific character highlighting
+* Integrating with existing tools
+
+**Keep defaults when**:
+
+* Working with CJK text
+* Maximum compatibility needed
+* Standard behavior preferred
+
+== Configuration
+
+Character visualization is automatically enabled when `use_color: true` and
+applies across all Canon interfaces.
+
+=== Enabling/disabling
+
+Visualization is tied to color output:
+
+[source,ruby]
+----
+# Enable (visualization active)
+diff: { use_color: true }
+
+# Disable (no visualization)
+diff: { use_color: false }
+----
+
+=== Interface configuration
+
+.Ruby API
+[example]
+====
+[source,ruby]
+----
+# Visualization enabled by default
+Canon::Comparison.equivalent?(doc1, doc2,
+  verbose: true,
+  diff: { use_color: true }  # Visualization active
+)
+
+# Disable for plain text
+Canon::Comparison.equivalent?(doc1, doc2,
+  verbose: true,
+  diff: { use_color: false }  # No visualization
+)
+----
+====
+
+.CLI
+[example]
+====
+[source,bash]
+----
+# Enable (default)
+$ canon diff file1.xml file2.xml --verbose
+
+# Disable
+$ canon diff file1.xml file2.xml --no-color --verbose
+----
+====
+
+.RSpec
+[example]
+====
+[source,ruby]
+----
+Canon::RSpecMatchers.configure do |config|
+  # Enable for local development
+  config.xml.diff.use_color = !ENV['CI']
+end
+----
+====
+
+== Troubleshooting
+
+=== Visualization not showing
+
+**Problem**: Invisible characters not visualized.
+
+**Solutions**:
+
+* Ensure `use_color: true`
+* Check terminal supports Unicode
+* Verify the characters are in diff lines (not context lines)
+
+=== Wrong symbols displayed
+
+**Problem**: Symbols appear garbled or as boxes.
+
+**Solutions**:
+
+* Use terminal with Unicode support
+* Install Unicode-compatible font
+* Check terminal encoding (should be UTF-8)
+
+=== CJK text affected
+
+**Problem**: Visualization conflicts with CJK text.
+
+**Solution**: Canon's defaults are CJK-safe. If using custom map, avoid the
+characters listed in "CJK safety" section.
+
+== See also
+
+* link:DIFF_FORMATTING[Diff formatting]
+* link:MODES[Diff modes]
+* link:MATCH_ARCHITECTURE[Match architecture]
+* link:RUBY_API[Ruby API documentation]