canon 0.1.7 → 0.1.8

data/docs/internals/index.adoc

@@ -0,0 +1,251 @@
+---
+title: Internals
+parent: Advanced
+nav_order: 7
+has_children: true
+---
+= Internals
+
+== Purpose
+
+This section contains detailed implementation documentation for Canon's internal systems. These documents explain how Canon works under the hood, covering data structures, algorithms, and architectural patterns.
+
+== Audience
+
+These documents are intended for:
+
+* Canon contributors and maintainers
+* Developers extending Canon with custom functionality
+* Anyone debugging complex comparison issues
+* Users wanting to understand implementation details
+
+== Topics
+
+link:diffnode-enrichment[**DiffNode Enrichment**]::
+How DiffNode objects carry location, serialized content, and attribute metadata through the comparison pipeline. Covers PathBuilder (canonical paths with ordinal indices), NodeSerializer (library-agnostic serialization), and how Layer 2 algorithms populate metadata for Layer 4 rendering.
+
+link:../advanced/dom-diff-internals[**DOM Diff Internals**]::
+Deep dive into Canon's default DOM diff algorithm: position-based matching, operation detection, and diff generation.
+
+link:../advanced/semantic-tree-diff-internals[**Semantic Tree Diff Internals**]::
+How the experimental semantic tree diff works: signature calculation, similarity matching, and operation classification.
+
+link:../advanced/verbose-mode-architecture[**Verbose Mode Architecture**]::
+The two-tier diff output system: normative vs informative diffs, and how verbose mode enriches output.
+
+link:../advanced/diff-classification[**Diff Classification System**]::
+How Canon classifies differences as normative (structural) or informative (presentational).
+
+link:../advanced/diff-pipeline[**Diff Pipeline Architecture**]::
+The six-layer technical pipeline from input to formatted output.
+
+link:../advanced/extending-canon[**Extending Canon**]::
+How to create custom comparators, formatters, and match strategies.
+
+== Core Concepts
+
+=== Library Agnosticism
+
+Canon is designed to work with multiple parsing libraries (Nokogiri, Moxml, Canon::Xml::Node) without being tied to any specific implementation. This is achieved through:
+
+* **Adapter pattern**: Format-specific adapters normalize different node types
+* **Utility classes**: PathBuilder and NodeSerializer work with any library
+* **Interface-based design**: Code depends on behavior (respond_to?) not concrete types
+
+This allows Canon to:
+* Support new parsing libraries without major refactoring
+* Switch libraries for better performance or features
+* Remain compatible as libraries evolve
+
+=== The 4-Layer Architecture
+
+Canon separates comparison concerns into four layers:
+
+* **Layer 1**: Preprocessing - Normalize documents before comparison
+* **Layer 2**: Algorithm - Choose comparison strategy (DOM vs Semantic)
+* **Layer 3**: Match Options - Configure what to compare
+* **Layer 4**: Diff Formatting - Control output presentation
+
+Only Layer 2 differs between algorithms, and the enriched DiffNode structure ensures clean communication between layers.
+
+See link:../understanding/architecture.adoc[Architecture] for the complete overview.
+
+=== Enriched Metadata Flow
+
+[mermaid]
+----
+graph LR
+    A[Layer 2: Algorithm] -->|Creates DiffNode| B[PathBuilder]
+    A -->|Creates DiffNode| C[NodeSerializer]
+    B -->|Enriches| D[DiffNode.path]
+    C -->|Enriches| E[DiffNode.serialized_before/after]
+    C -->|Enriches| F[DiffNode.attributes_before/after]
+    D --> G[Layer 4: Rendering]
+    E --> G
+    F --> G
+    G --> H[Accurate diff output]
+
+    style A fill:#fff4e1
+    style D fill:#e1f5ff
+    style G fill:#e1ffe1
+----
+
+Key insight: Metadata is captured at diff creation time (Layer 2) and carried through to rendering (Layer 4), ensuring accurate display even if nodes are modified during comparison.
+
+== Data Structures
+
+=== DiffNode
+
+Represents a semantic difference between two nodes:
+
+[source,ruby]
+----
+class DiffNode
+  # Core properties
+  attr_reader :node1, :node2           # Raw node references
+  attr_accessor :dimension, :reason    # What changed and why
+  attr_accessor :normative, :formatting # Classification
+
+  # Enriched metadata for Layer 4 rendering
+  attr_accessor :path                  # Canonical path with ordinal indices
+  attr_accessor :serialized_before     # Serialized "before" content
+  attr_accessor :serialized_after      # Serialized "after" content
+  attr_accessor :attributes_before     # Normalized "before" attributes
+  attr_accessor :attributes_after      # Normalized "after" attributes
+end
+----
+
+See link:diffnode-enrichment[DiffNode Enrichment] for details.
+
+=== TreeNode (Semantic Diff)
+
+Canonical node representation from semantic diff:
+
+[source,ruby]
+----
+class TreeNode
+  attr_reader :label          # Element name (e.g., "div", "span")
+  attr_reader :parent         # Parent TreeNode
+  attr_reader :children       # Array of child TreeNodes
+  attr_reader :attributes     # Normalized attribute hash
+  attr_reader :source_node    # Original parsing library node
+  attr_reader :signature      # Calculated signature for matching
+end
+----
+
+See link:../advanced/semantic-tree-diff-internals[Semantic Tree Diff Internals] for details.
+
+=== ComparisonResult
+
+Result object from verbose comparison:
+
+[source,ruby]
+----
+class ComparisonResult
+  attr_reader :differences           # Array of DiffNode objects
+  attr_reader :preprocessed_strings  # Preprocessed document strings
+  attr_reader :original_strings      # Original document strings
+  attr_reader :format               # :xml, :html, :json, :yaml
+  attr_reader :match_options         # Resolved match options
+  attr_reader :algorithm            # :dom or :semantic
+end
+----
+
+== Utility Classes
+
+=== PathBuilder
+
+Generates canonical XPath-like paths with ordinal indices:
+
+[source,ruby]
+----
+# Build path for any node type
+path = Canon::Diff::PathBuilder.build(node)
+# => "/#document/div[0]/body[0]/p[1]/span[2]"
+
+# Build human-readable path
+human = Canon::Diff::PathBuilder.human_path(node)
+# => "#document → div[0] → body[0] → p[1] → span[2]"
+----
+
+**Features**:
+* Library-agnostic: works with TreeNodes, Canon::Xml::Node, Nokogiri nodes
+* Ordinal indices: uniquely identifies nodes among siblings
+* Traverses parent hierarchy: builds complete path from root to node
+
+See link:diffnode-enrichment#pathbuilder-canonical-paths-with-ordinal-indices[PathBuilder documentation] for details.
+
+=== NodeSerializer
+
+Serializes nodes and extracts attributes regardless of parsing library:
+
+[source,ruby]
+----
+# Serialize any node
+serialized = Canon::Diff::NodeSerializer.serialize(node)
+
+# Extract normalized attributes
+attrs = Canon::Diff::NodeSerializer.extract_attributes(node)
+# => {"lang" => "EN-GB", "xml:lang" => "EN-GB", "id" => "example"}
+----
+
+**Features**:
+* Library-agnostic: handles Canon::Xml::Node, Nokogiri, Moxml
+* Normalized output: consistent format regardless of source library
+* Attribute extraction: returns hash of name-value pairs
+
+See link:diffnode-enrichment#nodeserializer-library-agnostic-serialization[NodeSerializer documentation] for details.
+
+== Algorithm Integration
+
+=== DOM Algorithm
+
+Enriches DiffNodes during positional comparison in `lib/canon/comparison/xml_comparator.rb`:
+
+[source,ruby]
+----
+def add_difference(node1, node2, diff1, diff2, dimension, opts, differences)
+  metadata = enrich_diff_metadata(node1, node2)
+  diff_node = Canon::Diff::DiffNode.new(
+    node1: node1,
+    node2: node2,
+    dimension: dimension,
+    reason: build_difference_reason(node1, node2, diff1, diff2, dimension),
+    **metadata  # Enriched from raw Nokogiri/Canon nodes
+  )
+  differences << diff_node
+end
+----
+
+See link:../understanding/algorithms/dom-diff.adoc[DOM Diff Algorithm] for details.
+
+=== Semantic Algorithm
+
+Enriches DiffNodes during operation conversion in `lib/canon/tree_diff/operation_converter.rb`:
+
+[source,ruby]
+----
+def convert_insert(operation)
+  tree_node2 = operation[:node]
+  node2 = extract_source_node(tree_node2)
+  metadata = enrich_diff_metadata(nil, tree_node2)
+  diff_node = Canon::Diff::DiffNode.new(
+    node1: nil,
+    node2: node2,
+    dimension: :element_structure,
+    reason: build_insert_reason(operation),
+    **metadata  # Enriched from TreeNode
+  )
+  diff_node.normative = determine_normative(:element_structure)
+  diff_node
+end
+----
+
+See link:../understanding/algorithms/semantic-tree-diff.adoc[Semantic Tree Diff Algorithm] for details.
+
+== See Also
+
+* link:../understanding/architecture.adoc[Architecture] - 4-layer architecture overview
+* link:../understanding/algorithms/[Algorithms] - DOM and Semantic algorithm details
+* link:../features/diff-formatting/[Diff Formatting] - Layer 4 rendering options
+* link:../advanced/[Advanced Topics] - Deep technical documentation

data/docs/lychee.toml

@@ -12,9 +12,11 @@ include_verbatim = true
 # Recursively check all files
 recursive = true
 
-# File types to check
+# File types to check (regex patterns)
 include = [
-  "_site/**/*.html"
+  "_site/**/*.html",
+  ".*\\.adoc$",
+  ".*\\.md$"
 ]
 
 # Excluded paths
@@ -25,7 +27,9 @@ exclude = [
   "vendor",
   ".bundle",
   ".sass-cache",
-  ".jekyll-cache"
+  ".jekyll-cache",
+  "_site/.jekyll-cache",
+  "Gemfile.lock"
 ]
 
 # Link checking behavior
@@ -56,10 +60,13 @@ include_mail = false  # Don't check mailto: links
 max_concurrency = 10
 
 # Verbose output for debugging
-verbose = "info"
+verbose = "warn"
 
 # Require HTTPS where possible
 require_https = false  # Don't enforce
 
-# Index files
-index_files = ["index.html"]
+# Index files for directory URLs
+index_files = ["index.html"]
+
+# Ignore patterns file
+ignore_file = ".lycheeignore"

data/docs/plans/2025-01-17-html-parser-selection-fix.adoc

@@ -0,0 +1,250 @@
+= HTML Parser Selection Fix Design
+:doctype: article
+:date: 2025-01-17
+:status: Approved
+
+== Problem Statement
+
+When comparing HTML documents with `lang` and `xml:lang` attributes, users see false attribute differences:
+
+----
+⊖ Expected (File 1):
+  <span> with 1 attribute: xml:lang
+
+⊕ Actual (File 2):
+  <span> with 2 attributes: lang, xml:lang
+----
+
+Both HTML strings have identical attributes (`lang="EN-GB" xml:lang="EN-GB"`), but the comparison shows different attribute counts. This happens because:
+
+. *DOM path* uses `Nokogiri::XML.fragment` for all HTML, which treats `lang` and `xml:lang` as the same attribute (XML namespace behavior)
+. *Semantic path* uses `Nokogiri::HTML5.fragment` or `Nokogiri::HTML4.fragment`, which correctly treats them as distinct
+. *The `parse_html` method ignores the format parameter* and returns raw strings, causing inconsistent parsing
+
+== Root Cause
+
+In `lib/canon/comparison.rb`, the `parse_html` method at line 374:
+
+[source,ruby]
+----
+def parse_html(content, _format)  # format is IGNORED!
+  return content unless content.is_a?(String)
+  # ... returns raw string instead of parsing
+end
+----
+
+This causes HTML version information to be lost, and `HtmlComparator#parse_node` ends up using `XML.fragment` for all HTML content.
+
+== Solution
+
+=== Architecture
+
+Fix the 4-layer architecture to respect user's parser choice:
+
+----
+User specifies format: :html5
+         |
+         v
+Level 1: Preprocessing
+  parse_html(html, :html5) -> Nokogiri::HTML5.fragment ✓
+         |
+         v
+Level 2: Diff Algorithm (DiffNode creation)
+  Parsed nodes have accurate attributes ✓
+         |
+         v
+Level 3: Diff Report
+  Enriched metadata is correct ✓
+         |
+         v
+Level 4: Diff Rendering
+  Accurate attribute counts in output ✓
+----
+
+=== Component Changes
+
+==== 1. `parse_html` Method (`lib/canon/comparison.rb`)
+
+*Current behavior:* Ignores format parameter, returns raw string
+
+*New behavior:* Parse with correct Nokogiri parser based on format
+
+[source,ruby]
+----
+def parse_html(content, format)
+  return content unless content.is_a?(String)
+  return content if already_parsed?(content)
+
+  begin
+    case format
+    when :html5
+      Nokogiri::HTML5.fragment(content)
+    when :html4
+      Nokogiri::HTML4.fragment(content)
+    when :html
+      detect_and_parse_html(content)
+    else
+      content
+    end
+  rescue StandardError
+    content
+  end
+end
+
+private
+
+def already_parsed?(content)
+  content.is_a?(Nokogiri::HTML::Document) ||
+    content.is_a?(Nokogiri::HTML5::Document) ||
+    content.is_a?(Nokogiri::HTML::DocumentFragment) ||
+    content.is_a?(Nokogiri::HTML5::DocumentFragment) ||
+    content.is_a?(Nokogiri::XML::DocumentFragment)
+end
+
+def detect_and_parse_html(content)
+  version = detect_html_version(content)
+  version == :html5 ?
+    Nokogiri::HTML5.fragment(content) :
+    Nokogiri::HTML4.fragment(content)
+end
+
+def detect_html_version(content)
+  content.include?('<!DOCTYPE html>') ? :html5 : :html4
+end
+----
+
+==== 2. `dom_diff` Method (`lib/canon/comparison.rb`)
+
+*Current behavior:* Normalizes `html4`/`html5` to `:html` at line 320
+
+*New behavior:* Preserve format information
+
+Remove or modify line 320:
+
+[source,ruby]
+----
+# OLD: format1 = format2 = :html
+# NEW: Keep format1, format2 as html4 or html5
+----
+
+This ensures the format is passed through to `HtmlComparator` and used consistently.
+
+=== Error Handling
+
+. *Parse failures:* Fall back to raw string (maintains backward compatibility)
+. *Already-parsed documents:* Return as-is, don't re-parse
+. *Mixed input types:* Both documents parsed with consistent parser based on format parameter
+
+=== Testing Strategy
+
+==== Unit Tests (`spec/canon/comparison_spec.rb`)
+
+[source,ruby]
+----
+context "parse_html with format parameter" do
+  it "parses HTML5 with HTML5.fragment when format is :html5" do
+    html = '<span lang="en" xml:lang="en">text</span>'
+    result = Canon::Comparison.send(:parse_html, html, :html5)
+
+    expect(result).to be_a(Nokogiri::HTML5::DocumentFragment)
+    expect(result.at_css('span').attributes.keys).to eq(['lang', 'xml:lang'])
+  end
+
+  it "parses HTML4 with HTML4.fragment when format is :html4" do
+    html = '<span lang="en" xml:lang="en">text</span>'
+    result = Canon::Comparison.send(:parse_html, html, :html4)
+
+    expect(result).to be_a(Nokogiri::HTML4::DocumentFragment)
+  end
+
+  it "returns already-parsed documents as-is" do
+    frag = Nokogiri::HTML5.fragment('<span>text</span>')
+    result = Canon::Comparison.send(:parse_html, frag, :html5)
+
+    expect(result).to eq(frag)
+  end
+end
+----
+
+==== Integration Tests (`spec/canon/html_comparison_spec.rb`)
+
+[source,ruby]
+----
+context "HTML5 lang and xml:lang attributes" do
+  it "treats lang and xml:lang as distinct attributes in HTML5" do
+    html1 = '<span lang="EN-GB" xml:lang="EN-GB">text</span>'
+    html2 = '<span lang="EN-GB" xml:lang="EN-GB">text</span>'
+
+    result = Canon::Comparison.equivalent?(
+      html1, html2,
+      format: :html5,
+      verbose: true
+    )
+
+    expect(result).to be_equivalent
+  end
+
+  it "does NOT show false attribute differences" do
+    html1 = '<span lang="EN-GB" xml:lang="EN-GB">&#xA0;</span>'
+    html2 = '<span lang="EN-GB" xml:lang="EN-GB">␣</span>'
+
+    result = Canon::Comparison.equivalent?(
+      html1, html2,
+      format: :html5,
+      verbose: true
+    )
+
+    # Only difference should be the non-breaking space encoding
+    # No attribute differences should be reported
+    attr_diffs = result.differences.select { |d| d.dimension == :attribute_values }
+    expect(attr_diffs).to be_empty
+  end
+end
+----
+
+==== Backward Compatibility Tests
+
+[source,ruby]
+----
+context "backward compatibility" do
+  it "works when format is not specified (auto-detect)" do
+    html1 = '<span>text</span>'
+    html2 = '<span>text</span>'
+
+    expect(Canon::Comparison.equivalent?(html1, html2)).to be true
+  end
+
+  it "handles strings with :html format (legacy behavior)" do
+    html1 = '<span>text</span>'
+    html2 = '<span>text</span>'
+
+    expect(Canon::Comparison.equivalent?(html1, html2, format: :html)).to be true
+  end
+end
+----
+
+== Implementation Checklist
+
+* [ ] Modify `parse_html` in `lib/canon/comparison.rb`
+* [ ] Add helper methods: `already_parsed?`, `detect_and_parse_html`, `detect_html_version`
+* [ ] Update `dom_diff` to preserve format (line 320)
+* [ ] Add unit tests for `parse_html` method
+* [ ] Add integration tests for lang/xml:lang
+* [ ] Add backward compatibility tests
+* [ ] Run full test suite to ensure no regressions
+
+== Expected Outcomes
+
+After this fix:
+
+. *`lang` and `xml:lang` are treated as distinct attributes in HTML5/HTML4*
+. *No false attribute differences when both documents have identical attributes*
+. *User can explicitly control parser via `format: :html5` or `format: :html4`*
+. *Backward compatible with existing code (auto-detect still works)*
+. *Consistent parsing regardless of input format (string vs DocumentFragment)*
+
+== Notes
+
+. HTML entity normalization (`&#xa0;` vs `␣`) is intentionally NOT changed - these are semantically equivalent but different serializations, and the diff correctly shows this difference
+. XML comparison continues to use `XML.fragment` - this fix only affects HTML parsing
+. The semantic path already works correctly via `Canon::Html::DataModel.from_html`