@ansonlai/docx-redline-js 0.2.0 → 0.4.0

package/docs/plans/2026-09-01-performance-and-complexity-reduction.md

@@ -0,0 +1,210 @@
+# Performance and Complexity Reduction Plan
+
+**Status:** Proposed  
+**Date:** 2026-09-01  
+
+This plan outlines opportunities to reduce architectural complexity, eliminate redundant diffing engines, and resolve critical performance bottlenecks across `@ansonlai/docx-redline-js`.
+
+---
+
+## 1. Context & Motivation
+
+Following the completion of reliability improvements (Rounds 1–3), the library has achieved strong correctness invariants (38/38 test suites passing, Word COM differential parity, and round-trip invariant checks).
+
+However, feature additions over time (tables, lists, comments, formatting removal, and batch operation orchestration) have introduced structural debt:
+1. **Three competing diff-and-patch paradigms**: [Reconstruction Mode](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/engine/reconstruction-mode.js), [Surgical Mode](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/engine/surgical-mode.js), and [Pipeline Mode](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/pipeline.js) each maintain distinct run models and patching strategies.
+2. **Monolithic Coordination**: [`services/standalone-operation-runner.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/services/standalone-operation-runner.js) has expanded to 1,451 lines (63 KB), entangling batch scheduling, atomic rollback, DOM mutation, range resolution, and ad-hoc heuristics.
+3. **Parse $\leftrightarrow$ Serialize Thrashing in Batches**: In multi-operation document turns, full-document XML is repeatedly parsed and serialized for each operation ($O(N \times \text{document size})$), consuming seconds of CPU time.
+4. **Scattered Domain Logic**: List markers, list parsing, numbering fallback, and list targeting are fragmented across 6+ separate files.
+5. **Hot Path Inefficiencies**: Linear scans in target detection, universal element traversal (`getElementsByTagName('*')`) during revision ID seeding, and excessive heap allocations (`Array.from`) in tree-traversal loops.
+
+---
+
+## 2. Guiding Principles
+
+This plan strictly follows the project's core philosophy:
+> **"Readability and Order > Speed and Complex Interconnections"**
+> This project prioritizes maintainability and clarity over clever, hyper-optimized code. Even if a solution is slightly less performant but significantly easier to read, choose the readable one (within reason).
+
+- **SOLID & DRY**: Each module should have one clear responsibility. Avoid maintaining duplicate diff/patch representations.
+- **KISS**: Favor simple, direct DOM operations over multi-layered abstractions.
+- **Strict Backward Compatibility**: Public APIs exported from [`index.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/index.js) and the package `exports` map must remain intact. All structural invariants verified by [`validateRedlineOoxml`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/core/redline-validation.js) and the round-trip tests must continue to pass.
+
+---
+
+## 3. Baseline Metrics
+
+As of 2026-09-01:
+- `npm test`: 38/38 test files pass (~6.0s execution time due to sequential process spawning).
+- Core package exports remain centralized in [`index.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/index.js).
+- For an $N$-operation batch on `word/document.xml`, the runner currently performs up to $3N$ full-document XML parses and $N$ full-document XML serializations using `@xmldom/xmldom`.
+- Four divergent text-extraction functions exist across targeting, engine, and ingestion modules.
+
+---
+
+## 4. Production API Compatibility
+
+| Phase | Expected Impact |
+|---|---|
+| **Phase 1: In-Memory Batch DOM & Hot Path Optimization** | **Non-breaking.** Pure internal performance optimization for [`applyOperationsToDocumentXml`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/services/standalone-operation-runner.js#L1356), target searches, and revision ID allocation. Output OOXML remains identical. |
+| **Phase 2: Decomposing `standalone-operation-runner.js`** | **Non-breaking.** Public exports preserved through [`index.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/index.js) and package exports; internal subroutines refactored into focused single-responsibility modules. |
+| **Phase 3: Diff Engine Consolidation & Pipeline Retirement** | **Non-breaking.** Unifies paragraph and list generation onto Reconstruction mode; deprecates/retires redundant AST patching in [`pipeline/pipeline.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/pipeline.js). |
+| **Phase 4: Consolidating Lists & Canonical Text Extraction** | **Non-breaking.** Centralizes list markers/heuristics into `services/list-service.js` and creates an authoritative canonical text extractor in [`core/word-xml.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/core/word-xml.js). |
+| **Phase 5: Developer Ergonomics & Fast Test Runner** | **Development-only.** In-process test execution cuts test suite run time from ~6.0s down to <1.0s. |
+
+---
+
+## Phase 1 — In-Memory Batch DOM & Hot Path Optimization
+
+### Problem
+1. In [`applyOperationsToDocumentXml`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/services/standalone-operation-runner.js#L1356-L1449), every scheduled operation causes a full serialize-and-reparse cycle:
+   - `parseOoxmlSafe(documentXml)` parses the document.
+   - `applyOperationToDocumentXml` parses it again.
+   - `applyToParagraphByExactText` parses it a third time.
+   - The scoped paragraph is serialized to a string, passed to `applyRedlineToOxml`, and parsed again.
+   - The result is serialized to a string, parsed again by `extractReplacementNodesFromOoxml`, imported back into `xmlDoc`, and the **entire document** is serialized back to string.
+   - In `@xmldom/xmldom` (pure JS DOM implementation), this creates massive CPU overhead for multi-operation turns.
+2. In [`oxml-engine.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/engine/oxml-engine.js#L183-L198), target detection uses an $O(\text{lines} \times \text{paragraphs} \times \text{spans})$ nested loop (`textSpans.filter(...)` inside `paragraphs.some(...)` for every line).
+3. In [`RevisionIdAllocator.prototype.seed`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/core/types.js#L203-L222), `getElementsByTagName('*')` copies every single element in the document into an array to find revision IDs, even though only 9 revision tags can carry `w:id`.
+
+### Implementation Steps
+1. **In-Memory Batch Execution**:
+   - Parse `documentXml` once at the beginning of `applyOperationsToDocumentXml`.
+   - Maintain a live `xmlDoc` reference across operations.
+   - Apply operations directly to `xmlDoc` without intermediate full-document string serialization.
+   - Serialize `xmlDoc` once at the end.
+   - Retain the initial `documentXml` string for atomic rollback; if any operation fails when `atomic: true`, simply return the untouched original string.
+2. **Pre-Group Spans in Target Check**:
+   - In `oxml-engine.js`, pre-aggregate text by paragraph into a `Map<Element, string>` in a single $O(\text{spans})$ pass (or reuse [`buildParagraphInfos`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/engine/format-paragraph-targeting.js#L27)), eliminating the quadratic scan.
+3. **Targeted Revision Tag Query**:
+   - In `RevisionIdAllocator.seed`, replace `getElementsByTagName('*')` with targeted queries for known revision element names (`w:ins`, `w:del`, `w:moveFrom`, `w:moveTo`, `w:rPrChange`, `w:pPrChange`, `w:cellIns`, `w:cellDel`, `w:comment`).
+4. **Common Prefix/Suffix Diff Short-Circuit**:
+   - In [`diff-engine.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/diff-engine.js#L206), add a fast character-level common prefix/suffix trim before DMP tokenization to short-circuit edits that only touch a few words in a large paragraph.
+
+### Acceptance
+- Multi-operation batch benchmarks demonstrate a **5x–15x speedup** on documents with 10+ operations.
+- All 38 existing test suites pass with zero regressions.
+
+---
+
+## Phase 2 — Modularizing `standalone-operation-runner.js`
+
+### Problem
+[`standalone-operation-runner.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/services/standalone-operation-runner.js) is 1,451 lines long. It acts as an orchestrator, batch scheduler, DOM updater, and heuristic router all at once, making navigation and maintenance difficult.
+
+### Implementation Steps
+1. Extract batch scheduling, priority sorting, dependency tracking, and atomic rollback into `services/batch-operation-orchestrator.js`:
+   - `applyOperationsToDocumentXml`
+   - `operationTargetPriority`
+   - Context cloning and commit helpers (`cloneBatchRuntimeContext`, `commitBatchRuntimeContext`)
+2. Extract single-operation execution and target paragraph scoping into `services/document-operation-applier.js`:
+   - `applyOperationToDocumentXml`
+   - `applyToParagraphByExactText`
+   - `applyHighlightToParagraphByExactText`
+   - `applyCommentToParagraphByExactText`
+3. Extract ad-hoc structural heuristics into `services/operation-heuristics.js`:
+   - Adjacency insertion heuristics (`deriveSingleParagraphListAdjacencyInsertion`, `deriveSingleParagraphPlainAdjacencyInsertion`, `buildInsertedListParagraph`, `buildInsertedPlainParagraph`)
+   - Insertion-only heuristics (`planListInsertionOnlyEdit`, `buildExplicitRangeInsertionEntries`, `applyExplicitRangeListInsertions`)
+   - Scope expansion (`synthesizeExpandedListScopeEdit`, `synthesizeTableMarkdownFromMultilineCellEdit`)
+4. Keep [`services/standalone-operation-runner.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/services/standalone-operation-runner.js) as a clean façade re-exporting the public contract.
+
+### Acceptance
+- No single file exceeds 500 lines.
+- Each module has a single, testable responsibility.
+- Existing tests in `tests/standalone_operation_runner_tests.mjs` pass without modification.
+
+---
+
+## Phase 3 — Diff Engine Consolidation & Pipeline Retirement
+
+### Problem
+The codebase currently supports three reconciliation execution engines:
+- **Reconstruction Mode**: Maps characters to runs, properties, and sentinels; rebuilds OOXML structure. (Primary, actively maintained, tab/field safe).
+- **Surgical Mode**: In-place mutation of runs in existing DOM. (Required for tables).
+- **Pipeline Mode** ([`pipeline/pipeline.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/pipeline.js)): Converts OOXML to an AST (`runModel`), splits runs at diff boundaries, and patches via string templates. It is now only invoked when `isTargetList` is true.
+
+Maintaining three diffing and patching engines increases maintenance burden and means bug fixes (such as `w:tab`, complex fields, or namespace handling) must be replicated across separate systems.
+
+### Implementation Steps
+1. Enhance Reconstruction Mode to handle list structural conversions natively:
+   - When marker-prefixed list text is detected, emit Word list properties (`w:numPr`) directly during reconstruction writing.
+2. Route `isTargetList` operations in [`oxml-engine.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/engine/oxml-engine.js#L371) through Reconstruction Mode instead of `ReconciliationPipeline`.
+3. Deprecate and remove redundant AST patching modules:
+   - [`pipeline/patching.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/patching.js)
+   - [`pipeline/serialization.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/serialization.js) (retaining only document fragment wrapping helpers)
+   - [`pipeline/pipeline.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/pipeline.js)
+4. Standardize on **Reconstruction Mode** for paragraphs/lists and **Surgical Mode** for tables.
+
+### Acceptance
+- ~3,000 lines of redundant code removed.
+- All list generation and list fallback tests (`tests/list_tests.mjs`, `tests/phase3_list_structural_fallback_tests.mjs`) pass with equal or better fidelity.
+- Output validity confirmed via `validateRedlineOoxml`.
+
+---
+
+## Phase 4 — Domain Consolidation: Lists & Canonical Text Extraction
+
+### Problem
+1. List logic is currently dispersed across 6+ separate files:
+   - [`pipeline/list-generation.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/list-generation.js)
+   - [`pipeline/list-markers.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/pipeline/list-markers.js)
+   - [`orchestration/list-parsing.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/orchestration/list-parsing.js)
+   - [`orchestration/list-markdown.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/orchestration/list-markdown.js)
+   - [`orchestration/list-structural-fallback.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/orchestration/list-structural-fallback.js)
+   - [`core/list-targeting.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/core/list-targeting.js)
+2. Four distinct visible-text extraction functions exist across targeting, engine, and ingestion modules, creating risk of subtle behavioral differences around `w:tab`, `w:br`, `w:noBreakHyphen`, and `w:del`.
+3. Frequent `Array.from(nodeList)` calls inside recursive DOM traversals generate heavy GC pressure.
+
+### Implementation Steps
+1. **Canonical Text Extractor**:
+   - Establish a single, authoritative `getParagraphVisibleText(pElement)` in [`core/word-xml.js`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/core/word-xml.js).
+   - Standardize handling for `w:t`, `w:tab` (`\t`), `w:br`/`w:cr` (`\n`), `w:noBreakHyphen` (`\u2011`), and exclusion of `w:delText` / deleted runs.
+   - Refactor `paragraph-targeting.js`, `oxml-engine.js`, and `ingestion-paragraph.js` to use this canonical helper.
+2. **Consolidate List Domain Modules**:
+   - Combine list marker matching and markdown list parsing into `services/list-service.js`.
+   - Simplify fallback orchestration so single-line structural fallback shares the same list definitions as multi-line lists.
+3. **Reduce DOM Traversal Allocations**:
+   - Replace `Array.from(node.childNodes)` in hot traversal loops with pointer iteration (`for (let child = node.firstChild; child; child = child.nextSibling)`).
+   - Use `node.getAttributeNS(NS_W, 'author')` directly instead of iterating through `Array.from(node.attributes)`.
+
+### Acceptance
+- Single source of truth for paragraph text extraction across all modules.
+- List parsing and numbering fallback unified under a cohesive domain service.
+- Reduced memory churn during large document traversals.
+
+---
+
+## Phase 5 — Developer Ergonomics & Fast Test Runner
+
+### Problem
+[`scripts/run-tests.mjs`](file:///c:/Users/Phara/Desktop/Projects/Docx%20Redline%20JS/scripts/run-tests.mjs#L21) currently runs 38 test files by invoking `execSync('node "${filePath}"')` sequentially. On Windows, spawning 38 separate Node.js processes accounts for ~5 seconds of the 6-second test run.
+
+### Implementation Steps
+1. Update `scripts/run-tests.mjs` to dynamically import and execute test files in-process (`await import(filePath)`), or leverage Node's native test runner (`node --test tests/*.mjs`).
+2. Retain process isolation checks in `npm run test:isolation`.
+
+### Acceptance
+- `npm test` execution time reduced from ~6.0s down to <1.0s.
+- Contributor iteration speed significantly improved.
+
+---
+
+## 5. Execution Roadmap & Prioritization
+
+```mermaid
+graph TD
+    P1[Phase 1: In-Memory Batch DOM & Hot Paths] --> P2[Phase 2: Modularize standalone-operation-runner]
+    P1 --> P5[Phase 5: Fast Test Runner]
+    P2 --> P3[Phase 3: Unify Diff Engines & Retire Pipeline]
+    P3 --> P4[Phase 4: List & Text Extraction Consolidation]
+```
+
+1. **Sprint 1 (Immediate Performance Wins)**:
+   - Phase 1 (In-memory DOM batch execution + target search optimization + revision allocator query).
+   - Phase 5 (In-process test runner).
+2. **Sprint 2 (Architecture Cleanliness & Maintainability)**:
+   - Phase 2 (Decompose `standalone-operation-runner.js`).
+   - Phase 4 (Canonical visible text extractor).
+3. **Sprint 3 (Structural Unification)**:
+   - Phase 3 (Retire legacy pipeline; unify on Reconstruction & Surgical modes).
+   - Complete list logic consolidation.

package/docs/plans/{2026-03-01-release-0.1.4-design.md → completed/2026-03-01-release-0.1.4-design.md}

@@ -1,5 +1,7 @@
 # 0.1.4 Release Prep Design
 
+**Status:** Completed
+
 **Objective:** Prepare the repository for a `0.1.4` patch release without publishing from this session.
 
 ## Scope

package/docs/plans/{2026-03-01-release-0.1.4.md → completed/2026-03-01-release-0.1.4.md}

@@ -1,5 +1,7 @@
 # 0.1.4 Release Prep Implementation Plan
 
+**Status:** Completed
+
 > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.
 
 **Goal:** Prepare the repository for a local/manual `0.1.4` release and verify it is ready to publish.
@@ -10,7 +12,7 @@
 
 ---
 
-### Task 1: Update Release Metadata
+### Task 1: Update Release Metadata (Completed)
 
 **Files:**
 - Create: `docs/plans/2026-03-01-release-0.1.4-design.md`
@@ -46,7 +48,7 @@ git add docs/plans/2026-03-01-release-0.1.4-design.md docs/plans/2026-03-01-rele
 git commit -m "release: prep v0.1.4"
 ```
 
-### Task 2: Run Release Preflight
+### Task 2: Run Release Preflight (Completed)
 
 **Files:**
 - Modify: `dist/docx-redline-js.esm.js`
@@ -81,7 +83,7 @@ git add dist/docx-redline-js.esm.js
 git commit -m "build: refresh dist for v0.1.4"
 ```
 
-### Task 3: Publish Handoff
+### Task 3: Publish Handoff (Completed)
 
 **Files:**
 - No file changes

package/docs/plans/{2026-05-31-architectural changes.md → completed/2026-05-31-architectural changes.md }

@@ -1,5 +1,7 @@
 # Redline Reliability Improvement Plan
 
+**Status:** Completed
+
 This plan hardens the library against the ways Microsoft Word redlines are brittle.
 It is written to be executed phase by phase, in order. Each phase is independently
 shippable and ends with a green `npm test`.