@ansonlai/docx-redline-js 0.5.4 → 0.6.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. package/AGENTS.md +82 -697
  2. package/ARCHITECTURE.md +13 -1
  3. package/CHANGELOG.md +8 -0
  4. package/README.md +177 -45
  5. package/core/paragraph-targeting.js +14 -2
  6. package/dist/docx-redline-js.esm.js +184 -51
  7. package/dist/docx-redline-js.esm.js.map +3 -3
  8. package/dist/docx-redline-js.esm.min.js +77 -77
  9. package/dist/docx-redline-js.esm.min.js.map +4 -4
  10. package/docs/AGENT_FAST_START.md +59 -0
  11. package/docs/AGENT_KNOWLEDGE_BASE.md +878 -0
  12. package/docs/SKILL_AUTHORING.md +126 -0
  13. package/docs/TESTING.md +35 -1
  14. package/docs/schemas/document-operations.schema.json +5 -1
  15. package/docs/validation-reports/2026-09-12-agent-cli-discovery-baseline.md +56 -0
  16. package/docs/validation-reports/2026-09-12-agent-protocol-rollout.md +86 -0
  17. package/docs/validation-reports/2026-09-13-agent-cli-efficiency-rollout.md +86 -0
  18. package/engine/oxml-engine.js +80 -13
  19. package/engine/run-builders.js +5 -15
  20. package/index.d.ts +28 -3
  21. package/node/cli-help.js +209 -0
  22. package/node/cli.js +323 -65
  23. package/node/docx-document.js +120 -69
  24. package/node/index.d.ts +6 -2
  25. package/package.json +15 -3
  26. package/scripts/generate-cross-author-slicing-fixtures.ps1 +25 -25
  27. package/services/batch-operation-orchestrator.js +215 -120
  28. package/services/document-inspection.js +89 -11
  29. package/services/document-operation-applier.js +52 -34
  30. package/services/document-operation-contract.js +10 -6
  31. package/services/document-operation-mutations.js +51 -5
  32. package/services/document-operation-session.js +4 -0
  33. package/services/error-recovery.js +174 -0
  34. package/services/operation-batch-compiler.js +394 -0
  35. package/services/operation-preflight.js +91 -72
  36. package/services/standalone-operation-runner.d.ts +17 -1
  37. package/docs/plans/2026-09-05-structural-revisions-and-fidelity-oracles.md +0 -1669
  38. package/docs/plans/2026-09-08-cross-author-revision-slicing.md +0 -1399
  39. package/docs/plans/completed/2026-03-01-release-0.1.4-design.md +0 -33
  40. package/docs/plans/completed/2026-03-01-release-0.1.4.md +0 -110
  41. package/docs/plans/completed/2026-05-31-architectural changes.md +0 -593
  42. package/docs/plans/completed/2026-08-02-reliability-improvements.md +0 -1155
  43. package/docs/plans/completed/2026-08-30-reliability-testing-improvements.md +0 -488
  44. package/docs/plans/completed/2026-09-01-performance-and-complexity-reduction.md +0 -669
  45. package/docs/plans/completed/2026-09-03-agent-friendly-document-workflows.md +0 -427
  46. package/docs/plans/completed/2026-09-04-comment-anchor-and-cli-reliability.md +0 -519
  47. package/docs/plans/completed/PERFORMANCE-CONSOLIDATION.md +0 -69
  48. package/docs/plans/completed/structural-revision-capability-matrix.md +0 -115
  49. package/docs/test-comparison-dashboard.html +0 -4338
  50. package/docs/validation-reports/2026-08-30-phase-1-word-visual-preflight.md +0 -22
  51. package/docs/validation-reports/2026-08-30-phase-2-word-visual-preflight.md +0 -24
  52. package/docs/validation-reports/2026-08-30-phase-3-coverage.md +0 -73
  53. package/docs/validation-reports/2026-09-02-multilevel-bullets-visual-review.md +0 -82
  54. package/docs/validation-reports/2026-09-02-multimodal-visual-samples.md +0 -114
  55. package/docs/validation-reports/2026-09-02-visual-failures-preflight.md +0 -79
@@ -1,669 +0,0 @@
1
- # Performance and Complexity Reduction Plan
2
-
3
- **Status:** Complete — Phases 1 through 6 finished (2026-09-04)
4
- **Original date:** 2026-09-01
5
- **Revised:** 2026-09-04
6
-
7
- This plan reduces batch latency and internal complexity without weakening the
8
- public operation contracts, target safety, package transactions, or exact-text
9
- workflow added by the 2026-09-03 agent-friendly document plan.
10
-
11
- The performance work must optimize the implementation behind those boundaries;
12
- it must not create a second runner, inspector, text model, or package mutation
13
- path.
14
-
15
- Accuracy is the governing constraint. A performance target is observational,
16
- not permission to weaken tracked-change fidelity, target safety, operation
17
- isolation, accepted/rejected text, or exact rollback. When speed and redline
18
- quality conflict, keep the accuracy safeguard and record the measured cost.
19
-
20
- ---
21
-
22
- ## 1. Current Baseline
23
-
24
- As of 2026-09-04:
25
-
26
- - Before Phase 2, `npm test` ran 55 isolated test files successfully. A local
27
- sequential run took roughly 29 seconds, with process startup a material part
28
- of the total. The Phase 1 and 2 regressions raise the suite to 57 files.
29
- - The pre-Phase-2 coverage baseline was 89.05% statements/lines, 77.04%
30
- branches, and 92.63% functions. The checked Phase 1 result is 89.32%
31
- statements/lines, 77.09% branches, and 92.78% functions.
32
- - Before Phase 2, `services/standalone-operation-runner.js` was 1,564 lines /
33
- 67,994 bytes. It is now a 13-line compatibility facade; the historical size
34
- remains the baseline for the decomposition work.
35
- - `applyOperationsToDocumentXml(...)` now parses the full document once and
36
- serializes it once after a successful changed batch. Scoped engine payloads
37
- still use their existing parse/serialize paths to preserve behavior.
38
- - `RevisionIdAllocator.seed(...)` performs one universal element query and
39
- copies the result with `Array.from(...)`.
40
- - `core/paragraph-text.js` is now the authoritative accepted/rejected/current
41
- paragraph-text API used by targeting, export ingestion, and structured
42
- inspection.
43
- - Some engine walkers remain intentionally specialized because they also build
44
- offset maps, formatting spans, structural sentinels, and reference maps.
45
- - The stable operation contract lives in
46
- `services/document-operation-contract.js`; strict read-only diagnostics live
47
- in `services/operation-preflight.js`.
48
- - Complete DOCX mutation is owned by the separate Node facade in `node/`; the
49
- CLI builds on that facade and requires strict, atomic, validated writes.
50
-
51
- These replace the older 38-test, four-extractor, and 1,451-line assumptions in
52
- the original plan.
53
-
54
- ## 2. Non-Regression Contracts
55
-
56
- Every phase must preserve the following behavior:
57
-
58
- 1. The root API, `./standalone-runner`, and `./node` package exports and their
59
- declarations remain stable.
60
- 2. Operation normalization, `INVALID_OPERATION`, per-operation author
61
- precedence, original operation indexes, `resolvedBy`, `resolvedTarget`, and
62
- `authorsUsed` remain unchanged.
63
- 3. Strict targeting never silently selects one of multiple exact matches.
64
- Fingerprints, paragraph IDs, occurrences, table context, and transient
65
- indexes retain their documented resolution semantics.
66
- 4. `preflightOperations(...)` remains read-only, deterministic, serializable,
67
- and consistent with the subsequent mutation path.
68
- 5. Comments still execute before text-changing operations when required for
69
- stable anchors. Atomic rollback must not commit allocator, numbering,
70
- comment, target-snapshot, or runtime-context state.
71
- 6. `inspectDocumentParts(...)`, targeting, ingestion export, and CLI `extract`
72
- continue to use canonical exact text. Performance work must not normalize
73
- whitespace or change accepted/rejected/current-view semantics.
74
- 7. Package operations keep comment IDs package-scoped, merge numbering by
75
- schema order, preserve unrelated part contents, validate before commit, and
76
- return original bytes on atomic failure or no-op.
77
- 8. The browser/root dependency graph remains free of Node filesystem/ZIP
78
- dependencies. Node-specific optimizations stay under `node/`.
79
- 9. Public pipeline exports such as `ReconciliationPipeline`,
80
- `serializeToOoxml`, and `wrapInDocumentFragment` cannot be removed in a
81
- minor release merely because the main engine stops using them internally.
82
-
83
- ## 3. Measurement Before Refactoring
84
-
85
- Add deterministic performance fixtures before changing architecture:
86
-
87
- - document sizes: approximately 100, 1,000, and 10,000 paragraphs;
88
- - operation counts: 1, 10, 50, and 100;
89
- - target shapes: unique text, duplicate text with stable descriptors, table
90
- cells, lists, comments, and mixed batches;
91
- - operation outcomes: success, no-op, strict ambiguity, and atomic rollback;
92
- - package cases: no artifacts, existing comments, existing numbering, and both.
93
-
94
- Record wall time, parse count, serialize count, peak heap, and output size. Run
95
- warm-up iterations and report median plus p95 rather than one timing. Store
96
- machine-readable results under ignored `tmp/benchmarks/`; keep fixture builders
97
- and thresholds in version control.
98
-
99
- Performance acceptance should use relative comparisons on the same machine.
100
- Initial targets are aspirational after all correctness gates pass:
101
-
102
- - at least 5x lower median latency for 10+ paragraph operations on a large
103
- document;
104
- - one full-document parse and one final full-document serialization for a
105
- successful batch, excluding validation/package reads;
106
- - no more than 10% regression for single-operation calls;
107
- - no material peak-heap regression after removing parse/serialize churn.
108
-
109
- The 2026-09-04 Phase 1 benchmark (`npm run benchmark:session`) used 1,000
110
- paragraphs, 10 replacements, two warmups, and seven measured iterations on
111
- Node 24/Windows x64. The live batch recorded a 167.19 ms median and 224.25 ms
112
- p95 versus 264.96 ms and 306.17 ms for sequential single-operation execution,
113
- or a 1.58x median speedup. Full-document parse/serialize counts fell from 10/10
114
- to 1/1 and output sizes matched. The 5x aspiration was not met because each
115
- operation retains a full DOM savepoint so a thrown error or false no-op cannot
116
- leak a partial mutation. That is an intentional accuracy tradeoff, not a reason
117
- to remove the safeguard. Heap deltas were too GC-sensitive in this short run to
118
- support a claim; retain them as observational output only.
119
-
120
- Do not require byte-identical XML serialization. Require semantic equivalence,
121
- accepted/rejected text parity, structural validation, and unchanged package
122
- parts; namespace and attribute ordering may legitimately differ.
123
-
124
- ---
125
-
126
- ## Phase 1 — In-Memory Document Operation Session
127
-
128
- **Status:** Complete (2026-09-04). Phase 2 created the boundary first; Phase 1
129
- then added live-DOM execution with accuracy-preserving savepoints.
130
-
131
- ### Problem
132
-
133
- Before Phase 1, the batch runner parsed the complete document at the batch
134
- boundary, reparsed inside every single-operation call, serialized after each
135
- mutation, and then parsed again to continue. The new agent APIs add more
136
- metadata and preflight requirements, so the live DOM must also preserve target
137
- identity and transaction state.
138
-
139
- ### Design
140
-
141
- The internal `DocumentOperationSession` owned by the batch orchestrator
142
- contains:
143
-
144
- - the live `xmlDoc` and serializer;
145
- - the original input string for rollback;
146
- - one document-scoped revision allocator;
147
- - the immutable initial target-reference snapshot;
148
- - lazily built paragraph/ID/text indexes with explicit invalidation after
149
- mutations;
150
- - numbering/list fallback context;
151
- - accumulated comment and numbering payloads;
152
- - operation results, authors, warnings, and execution order.
153
-
154
- This is an internal execution object, not a second public API. Existing
155
- `applyOperationToDocumentXml(...)` creates a one-operation session;
156
- `applyOperationsToDocumentXml(...)` creates one session for the entire batch.
157
- The Node facade continues to call the stable runner and then commits package
158
- artifacts transactionally.
159
-
160
- ### Work
161
-
162
- 1. Split DOM-native operation functions from string compatibility wrappers.
163
- 2. Resolve targets against the live DOM while retaining the initial snapshot
164
- used by transient references.
165
- 3. Apply paragraph/table replacements by importing scoped results into the
166
- live document without serializing the entire document between operations.
167
- Scoped paragraph serialization is acceptable until individual engines gain
168
- DOM-native entry points.
169
- 4. Invalidate only affected paragraph indexes after replacement/insertion;
170
- never reuse stale node references.
171
- 5. Serialize once after the batch succeeds. On atomic failure, discard the
172
- session and return the exact original string and no generated artifacts.
173
- 6. Commit external runtime context only after the batch and package transaction
174
- succeed.
175
-
176
- ### Acceptance
177
-
178
- - The benchmark targets above are met or the measured limitation is recorded.
179
- - Existing operation, preflight, inspection, CLI, comment, numbering, and
180
- rollback tests pass unchanged.
181
- - A new parse/serialize instrumentation test proves the full document is not
182
- reparsed per operation.
183
- - Mixed batches retain original indexes, comment-first execution, resolution
184
- metadata, and per-operation authors.
185
-
186
- ### Implementation record
187
-
188
- - A batch parses `word/document.xml` once, keeps the initial target-reference
189
- snapshot and one revision allocator, mutates one live DOM, and serializes
190
- once only after a successful changed batch.
191
- - The existing scoped redline, list, table, highlight, and comment engines are
192
- unchanged. They still serialize only the selected scope and import their
193
- result into the live document, preserving their established accuracy.
194
- - Every operation starts with a cloned-DOM and revision-allocator savepoint.
195
- Errors and reported no-ops restore that savepoint. This is deliberately more
196
- conservative than maximum-throughput mutation and prevents partial state or
197
- consumed revision IDs from leaking across operations.
198
- - Atomic failure and all-no-op batches return the exact original XML without a
199
- final serialization. Non-atomic batches serialize only successfully
200
- committed mutations. Runtime context is committed only after successful
201
- batch serialization; the Node facade continues to isolate package context in
202
- its package transaction.
203
- - The session owns lazy paragraph metadata and invalidates it after every
204
- committed mutation or restore, so future indexing work cannot reuse stale DOM
205
- nodes.
206
- - `performance_phase1_session_tests.mjs` verifies real provider parse/serialize
207
- counts, expected accepted and rejected text, structural redline validation,
208
- list/table/comment/highlight behavior, equivalence with sequential execution,
209
- byte-exact no-op output, and zero-serialization atomic rollback.
210
- - The checked benchmark reports the accuracy-first 1.58x result and the
211
- documented 10/10 to 1/1 parse/serialize reduction above.
212
- - All 57 isolated tests and coverage pass. Coverage is 89.32%
213
- statements/lines, 77.09% branches, and 92.78% functions. Isolation, types,
214
- lint, build, the 143-file package dry run, package self-imports, and
215
- `git diff --check` also pass.
216
- - Desktop Word, corpus, visual, XSD, and LibreOffice lanes were not rerun. The
217
- scoped reconciliation engines and emitted revision construction were not
218
- changed, while the new equivalence test validates exact accepted/rejected
219
- outcomes and the existing package/OOXML suite exercises the integrated path.
220
-
221
- ---
222
-
223
- ## Phase 2 — Modularize the Runner Around Existing Boundaries
224
-
225
- **Status:** Complete (2026-09-04), intentionally completed before the Phase 1
226
- in-memory optimization.
227
-
228
- ### Problem
229
-
230
- Before this phase, `standalone-operation-runner.js` mixed compatibility
231
- exports, scheduling, targeting, DOM mutation, list/table heuristics, and result
232
- assembly. The operation-contract and preflight modules already established
233
- boundaries that the decomposition reused.
234
-
235
- ### Work
236
-
237
- 1. Add `services/document-operation-session.js` for live DOM state, indexes,
238
- allocators, invalidation, serialization, and rollback.
239
- 2. Add `services/batch-operation-orchestrator.js` for scheduling, dependency
240
- order, context commit, atomic policy, and batch result assembly.
241
- 3. Add `services/document-operation-applier.js` for canonical single-operation
242
- dispatch and result metadata.
243
- 4. Add `services/operation-heuristics.js` for adjacency insertion, explicit
244
- range insertion, and list/table scope-expansion heuristics.
245
- 5. Keep `services/document-operation-contract.js` and
246
- `services/operation-preflight.js` authoritative; do not duplicate their
247
- normalization or target diagnostics in the new modules.
248
- 6. Turn `services/standalone-operation-runner.js` into a small compatibility
249
- facade re-exporting the same runtime functions. Preserve
250
- `services/standalone-operation-runner.d.ts` and the package subpath.
251
- 7. Replace the runner's import from `../index.js` with direct leaf-module
252
- imports. This removes the current circular dependency without changing the
253
- public root exports.
254
-
255
- ### Acceptance
256
-
257
- - The facade is preferably under 250 lines and new responsibility modules are
258
- preferably under 500 lines; exceptions require a short rationale.
259
- - No operation-normalization, preflight, or package-transaction fork exists.
260
- - Root, standalone-runner, and Node self-import type fixtures pass.
261
- - Coverage does not fall below the checked baseline for extracted behavior.
262
-
263
- ### Implementation record
264
-
265
- - `standalone-operation-runner.js` is a compatibility-only facade and preserves
266
- the same runtime exports, declaration file, and package subpath.
267
- - `document-operation-applier.js` owns validation, author resolution, dispatch,
268
- and uniform result metadata; `batch-operation-orchestrator.js` owns stable
269
- scheduling, atomic policy, artifact aggregation, and context commit.
270
- - `document-operation-session.js` owns parse state, exact rollback input,
271
- revision allocation, and isolated runtime-context clone/commit helpers. Its
272
- live-DOM optimization remains Phase 1 work; Phase 2 does not claim the batch
273
- parse/serialize performance target.
274
- - `operation-heuristics.js` owns the extracted adjacency and explicit-range
275
- decisions. Existing shared list/table scope decisions remain authoritative in
276
- `core/list-targeting.js` and `core/table-targeting.js` rather than being
277
- duplicated.
278
- - Coupled OOXML construction and replacement helpers remain together in
279
- `document-operation-mutations.js` (about 1,000 lines). This is the documented
280
- size exception: splitting those interdependent mutation paths before the
281
- live-session work would add interfaces and semantic churn without reducing
282
- reparsing. The public facade, applier, orchestrator, session, and heuristics
283
- modules are each below 250 lines.
284
- - The operation implementation no longer imports `../index.js`; it imports leaf
285
- modules directly. `performance_phase2_boundary_tests.mjs` locks the facade
286
- identities, leaf-import rule, exact rollback, isolated context, and stable
287
- scheduling.
288
- - All 56 isolated tests and coverage pass. The post-refactor coverage totals
289
- are 89.16% statements/lines, 77.18% branches, and 92.71% functions, meeting
290
- or exceeding every recorded baseline dimension.
291
- - `npm run test:isolation`, `npm run check:types`, `npm run lint`, `npm run
292
- build`, `npm pack --dry-run`, package self-import checks, and `git diff
293
- --check` pass. The dry-run package contains all new service modules while the
294
- stable standalone subpath still exposes exactly its four prior functions.
295
- - Desktop Word, corpus, visual, XSD, and LibreOffice lanes were not rerun for
296
- this boundary-only phase because reconciliation routing and emitted OOXML
297
- behavior did not change; the existing script-level and package tests cover
298
- the moved behavior.
299
-
300
- ---
301
-
302
- ## Phase 3 — Target and Traversal Hot Paths
303
-
304
- **Status:** Complete (2026-09-04). The session and preflight paths now share
305
- one document-scoped metadata representation, while all caches remain bounded
306
- to their owning operation or preflight call.
307
-
308
- ### Work
309
-
310
- 1. Build paragraph metadata once per operation session: canonical text,
311
- normalized text, paragraph ID, fingerprint inputs, table context, and
312
- document index. Share the immutable representation with preflight where
313
- possible.
314
- 2. Replace repeated `textSpans.filter(...)`/paragraph scans in target detection
315
- with a single grouped map, reusing existing paragraph-info builders where
316
- their semantics match.
317
- 3. Change revision-ID seeding to a single pointer-based tree traversal that
318
- recognizes revision-bearing elements without allocating an array of every
319
- element. Do not replace one full traversal with nine independent tag
320
- traversals unless benchmarks show that is faster in both browser DOM and
321
- `@xmldom/xmldom`.
322
- 4. Replace `Array.from(...)` only in measured hot recursive loops. Keep it where
323
- it improves clarity and does not dominate profiles.
324
- 5. Benchmark diff preprocessing before adding a manual common-prefix/suffix
325
- trim: `diff-match-patch` already performs that optimization internally.
326
- Add another fast path only if tokenization before DMP is proven to dominate.
327
- 6. Bound caches to one session. Do not retain DOM nodes or document-derived text
328
- in process-global maps.
329
-
330
- ### Acceptance
331
-
332
- - Target resolution remains byte-for-byte deterministic in its JSON metadata.
333
- - Canonical text and fingerprint compatibility tests pass.
334
- - Profiles demonstrate reduced traversal/allocation cost on large fixtures.
335
- - Browser and Node benchmarks show no meaningful single-operation regression.
336
-
337
- ### Implementation record
338
-
339
- - `buildParagraphMetadataIndex(...)` computes canonical text, normalized text,
340
- paragraph ID, fingerprint, table context, and the one-based document index in
341
- one pass. It exposes grouped paragraph-ID and normalized-text maps plus a
342
- node-keyed metadata map; entries are immutable and retain document order.
343
- - `DocumentOperationSession` constructs the index once and uses it to build the
344
- initial target-reference snapshot. It preserves the earlier paragraph-array
345
- accessor for internal compatibility, exposes the richer index separately,
346
- and invalidates both after any committed mutation or savepoint restore.
347
- - Batch preflight builds one metadata index and shares it with every strict
348
- target resolution. Mutation targeting and resolution metadata use the same
349
- cached entries, avoiding repeated canonical-text, ID, table-context, and
350
- fingerprint traversal without changing the resolver's public JSON shapes.
351
- - Strict duplicate detection uses the grouped normalized-text map. Exact-text
352
- filtering still occurs within the matched group, so whitespace normalization
353
- does not broaden strict selection and duplicate targets remain ambiguous.
354
- - `RevisionIdAllocator.seed(...)` now uses a pointer-based depth-first walk and
355
- examines revision-bearing elements without materializing every element into
356
- an array. A regression deliberately disables universal tag queries to prove
357
- the allocator uses the pointer traversal.
358
- - `performance_phase3_target_hot_paths_tests.mjs` covers cached/uncached
359
- resolution parity for exact text, duplicate occurrences, paragraph IDs,
360
- fingerprints, transient indexes, and table context; it also checks snapshot
361
- construction, session reuse/invalidation, and revision-ID allocation.
362
- - `npm run benchmark:targeting` is a deterministic 10,000-paragraph benchmark
363
- with two warmups and seven measured iterations. On Node 24/Windows x64, one
364
- operation improved from a 45.10 ms median to 36.53 ms (1.23x), satisfying
365
- the no-regression constraint. One hundred resolutions improved from
366
- 1,948.49 ms to 41.31 ms (47.16x). The uncached comparison includes the same
367
- initial target snapshot required before this phase, so the comparison does
368
- not omit session setup work.
369
- - `scripts/benchmark-targeting-browser.html` runs the same fixture, warmups,
370
- percentiles, snapshot work, and resolver comparison against a native browser
371
- DOM. Query parameters select paragraph, operation, and iteration counts so
372
- single-operation and batch profiles can be reproduced without a Node DOM.
373
- On Chrome 151/Windows x64, the 10,000-paragraph single-operation median moved
374
- from 43.10 ms to 42.30 ms (1.02x), satisfying the browser no-regression
375
- constraint. One hundred resolutions moved from 1,363.20 ms to 37.40 ms
376
- (36.45x). Both browser measurements used two warmups and seven measured runs.
377
- - No extra diff prefix/suffix fast path was introduced: the work did not show
378
- token preprocessing to be the dominant cost, and `diff-match-patch` already
379
- performs prefix/suffix trimming internally.
380
-
381
- ---
382
-
383
- ## Phase 4 — List-Domain Consolidation Without Replacing Canonical Text
384
-
385
- **Status:** Complete (2026-09-04). Canonical consumers now use the established
386
- paragraph-text API, specialized offset/sentinel walkers remain specialized,
387
- and all list-facing parsers share one dependency-light grammar.
388
-
389
- ### Current state
390
-
391
- Canonical extraction is already implemented in `core/paragraph-text.js`; this
392
- phase must not create `getParagraphVisibleText(...)` in `core/word-xml.js` or
393
- move the source of truth again.
394
-
395
- Engine mapping walkers may remain specialized when they produce more than
396
- visible text. The goal is shared semantics, not forcing offset/sentinel logic
397
- through a string-only helper.
398
-
399
- ### Work
400
-
401
- 1. Inventory every remaining paragraph-text walker and label it as either:
402
- - canonical visible-text consumption; or
403
- - specialized mapping with offsets, formatting, fields, references, or
404
- structural sentinels.
405
- 2. Replace only the first category with `extractCanonicalParagraphText(...)`.
406
- For specialized walkers, add parity tests for the visible projection while
407
- retaining their richer data models.
408
- 3. Extract shared list-token grammar and marker classification into a small
409
- dependency-light module used by markdown parsing, routing, and fallback.
410
- 4. Keep targeting, structural planning, OOXML writing, numbering allocation,
411
- and inspection numbering resolution in focused modules. Avoid replacing six
412
- files with one oversized `list-service.js`.
413
- 5. Make single-line and multi-line list paths consume the same parsed list-item
414
- representation and numbering-style vocabulary.
415
- 6. Preserve `inspectDocumentParts(...)` labels, exact excerpts, target
416
- descriptors, numbering merge behavior, and CLI JSON shapes.
417
-
418
- ### Acceptance
419
-
420
- - `core/paragraph-text.js` remains the only public canonical text definition.
421
- - Specialized walkers have explicit parity tests for tabs, breaks, moves,
422
- fields, notes, and revision views.
423
- - List syntax is defined once without merging unrelated package/targeting/writer
424
- responsibilities.
425
- - All list, numbering, inspection, package, and Word differential fixtures pass.
426
-
427
- ### Implementation record
428
-
429
- - `docs/PERFORMANCE-CONSOLIDATION.md` inventories the remaining text walkers as
430
- canonical consumers or specialized mappings and records the semantic boundary
431
- of each. `core/paragraph-text.js` remains the sole canonical visible-text
432
- definition.
433
- - `engine/table-cell-context.js`, a string-only consumer, now calls
434
- `extractCanonicalParagraphText(...)`. Mapping walkers in ingestion, surgical
435
- editing, formatting, reconstruction, and comment anchoring keep their richer
436
- offset, run, sentinel, or reference models.
437
- - The accepted visible projections in ingestion and surgical span mapping now
438
- include soft hyphens consistently with the canonical walker. Existing
439
- structural ownership and zero-width handling for fields and note references
440
- are retained.
441
- - `pipeline/list-markers.js` now owns list-marker matching, stripping,
442
- classification, numbering-style inference, outline-depth parsing, and the
443
- common parsed list-item representation. Pipeline analysis, orchestration
444
- parsing/normalization, and list targeting consume those helpers; numbering
445
- allocation, structural planning, OOXML writing, and package merging remain
446
- in their focused modules.
447
- - The shared vocabulary recognizes bullet markers (`-`, `*`, `+`, and common
448
- Unicode bullets), decimal/composite decimal markers, alphabetic markers, and
449
- Roman markers. It reports `bullet`/`numbered`, the established numbering
450
- style, indentation level, and optional outline level.
451
- - `performance_phase4_list_and_text_parity_tests.mjs` exercises marker parity
452
- across every consumer, including redundant marker stripping, and compares
453
- canonical, ingestion, and surgical visible projections for tabs, breaks,
454
- hyperlinks, soft/non-breaking hyphens, fields, notes, insertions, deletions,
455
- and moves.
456
- - The complete automated suite and the 47-fixture Microsoft Word differential
457
- pass. The Word set includes ten list cases plus tables, comments, hyperlinks,
458
- fields, footnotes/endnotes, formatting-only edits, and independent Accept All
459
- and Reject All expectations.
460
-
461
- ---
462
-
463
- ## Phase 5 — Diff-Engine Consolidation With a Compatibility Window
464
-
465
- **Status:** Complete (2026-09-04). The preferred focused list generator now
466
- serves the safe single-paragraph route, while multi-paragraph marked-list edits
467
- remain on the compatibility pipeline until equally strong structural evidence
468
- supports moving them.
469
-
470
- ### Problem
471
-
472
- Reconstruction, surgical, table, and legacy pipeline paths overlap, but they do
473
- not yet have identical structural capabilities. The legacy pipeline also has
474
- public exports, so deleting its files in a nominally non-breaking phase would
475
- conflict with the current API contract.
476
-
477
- ### Work
478
-
479
- 1. Create a capability matrix for paragraphs, lists, tables, hyperlinks,
480
- fields, comments, notes, formatting-only edits, and structural insertions.
481
- 2. Instrument route selection and establish corpus frequency before changing
482
- defaults.
483
- 3. Add missing list/numbering behavior to the preferred engine behind existing
484
- result shapes, including `numberingXml`, `sourceType`, warnings, and
485
- `useNativeApi` behavior.
486
- 4. Move internal list routing off `ReconciliationPipeline` only after focused,
487
- fuzz, package, Word accept/reject, and visual tests demonstrate parity.
488
- 5. Keep `ReconciliationPipeline`, `serializeToOoxml`, and
489
- `wrapInDocumentFragment` as compatibility exports for the remainder of the
490
- current major version. Mark genuinely obsolete APIs deprecated in docs and
491
- the changelog before removal.
492
- 6. Remove public legacy modules only in a planned major release. Internal dead
493
- code may be removed earlier when no export or deep-import contract is
494
- affected.
495
-
496
- ### Acceptance
497
-
498
- - Route changes are supported by benchmark and correctness data, not line-count
499
- targets alone.
500
- - Accepted/rejected text, revision metadata, structural references, formatting,
501
- comments, tables, and numbering match the established oracle fixtures.
502
- - No public export disappears in a minor release.
503
- - Any eventual major-version removal has a migration example and deprecation
504
- period.
505
-
506
- ### Implementation record
507
-
508
- - `engine/route-selection.js` contains an executable capability matrix for
509
- format-only, surgical, reconstruction, table, direct-list, and compatibility-
510
- pipeline routes. It records primary structures, retained structures, artifact
511
- behavior, and the reason each route remains distinct.
512
- - Route instrumentation is opt-in and internal: `_routeInstrumentation.onRoute`
513
- observes the selected route and context without changing public result
514
- shapes or providing a routing-policy override. `npm run profile:routes`
515
- provides a deterministic synthetic frequency profile; its checked four-case
516
- sample selects table, reconstruction, direct-list, and compatibility-list
517
- exactly once each.
518
- - A list edit sourced from one paragraph now calls the focused
519
- `executeListGeneration(...)` path directly with the established numbering
520
- service and document-scoped revision allocator. This preserves wrapping,
521
- `numberingXml`, `sourceType`, warnings, and native-API signaling.
522
- - Multi-paragraph marked-list edits continue through `ReconciliationPipeline`.
523
- That path performs run-aware patching across existing paragraph boundaries;
524
- retaining it is an intentional accuracy boundary, not unfinished deletion.
525
- - The structural list fallback uses the focused generator directly during
526
- normal execution but continues to honor an explicitly injected pipeline for
527
- compatibility tests and existing internal integrations.
528
- - `performance_phase5_route_consolidation_tests.mjs` compares direct and legacy
529
- list generation with independent accepted/rejected extraction, verifies
530
- numbering-package output, locks single- versus multi-paragraph route choices,
531
- checks the capability matrix, and proves the public compatibility export
532
- identities remain available.
533
- - `ReconciliationPipeline`, `serializeToOoxml`, and `wrapInDocumentFragment`
534
- remain public and supported throughout the current major version. No public
535
- API was deprecated or removed by this phase; future removal still requires a
536
- major version, a prior deprecation notice, and a migration example.
537
- - All automated, coverage, isolation, type, lint, build, package-dry-run, and
538
- fixture-export checks pass. The 62-file coverage result is 89.81% statements/
539
- lines, 77.34% branches, and 93.04% functions, improving every recorded
540
- baseline dimension. The 47/47 Microsoft Word differential also passes.
541
- - The 60-scenario SuperDoc Microsoft Word corpus passes after the route change,
542
- including multi-paragraph and nested lists, list insertions/deletions, table
543
- reconciliation, page headers, long documents, and mixed batches. Its package
544
- guard also verifies that every unrelated DOCX part remains byte-identical.
545
- - The local ECMA-376 XSD command could not run because this Windows environment
546
- has the WSL launcher but no usable `/bin/bash`. Schema validation remains in
547
- the nightly independent-oracle workflow; runtime structural validation, the
548
- synthetic Word differential, and the real-document Word corpus all passed.
549
-
550
- ---
551
-
552
- ## Phase 6 — Faster Tests While Preserving Isolation
553
-
554
- **Status:** Complete (2026-09-04). The JavaScript suite now uses bounded
555
- subprocess concurrency while every test file retains a fresh Node process.
556
-
557
- ### Problem
558
-
559
- The sequential runner starts 57 Node processes. Importing every test into one
560
- process would be faster, but it would also combine global XML providers,
561
- loggers, default authors, revision counters, module caches, environment changes,
562
- and top-level test side effects that are currently isolated.
563
-
564
- ### Work
565
-
566
- 1. Replace synchronous shell-string execution with bounded parallel
567
- `child_process.execFile`/`spawn` calls. Keep one process per test file.
568
- 2. Default concurrency conservatively (for example, CPU count capped at 4) and
569
- allow `DOCX_TEST_CONCURRENCY=1` for deterministic reproduction.
570
- 3. Preserve sorted result reporting, per-file timeout handling, captured
571
- stdout/stderr, failure-marker detection, and nonzero exit behavior.
572
- 4. Keep Word COM, visual, corpus, and filesystem-collision-sensitive lanes
573
- serial unless their runners explicitly coordinate resources.
574
- 5. Confirm `c8` still merges subprocess coverage correctly. If it does not,
575
- retain a coverage-specific serial command.
576
- 6. Consider Node's native test runner only after tests migrate from top-level
577
- script assertions and their global-state assumptions are documented.
578
-
579
- ### Acceptance
580
-
581
- - Median `npm test` time improves by at least 40% on the same machine.
582
- - Serial and parallel modes execute the same files and report the same failures.
583
- - No flaky failures occur across at least 20 repeated parallel runs.
584
- - `npm run test:coverage`, isolation checks, and Windows paths remain valid.
585
-
586
- ### Implementation record
587
-
588
- - `scripts/run-tests.mjs` now launches test files with asynchronous
589
- `execFile(...)` workers instead of synchronous shell command strings. It uses
590
- `process.execPath` plus an argument array, avoiding shell quoting and keeping
591
- Windows paths safe.
592
- - The default worker count is the runtime's available parallelism capped at
593
- four. `DOCX_TEST_CONCURRENCY=1` provides deterministic serial reproduction;
594
- any positive integer is accepted explicitly and is bounded by the discovered
595
- test count. Invalid values fail before starting tests.
596
- - Test discovery and final reporting remain filename-sorted regardless of
597
- completion order. Every child retains its own process, 30-second timeout,
598
- captured stdout/stderr, failure-marker detection, and nonzero-exit handling.
599
- Success summaries stay on stdout and detailed failure output stays on stderr.
600
- - The Word COM, synthetic visual, corpus, corpus-visual, and other independent
601
- oracle commands remain separate serial runners. JavaScript tests that write
602
- fixtures use distinct temporary directories; the runner does not combine
603
- their module state or files into one process.
604
- - `performance_phase6_parallel_runner_tests.mjs` locks sorted discovery,
605
- setup-helper exclusion, the concurrency bound, separate child PIDs, serial/
606
- parallel result parity, successful-exit failure markers, nonzero exits,
607
- stdout/stderr capture, timeouts, and deterministic reporting.
608
- - `npm run benchmark:tests` runs one warmup and three paired measurements and
609
- writes machine-readable output to ignored
610
- `tmp/benchmarks/test-runner-latest.json`. On Node 24/Windows x64, concurrency
611
- four reduced the median from 18,018.68 ms to 7,964.47 ms: a 55.80% reduction
612
- and 2.26x speedup, exceeding the 40% acceptance target.
613
- - Twenty consecutive four-worker runs passed all 63 test files with no flaky
614
- failure. Their observed range was 7.49–8.24 seconds, with a 7.86-second
615
- average. A separate `DOCX_TEST_CONCURRENCY=1` run passed the same 63 files.
616
- - c8 successfully merged the parallel subprocess data, so the regular coverage
617
- command remains parallel. The checked result is 89.90% statements/lines,
618
- 77.40% branches, and 93.10% functions, meeting or improving the Phase 5
619
- totals. No coverage-specific serial fork was added.
620
- - Node's native test runner was not adopted: these files remain top-level
621
- assertion scripts with intentional process-global isolation. A future
622
- migration should first document and change that execution contract.
623
-
624
- ---
625
-
626
- ## 4. Revised Execution Roadmap
627
-
628
- 1. **Measurement and safety baseline**
629
- - Add benchmarks, parse/serialize instrumentation, and result-equivalence
630
- fixtures.
631
- 2. **In-memory session foundation — complete 2026-09-04**
632
- - Introduce the focused operation-session module behind the existing runner,
633
- retain compatibility wrappers, and measure the improvement. Per-operation
634
- savepoints remain mandatory unless an equally strong isolation proof
635
- replaces them.
636
- 3. **Boundary-preserving decomposition — complete 2026-09-04**
637
- - Extract the applier, orchestrator, and heuristics around the proven session
638
- without changing public behavior. This step was completed first; the
639
- session boundary exists but still uses legacy per-operation serialization.
640
- 4. **Measured hot paths and parallel test runner — complete 2026-09-04**
641
- - Optimize indexes/traversals and add bounded subprocess concurrency.
642
- 5. **List-domain cleanup**
643
- - Share syntax and visible-text semantics without collapsing distinct
644
- responsibilities.
645
- 6. **Engine consolidation**
646
- - Change internal routing only after capability, corpus, and Word parity;
647
- defer public removals to a major release.
648
-
649
- ## 5. Required Validation Per Phase
650
-
651
- At minimum run:
652
-
653
- ```bash
654
- npm test
655
- npm run test:coverage
656
- npm run test:isolation
657
- npm run check:types
658
- npm run lint
659
- npm run build
660
- npm pack --dry-run
661
- ```
662
-
663
- For phases that change reconciliation routes, list construction, comments,
664
- numbering, or package output, also run the applicable Word differential, corpus,
665
- visual, XSD, and LibreOffice lanes described in `docs/TESTING.md` and
666
- `docs/VALIDATION.md`.
667
-
668
- Each phase should update benchmark results, `ARCHITECTURE.md`, `CHANGELOG.md`,
669
- `docs/TESTING.md`, and this plan as implementation decisions become concrete.