@intlayer/cli 7.0.6 → 7.0.8-canary.0

package/dist/assets/translation-alignment/ARCHITECTURE.md

@@ -0,0 +1,518 @@
+# Architecture Diagram
+
+This document provides visual representations of the block-aware translation alignment system.
+
+## System Overview
+
+```
+┌─────────────────────────────────────────────────────────────────────┐
+│                     Block-Aware Translation System                  │
+└─────────────────────────────────────────────────────────────────────┘
+                                    │
+                ┌───────────────────┴───────────────────┐
+                │                                       │
+         ┌──────▼──────┐                        ┌──────▼──────┐
+         │   English   │                        │   French    │
+         │  Document   │                        │  Document   │
+         │   (Base)    │                        │ (Translation)│
+         └──────┬──────┘                        └──────┬──────┘
+                │                                       │
+                │         ┌─────────────────┐          │
+                └────────►│  Segmentation   │◄─────────┘
+                          └────────┬────────┘
+                                   │
+                          ┌────────▼────────┐
+                          │  Normalization  │
+                          │  - Semantic     │
+                          │  - Anchor Text  │
+                          └────────┬────────┘
+                                   │
+                          ┌────────▼────────┐
+                          │ Fingerprinting  │
+                          │  - SHA-256      │
+                          │  - Context      │
+                          └────────┬────────┘
+                                   │
+                ┌──────────────────┴──────────────────┐
+                │                                     │
+         ┌──────▼──────┐                      ┌──────▼──────┐
+         │   English   │                      │   French    │
+         │   Blocks    │                      │   Blocks    │
+         │ (Fingerpr.) │                      │ (Fingerpr.) │
+         └──────┬──────┘                      └──────┬──────┘
+                │                                     │
+                └──────────────┬──────────────────────┘
+                               │
+                      ┌────────▼────────┐
+                      │    Alignment    │
+                      │ (Needleman-     │
+                      │  Wunsch)        │
+                      └────────┬────────┘
+                               │
+                      ┌────────▼────────┐
+                      │  Alignment      │
+                      │  Pairs          │
+                      └────────┬────────┘
+                               │
+                ┌──────────────┴──────────────┐
+                │                             │
+         ┌──────▼──────┐            ┌─────────▼────────┐
+         │    Git      │            │   Similarity     │
+         │  Changed    │            │   Options        │
+         │   Lines     │            │                  │
+         └──────┬──────┘            └─────────┬────────┘
+                │                             │
+                └──────────┬──────────────────┘
+                           │
+                  ┌────────▼────────┐
+                  │  Action         │
+                  │  Planning       │
+                  └────────┬────────┘
+                           │
+                  ┌────────▼────────┐
+                  │  Action Plan    │
+                  │  - reuse        │
+                  │  - review       │
+                  │  - insert_new   │
+                  │  - delete       │
+                  └────────┬────────┘
+                           │
+         ┌─────────────────┴─────────────────┐
+         │                                   │
+         │        ┌──────────────────┐       │
+         └───────►│   Identify       │       │
+                  │   Segments to    │       │
+                  │   Review         │       │
+                  └────────┬─────────┘       │
+                           │                 │
+                  ┌────────▼─────────┐       │
+                  │  AI Translation  │       │
+                  │  (Your Logic)    │       │
+                  └────────┬─────────┘       │
+                           │                 │
+                  ┌────────▼─────────┐       │
+                  │   Reviewed       │       │
+                  │   Segments       │       │
+                  │   (Map)          │       │
+                  └────────┬─────────┘       │
+                           │                 │
+                           └────────┬────────┘
+                                    │
+                           ┌────────▼────────┐
+                           │  Merge          │
+                           │  Reviewed       │
+                           │  Segments       │
+                           └────────┬────────┘
+                                    │
+                           ┌────────▼────────┐
+                           │  Final French   │
+                           │  Document       │
+                           └─────────────────┘
+```
+
+## Data Flow
+
+```
+Input:
+  englishText: string
+  frenchText: string (existing translation or "")
+  changedLines: number[] (optional, from Git)
+  similarityOptions: { minimumMatchForReuse: 0.9 }
+
+Output:
+  finalFrenchDocument: string (updated translation)
+```
+
+## Module Dependencies
+
+```
+types.ts
+  ↓
+segmentDocument.ts ────────┐
+  ↓                        │
+normalizeBlock.ts ─────────┤
+  ↓                        │
+fingerprintBlock.ts ───────┤
+                           ├──→ pipeline.ts ──→ reviewDocBlockAware.ts
+computeSimilarity.ts ──────┤         ↑
+  ↓                        │         │
+alignBlocks.ts ────────────┤         │
+                           │         │
+mapChangedLinesToBlocks.ts ┤         │
+  ↓                        │         │
+planActions.ts ────────────┤         │
+  ↓                        │         │
+rebuildDocument.ts ────────┘         │
+                                     │
+                              index.ts (exports)
+```
+
+## Alignment Algorithm (Needleman-Wunsch)
+
+```
+English Blocks:  [E1] [E2] [E3] [E4] [E5]
+French Blocks:   [F1] [F2] [F3] [F4]
+
+Score Matrix (5x4):
+       F1   F2   F3   F4
+E1   [ 8  ][  2 ][  1 ][  0 ]
+E2   [ 2  ][ 9  ][  3 ][  1 ]
+E3   [ 1  ][ 3  ][ 8  ][  2 ]
+E4   [ 0  ][ 1  ][ 2  ][ 7  ]
+E5   [ 0  ][ 0  ][ 1  ][ 2  ]  ← E5 is new
+
+Traceback (diagonal = match, up = delete, left = insert):
+  E1 ──→ F1  (match, score 8)
+  E2 ──→ F2  (match, score 9)
+  E3 ──→ F3  (match, score 8)
+  E4 ──→ F4  (match, score 7)
+  E5 ──→ NULL (new block, no match)
+
+Alignment Pairs:
+  [ (E1→F1, sim=0.95), (E2→F2, sim=0.98), (E3→F3, sim=0.93), (E4→F4, sim=0.91), (E5→NULL, sim=0) ]
+```
+
+## Action Planning Logic
+
+```
+For each alignment pair:
+
+  IF englishIndex = -1 AND frenchIndex != null:
+    → Action: DELETE (French block with no English match)
+
+  ELSE IF englishIndex >= 0 AND frenchIndex = null:
+    → Action: INSERT_NEW (New English block)
+
+  ELSE IF englishIndex >= 0 AND frenchIndex != null:
+    IF block changed (from Git) OR similarity < threshold:
+      → Action: REVIEW (Changed block, needs AI)
+    ELSE:
+      → Action: REUSE (Unchanged block, copy existing)
+```
+
+## Similarity Calculation (Jaccard)
+
+```
+Block 1: "[Click here](https://example.com) - see section 2.1"
+Block 2: "[Cliquez ici](https://example.com) - voir section 2.1"
+
+Anchor Text Extraction:
+  Block 1 anchor: "[](://.)-2.1"
+  Block 2 anchor: "[](://.)-2.1"
+
+Character Shingles (length=3):
+  Set A: {"[](" "](:" "]:/" "://", "//." "//.)", ".)-", ")-2", "-2.", "2.1"}
+  Set B: {"[](" "](:" "]:/" "://", "//." "//.)", ".)-", ")-2", "-2.", "2.1"}
+
+Jaccard Similarity:
+  Intersection: 10 shingles
+  Union: 10 shingles
+  Similarity = 10/10 = 1.0 (100% match!)
+
+Result: REUSE (despite different languages!)
+```
+
+## State Machine
+
+```
+┌─────────────┐
+│   Start     │
+└──────┬──────┘
+       │
+       ▼
+┌─────────────────┐
+│ Segment         │
+│ Documents       │
+└──────┬──────────┘
+       │
+       ▼
+┌─────────────────┐
+│ Normalize       │
+│ Blocks          │
+└──────┬──────────┘
+       │
+       ▼
+┌─────────────────┐
+│ Fingerprint     │
+│ Blocks          │
+└──────┬──────────┘
+       │
+       ▼
+┌─────────────────┐
+│ Align           │
+│ Blocks          │
+└──────┬──────────┘
+       │
+       ▼
+┌─────────────────┐      YES   ┌─────────────┐
+│ Changed Lines?  ├────────────►│ Map Lines   │
+└──────┬──────────┘             │ to Blocks   │
+       │ NO                     └──────┬──────┘
+       │                               │
+       └───────────────┬───────────────┘
+                       │
+                       ▼
+              ┌─────────────────┐
+              │ Plan Actions    │
+              └────────┬─────────┘
+                       │
+                       ▼
+              ┌─────────────────┐
+              │ For Each Action │
+              └────────┬─────────┘
+                       │
+       ┌───────────────┼───────────────┐
+       │               │               │
+       ▼               ▼               ▼
+┌─────────┐   ┌─────────────┐   ┌─────────┐
+│ REUSE   │   │   REVIEW    │   │  DELETE │
+│ Copy    │   │   Translate │   │  Skip   │
+│ Existing│   │   with AI   │   │         │
+└────┬────┘   └──────┬──────┘   └────┬────┘
+     │               │               │
+     └───────────────┼───────────────┘
+                     │
+                     ▼
+            ┌─────────────────┐
+            │ Merge Segments  │
+            └────────┬─────────┘
+                     │
+                     ▼
+            ┌─────────────────┐
+            │ Write Output    │
+            └────────┬─────────┘
+                     │
+                     ▼
+              ┌─────────────┐
+              │    Done     │
+              └─────────────┘
+```
+
+## Block Types Hierarchy
+
+```
+Block (abstract)
+  │
+  ├── StructuralBlock
+  │     ├── heading (# ## ###)
+  │     ├── horizontal_rule (---)
+  │     └── html (<div>...</div>)
+  │
+  ├── ContentBlock
+  │     ├── paragraph
+  │     ├── blockquote (> text)
+  │     └── list_item (- item)
+  │
+  ├── CodeBlock
+  │     └── code_block (```...```)
+  │
+  ├── TableBlock
+  │     └── table (| col |)
+  │
+  └── UnknownBlock
+        └── unknown (blank lines, etc.)
+```
+
+## Fingerprinting Strategy
+
+```
+Block Content:
+  "# Introduction\n\nWelcome to [our guide](https://example.com)!"
+
+Step 1: Normalize
+  semanticText: "introduction welcome to our guide"
+  anchorText: "#[](://.)!"
+
+Step 2: Hash
+  semanticDigest: sha256(semanticText) = "a1b2c3..."
+  anchorDigest: sha256(anchorText) = "d4e5f6..."
+
+Step 3: Composite Key
+  compositeKey: "a1b2c3...:d4e5f6..."
+
+Step 4: Context (surrounding blocks)
+  previousDigest: sha256(previousBlock.semanticText)
+  nextDigest: sha256(nextBlock.semanticText)
+  contextKey: sha256(previousDigest + ":" + nextDigest)
+
+Result: Unique fingerprint that:
+  - Identifies content semantically
+  - Preserves structure (anchors)
+  - Considers context (neighbors)
+```
+
+## Performance Characteristics
+
+```
+Time Complexity:
+  Segmentation:      O(n)        n = lines in document
+  Normalization:     O(b*m)      b = blocks, m = avg block length
+  Fingerprinting:    O(b)        b = blocks
+  Alignment:         O(b_en * b_fr)  Needleman-Wunsch
+  Planning:          O(p)        p = alignment pairs
+  Reconstruction:    O(a)        a = actions
+
+Total: O(b_en * b_fr) dominant for large documents
+
+Space Complexity:
+  Blocks storage:    O(b*m)
+  Score matrix:      O(b_en * b_fr)
+  Action plan:       O(max(b_en, b_fr))
+
+Total: O(b*m + b_en*b_fr)
+
+Optimization:
+  Blocks << Lines (typically 10x fewer)
+  So O(blocks^2) << O(lines^2)
+```
+
+## Example Trace
+
+```
+Input:
+  English: "# Title\n\nPara 1\n\nPara 2\n"
+  French:  "# Titre\n\nPara 1 (fr)\n\nPara 2 (fr)\n"
+  Changed: [3] (line 3 changed)
+
+Segmentation:
+  English Blocks: [E1: heading, E2: paragraph, E3: paragraph]
+  French Blocks:  [F1: heading, F2: paragraph, F3: paragraph]
+
+Normalization:
+  E1: semantic="title",        anchor="#"
+  E2: semantic="para 1",       anchor=""
+  E3: semantic="para 2",       anchor=""
+  F1: semantic="titre",        anchor="#"
+  F2: semantic="para 1 (fr)",  anchor="()"
+  F3: semantic="para 2 (fr)",  anchor="()"
+
+Fingerprinting:
+  E1: compositeKey="hash1:hash2"
+  E2: compositeKey="hash3:hash4"
+  E3: compositeKey="hash5:hash6"
+
+Alignment:
+  Pairs: [(E1→F1, 0.82), (E2→F2, 0.95), (E3→F3, 0.94)]
+
+Change Detection:
+  Line 3 is in block E2 (lines 3-3)
+  Changed blocks: {E2}
+
+Action Planning:
+  E1→F1: similarity=0.82 < 0.90, but not changed → REUSE
+  E2→F2: similarity=0.95, but CHANGED → REVIEW
+  E3→F3: similarity=0.94, not changed → REUSE
+
+Reconstruction:
+  Output:
+    F1 content (reused)
+    F2 content (from AI)
+    F3 content (reused)
+
+Final:
+  "# Titre\n\nPara 1 (translated by AI)\n\nPara 2 (fr)\n"
+```
+
+## Integration Points
+
+```
+┌───────────────────────────────────────────────────────┐
+│                    Your Application                   │
+└───────────────────────┬───────────────────────────────┘
+                        │
+           ┌────────────┴────────────┐
+           │                         │
+    ┌──────▼──────┐          ┌──────▼──────┐
+    │  reviewDoc  │          │   Direct    │
+    │   (main)    │          │  Pipeline   │
+    └──────┬──────┘          │    API      │
+           │                 └──────┬──────┘
+           │                        │
+    ┌──────▼──────────────────────┬─┘
+    │                             │
+    │  reviewFileBlockAware       │
+    │  (integration layer)        │
+    └──────┬──────────────────────┘
+           │
+    ┌──────▼──────────────────────┐
+    │  buildAlignmentPlan         │
+    │  (pipeline)                 │
+    └──────┬──────────────────────┘
+           │
+    ┌──────▼──────────────────────┐
+    │  Core Translation           │
+    │  Alignment Modules          │
+    └─────────────────────────────┘
+```
+
+## Testing Strategy
+
+```
+Unit Tests:
+  ├── segmentDocument.test.ts
+  │     ├── Should segment headings
+  │     ├── Should segment paragraphs
+  │     ├── Should segment code blocks
+  │     └── Should handle edge cases
+  │
+  ├── normalizeBlock.test.ts
+  │     ├── Should extract semantic text
+  │     ├── Should extract anchor text
+  │     └── Should handle special chars
+  │
+  ├── computeSimilarity.test.ts
+  │     ├── Should compute Jaccard similarity
+  │     └── Should handle edge cases
+  │
+  └── alignBlocks.test.ts
+        ├── Should align matching blocks
+        ├── Should detect insertions
+        ├── Should detect deletions
+        └── Should detect reordering
+
+Integration Tests:
+  ├── Full pipeline test
+  ├── Reordering scenario
+  ├── Insertion scenario
+  ├── Deletion scenario
+  └── Mixed changes scenario
+
+E2E Tests:
+  ├── Real markdown documents
+  ├── Multiple locales
+  └── Git integration
+```
+
+## Configuration Flow
+
+```
+User Config
+  │
+  ├── minimumMatchForReuse (default: 0.90)
+  ├── minimumMatchForNearDuplicate (default: 0.80)
+  └── AI options (provider, model, etc.)
+  │
+  └──→ pipeline.ts
+       │
+       ├──→ buildAlignmentPlan()
+       │    │
+       │    ├──→ planActions(similarityOptions)
+       │    │
+       │    └──→ Result: Plan + Segments
+       │
+       └──→ reviewFileBlockAware()
+            │
+            ├──→ chunkInference(AI options)
+            │
+            └──→ Final output
+```
+
+---
+
+This architecture ensures:
+- ✅ Separation of concerns
+- ✅ Testability
+- ✅ Extensibility
+- ✅ Performance
+- ✅ Maintainability
+