@intlayer/cli 7.0.7 → 7.0.8-canary.0

package/dist/assets/translation-alignment/README.md

@@ -0,0 +1,485 @@
+# Translation Alignment System
+
+A sophisticated block-aware alignment system for maintaining translations of markdown documents. This system intelligently detects changes, handles reordering, and minimizes unnecessary AI translation calls.
+
+## Overview
+
+When you have an English document and its French translation, and you make changes to the English version, this system:
+
+1. **Segments** both documents into semantic blocks (headings, paragraphs, lists, code blocks, etc.)
+2. **Fingerprints** each block using structural anchors (special characters, numbers, punctuation)
+3. **Aligns** blocks between English and French using the Needleman-Wunsch algorithm
+4. **Detects** which blocks changed, were added, deleted, or reordered
+5. **Optimizes** by only sending changed/new blocks to AI for translation
+6. **Reconstructs** the French document with proper ordering and spacing
+
+## Key Features
+
+### Language-Agnostic Alignment
+
+Uses **anchor text** (special characters like `[]`124567890-=!@#$%^&*()><`) rather than language-specific content. This makes alignment robust across different languages.
+
+```typescript
+// Example anchor text extraction:
+"Hello [World](https://example.com) - 2024"
+// Becomes: "[ ](://.)-2024"
+```
+
+### Automatic Reordering Detection
+
+If you swap two paragraphs in the English document and both remain unchanged in content, the system automatically reorders them in the French translation without re-translating.
+
+### Smart Change Detection
+
+Uses Git changed lines (if available) to precisely identify which blocks need review. Falls back to content comparison if Git info is unavailable.
+
+### Block Types Supported
+
+- `heading` - Markdown headings (# ## ###)
+- `paragraph` - Regular text paragraphs
+- `list_item` - Bullet and numbered lists
+- `code_block` - Fenced code blocks
+- `blockquote` - Quote blocks
+- `table` - Markdown tables
+- `horizontal_rule` - Horizontal rules (---)
+- `html` - Embedded HTML
+- `unknown` - Fallback for blank lines
+
+## Architecture
+
+### Core Components
+
+```
+translation-alignment/
+├── types.ts                    # Type definitions
+├── segmentDocument.ts          # Document → Blocks
+├── normalizeBlock.ts           # Block → Normalized (semantic + anchor)
+├── fingerprintBlock.ts         # Normalized → Fingerprinted (with digests)
+├── computeSimilarity.ts        # Jaccard similarity calculation
+├── alignBlocks.ts              # Needleman-Wunsch alignment
+├── mapChangedLinesToBlocks.ts  # Git lines → Block indexes
+├── planActions.ts              # Alignment → Action plan
+├── rebuildDocument.ts          # Action plan → Segments + Merge
+└── pipeline.ts                 # High-level orchestration
+```
+
+### Data Flow
+
+```
+English Text ─────────┐
+                      │
+                      ├─→ Segment → Normalize → Fingerprint
+                      │
+French Text ──────────┘
+                      │
+                      ↓
+              Align (Needleman-Wunsch)
+                      │
+                      ↓
+              Map Changed Lines
+                      │
+                      ↓
+              Plan Actions
+                      │
+                      ↓
+    ┌─────────────────┴─────────────────┐
+    │                                   │
+    ↓                                   ↓
+Segments to Review              Reusable Segments
+    │                                   │
+    ↓                                   │
+AI Translation                          │
+    │                                   │
+    └─────────────────┬─────────────────┘
+                      │
+                      ↓
+              Merge & Output
+```
+
+## How It Works
+
+### 1. Segmentation
+
+```typescript
+const blocks = segmentDocument(text);
+// Input: "# Title\n\nParagraph 1\n\nParagraph 2"
+// Output: [
+//   { type: "heading", content: "# Title\n", lineStart: 1, lineEnd: 1 },
+//   { type: "paragraph", content: "Paragraph 1\n", lineStart: 3, lineEnd: 3 },
+//   { type: "paragraph", content: "Paragraph 2\n", lineStart: 5, lineEnd: 5 }
+// ]
+```
+
+### 2. Normalization
+
+```typescript
+const normalized = normalizeBlock(block);
+// Creates two representations:
+// - semanticText: lowercase, no markdown formatting
+// - anchorText: only special chars, numbers, symbols
+```
+
+### 3. Fingerprinting
+
+```typescript
+const fingerprinted = fingerprintBlock(normalized, prev, next);
+// Adds:
+// - semanticDigest: hash of semantic content
+// - anchorDigest: hash of structural anchors
+// - compositeKey: combined unique identifier
+// - contextKey: hash of surrounding blocks
+```
+
+### 4. Alignment
+
+Uses Needleman-Wunsch algorithm with custom scoring:
+
+- **Match score**: Based on type similarity, anchor similarity, and length ratio
+- **Gap penalty**: Penalizes insertions/deletions
+- **Type bonus**: +2 if block types match
+- **Length bonus**: +1 if lengths within 75%
+- **Anchor similarity**: 0-1 Jaccard score × 8
+
+### 5. Action Planning
+
+Produces one of four actions for each block:
+
+```typescript
+type PlannedAction =
+  | { kind: "reuse" }        // Unchanged, use existing translation
+  | { kind: "review" }       // Changed, needs AI review
+  | { kind: "insert_new" }   // New block, needs translation
+  | { kind: "delete" };      // Removed from source, skip
+```
+
+### 6. Reconstruction
+
+```typescript
+// Only segments needing review are sent to AI
+const reviewedMap = new Map<actionIndex, translatedText>();
+
+// Final document merges reused + reviewed segments in correct order
+const output = mergeReviewedSegments(plan, frenchBlocks, reviewedMap);
+```
+
+## Integration
+
+### Basic Usage
+
+```typescript
+import { reviewFileBlockAware } from "./reviewDocBlockAware";
+
+await reviewFileBlockAware(
+  "docs/en/guide.md",      // English source
+  "docs/fr/guide.md",      // French translation
+  Locales.FRENCH,          // Target locale
+  Locales.ENGLISH,         // Base locale
+  aiOptions,               // AI configuration
+  configOptions,           // App configuration
+  customInstructions,      // Optional AI instructions
+  [10, 15, 20]            // Git changed lines (optional)
+);
+```
+
+### Replace Existing Implementation
+
+In your main review script, replace:
+
+```typescript
+// Old:
+await reviewFile(
+  baseFilePath,
+  outputFilePath,
+  locale,
+  baseLocale,
+  aiOptions,
+  configOptions,
+  customInstructions,
+  changedLines
+);
+
+// New:
+await reviewFileBlockAware(
+  baseFilePath,
+  outputFilePath,
+  locale,
+  baseLocale,
+  aiOptions,
+  configOptions,
+  customInstructions,
+  changedLines
+);
+```
+
+### Configuration Options
+
+```typescript
+const { plan, segmentsToReview } = buildAlignmentPlan({
+  englishText,
+  frenchText,
+  changedLines: [10, 15, 20],  // Optional Git changed lines
+  similarityOptions: {
+    minimumMatchForReuse: 0.9,        // Default: 0.9 (90% similar)
+    minimumMatchForNearDuplicate: 0.8  // Default: 0.8 (80% similar)
+  }
+});
+```
+
+## Examples
+
+### Example 1: Simple Change
+
+**English v1:**
+```markdown
+# Introduction
+Hello world
+```
+
+**French v1:**
+```markdown
+# Introduction
+Bonjour le monde
+```
+
+**English v2 (changed second line):**
+```markdown
+# Introduction
+Hello universe
+```
+
+**Result:**
+- Block 1 (heading): **reuse** existing translation
+- Block 2 (paragraph): **review** with AI
+- Final output preserves structure, only paragraph updated
+
+### Example 2: Reordering
+
+**English v1:**
+```markdown
+## Section A
+Content A
+
+## Section B
+Content B
+```
+
+**English v2 (swapped order):**
+```markdown
+## Section B
+Content B
+
+## Section A
+Content A
+```
+
+**Result:**
+- All blocks identified as unchanged
+- Automatically reordered in French
+- No AI calls needed
+
+### Example 3: Insertion
+
+**English v1:**
+```markdown
+Para 1
+
+Para 3
+```
+
+**English v2:**
+```markdown
+Para 1
+
+Para 2
+
+Para 3
+```
+
+**Result:**
+- Para 1: **reuse**
+- Para 2: **insert_new** (translate with AI)
+- Para 3: **reuse**
+
+### Example 4: Deletion
+
+**English v1:**
+```markdown
+Para 1
+
+Para 2
+
+Para 3
+```
+
+**English v2:**
+```markdown
+Para 1
+
+Para 3
+```
+
+**Result:**
+- Para 1: **reuse**
+- Para 2: **delete** (skip in output)
+- Para 3: **reuse**
+
+## Performance Optimization
+
+### With Git Integration
+
+When Git changed lines are available, the system:
+1. Maps changed lines to block indexes
+2. Only marks those blocks as needing review
+3. Dramatically reduces AI calls for large documents
+
+```typescript
+// With git info: only reviews blocks 5, 6, 7 (3 AI calls)
+changedLines: [45, 46, 47, 48, 49, 50]
+
+// Without git info: reviews all changed blocks by content comparison
+changedLines: undefined
+```
+
+### Reuse Rate
+
+Typical reuse rates:
+- **Minor edits**: 90-95% blocks reused
+- **Moderate changes**: 70-80% blocks reused
+- **Major refactor**: 40-60% blocks reused
+- **New document**: 0% blocks reused (but alignment helps with structure)
+
+## Best Practices
+
+### 1. Preserve Structure
+
+Keep consistent markdown structure between versions. The alignment algorithm works best when block types remain stable.
+
+✅ Good:
+```markdown
+# Heading
+Paragraph
+
+# Heading
+Paragraph (updated content)
+```
+
+❌ Avoid:
+```markdown
+# Heading
+Paragraph
+
+## Different heading level
+List instead of paragraph
+- Item 1
+```
+
+### 2. Use Git Integration
+
+Always pass `changedLines` when available for maximum efficiency:
+
+```typescript
+const changedLines = await listGitLines(filePath, gitOptions);
+await reviewFileBlockAware(..., changedLines);
+```
+
+### 3. Adjust Similarity Thresholds
+
+For different document types:
+
+- **Technical docs** (lots of code/numbers): Lower threshold (0.85)
+- **Marketing content** (more text): Higher threshold (0.95)
+- **Mixed content**: Default (0.90)
+
+### 4. Monitor Action Distribution
+
+Log action distribution to understand system behavior:
+
+```typescript
+const stats = {
+  reuse: plan.actions.filter(a => a.kind === "reuse").length,
+  review: plan.actions.filter(a => a.kind === "review").length,
+  insertNew: plan.actions.filter(a => a.kind === "insert_new").length,
+  delete: plan.actions.filter(a => a.kind === "delete").length,
+};
+console.log(`Efficiency: ${(stats.reuse / plan.actions.length * 100).toFixed(1)}% reused`);
+```
+
+## Troubleshooting
+
+### Issue: Too Many Blocks Marked for Review
+
+**Cause**: Similarity threshold too high
+**Solution**: Lower `minimumMatchForReuse` to 0.85 or 0.80
+
+### Issue: Blocks Not Aligning Correctly
+
+**Cause**: Document structure changed significantly
+**Solution**: 
+- Ensure block types are consistent
+- Check that special characters/numbers are preserved
+- Verify the anchor text is being extracted correctly
+
+### Issue: Translations Not Merging
+
+**Cause**: Action indexes not matching
+**Solution**: Ensure `reviewedSegmentsMap` uses `segment.actionIndex` as key
+
+### Issue: Extra Blank Lines in Output
+
+**Cause**: Block content doesn't preserve trailing newlines consistently
+**Solution**: Review `trimTrailingNewlines` in segmentation logic
+
+## Testing
+
+### Unit Tests
+
+```typescript
+import { segmentDocument } from "./segmentDocument";
+import { alignEnglishAndFrenchBlocks } from "./alignBlocks";
+
+describe("Block Alignment", () => {
+  it("should detect reordered paragraphs", () => {
+    const english = "A\n\nB\n\nC";
+    const french = "B\n\nA\n\nC";
+    // ... test alignment
+  });
+});
+```
+
+### Integration Tests
+
+```typescript
+it("should handle complete document translation workflow", async () => {
+  const result = await reviewFileBlockAware(
+    testEnglishPath,
+    testFrenchPath,
+    Locales.FRENCH,
+    Locales.ENGLISH
+  );
+  // Verify output structure, reuse rate, etc.
+});
+```
+
+## Future Enhancements
+
+- [ ] Support for more block types (footnotes, definition lists)
+- [ ] Fuzzy matching for minor structural variations
+- [ ] Parallel AI translation for multiple segments
+- [ ] Caching of fingerprints for repeated processing
+- [ ] Visual diff tool for reviewing alignment decisions
+- [ ] Machine learning for optimal threshold selection
+
+## Contributing
+
+When contributing to this system:
+
+1. **Follow naming conventions**: No abbreviations, descriptive names
+2. **Use arrow functions**: Consistent function syntax
+3. **Add JSDoc comments**: Document complex logic
+4. **Write tests**: Cover edge cases
+5. **Maintain backwards compatibility**: Ensure existing integrations work
+
+## License
+
+See main project LICENSE file.
+