@intlayer/cli 7.1.4 → 7.1.5

package/dist/assets/translation-alignment/README.md

@@ -1,485 +0,0 @@
-# Translation Alignment System
-
-A sophisticated block-aware alignment system for maintaining translations of markdown documents. This system intelligently detects changes, handles reordering, and minimizes unnecessary AI translation calls.
-
-## Overview
-
-When you have an English document and its French translation, and you make changes to the English version, this system:
-
-1. **Segments** both documents into semantic blocks (headings, paragraphs, lists, code blocks, etc.)
-2. **Fingerprints** each block using structural anchors (special characters, numbers, punctuation)
-3. **Aligns** blocks between English and French using the Needleman-Wunsch algorithm
-4. **Detects** which blocks changed, were added, deleted, or reordered
-5. **Optimizes** by only sending changed/new blocks to AI for translation
-6. **Reconstructs** the French document with proper ordering and spacing
-
-## Key Features
-
-### Language-Agnostic Alignment
-
-Uses **anchor text** (special characters like `[]`124567890-=!@#$%^&*()><`) rather than language-specific content. This makes alignment robust across different languages.
-
-```typescript
-// Example anchor text extraction:
-"Hello [World](https://example.com) - 2024"
-// Becomes: "[ ](://.)-2024"
-```
-
-### Automatic Reordering Detection
-
-If you swap two paragraphs in the English document and both remain unchanged in content, the system automatically reorders them in the French translation without re-translating.
-
-### Smart Change Detection
-
-Uses Git changed lines (if available) to precisely identify which blocks need review. Falls back to content comparison if Git info is unavailable.
-
-### Block Types Supported
-
-- `heading` - Markdown headings (# ## ###)
-- `paragraph` - Regular text paragraphs
-- `list_item` - Bullet and numbered lists
-- `code_block` - Fenced code blocks
-- `blockquote` - Quote blocks
-- `table` - Markdown tables
-- `horizontal_rule` - Horizontal rules (---)
-- `html` - Embedded HTML
-- `unknown` - Fallback for blank lines
-
-## Architecture
-
-### Core Components
-
-```
-translation-alignment/
-├── types.ts                    # Type definitions
-├── segmentDocument.ts          # Document → Blocks
-├── normalizeBlock.ts           # Block → Normalized (semantic + anchor)
-├── fingerprintBlock.ts         # Normalized → Fingerprinted (with digests)
-├── computeSimilarity.ts        # Jaccard similarity calculation
-├── alignBlocks.ts              # Needleman-Wunsch alignment
-├── mapChangedLinesToBlocks.ts  # Git lines → Block indexes
-├── planActions.ts              # Alignment → Action plan
-├── rebuildDocument.ts          # Action plan → Segments + Merge
-└── pipeline.ts                 # High-level orchestration
-```
-
-### Data Flow
-
-```
-English Text ─────────┐
-                      │
-                      ├─→ Segment → Normalize → Fingerprint
-                      │
-French Text ──────────┘
-                      │
-                      ↓
-              Align (Needleman-Wunsch)
-                      │
-                      ↓
-              Map Changed Lines
-                      │
-                      ↓
-              Plan Actions
-                      │
-                      ↓
-    ┌─────────────────┴─────────────────┐
-    │                                   │
-    ↓                                   ↓
-Segments to Review              Reusable Segments
-    │                                   │
-    ↓                                   │
-AI Translation                          │
-    │                                   │
-    └─────────────────┬─────────────────┘
-                      │
-                      ↓
-              Merge & Output
-```
-
-## How It Works
-
-### 1. Segmentation
-
-```typescript
-const blocks = segmentDocument(text);
-// Input: "# Title\n\nParagraph 1\n\nParagraph 2"
-// Output: [
-//   { type: "heading", content: "# Title\n", lineStart: 1, lineEnd: 1 },
-//   { type: "paragraph", content: "Paragraph 1\n", lineStart: 3, lineEnd: 3 },
-//   { type: "paragraph", content: "Paragraph 2\n", lineStart: 5, lineEnd: 5 }
-// ]
-```
-
-### 2. Normalization
-
-```typescript
-const normalized = normalizeBlock(block);
-// Creates two representations:
-// - semanticText: lowercase, no markdown formatting
-// - anchorText: only special chars, numbers, symbols
-```
-
-### 3. Fingerprinting
-
-```typescript
-const fingerprinted = fingerprintBlock(normalized, prev, next);
-// Adds:
-// - semanticDigest: hash of semantic content
-// - anchorDigest: hash of structural anchors
-// - compositeKey: combined unique identifier
-// - contextKey: hash of surrounding blocks
-```
-
-### 4. Alignment
-
-Uses Needleman-Wunsch algorithm with custom scoring:
-
-- **Match score**: Based on type similarity, anchor similarity, and length ratio
-- **Gap penalty**: Penalizes insertions/deletions
-- **Type bonus**: +2 if block types match
-- **Length bonus**: +1 if lengths within 75%
-- **Anchor similarity**: 0-1 Jaccard score × 8
-
-### 5. Action Planning
-
-Produces one of four actions for each block:
-
-```typescript
-type PlannedAction =
-  | { kind: "reuse" }        // Unchanged, use existing translation
-  | { kind: "review" }       // Changed, needs AI review
-  | { kind: "insert_new" }   // New block, needs translation
-  | { kind: "delete" };      // Removed from source, skip
-```
-
-### 6. Reconstruction
-
-```typescript
-// Only segments needing review are sent to AI
-const reviewedMap = new Map<actionIndex, translatedText>();
-
-// Final document merges reused + reviewed segments in correct order
-const output = mergeReviewedSegments(plan, frenchBlocks, reviewedMap);
-```
-
-## Integration
-
-### Basic Usage
-
-```typescript
-import { reviewFileBlockAware } from "./reviewDocBlockAware";
-
-await reviewFileBlockAware(
-  "docs/en/guide.md",      // English source
-  "docs/fr/guide.md",      // French translation
-  Locales.FRENCH,          // Target locale
-  Locales.ENGLISH,         // Base locale
-  aiOptions,               // AI configuration
-  configOptions,           // App configuration
-  customInstructions,      // Optional AI instructions
-  [10, 15, 20]            // Git changed lines (optional)
-);
-```
-
-### Replace Existing Implementation
-
-In your main review script, replace:
-
-```typescript
-// Old:
-await reviewFile(
-  baseFilePath,
-  outputFilePath,
-  locale,
-  baseLocale,
-  aiOptions,
-  configOptions,
-  customInstructions,
-  changedLines
-);
-
-// New:
-await reviewFileBlockAware(
-  baseFilePath,
-  outputFilePath,
-  locale,
-  baseLocale,
-  aiOptions,
-  configOptions,
-  customInstructions,
-  changedLines
-);
-```
-
-### Configuration Options
-
-```typescript
-const { plan, segmentsToReview } = buildAlignmentPlan({
-  englishText,
-  frenchText,
-  changedLines: [10, 15, 20],  // Optional Git changed lines
-  similarityOptions: {
-    minimumMatchForReuse: 0.9,        // Default: 0.9 (90% similar)
-    minimumMatchForNearDuplicate: 0.8  // Default: 0.8 (80% similar)
-  }
-});
-```
-
-## Examples
-
-### Example 1: Simple Change
-
-**English v1:**
-```markdown
-# Introduction
-Hello world
-```
-
-**French v1:**
-```markdown
-# Introduction
-Bonjour le monde
-```
-
-**English v2 (changed second line):**
-```markdown
-# Introduction
-Hello universe
-```
-
-**Result:**
-- Block 1 (heading): **reuse** existing translation
-- Block 2 (paragraph): **review** with AI
-- Final output preserves structure, only paragraph updated
-
-### Example 2: Reordering
-
-**English v1:**
-```markdown
-## Section A
-Content A
-
-## Section B
-Content B
-```
-
-**English v2 (swapped order):**
-```markdown
-## Section B
-Content B
-
-## Section A
-Content A
-```
-
-**Result:**
-- All blocks identified as unchanged
-- Automatically reordered in French
-- No AI calls needed
-
-### Example 3: Insertion
-
-**English v1:**
-```markdown
-Para 1
-
-Para 3
-```
-
-**English v2:**
-```markdown
-Para 1
-
-Para 2
-
-Para 3
-```
-
-**Result:**
-- Para 1: **reuse**
-- Para 2: **insert_new** (translate with AI)
-- Para 3: **reuse**
-
-### Example 4: Deletion
-
-**English v1:**
-```markdown
-Para 1
-
-Para 2
-
-Para 3
-```
-
-**English v2:**
-```markdown
-Para 1
-
-Para 3
-```
-
-**Result:**
-- Para 1: **reuse**
-- Para 2: **delete** (skip in output)
-- Para 3: **reuse**
-
-## Performance Optimization
-
-### With Git Integration
-
-When Git changed lines are available, the system:
-1. Maps changed lines to block indexes
-2. Only marks those blocks as needing review
-3. Dramatically reduces AI calls for large documents
-
-```typescript
-// With git info: only reviews blocks 5, 6, 7 (3 AI calls)
-changedLines: [45, 46, 47, 48, 49, 50]
-
-// Without git info: reviews all changed blocks by content comparison
-changedLines: undefined
-```
-
-### Reuse Rate
-
-Typical reuse rates:
-- **Minor edits**: 90-95% blocks reused
-- **Moderate changes**: 70-80% blocks reused
-- **Major refactor**: 40-60% blocks reused
-- **New document**: 0% blocks reused (but alignment helps with structure)
-
-## Best Practices
-
-### 1. Preserve Structure
-
-Keep consistent markdown structure between versions. The alignment algorithm works best when block types remain stable.
-
-✅ Good:
-```markdown
-# Heading
-Paragraph
-
-# Heading
-Paragraph (updated content)
-```
-
-❌ Avoid:
-```markdown
-# Heading
-Paragraph
-
-## Different heading level
-List instead of paragraph
-- Item 1
-```
-
-### 2. Use Git Integration
-
-Always pass `changedLines` when available for maximum efficiency:
-
-```typescript
-const changedLines = await listGitLines(filePath, gitOptions);
-await reviewFileBlockAware(..., changedLines);
-```
-
-### 3. Adjust Similarity Thresholds
-
-For different document types:
-
-- **Technical docs** (lots of code/numbers): Lower threshold (0.85)
-- **Marketing content** (more text): Higher threshold (0.95)
-- **Mixed content**: Default (0.90)
-
-### 4. Monitor Action Distribution
-
-Log action distribution to understand system behavior:
-
-```typescript
-const stats = {
-  reuse: plan.actions.filter(a => a.kind === "reuse").length,
-  review: plan.actions.filter(a => a.kind === "review").length,
-  insertNew: plan.actions.filter(a => a.kind === "insert_new").length,
-  delete: plan.actions.filter(a => a.kind === "delete").length,
-};
-console.log(`Efficiency: ${(stats.reuse / plan.actions.length * 100).toFixed(1)}% reused`);
-```
-
-## Troubleshooting
-
-### Issue: Too Many Blocks Marked for Review
-
-**Cause**: Similarity threshold too high
-**Solution**: Lower `minimumMatchForReuse` to 0.85 or 0.80
-
-### Issue: Blocks Not Aligning Correctly
-
-**Cause**: Document structure changed significantly
-**Solution**: 
-- Ensure block types are consistent
-- Check that special characters/numbers are preserved
-- Verify the anchor text is being extracted correctly
-
-### Issue: Translations Not Merging
-
-**Cause**: Action indexes not matching
-**Solution**: Ensure `reviewedSegmentsMap` uses `segment.actionIndex` as key
-
-### Issue: Extra Blank Lines in Output
-
-**Cause**: Block content doesn't preserve trailing newlines consistently
-**Solution**: Review `trimTrailingNewlines` in segmentation logic
-
-## Testing
-
-### Unit Tests
-
-```typescript
-import { segmentDocument } from "./segmentDocument";
-import { alignEnglishAndFrenchBlocks } from "./alignBlocks";
-
-describe("Block Alignment", () => {
-  it("should detect reordered paragraphs", () => {
-    const english = "A\n\nB\n\nC";
-    const french = "B\n\nA\n\nC";
-    // ... test alignment
-  });
-});
-```
-
-### Integration Tests
-
-```typescript
-it("should handle complete document translation workflow", async () => {
-  const result = await reviewFileBlockAware(
-    testEnglishPath,
-    testFrenchPath,
-    Locales.FRENCH,
-    Locales.ENGLISH
-  );
-  // Verify output structure, reuse rate, etc.
-});
-```
-
-## Future Enhancements
-
-- [ ] Support for more block types (footnotes, definition lists)
-- [ ] Fuzzy matching for minor structural variations
-- [ ] Parallel AI translation for multiple segments
-- [ ] Caching of fingerprints for repeated processing
-- [ ] Visual diff tool for reviewing alignment decisions
-- [ ] Machine learning for optimal threshold selection
-
-## Contributing
-
-When contributing to this system:
-
-1. **Follow naming conventions**: No abbreviations, descriptive names
-2. **Use arrow functions**: Consistent function syntax
-3. **Add JSDoc comments**: Document complex logic
-4. **Write tests**: Cover edge cases
-5. **Maintain backwards compatibility**: Ensure existing integrations work
-
-## License
-
-See main project LICENSE file.
-