@intlayer/cli 7.0.6 → 7.0.8-canary.0

package/dist/assets/translation-alignment/QUICK_START.md

@@ -0,0 +1,494 @@
+# Quick Start Guide
+
+Get started with the block-aware translation alignment system in 5 minutes.
+
+## Installation
+
+The system is already integrated into your CLI package. No additional dependencies needed.
+
+## Basic Usage
+
+### 1. Simple Translation Review
+
+```typescript
+import { reviewFileBlockAware } from './reviewDocBlockAware';
+import { Locales } from '@intlayer/types';
+
+await reviewFileBlockAware(
+  'docs/en/getting-started.md',   // Source file
+  'docs/fr/getting-started.md',   // Target file
+  Locales.FRENCH,                 // Target locale
+  Locales.ENGLISH                 // Source locale
+);
+```
+
+### 2. With Git Integration
+
+```typescript
+import { listGitLines } from '@intlayer/chokidar';
+
+const changedLines = await listGitLines(
+  'docs/en/getting-started.md',
+  { sinceDays: 7 }  // Changes in last 7 days
+);
+
+await reviewFileBlockAware(
+  'docs/en/getting-started.md',
+  'docs/fr/getting-started.md',
+  Locales.FRENCH,
+  Locales.ENGLISH,
+  aiOptions,
+  configOptions,
+  undefined,
+  changedLines  // Only review changed blocks!
+);
+```
+
+### 3. With Custom AI Instructions
+
+```typescript
+const customInstructions = `
+- Use formal French (vous, not tu)
+- Preserve technical terminology in English
+- Maintain consistent tone
+`;
+
+await reviewFileBlockAware(
+  'docs/en/api.md',
+  'docs/fr/api.md',
+  Locales.FRENCH,
+  Locales.ENGLISH,
+  aiOptions,
+  configOptions,
+  customInstructions
+);
+```
+
+## Using the Pipeline Directly
+
+For more control, use the pipeline API:
+
+```typescript
+import { buildAlignmentPlan, mergeReviewedSegments } from './translation-alignment/pipeline';
+
+// 1. Build alignment plan
+const { englishBlocks, frenchBlocks, plan, segmentsToReview } = buildAlignmentPlan({
+  englishText: await readFile('docs/en/guide.md', 'utf-8'),
+  frenchText: await readFile('docs/fr/guide.md', 'utf-8'),
+  changedLines: [10, 15, 20],
+  similarityOptions: {
+    minimumMatchForReuse: 0.92,
+  },
+});
+
+// 2. Log plan details
+console.log(`Blocks to reuse: ${plan.actions.filter(a => a.kind === 'reuse').length}`);
+console.log(`Blocks to review: ${segmentsToReview.length}`);
+
+// 3. Translate segments (your AI logic here)
+const reviewedMap = new Map();
+for (const segment of segmentsToReview) {
+  const translation = await yourTranslationFunction(segment.englishBlock.content);
+  reviewedMap.set(segment.actionIndex, translation);
+}
+
+// 4. Merge and output
+const finalOutput = mergeReviewedSegments(plan, frenchBlocks, reviewedMap);
+await writeFile('docs/fr/guide.md', finalOutput);
+```
+
+## Common Scenarios
+
+### Scenario 1: New Document (No Translation Exists)
+
+```typescript
+// French file doesn't exist yet
+await reviewFileBlockAware(
+  'docs/en/new-feature.md',
+  'docs/fr/new-feature.md',  // Will be created
+  Locales.FRENCH,
+  Locales.ENGLISH
+);
+
+// Result: All blocks translated from scratch
+```
+
+### Scenario 2: Minor Edit (2 paragraphs changed)
+
+```typescript
+// Git shows lines 45-60 changed
+const changedLines = [45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60];
+
+await reviewFileBlockAware(
+  'docs/en/guide.md',
+  'docs/fr/guide.md',
+  Locales.FRENCH,
+  Locales.ENGLISH,
+  aiOptions,
+  configOptions,
+  undefined,
+  changedLines
+);
+
+// Result: Only 2 blocks sent to AI, 98% reused
+```
+
+### Scenario 3: Reordered Sections
+
+```markdown
+<!-- Before -->
+## Installation
+## Configuration
+## Usage
+
+<!-- After -->
+## Installation
+## Usage
+## Configuration
+```
+
+```typescript
+await reviewFileBlockAware(
+  'docs/en/guide.md',
+  'docs/fr/guide.md',
+  Locales.FRENCH,
+  Locales.ENGLISH
+);
+
+// Result: Sections automatically reordered, 0 AI calls needed
+```
+
+### Scenario 4: Added Section
+
+```markdown
+<!-- Before -->
+## Installation
+## Usage
+
+<!-- After -->
+## Installation
+## Configuration  ← NEW
+## Usage
+```
+
+```typescript
+await reviewFileBlockAware(
+  'docs/en/guide.md',
+  'docs/fr/guide.md',
+  Locales.FRENCH,
+  Locales.ENGLISH
+);
+
+// Result: 
+// - Installation: reused
+// - Configuration: translated (1 AI call)
+// - Usage: reused
+```
+
+### Scenario 5: Deleted Section
+
+```markdown
+<!-- Before -->
+## Installation
+## Deprecated  ← DELETE
+## Usage
+
+<!-- After -->
+## Installation
+## Usage
+```
+
+```typescript
+await reviewFileBlockAware(
+  'docs/en/guide.md',
+  'docs/fr/guide.md',
+  Locales.FRENCH,
+  Locales.ENGLISH
+);
+
+// Result: 
+// - Installation: reused
+// - Deprecated: deleted from output
+// - Usage: reused
+```
+
+## Batch Processing
+
+Process multiple files:
+
+```typescript
+import fg from 'fast-glob';
+
+const englishFiles = await fg('docs/en/**/*.md');
+
+for (const englishFile of englishFiles) {
+  const frenchFile = englishFile.replace('/en/', '/fr/');
+  
+  await reviewFileBlockAware(
+    englishFile,
+    frenchFile,
+    Locales.FRENCH,
+    Locales.ENGLISH,
+    aiOptions,
+    configOptions
+  );
+}
+```
+
+## Monitoring & Debugging
+
+### Enable Verbose Logging
+
+```typescript
+import { getConfiguration, getAppLogger } from '@intlayer/config';
+
+const configuration = getConfiguration();
+const logger = getAppLogger(configuration);
+
+// Logger will output:
+// - Block segmentation results
+// - Alignment statistics
+// - Action plan details
+// - Token usage per block
+// - Efficiency metrics
+```
+
+### Inspect Alignment Plan
+
+```typescript
+const { plan, segmentsToReview } = buildAlignmentPlan({
+  englishText,
+  frenchText,
+  changedLines,
+});
+
+// Analyze plan
+plan.actions.forEach((action, index) => {
+  console.log(`Action ${index}: ${action.kind}`);
+  if (action.kind === 'reuse') {
+    console.log(`  English block ${action.englishIndex} → French block ${action.frenchIndex}`);
+  } else if (action.kind === 'review') {
+    console.log(`  Reviewing English block ${action.englishIndex}`);
+  }
+});
+```
+
+### Debug Block Segmentation
+
+```typescript
+import { segmentDocument } from './translation-alignment/segmentDocument';
+
+const blocks = segmentDocument(englishText);
+
+blocks.forEach((block, index) => {
+  console.log(`Block ${index}:`);
+  console.log(`  Type: ${block.type}`);
+  console.log(`  Lines: ${block.lineStart}-${block.lineEnd}`);
+  console.log(`  Content preview: ${block.content.slice(0, 50)}...`);
+});
+```
+
+### Analyze Similarity Scores
+
+```typescript
+import { computeJaccardSimilarity } from './translation-alignment/computeSimilarity';
+
+const similarity = computeJaccardSimilarity(
+  "Hello [World](https://example.com)",
+  "Bonjour [Monde](https://example.com)",
+  3
+);
+
+console.log(`Similarity: ${(similarity * 100).toFixed(1)}%`);
+// Output: Similarity: 75.3%
+```
+
+## Configuration Tuning
+
+### Adjust Similarity Thresholds
+
+```typescript
+const { plan } = buildAlignmentPlan({
+  englishText,
+  frenchText,
+  changedLines,
+  similarityOptions: {
+    // Higher = more conservative (more blocks reviewed)
+    minimumMatchForReuse: 0.95,
+    
+    // For future near-duplicate detection
+    minimumMatchForNearDuplicate: 0.85,
+  },
+});
+```
+
+### Finding Optimal Thresholds
+
+Run experiments:
+
+```typescript
+const thresholds = [0.85, 0.87, 0.90, 0.92, 0.95, 0.97];
+const results = [];
+
+for (const threshold of thresholds) {
+  const { plan } = buildAlignmentPlan({
+    englishText,
+    frenchText,
+    changedLines,
+    similarityOptions: { minimumMatchForReuse: threshold },
+  });
+  
+  const reusedCount = plan.actions.filter(a => a.kind === 'reuse').length;
+  const reviewCount = plan.actions.filter(a => a.kind === 'review').length;
+  
+  results.push({
+    threshold,
+    reused: reusedCount,
+    reviewed: reviewCount,
+    efficiency: (reusedCount / plan.actions.length * 100).toFixed(1) + '%',
+  });
+}
+
+console.table(results);
+```
+
+Example output:
+
+```
+┌─────────┬───────────┬────────┬──────────┬────────────┐
+│ (index) │ threshold │ reused │ reviewed │ efficiency │
+├─────────┼───────────┼────────┼──────────┼────────────┤
+│    0    │   0.85    │   92   │    8     │  '92.0%'   │
+│    1    │   0.87    │   90   │    10    │  '90.0%'   │
+│    2    │   0.90    │   88   │    12    │  '88.0%'   │
+│    3    │   0.92    │   85   │    15    │  '85.0%'   │
+│    4    │   0.95    │   80   │    20    │  '80.0%'   │
+│    5    │   0.97    │   72   │    28    │  '72.0%'   │
+└─────────┴───────────┴────────┴──────────┴────────────┘
+```
+
+Choose based on your priorities:
+- **High accuracy**: Use 0.95+ (more AI reviews)
+- **High efficiency**: Use 0.85-0.90 (fewer AI reviews)
+- **Balanced**: Use 0.90-0.92 (default)
+
+## Testing
+
+### Unit Test Example
+
+```typescript
+import { segmentDocument } from './translation-alignment/segmentDocument';
+
+describe('Block Segmentation', () => {
+  it('should segment markdown correctly', () => {
+    const input = '# Title\n\nParagraph 1\n\n## Subtitle\n\nParagraph 2\n';
+    const blocks = segmentDocument(input);
+    
+    expect(blocks).toHaveLength(4);
+    expect(blocks[0].type).toBe('heading');
+    expect(blocks[0].content).toBe('# Title\n');
+    expect(blocks[1].type).toBe('paragraph');
+    expect(blocks[2].type).toBe('heading');
+    expect(blocks[3].type).toBe('paragraph');
+  });
+});
+```
+
+### Integration Test Example
+
+```typescript
+import { buildAlignmentPlan } from './translation-alignment/pipeline';
+
+describe('Reordering Detection', () => {
+  it('should detect reordered paragraphs', () => {
+    const englishV1 = 'Para A\n\nPara B\n\nPara C\n';
+    const englishV2 = 'Para B\n\nPara A\n\nPara C\n';
+    const frenchV1 = 'Para A (fr)\n\nPara B (fr)\n\nPara C (fr)\n';
+    
+    const { plan, segmentsToReview } = buildAlignmentPlan({
+      englishText: englishV2,
+      frenchText: frenchV1,
+      changedLines: undefined,
+    });
+    
+    // All blocks should be reused (just reordered)
+    expect(segmentsToReview).toHaveLength(0);
+    const reusedCount = plan.actions.filter(a => a.kind === 'reuse').length;
+    expect(reusedCount).toBe(3);
+  });
+});
+```
+
+## Troubleshooting
+
+### Problem: Too Many Blocks Marked for Review
+
+```typescript
+// Check if threshold is too high
+const { plan } = buildAlignmentPlan({
+  englishText,
+  frenchText,
+  changedLines,
+  similarityOptions: { minimumMatchForReuse: 0.85 }, // Lower threshold
+});
+```
+
+### Problem: Blocks Not Aligning Correctly
+
+```typescript
+// Debug anchor text extraction
+import { normalizeBlock } from './translation-alignment/normalizeBlock';
+import { segmentDocument } from './translation-alignment/segmentDocument';
+
+const blocks = segmentDocument(englishText);
+const normalized = blocks.map(normalizeBlock);
+
+normalized.forEach((block, index) => {
+  console.log(`Block ${index}:`);
+  console.log(`  Semantic: ${block.semanticText.slice(0, 50)}`);
+  console.log(`  Anchor: ${block.anchorText}`);
+});
+```
+
+### Problem: Output Has Extra Blank Lines
+
+```typescript
+// Check block content preservation
+const { englishBlocks } = buildAlignmentPlan({
+  englishText,
+  frenchText,
+  changedLines,
+});
+
+englishBlocks.forEach((block, index) => {
+  const hasTrailingNewline = block.content.endsWith('\n');
+  console.log(`Block ${index}: trailing newline = ${hasTrailingNewline}`);
+});
+```
+
+## Performance Tips
+
+1. **Always use Git integration** when available for maximum efficiency
+2. **Batch process during off-peak hours** to reduce API rate limit issues
+3. **Cache fingerprints** for documents that don't change often
+4. **Use parallel processing** for multiple locales (built into `reviewDoc`)
+5. **Monitor token usage** and adjust thresholds if costs are too high
+
+## Next Steps
+
+- Read [README.md](./README.md) for detailed architecture
+- Review [IMPROVEMENTS.md](./IMPROVEMENTS.md) for comparison with old system
+- Follow [INTEGRATION_EXAMPLE.md](./INTEGRATION_EXAMPLE.md) for production deployment
+- Check [TESTING.md](./TESTING.md) for comprehensive test examples
+
+## Support
+
+For issues or questions:
+1. Check logs for alignment details
+2. Verify input markdown is valid
+3. Test with simplified example
+4. Review similarity scores
+5. Open an issue with sample files
+
+Happy translating! 🌍
+