@llm-translate/cli 1.0.0-next.1

package/docs/guide/chunking.md

@@ -0,0 +1,272 @@
+# Chunking Strategy
+
+::: info Translations
+All non-English documentation is automatically translated using Claude Sonnet 4.
+:::
+
+Large documents are split into chunks for translation. Understanding chunking helps optimize quality and cost.
+
+## Why Chunking?
+
+LLMs have context limits and perform better on focused content:
+
+| Reason | Description |
+|--------|-------------|
+| **Context limits** | Models have maximum input sizes |
+| **Quality** | Smaller chunks get more focused attention |
+| **Cost** | Allows caching of repeated content |
+| **Progress** | Enables progress tracking and resumption |
+
+## Default Configuration
+
+```json
+{
+  "chunking": {
+    "maxTokens": 1024,
+    "overlapTokens": 150
+  }
+}
+```
+
+## Chunk Size Options
+
+### maxTokens
+
+Maximum tokens per chunk (excluding prompt overhead).
+
+| Size | Best For | Trade-off |
+|------|----------|-----------|
+| 512 | High quality requirements | More API calls |
+| **1024** | General use (default) | Balanced |
+| 2048 | Cost optimization | May reduce quality |
+
+### overlapTokens
+
+Context from previous chunk ensures continuity across boundaries.
+
+```
+Chunk 1: [Content A                    ]
+Chunk 2:            [overlap][Content B                    ]
+Chunk 3:                              [overlap][Content C  ]
+```
+
+::: tip Recommended Overlap
+Use 10-15% of your `maxTokens` value. For 1024 tokens, 100-150 overlap tokens work well.
+:::
+
+## Markdown-Aware Chunking
+
+llm-translate uses AST-based chunking that respects document structure.
+
+### Preserved Boundaries
+
+The chunker never splits these elements:
+
+| Element | Behavior |
+|---------|----------|
+| Headers | Section boundaries preserved |
+| Code blocks | Always kept intact |
+| Lists | Items grouped when possible |
+| Tables | Never split across chunks |
+| Paragraphs | Split at natural boundaries |
+
+### Example
+
+::: details Click to see chunking example
+
+**Input document:**
+
+```markdown
+# Introduction
+
+This is the introduction paragraph that explains
+the purpose of the document.
+
+## Getting Started
+
+### Prerequisites
+
+- Node.js 20+
+- npm or yarn
+
+### Installation
+
+npm install @llm-translate/cli
+```
+
+**Result:**
+
+```
+Chunk 1: # Introduction + paragraph
+Chunk 2: ## Getting Started + ### Prerequisites + list
+Chunk 3: ### Installation + code block
+```
+
+:::
+
+## Configuration
+
+::: code-group
+
+```bash [CLI]
+llm-translate file doc.md --target ko --chunk-size 2048
+```
+
+```json [.translaterc.json]
+{
+  "chunking": {
+    "maxTokens": 2048,
+    "overlapTokens": 200,
+    "preservePatterns": [
+      "```[\\s\\S]*?```",
+      "\\|[^\\n]+\\|"
+    ]
+  }
+}
+```
+
+```typescript [Programmatic]
+import { chunkContent } from '@llm-translate/cli';
+
+const chunks = chunkContent(content, {
+  maxTokens: 1024,
+  overlapTokens: 150,
+});
+```
+
+:::
+
+## Optimization Presets
+
+Choose based on your priority:
+
+::: code-group
+
+```json [Quality Focus]
+{
+  "chunking": {
+    "maxTokens": 512,
+    "overlapTokens": 100
+  }
+}
+```
+
+```json [Cost Focus]
+{
+  "chunking": {
+    "maxTokens": 2048,
+    "overlapTokens": 50
+  }
+}
+```
+
+```json [Long Documents]
+{
+  "chunking": {
+    "maxTokens": 1500,
+    "overlapTokens": 150
+  },
+  "translation": {
+    "maxIterations": 3
+  }
+}
+```
+
+:::
+
+::: info When to use each preset
+- **Quality Focus**: Technical documentation, legal content
+- **Cost Focus**: Blog posts, general content
+- **Long Documents**: Books, comprehensive guides
+:::
+
+## Content Preservation
+
+### What Gets Protected
+
+llm-translate automatically protects certain content from translation:
+
+| Content Type | Example | Behavior |
+|--------------|---------|----------|
+| Code blocks | ` ```js ... ``` ` | Never translated |
+| Inline code | `` `variable` `` | Preserved |
+| URLs | `https://...` | Preserved |
+| File paths | `./path/to/file` | Preserved |
+
+### Link Handling
+
+Link URLs are preserved, but link text is translated:
+
+```markdown
+[Getting Started](./getting-started.md)
+↓
+[시작하기](./getting-started.md)
+```
+
+## Debugging
+
+### Preview Chunks
+
+Use `--dry-run` to see how your document will be chunked:
+
+```bash
+llm-translate file doc.md --target ko --dry-run --verbose
+```
+
+Output:
+```
+Document Analysis:
+  Total tokens: ~5,200
+  Chunks: 6
+  Average chunk size: ~867 tokens
+
+Chunk breakdown:
+  [1] Lines 1-45 (Introduction) - 823 tokens
+  [2] Lines 46-89 (Getting Started) - 912 tokens
+  [3] Lines 90-134 (Configuration) - 878 tokens
+  ...
+```
+
+### Programmatic Inspection
+
+```typescript
+import { chunkContent, getChunkStats } from '@llm-translate/cli';
+
+const chunks = chunkContent(content, { maxTokens: 1024 });
+const stats = getChunkStats(chunks);
+
+console.log(`Total chunks: ${stats.count}`);
+console.log(`Average size: ${stats.avgTokens} tokens`);
+```
+
+## Troubleshooting
+
+::: warning Chunks Too Small
+**Symptom**: Many small chunks, excessive API calls
+
+**Solution**: Increase `maxTokens`
+```json
+{ "chunking": { "maxTokens": 2048 } }
+```
+:::
+
+::: warning Lost Context Between Chunks
+**Symptom**: Inconsistent terminology across sections
+
+**Solution**: Increase overlap or use glossary
+```json
+{ "chunking": { "overlapTokens": 300 } }
+```
+:::
+
+::: danger Code Blocks Being Split
+**Symptom**: Syntax errors in output
+
+**Cause**: This should never happen. If it does, please [report an issue](https://github.com/selenehyun/llm-translate/issues).
+:::
+
+::: warning Tables Being Mangled
+**Symptom**: Broken table formatting
+
+**Solution**: Tables should be kept intact automatically. For very large tables (100+ rows), consider splitting them manually.
+:::

package/docs/guide/configuration.md

@@ -0,0 +1,139 @@
+# Configuration
+
+::: info Translations
+All non-English documentation is automatically translated using Claude Sonnet 4.
+:::
+
+llm-translate uses a layered configuration system. Settings are applied in this order (later overrides earlier):
+
+1. Built-in defaults
+2. Configuration file (`.translaterc.json`)
+3. Environment variables
+4. CLI arguments
+
+## Configuration File
+
+Create `.translaterc.json` in your project root:
+
+```json
+{
+  "provider": {
+    "name": "claude",
+    "model": "claude-haiku-4-5-20251001",
+    "apiKey": null
+  },
+  "translation": {
+    "qualityThreshold": 85,
+    "maxIterations": 4,
+    "preserveFormatting": true
+  },
+  "chunking": {
+    "maxTokens": 1024,
+    "overlapTokens": 150
+  },
+  "paths": {
+    "glossary": "./glossary.json",
+    "cache": "./.translate-cache"
+  }
+}
+```
+
+### Provider Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `name` | string | `"claude"` | Provider name: `claude`, `openai`, `ollama` |
+| `model` | string | varies | Model identifier |
+| `apiKey` | string | null | API key (prefer env var) |
+| `baseUrl` | string | null | Custom API endpoint |
+
+### Translation Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `qualityThreshold` | number | `85` | Minimum quality score (0-100) |
+| `maxIterations` | number | `4` | Maximum refinement iterations |
+| `preserveFormatting` | boolean | `true` | Preserve markdown/HTML structure |
+
+### Chunking Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `maxTokens` | number | `1024` | Maximum tokens per chunk |
+| `overlapTokens` | number | `150` | Context overlap between chunks |
+
+### Path Settings
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `glossary` | string | null | Path to glossary file |
+| `cache` | string | null | Path to translation cache |
+
+## Environment Variables
+
+```bash
+# API Keys
+ANTHROPIC_API_KEY=sk-ant-xxxxx
+OPENAI_API_KEY=sk-xxxxx
+OLLAMA_BASE_URL=http://localhost:11434
+
+# Default Settings
+LLM_TRANSLATE_PROVIDER=claude
+LLM_TRANSLATE_MODEL=claude-haiku-4-5-20251001
+LLM_TRANSLATE_QUALITY_THRESHOLD=85
+```
+
+## CLI Override Examples
+
+```bash
+# Override provider
+llm-translate file doc.md -o doc.ko.md --target ko --provider openai
+
+# Override model
+llm-translate file doc.md -o doc.ko.md --target ko --model claude-sonnet-4-5-20250929
+
+# Override quality threshold
+llm-translate file doc.md -o doc.ko.md --target ko --quality 90
+
+# Override max iterations
+llm-translate file doc.md -o doc.ko.md --target ko --max-iterations 6
+```
+
+## Per-Project Configuration
+
+For monorepos or multi-project setups, place `.translaterc.json` in each project directory:
+
+```
+my-monorepo/
+├── packages/
+│   ├── frontend/
+│   │   ├── .translaterc.json  # Frontend-specific terms
+│   │   └── docs/
+│   └── backend/
+│       ├── .translaterc.json  # Backend-specific terms
+│       └── docs/
+└── .translaterc.json          # Shared defaults
+```
+
+llm-translate searches for config files from the current directory upward.
+
+## Model Selection Guide
+
+| Model | Speed | Quality | Cost | Best For |
+|-------|-------|---------|------|----------|
+| `claude-haiku-4-5-20251001` | Fast | Good | Low | General docs, high volume |
+| `claude-sonnet-4-5-20250929` | Medium | Excellent | Medium | Technical docs, quality-critical |
+| `claude-opus-4-5-20251101` | Slow | Best | High | Complex content, nuanced text |
+| `gpt-4o-mini` | Fast | Good | Low | Alternative to Haiku |
+| `gpt-4o` | Medium | Excellent | Medium | Alternative to Sonnet |
+
+## Quality Threshold Guidelines
+
+| Threshold | Use Case |
+|-----------|----------|
+| 70-75 | Draft translations, internal docs |
+| 80-85 | Standard documentation (default) |
+| 90-95 | Public-facing, marketing content |
+| 95+ | Legal, medical, regulated content |
+
+Higher thresholds require more iterations and cost more.

package/docs/guide/cost-optimization.md

@@ -0,0 +1,237 @@
+# Cost Optimization
+
+::: info Translations
+All non-English documentation is automatically translated using Claude Sonnet 4.
+:::
+
+This guide covers strategies to minimize API costs while maintaining translation quality.
+
+## Cost Structure
+
+### Token Pricing (as of 2025)
+
+| Model | Input (1K) | Output (1K) | Cache Read | Cache Write |
+|-------|-----------|-------------|------------|-------------|
+| Claude Haiku 4.5 | $0.001 | $0.005 | $0.0001 | $0.00125 |
+| Claude Sonnet 4.5 | $0.003 | $0.015 | $0.0003 | $0.00375 |
+| Claude Opus 4.5 | $0.015 | $0.075 | $0.0015 | $0.01875 |
+| GPT-4o-mini | $0.00015 | $0.0006 | Auto | Auto |
+| GPT-4o | $0.0025 | $0.01 | Auto | Auto |
+
+### Cost Factors
+
+1. **Input tokens**: Source text + glossary + prompts
+2. **Output tokens**: Translated text
+3. **Iterations**: Quality refinement cycles (multiply by iterations)
+4. **Cache efficiency**: Savings from prompt caching
+
+## Optimization Strategies
+
+### 1. Choose the Right Model
+
+```bash
+# Most cost-effective for standard docs
+llm-translate file doc.md --model claude-haiku-4-5-20251001
+
+# Better quality when needed
+llm-translate file important.md --model claude-sonnet-4-5-20250929
+```
+
+**Model selection guide:**
+
+| Content Type | Recommended Model |
+|--------------|-------------------|
+| README, guides | Haiku |
+| API reference | Haiku |
+| User-facing docs | Sonnet |
+| Marketing content | Sonnet/Opus |
+| Legal/compliance | Opus |
+
+### 2. Optimize Quality Settings
+
+Lower threshold = fewer iterations = lower cost
+
+```bash
+# Draft quality (faster, cheaper)
+llm-translate file doc.md --quality 70 --max-iterations 2
+
+# Standard quality
+llm-translate file doc.md --quality 85 --max-iterations 4
+
+# High quality (slower, more expensive)
+llm-translate file doc.md --quality 95 --max-iterations 6
+```
+
+**Cost impact:**
+
+| Setting | Avg Iterations | Relative Cost |
+|---------|---------------|---------------|
+| quality=70, iter=2 | 1.5 | 0.5x |
+| quality=85, iter=4 | 2.5 | 1.0x |
+| quality=95, iter=6 | 4.0 | 1.6x |
+
+### 3. Maximize Prompt Caching
+
+Enable caching and process files in batches:
+
+```bash
+# Process all files together to share cache
+llm-translate dir ./docs ./docs-ko --target ko
+
+# Not: Process one file at a time
+```
+
+See [Prompt Caching](./prompt-caching) for details.
+
+### 4. Optimize Glossary Size
+
+Large glossaries increase costs. Keep only necessary terms:
+
+```bash
+# Check glossary token count
+llm-translate glossary stats --glossary glossary.json
+```
+
+**Best practices:**
+- Remove unused terms regularly
+- Use `doNotTranslate` sparingly
+- Split large glossaries by domain
+
+### 5. Chunk Size Optimization
+
+Larger chunks = fewer API calls = lower overhead
+
+```json
+{
+  "chunking": {
+    "maxTokens": 2048,
+    "overlapTokens": 200
+  }
+}
+```
+
+**Trade-offs:**
+
+| Chunk Size | API Calls | Quality | Cost |
+|------------|-----------|---------|------|
+| 512 tokens | Many | Higher | Higher |
+| 1024 tokens | Medium | Good | Medium |
+| 2048 tokens | Fewer | Acceptable | Lower |
+
+### 6. Use Translation Cache
+
+Cache translated chunks to avoid re-translating:
+
+```json
+{
+  "paths": {
+    "cache": "./.translate-cache"
+  }
+}
+```
+
+Benefits:
+- Skip unchanged content on re-runs
+- Reduce costs for incremental updates
+- Faster subsequent translations
+
+## Cost Estimation
+
+### Before Translation
+
+```bash
+llm-translate estimate doc.md --target ko --glossary glossary.json
+```
+
+Output:
+```
+Estimated costs for doc.md:
+  Chunks: 15
+  Input tokens: ~18,000
+  Output tokens: ~20,000 (estimated)
+  Iterations: 2-3 (estimated)
+
+  Model: claude-haiku-4-5-20251001
+  Without caching: $0.12 - $0.18
+  With caching: $0.05 - $0.08 (55-60% savings)
+```
+
+### After Translation
+
+```bash
+llm-translate file doc.md -o doc.ko.md --target ko --verbose
+```
+
+Output:
+```
+✓ Translation complete
+  Tokens: 18,234 input / 21,456 output
+  Cache: 12,000 read / 3,000 written
+  Cost: $0.067 (estimated)
+```
+
+## Batch Processing Economics
+
+### Single File vs Batch
+
+| Approach | Cache Efficiency | Total Cost |
+|----------|-----------------|------------|
+| 10 files sequentially | 0% | $1.00 |
+| 10 files with caching | 80% | $0.35 |
+| Batch directory | 85% | $0.30 |
+
+### Optimal Batch Size
+
+```bash
+# Process in batches of 20-50 files for best cache utilization
+llm-translate dir ./docs ./docs-ko --target ko --concurrency 5
+```
+
+## Cost Monitoring
+
+### Per-Project Tracking
+
+Create a cost log:
+
+```bash
+llm-translate file doc.md --cost-log ./costs.json
+```
+
+### Cost Alerts
+
+Set budget limits:
+
+```json
+{
+  "budget": {
+    "maxCostPerFile": 0.50,
+    "maxCostPerRun": 10.00,
+    "warnAt": 0.80
+  }
+}
+```
+
+## Cost Comparison by Language
+
+Output varies by target language:
+
+| Target | Relative Output Tokens |
+|--------|----------------------|
+| Korean | 0.9-1.1x source |
+| Japanese | 0.8-1.0x source |
+| Chinese | 0.7-0.9x source |
+| German | 1.1-1.3x source |
+| Spanish | 1.1-1.2x source |
+
+Factor this into cost estimates.
+
+## Summary: Quick Cost Reduction Checklist
+
+- [ ] Use Haiku for standard documentation
+- [ ] Set quality threshold appropriately (not higher than needed)
+- [ ] Enable and maximize prompt caching
+- [ ] Process files in batches
+- [ ] Keep glossary lean
+- [ ] Use translation cache for incremental updates
+- [ ] Monitor costs with verbose output
+- [ ] Estimate before large jobs