omni-localizer 0.1.0__tar.gz

omni_localizer-0.1.0/AGENT_USAGE.md

@@ -0,0 +1,312 @@
+# Omni-Localizer Agent Usage Guide
+
+## Overview
+
+Omni-Localizer is an AI-native localization pipeline that translates Markdown documents. It now supports agent integration via SKILL.md files for OpenCode and Hermes.
+
+---
+
+## Quick Start
+
+### For Agents
+
+1. **Discover the skill**
+   ```
+   Look for: src/.opencode/skills/ol-localizer/SKILL.md
+   Look for: src/.hermes/skills/ol-localizer/SKILL.md
+   ```
+
+2. **Read the SKILL.md** for instructions on how to invoke
+
+3. **Configure API keys** in environment:
+   ```
+   export OPENAI_API_KEY=sk-...
+   export MINIMAX_API_KEY=...    # if using MiniMax
+   export BAIDU_API_KEY=...    # if using Baidu
+   ```
+
+4. **Invoke via CLI**:
+   ```
+   python -m ol_cli translate-md <file.md> -c config/default.yaml -s en -t zh -o output/ --json
+   ```
+
+---
+
+## CLI Commands
+
+### Basic Translation
+
+```bash
+# Translate Markdown (human-readable output)
+python -m ol_cli translate-md input.md -c config/default.yaml -s en -t zh -o output/
+
+# Translate with JSON output (for agents)
+python -m ol_cli translate-md input.md -c config/default.yaml -s en -t zh -o output/ --json
+
+# Translate XLIFF
+python -m ol_cli translate-xliff input.xlf -c config/default.yaml -s en -t zh -o output/
+```
+
+### JSON Output Format
+
+When `--json` is used, output is:
+```json
+{
+  "success": true,
+  "input_file": "input.md",
+  "output_file": "output/input.md",
+  "source_lang": "en",
+  "target_lang": "zh"
+}
+```
+
+On error:
+```json
+{
+  "success": false,
+  "error": "Error message here"
+}
+```
+
+---
+
+## Configuration
+
+### Default Config Location
+`config/default.yaml`
+
+### Environment Variables
+Set in shell before running:
+```bash
+export OPENAI_API_KEY=sk-your-key
+export MINIMAX_API_KEY=your-minimax-key  # optional
+export BAIDU_API_KEY=your-baidu-key    # optional
+```
+
+### Config Structure
+```yaml
+project_id: "your-project"
+source_lang: "en"
+target_lang: "zh"
+llm_pool:
+  translation:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "translation"
+  judging:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "judging"
+  restoration:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "restoration"
+```
+
+---
+
+## Agent Integration
+
+### OpenCode
+
+1. Copy skill to project:
+   ```bash
+   cp -r src/.opencode/skills/ol-localizer <your-project>/.opencode/skills/
+   ```
+
+2. Read `SKILL.md` in that directory for detailed instructions
+
+### Hermes
+
+1. Copy skill to Hermes:
+   ```bash
+   cp -r src/.hermes/skills/ol-localizer ~/.hermes/skills/
+   ```
+
+2. Restart Hermes to activate
+
+---
+
+## Testing
+
+### Run All Skill Tests
+```bash
+pytest tests/test_opencode_skill.py tests/test_hermes_skill.py tests/test_skill_invocation.py -v
+```
+
+### Test Individual Skill
+```bash
+pytest tests/test_opencode_skill.py -v
+pytest tests/test_hermes_skill.py -v
+```
+
+### Verify JSON Output
+```bash
+python -m ol_cli translate-md nonexistent.md -o /tmp/out --json
+```
+
+Expected: JSON error output
+
+---
+
+## Common Scenarios
+
+### Scenario 1: Agent Needs to Translate a File
+
+**Agent action:**
+1. Read SKILL.md for instructions
+2. Check if API keys are set
+3. Write source text to temp .md file
+4. Run CLI with --json flag
+5. Parse JSON response
+6. Read translated file from output directory
+
+**Example invocation:**
+```bash
+# Agent writes temp file
+echo "# Hello" > /tmp/test.md
+
+# Agent runs translation
+python -m ol_cli translate-md /tmp/test.md -c config/default.yaml -s en -t zh -o /tmp/ --json
+
+# Agent parses JSON output
+# Reads /tmp/test.md for translated content
+```
+
+### Scenario 2: Agent Wants to Verify Skill is Available
+
+**Agent action:**
+```bash
+ls src/.opencode/skills/ol-localizer/SKILL.md
+ls src/.hermes/skills/ol-localizer/SKILL.md
+```
+
+### Scenario 3: Agent Needs to Configure API Keys
+
+**Agent action:**
+1. Read SKILL.md Configuration section
+2. Set required environment variables:
+```bash
+export OPENAI_API_KEY=sk-...
+```
+
+### Scenario 4: Translation Fails
+
+**Agent checks:**
+1. JSON output for error message:
+```bash
+python -m ol_cli translate-md input.md -c config/default.yaml -s en -t zh -o output/ --json 2>/dev/null
+```
+
+2. Common fixes:
+   - Missing API key → Set `OPENAI_API_KEY`
+   - Invalid config → Check `config/default.yaml` exists
+   - File not found → Verify input path
+   - Rate limit → Wait and retry
+
+### Scenario 5: Agent Wants to Use Different LLM Provider
+
+**Agent action:**
+1. Modify `config/default.yaml`:
+```yaml
+llm_pool:
+  translation:
+    - provider: "openai"        # or "anthropic", "deepseek", etc.
+      model: "gpt-4o"          # or "claude-3-sonnet", etc.
+      api_key: "${PROVIDER_API_KEY}"
+```
+
+2. Set appropriate API key:
+```bash
+export PROVIDER_API_KEY=sk-...
+```
+
+---
+
+## Troubleshooting
+
+### "No module named ol_cli"
+
+**Cause:** Running without `PYTHONPATH=src`
+
+**Fix:**
+```bash
+PYTHONPATH=src python -m ol_cli translate-md ...
+```
+
+### JSON Output Not Valid
+
+**Cause:** Error occurred before JSON could be generated
+
+**Fix:** Check stderr for error message, fix issue, retry
+
+### Tests Failing
+
+**Agent action:**
+```bash
+# Run specific failing test
+pytest tests/test_opencode_skill.py::TestOpenCodeSkill::test_opencode_skill_exists -v
+
+# Run all tests
+pytest tests/test_opencode_skill.py tests/test_hermes_skill.py -v
+```
+
+### Skill Not Discovered by Agent
+
+**Agent checks:**
+1. File exists:
+```bash
+ls src/.opencode/skills/ol-localizer/SKILL.md
+```
+
+2. YAML frontmatter valid:
+```bash
+python -c "import yaml; yaml.safe_load(open('src/.opencode/skills/ol-localizer/SKILL.md').read().split('---')[1])"
+```
+
+3. Required fields present:
+```bash
+grep -q "name:" src/.opencode/skills/ol-localizer/SKILL.md
+grep -q "description:" src/.opencode/skills/ol-localizer/SKILL.md
+```
+
+---
+
+## File Locations
+
+| Component | Path |
+|-----------|------|
+| OpenCode Skill | `src/.opencode/skills/ol-localizer/SKILL.md` |
+| Hermes Skill | `src/.hermes/skills/ol-localizer/SKILL.md` |
+| CLI Entry | `src/ol_cli.py` |
+| Default Config | `config/default.yaml` |
+| Test Helpers | `tests/skill_helpers.py` |
+| OpenCode Tests | `tests/test_opencode_skill.py` |
+| Hermes Tests | `tests/test_hermes_skill.py` |
+| Invocation Tests | `tests/test_skill_invocation.py` |
+
+---
+
+## Key Design Decisions
+
+1. **SKILL.md format** - Universal skill format supported by OpenCode and Hermes
+2. **JSON output** - Machine-readable for agent parsing
+3. **Environment variables** - API keys never in code or config files
+4. **Shell invocation** - Agents invoke via `python -m ol_cli` with CLI arguments
+5. **No daemon/server** - Stateless single-shot invocations
+6. **Failover** - Multiple LLM providers configured, automatic fallback
+
+---
+
+## Security Notes
+
+- API keys stored in environment, never in code
+- `PYTHONPATH=src` required when running from repo root
+- Temp files should be cleaned up after use
+- No persistent state - each invocation is independent

omni_localizer-0.1.0/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2025-01-01
+
+### Added
+- Initial release
+- Translate markdown files using LLM APIs
+- Translate XLIFF files using LLM APIs
+- Model pool failover with LiteLLM router
+- Content shielding for code blocks, links, images
+- 4-layer semantic repair pipeline
+- LLM-based translation quality judging
+- Translation memory integration via hypomnema
+- Span alignment for content preservation
+- Agent skill support for OpenCode and Hermes
+- JSON output mode for machine-readable results

omni_localizer-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 1StepMore
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

omni_localizer-0.1.0/MANIFEST.in

@@ -0,0 +1,24 @@
+# Documentation
+include README.md
+include LICENSE
+include CHANGELOG.md
+include AGENT_USAGE.md
+
+# Config files
+include config/*.yaml
+
+# OpenCode skill files (recursively include all files in skill directories)
+recursive-include src/.opencode *
+
+# Hermes skill files (recursively include all files in skill directories)
+recursive-include src/.hermes *
+
+# Jinja2 templates in ol_lqa
+recursive-include src/ol_lqa *.j2
+
+# Exclude unnecessary files
+exclude .gitignore
+exclude .env
+exclude .env.example
+exclude *.pyc
+exclude __pycache__

omni_localizer-0.1.0/PKG-INFO

@@ -0,0 +1,205 @@
+Metadata-Version: 2.4
+Name: omni-localizer
+Version: 0.1.0
+Summary: AI-native localization pipeline with automated quality control
+Author-email: 1StepMore <renanzai@foxmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/1StepMore/Omni_Localizer
+Project-URL: Documentation, https://github.com/1StepMore/Omni_Localizer#readme
+Project-URL: Repository, https://github.com/1StepMore/Omni_Localizer
+Project-URL: Issues, https://github.com/1StepMore/Omni_Localizer/issues
+Keywords: localization,translation,llm,ai,markdown,xliff,i18n,internationalization
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Localization
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: litellm>=1.84.0
+Requires-Dist: translate-toolkit>=3.19.9
+Requires-Dist: markdown-it-py>=3.0.0
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: PyYAML>=6.0.0
+Requires-Dist: pytest>=8.0.0
+Requires-Dist: span-aligner>=0.3.2
+Requires-Dist: hypomnema>=0.8
+Requires-Dist: openevalkit>=0.1.7
+Requires-Dist: typer[all]>=0.15.0
+Requires-Dist: jinja2>=3.1.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: ml
+Requires-Dist: torch>=2.0.0; extra == "ml"
+Requires-Dist: sentence-transformers>=3.0.0; extra == "ml"
+Requires-Dist: transformers>=4.41.0; extra == "ml"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: ruff>=0.9.0; extra == "dev"
+Requires-Dist: mypy>=1.14.0; extra == "dev"
+Requires-Dist: pip-audit>=2.8.0; extra == "dev"
+Dynamic: license-file
+
+# Omni-Localizer (OL)
+
+AI-native localization pipeline that translates documents through intelligent LLM routing with built-in quality control.
+
+## What It Does
+
+- **Translate documents** (Markdown, XLIFF) using LLM APIs
+- **Automatic failover** — switches to backup model if primary fails
+- **Quality preservation** — shields code blocks, links, images during translation
+- **LLM-based judging** — evaluates translation accuracy and fluency
+- **Restoration layer** — uses LLM to restore placeholders after translation
+
+## Quick Start
+
+### 1. Install
+
+```bash
+pip install -e .
+```
+
+### 2. Configure API Keys
+
+Create a `.bat` file (gitignored) with your API keys:
+
+```bat
+@echo off
+set OPENAI_API_KEY=your_api_key
+set PYTHONPATH=src
+python -m ol_cli translate-md %* -c config/default.yaml -s en -t zh
+```
+
+### 3. Run
+
+```cmd
+test_en_to_zh.bat your_document.md -o output/
+```
+
+## Configuration
+
+`config/default.yaml` — Example LLM pool configuration:
+
+```yaml
+llm_pool:
+  translation:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "translation"
+    - provider: "openai"
+      model: "gpt-4o"
+      priority: 2
+      api_key: "${OPENAI_API_KEY}"
+      role: "translation"
+  judging:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "judging"
+  restoration:
+    - provider: "openai"
+      model: "gpt-4o-mini"
+      priority: 1
+      api_key: "${OPENAI_API_KEY}"
+      role: "restoration"
+```
+
+## CLI Commands
+
+```bash
+# Translate markdown
+ol translate-md <file.md> -c <config.yaml> -s en -t zh -o output/
+
+# Translate XLIFF
+ol translate-xliff <file.xlf> -c <config.yaml> -s en -t zh -o output/
+
+# Extract warnings from file
+ol extract-warnings <file> -o warnings.md
+```
+
+## Key Features
+
+| Feature | Description |
+|---------|-------------|
+| **Model Pool Failover** | LiteLLM router with primary + backup models per role |
+| **Content Shielding** | Code blocks, links, images preserved during translation |
+| **4-Layer Repair** | Regex → Span alignment → LLM restoration → Safe fallback |
+| **Translation + Judging** | JudgeService evaluates quality (adequacy, fluency, terminology) |
+| **TM Integration** | hypomnema for translation memory lookups |
+
+## Architecture
+
+- **MD Channel**: Token Stream + 4-layer semantic repair
+- **XLIFF Channel**: translate-toolkit based
+- **LLM Routing**: LiteLLM with model pool failover
+- **LQA**: openevalkit Scorer→Judge + COMET
+- **TM**: hypomnema (TMX)
+- **Alignment**: span-aligner + VectorAlign
+
+## Agent Usage
+
+Omni-Localizer can be used as a **skill** by coding agents (OpenCode, Hermes). Agents read the SKILL.md file to understand how to invoke translation.
+
+### OpenCode
+
+1. Add the skill to your project:
+   ```bash
+   cp -r src/.opencode/skills/ol-localizer <your-project>/.opencode/skills/
+   ```
+
+2. Reference it in your OpenCode configuration if needed
+
+For detailed usage, see `src/.opencode/skills/ol-localizer/SKILL.md`
+
+### Hermes
+
+1. Copy or symlink the skill:
+   ```bash
+   cp -r src/.hermes/skills/ol-localizer ~/.hermes/skills/
+   ```
+
+2. Restart Hermes to activate
+
+For detailed usage, see `src/.hermes/skills/ol-localizer/SKILL.md`
+
+### Environment Variables
+
+Configure your LLM provider API keys in your shell environment.
+
+### Testing the Agent Integration
+
+**Verify skill files exist:**
+```bash
+ls src/.opencode/skills/ol-localizer/SKILL.md
+ls src/.hermes/skills/ol-localizer/SKILL.md
+```
+
+**Test JSON output (machine-readable for agents):**
+```bash
+python -m ol_cli translate-md input.md -c config/default.yaml -s en -t zh -o output/ --json
+```
+
+Expected JSON output:
+```json
+{"success": true, "input_file": "input.md", "output_file": "output/input.md", "source_lang": "en", "target_lang": "zh"}
+```
+
+**Run skill tests:**
+```bash
+pytest tests/test_opencode_skill.py tests/test_hermes_skill.py -v
+```
+
+**Verify --json flag in help:**
+```bash
+python -m ol_cli translate-md --help | grep json
+```
+
+## License
+
+MIT