contextweave 0.2.0__py3-none-any.whl

context_aware_translation/AGENTS.md

@@ -0,0 +1,134 @@
+<!-- Parent: ../AGENTS.md -->
+<!-- Generated: 2026-02-26 -->
+
+# context_aware_translation
+
+## Purpose
+Core package for LLM-powered document translation with context-aware glossary management. Implements hierarchical context trees (LSM-tree-like), multi-pass glossary extraction, SQLite storage with WAL mode, and PySide6 GUI. Supports text, PDF, scanned books, and manga documents.
+
+## Key Files
+
+| File | Description |
+|------|-------------|
+| `config.py` | All config dataclasses: `LLMConfig`, `ExtractorConfig`, `SummarizerConfig`, `TranslatorConfig`, `GlossaryConfig`, `ReviewConfig`, `OCRConfig`, `ImageReembeddingConfig`, `MangaTranslatorConfig`, `EndpointProfile`. Central config hub; note: `num_of_chunks_per_llm_call` must NOT exceed 10. |
+| `adapters/files/glossary_io.py` | Glossary import/export to/from JSON files; handles term consolidation and validation at the file boundary. |
+| `__init__.py` | Package initialization and logging configuration via `configure_logging()`. |
+
+## Subdirectories
+
+| Directory | Purpose |
+|-----------|---------|
+| `core/` | Context tree and translation strategies: `context_tree.py` (hierarchical summarization), `context_manager.py` (context lifecycle), `translation_strategies.py` (strategy patterns), `context_extractor.py`, `progress.py`, `models.py`. |
+| `documents/` | Document type implementations: `text.py`, `pdf.py`, `scanned_book.py`, `manga.py`, `base.py` (abstract Document class), plus EPUB support and alignment utilities. |
+| `llm/` | LLM integration layer: `client.py` (OpenAI client with retry/timeout), `translator.py`, `extractor.py`, `glossary_translator.py`, `summarizor.py`, `reviewer.py`, `ocr.py`, `manga_ocr.py`, `language_detector.py`, `token_tracker.py`, `image_backends/` (PIL, DALL-E, etc.), `batch_jobs/` (batch processing). |
+| `storage/` | SQLite persistence layer with `schema/` (raw DB/schema owners), `repositories/` (query/update services), `models/` (persisted records), `library/` (book lifecycle), and batch/task storage. |
+| `adapters/` | Boundary adapters: Qt event bridge, Qt task engine, and Qt workers live under `adapters/qt/`; file import/export adapters live under `adapters/files/`. |
+| `ui/` | PySide6 GUI surfaces: `main_window.py`, `features/`, `widgets/`, `resources/`, `translations/`, `i18n.py`. |
+| `workflow/` | Task orchestration and execution: `service.py` (WorkflowService), `runtime.py`, `bootstrap.py`, `session.py`, `tasks/` (EngineCore, task handlers, claims, execution), `__init__.py` (exports entry points). |
+| `utils/` | Helper utilities: `chunking.py`, `cjk_normalize.py`, `markdown_escape.py`, `image_utils.py`, `hashing.py`, `file_utils.py`, semantic chunking, string similarity, symbol checking. |
+| `resources/` | Static assets: tokenizer data and other resources. |
+
+## For AI Agents
+
+### Working In This Directory
+
+**Config Management:**
+- Central config hub is `config.py` (all dataclasses for 5 required + 3 optional steps)
+- `EndpointProfile` allows reusable API endpoint configs; step configs can reference profiles
+- **Critical:** `num_of_chunks_per_llm_call` must NOT exceed 10 (causes hallucinations; default 5)
+- `noise_filtering_threshold`: 0.5 default (0=lenient, 1=strict)
+- `max_gleaning`: 3 default (multi-pass extraction; more = thorough but costlier)
+- `ocr_dpi`: 150 default (72-300 range)
+
+**Architecture:**
+- `workflow/ops/*.py` contains workflow domain operations (translation/glossary/ocr/export/bootstrap)
+- `workflow/tasks/engine_core.py` is the pure-Python task scheduling engine (no Qt)
+- Task handlers in `workflow/tasks/handlers/` implement task_type-specific logic
+- Qt adapter workers live in `adapters/qt/workers/` and bridge UI signals to workflow operations
+- SQLite with WAL mode everywhere (registry.db global, book.db per-book, context_tree.db)
+
+**Type Safety:**
+- mypy strict mode enabled (excludes ui/ and tests/)
+- All core logic is type-checked
+- Config models are dataclass-based with validation in `__post_init__`
+
+**Entry Points:**
+- UI: `ui.main:main` (PySide6 GUI application)
+- Workflow ops: `workflow/ops/translation_ops.py` (programmatic translation operations)
+
+### Testing Requirements
+
+- Tests mirror main package structure in `tests/`
+- Run `uv run pytest tests/` or `make test`
+- Tests run in parallel via pytest-xdist
+- Async tests use pytest-asyncio with auto mode
+- Key test files: `test_context_tree.py`, `test_book_db.py`, `test_translation_view.py`, `test_glossary_view.py`, task worker tests
+
+### Common Patterns
+
+**Config and Dataclasses:**
+- All config classes inherit from `LLMConfig` or extend it
+- Use `to_dict()` / `from_dict()` for serialization
+- Profile references are resolved at config load time
+
+**SQLite Storage:**
+- All tables use WAL mode for concurrent access
+- Use transaction context managers (`with db.conn:`)
+- Term records include descriptions (multi-pass gleaning), occurrence counts, votes, timestamps
+
+**Document Abstraction:**
+- `Document` base class in `documents/base.py`
+- Subclasses: `TextDocument`, `PDFDocument`, `ScannedBookDocument`, `MangaDocument`
+- Document type determines available operations (OCR, image extraction, etc.)
+
+**Context Trees:**
+- Hierarchical summaries stored in `context_tree_db`
+- Reduces token usage by 99%+ via LSM-tree-like compression
+- Managed by `ContextManager` and `ContextTree` in core/
+
+**LLM Integration:**
+- `LLMClient` wraps OpenAI API with retry logic (tenacity)
+- All LLM calls go through `client.py` (single source of truth for API config)
+- Async execution with configurable concurrency
+- Token tracking via `TokenTracker`
+
+**PySide6 UI:**
+- App/project/document chrome is migrating to hybrid QML hosts under `ui/qml/`, `ui/viewmodels/`, and `ui/shell_hosts/`
+- `MainWindow` is moving toward composition and lifetime management; shell policy should live in hosts/viewmodels instead of widget-local navigation code
+- Translation strings live in `.ts` / `.qm` files (zh_CN), and QML-backed chrome still depends on the same Qt translation pipeline
+- Workers handle long-running operations (glossary extraction, OCR, translation)
+- Task status monitoring still flows through `TaskStatusCard`, `TaskActivityPanel`, and the queue drawer surfaces
+
+**Task Execution:**
+- Pure-Python `EngineCore` in `workflow/tasks/engine_core.py`
+- Task handlers implement `TaskTypeHandler` interface
+- Claims-based resource arbitration to prevent concurrent conflicts
+- Config snapshot capture for fault tolerance
+
+## Dependencies
+
+### Internal
+- `context_aware_translation.config` - central config models
+- `context_aware_translation.core.*` - context tree, strategies, progress tracking
+- `context_aware_translation.documents.*` - document type implementations
+- `context_aware_translation.llm.*` - LLM client, translator, extractor, OCR
+- `context_aware_translation.storage.*` - SQLite repos, book manager
+- `context_aware_translation.workflow.*` - task orchestration
+- `context_aware_translation.ui.*` - PySide6 GUI (type-checked separately)
+- `context_aware_translation.utils.*` - text/image processing utilities
+
+### External (Top-Level)
+- `pyside6` - Qt GUI framework (mypy strict excluded)
+- `openai` - OpenAI API client
+- `google-genai` - Google Gemini API client
+- `tenacity` - Exponential backoff retry logic
+- `torch`, `transformers` - ML models for embeddings, tokenization
+- `faiss-cpu` - Vector similarity for context relevance
+- `pikepdf`, `pypdfium2` - PDF parsing and extraction
+- `pypandoc-binary` - Document format conversion (Markdown, DOCX)
+- `semchunk` - Semantic text chunking
+- `yaml` - Config file parsing
+- `pillow` - Image processing
+- `pydantic` - Config validation (used selectively)
+
+<!-- MANUAL: -->

context_aware_translation/__init__.py

@@ -0,0 +1,114 @@
+"""
+Term disambiguation package.
+"""
+
+import logging
+import logging.handlers
+import os
+import sys
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from context_aware_translation.config import Config
+
+# Track if we've already configured logging to avoid duplicate handlers
+_logging_configured = False
+
+# Default log file location
+DEFAULT_LOG_MAX_BYTES = 10 * 1024 * 1024  # 10MB
+DEFAULT_LOG_BACKUP_COUNT = 50
+
+
+class SafeRotatingFileHandler(logging.handlers.RotatingFileHandler):
+    """Rotating file handler that tolerates disappearing parent directories.
+
+    Book directories can be removed while background threads still emit logs.
+    In that case, silently drop file writes instead of surfacing noisy
+    logging tracebacks to users.
+    """
+
+    def emit(self, record: logging.LogRecord) -> None:
+        parent = Path(self.baseFilename).parent
+        if not parent.exists():
+            return
+        try:
+            super().emit(record)
+        except (FileNotFoundError, OSError, ValueError):
+            # Runtime race: path removed, stream already closed, or rollover reopen failed.
+            # Keep console logging alive and avoid cascading logging exceptions.
+            return
+
+
+def configure_logging(config: "Config") -> None:
+    """
+    Configure logging with Config.
+
+    Uses config.log_dir for the log file location.
+    Can be called multiple times - if called again, it updates
+    the file handler location while preserving the console handler.
+
+    Args:
+        config: Config instance with log_dir set
+    """
+    global _logging_configured
+
+    # Skip configuration if we're in a test environment (pytest will handle it)
+    is_test_env = (
+        "pytest" in sys.modules
+        or "_pytest" in sys.modules
+        or any("pytest" in arg.lower() for arg in sys.argv)
+        or os.environ.get("PYTEST_CURRENT_TEST") is not None
+    )
+
+    if is_test_env:
+        _logging_configured = True
+        return
+
+    root_logger = logging.root
+
+    assert config.log_dir is not None, "log_dir should be set after Config.__post_init__"
+    log_file_path = config.log_dir / "app.log"
+    # Create formatters
+    detailed_formatter = logging.Formatter(
+        "%(asctime)s [%(levelname)s] %(name)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S"
+    )
+    console_formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s: %(message)s", datefmt="%H:%M:%S")
+
+    # Set up console handler (only if not already configured)
+    # Check if we already have a console handler (StreamHandler writing to stderr)
+    has_console_handler = any(
+        isinstance(h, logging.StreamHandler) and h.stream is sys.stderr for h in root_logger.handlers
+    )
+    if not has_console_handler:
+        console_handler = logging.StreamHandler(sys.stderr)
+        console_handler.setFormatter(console_formatter)
+        console_handler.setLevel(logging.INFO)  # Only INFO and above to console
+        root_logger.addHandler(console_handler)
+
+    # Remove existing file handlers (if any) to update location
+    file_handlers = [h for h in root_logger.handlers if isinstance(h, logging.handlers.RotatingFileHandler)]
+    for handler in file_handlers:
+        root_logger.removeHandler(handler)
+        handler.close()
+
+    # Create file handler with rotation
+    try:
+        log_file_path.parent.mkdir(parents=True, exist_ok=True)
+        file_handler = SafeRotatingFileHandler(
+            filename=str(log_file_path),
+            maxBytes=DEFAULT_LOG_MAX_BYTES,
+            backupCount=DEFAULT_LOG_BACKUP_COUNT,
+            encoding="utf-8",
+        )
+        file_handler.setFormatter(detailed_formatter)
+        file_handler.setLevel(logging.DEBUG)  # All levels to file
+
+        root_logger.setLevel(logging.DEBUG)
+        root_logger.addHandler(file_handler)
+        root_logger.info(f"Logging to file: {log_file_path.absolute()}")
+    except (PermissionError, OSError) as e:
+        root_logger.warning(f"Could not create log file at {log_file_path}: {e}")
+        root_logger.warning("Continuing with console logging only")
+
+    _logging_configured = True

context_aware_translation/adapters/__init__.py

@@ -0,0 +1 @@
+"""Qt adapter layer for UI-facing orchestration and workers."""

context_aware_translation/adapters/files/__init__.py

@@ -0,0 +1,3 @@
+"""File-format adapters."""
+
+__all__ = ["glossary_io"]

context_aware_translation/adapters/files/glossary_io.py

@@ -0,0 +1,203 @@
+"""Glossary import/export functionality."""
+
+from __future__ import annotations
+
+import json
+import time
+from pathlib import Path
+from typing import TYPE_CHECKING, cast
+
+from context_aware_translation.core.models import normalize_term_type, ordered_description_values
+
+if TYPE_CHECKING:
+    from context_aware_translation.storage.schema.book_db import SQLiteBookDB
+
+from context_aware_translation.storage.schema.book_db import TermRecord
+
+
+def _consolidate_description(descriptions: dict[str, str]) -> str:
+    return " ".join(ordered_description_values(descriptions))
+
+
+def export_glossary(
+    db: SQLiteBookDB,
+    output_path: Path,
+    summarized_descriptions: dict[str, str] | None = None,
+) -> int:
+    """Export all glossary terms to a JSON file.
+
+    Returns the number of terms exported.
+    """
+    terms = db.list_terms()
+    entries = []
+    for term in terms:
+        description = (
+            summarized_descriptions.get(term.key)
+            if summarized_descriptions is not None and term.key in summarized_descriptions
+            else _consolidate_description(term.descriptions)
+        )
+        entries.append(
+            {
+                "key": term.key,
+                "translated_name": term.translated_name,
+                "description": description,
+                "term_type": term.term_type,
+                "ignored": term.ignored,
+                "is_reviewed": term.is_reviewed,
+            }
+        )
+
+    data = {"version": 1, "terms": entries}
+    with open(output_path, "w", encoding="utf-8") as f:
+        json.dump(data, f, ensure_ascii=False, indent=2)
+
+    return len(entries)
+
+
+def _validate_glossary_json(data: object) -> None:
+    """Validate glossary JSON structure. Raises ValueError on invalid data."""
+    if not isinstance(data, dict):
+        raise ValueError("Invalid glossary file: expected a JSON object")
+
+    version = data.get("version")
+    if version is not None and version > 1:
+        raise ValueError(
+            f"Unsupported glossary format version: {version}. This version of the application supports version 1."
+        )
+
+    if "terms" not in data:
+        raise ValueError("Invalid glossary file: missing 'terms' key")
+
+    terms = data["terms"]
+    if not isinstance(terms, list):
+        raise ValueError("Invalid glossary file: 'terms' must be a list")
+
+    for i, entry in enumerate(terms):
+        if not isinstance(entry, dict):
+            raise ValueError(f"Invalid glossary file: entry {i} must be a dict")
+        if "key" not in entry or not entry["key"]:
+            raise ValueError(f"Invalid glossary file: entry {i} missing or empty 'key'")
+
+
+def _validate_simple_glossary_json(data: object) -> None:
+    """Validate flat term-to-translation JSON mappings."""
+    if not isinstance(data, dict):
+        raise ValueError("Invalid glossary file: expected a JSON object")
+    for key, value in data.items():
+        if not isinstance(key, str) or not key.strip():
+            raise ValueError("Invalid glossary file: flat mapping keys must be non-empty strings")
+        if not isinstance(value, str) or not value.strip():
+            raise ValueError("Invalid glossary file: flat mapping values must be non-empty strings")
+
+
+def _build_import_term_record(
+    key: str,
+    *,
+    translated_name: str | None = None,
+    description: str = "",
+    term_type: str | None = None,
+    ignored: bool = False,
+    is_reviewed: bool = False,
+) -> TermRecord:
+    now = time.time()
+    return TermRecord(
+        key=key,
+        descriptions={"imported": description} if description else {},
+        occurrence={},
+        votes=1,
+        total_api_calls=1,
+        term_type=normalize_term_type(term_type),
+        new_translation=None,
+        translated_name=translated_name,
+        ignored=ignored,
+        is_reviewed=is_reviewed,
+        created_at=now,
+        updated_at=now,
+    )
+
+
+def _import_structured_glossary(
+    db: SQLiteBookDB,
+    data: dict[str, object],
+    *,
+    include_translations: bool,
+) -> int:
+    # Wipe existing data
+    db.delete_all_term_memory(auto_commit=False)
+    db.conn.execute("DELETE FROM terms")
+
+    term_records = []
+    for entry in cast(list[object], data["terms"]):
+        if not isinstance(entry, dict):
+            continue
+        term_records.append(
+            _build_import_term_record(
+                str(entry["key"]),
+                translated_name=entry.get("translated_name") if include_translations else None,
+                description=str(entry.get("description", "")),
+                term_type=entry.get("term_type") if isinstance(entry.get("term_type"), str) else None,
+                ignored=bool(entry.get("ignored", False)),
+                is_reviewed=bool(entry.get("is_reviewed", False)),
+            )
+        )
+
+    db.upsert_terms(term_records)
+    return len(term_records)
+
+
+def _import_simple_glossary(
+    db: SQLiteBookDB,
+    data: dict[str, object],
+    *,
+    include_translations: bool,
+) -> int:
+    term_records: list[TermRecord] = []
+    now = time.time()
+    for raw_key, raw_value in data.items():
+        key = raw_key.strip()
+        translated_name = raw_value.strip() if isinstance(raw_value, str) else ""
+        if not key:
+            raise ValueError("Invalid glossary file: flat mapping keys must be non-empty strings")
+        existing = db.get_term(key)
+        if existing is not None:
+            existing.translated_name = translated_name if include_translations else None
+            existing.ignored = False
+            existing.is_reviewed = True
+            existing.updated_at = now
+            term_records.append(existing)
+            continue
+        term_records.append(
+            _build_import_term_record(
+                key,
+                translated_name=translated_name if include_translations else None,
+                ignored=False,
+                is_reviewed=True,
+            )
+        )
+
+    if term_records:
+        db.upsert_terms(term_records)
+    return len(term_records)
+
+
+def import_glossary(
+    db: SQLiteBookDB,
+    input_path: Path,
+    include_translations: bool = True,
+) -> int:
+    """Import glossary terms from a JSON file.
+
+    Structured glossary JSON preserves current replace-all behavior.
+    Flat term-to-translation JSON mappings are merged into the existing glossary.
+
+    Returns the number of terms processed.
+    """
+    with open(input_path, encoding="utf-8") as f:
+        data = json.load(f)
+
+    if isinstance(data, dict) and ("terms" in data or "version" in data):
+        _validate_glossary_json(data)
+        return _import_structured_glossary(db, data, include_translations=include_translations)
+
+    _validate_simple_glossary_json(data)
+    return _import_simple_glossary(db, data, include_translations=include_translations)

context_aware_translation/adapters/qt/__init__.py

@@ -0,0 +1 @@
+"""Qt-specific adapters used by the desktop UI."""

context_aware_translation/adapters/qt/application_event_bridge.py

@@ -0,0 +1,78 @@
+from __future__ import annotations
+
+from PySide6.QtCore import QObject, Qt, QThread, Signal, Slot
+
+from context_aware_translation.application.events import (
+    ApplicationEventKind,
+    ApplicationEventPayload,
+    ApplicationEventSubscriber,
+    DocumentInvalidatedEvent,
+    ProjectsInvalidatedEvent,
+    QueueChangedEvent,
+    SetupInvalidatedEvent,
+    TermsInvalidatedEvent,
+    WorkboardInvalidatedEvent,
+)
+
+
+class QtApplicationEventBridge(QObject):
+    """Adapt application events into Qt signals.
+
+    The application event bus is the system of record. This bridge only turns
+    those framework-agnostic events into Qt-friendly notifications.
+    """
+
+    event_received = Signal(object)
+    projects_invalidated = Signal(object)
+    queue_changed = Signal(object)
+    workboard_invalidated = Signal(object)
+    document_invalidated = Signal(object)
+    terms_invalidated = Signal(object)
+    setup_invalidated = Signal(object)
+
+    _enqueue_event = Signal(object)
+
+    def __init__(self, events: ApplicationEventSubscriber, parent: QObject | None = None) -> None:
+        super().__init__(parent)
+        self._subscription = events.subscribe(self._on_event_from_any_thread)
+        self._enqueue_event.connect(self._dispatch_event, Qt.ConnectionType.QueuedConnection)
+
+    def close(self) -> None:
+        self._subscription.close()
+
+    def __del__(self) -> None:
+        self.close()
+
+    def _on_event_from_any_thread(self, event: ApplicationEventPayload) -> None:
+        if QThread.currentThread() is self.thread():
+            self._dispatch_event(event)
+            return
+        self._enqueue_event.emit(event)
+
+    @Slot(object)
+    def _dispatch_event(self, event: ApplicationEventPayload) -> None:
+        self.event_received.emit(event)
+        if event.kind is ApplicationEventKind.PROJECTS_INVALIDATED:
+            self.projects_invalidated.emit(event)
+        elif event.kind is ApplicationEventKind.QUEUE_CHANGED:
+            self.queue_changed.emit(event)
+        elif event.kind is ApplicationEventKind.WORKBOARD_INVALIDATED:
+            self.workboard_invalidated.emit(event)
+        elif event.kind is ApplicationEventKind.DOCUMENT_INVALIDATED:
+            self.document_invalidated.emit(event)
+        elif event.kind is ApplicationEventKind.TERMS_INVALIDATED:
+            self.terms_invalidated.emit(event)
+        elif event.kind is ApplicationEventKind.SETUP_INVALIDATED:
+            self.setup_invalidated.emit(event)
+
+
+__all__ = [
+    "QtApplicationEventBridge",
+    "ApplicationEventPayload",
+    "ProjectsInvalidatedEvent",
+    "QueueChangedEvent",
+    "WorkboardInvalidatedEvent",
+    "DocumentInvalidatedEvent",
+    "TermsInvalidatedEvent",
+    "SetupInvalidatedEvent",
+]