locus-etl 0.0.1__tar.gz

locus_etl-0.0.1/.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Set Python version
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Sync dependencies
+        run: uv sync --extra dedup --extra pdf --extra serve
+
+      - name: Ruff (lint)
+        run: uv run ruff check src tests
+
+      - name: Mypy (types)
+        run: uv run mypy
+
+      - name: Pytest (with coverage)
+        run: uv run pytest --cov --cov-report=term-missing

locus_etl-0.0.1/.gitignore

@@ -0,0 +1,24 @@
+# Diagram-generation virtualenv (transient build tooling)
+.diagram-venv/
+
+# Python
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+*.egg-info/
+htmlcov/
+.coverage
+
+# OS
+.DS_Store
+
+# Local scratch / demo space (not committed)
+scratch/
+
+# Build artifacts
+dist/
+build/

locus_etl-0.0.1/.kiro/specs/locus-image-runtime/.config.kiro

@@ -0,0 +1 @@
+{"specId": "2c1324e3-b19f-554d-a057-476f4bf65262", "workflowType": "requirements-first", "specType": "feature"}

locus_etl-0.0.1/.kiro/specs/locus-image-runtime/architecture-notes.md

@@ -0,0 +1,37 @@
+# Architecture & Vision Notes — Layer 2 (locus-image-runtime)
+
+> The canonical, cross-layer decision log lives in the sibling spec:
+> `.kiro/specs/unstructured-to-tabular-etl/architecture-notes.md`
+> Read that file for the full decision trail (project name, runtime model, key handling,
+> LiteLLM, composition solutions, Hub/Harbor, privacy model, etc.). It spans BOTH layers.
+> This file only records what is specific to Layer 2 navigation.
+
+## What this spec owns (Layer 2 = packaging / distribution / runtime)
+- Client install + invocation (single CLI, local-process default, Docker optional backend).
+- Image pull + local cache; version resolution.
+- Locusfile run-configuration surface (source, volumes, ports, schema ref, llm, export,
+  review) — minimal required fields = source + image.
+- Local LLM credential handling (.env primary; env var / keyring fallbacks; raw-key
+  prohibition + gitignore guardrails).
+- Multi-image composition = pipeline DAG (NOT docker-compose semantics; compose-FAMILIAR
+  syntax only). `needs:`-style dependency edges, parallel branches, cycle detection,
+  stage output caching.
+- Stage interchange contract: single versioned `Locus_Artifact` envelope, fixed
+  Artifact_Type set, Arrow/Parquet payloads, static pre-run type check (fail fast).
+- Cross-stage provenance propagation: framework-managed (SDK) lineage, run-scoped
+  append-only Lineage_Store, conformance certification, strict/permissive non-conformant.
+- Result serving / preview UI / export (local, on mapped port).
+- Image authoring + building (Image_Manifest, dependency pinning, conformance cert).
+- Publishing to Locus Hub: public + private (Harbor), namespaces, RBAC, self-host path.
+- Image discovery; runtime privacy disclosure / consent.
+
+## What the SIBLING spec (`unstructured-to-tabular-etl`, Layer 1) owns
+- The processing engine an Image embeds: connectors, parser routing, IR, schema-driven
+  extraction, cleaning, dedup, the cell-level grounding/faithfulness contract, HITL review,
+  plugin architecture, observability, dual-engine (deterministic default + LLM opt-in),
+  LLM guardrails. Req 1-15 there.
+
+## Key cross-references
+- Layer 2 Req 6 (interchange) + Req 7 (cross-stage provenance) depend on Layer 1 Req 3 (IR)
+  and Req 8 (emission/provenance) being the STABLE VERSIONED contract.
+- `image-catalog.md` (this folder) = 65 candidate images / 9 tiers / locked build order.

locus_etl-0.0.1/.kiro/specs/locus-image-runtime/design.md

@@ -0,0 +1,342 @@
+# Design Document
+
+## Overview
+
+This document specifies the design of **Layer 2 of Locus: the image runtime, packaging, distribution, and registry**. It builds on the Layer 1 engine (`unstructured-to-tabular-etl`, now implemented as the `locus_engine` package) and turns engine capabilities into reusable, versioned **images** that users pull, point at their own data via a **Locusfile**, run locally, preview, and export. Multiple images compose into a **pipeline DAG** with a versioned interchange contract and cross-stage provenance.
+
+This layer is a separate installable package, `locus` (the CLI), which depends on `locus_engine`. The engine stays free of any packaging/registry concern; the runtime orchestrates engine-backed images.
+
+### Tech baseline (inherited + additions)
+
+- **Python 3.11+**, `uv` + `pyproject.toml`, Pydantic v2 (same as Layer 1).
+- **CLI:** Typer (argument parsing, subcommands) + Rich (terminal output).
+- **Registry transport:** ORAS-style OCI client (`oras` Python library) against Harbor; the CLI is registry-agnostic OCI so Harbor is swappable.
+- **Artifact serialization:** Apache Arrow / Parquet for tabular payloads (already produced by the engine's emitters).
+- **Result UI:** embedded FastAPI + a static front end, served locally via uvicorn.
+- **Optional Docker backend:** the `docker` Python SDK, used only when `--runtime=docker`.
+
+### Design principles
+
+1. **The CLI is the only install.** A single `pip install locus`. No daemon, no Linux VM by default (Req 1).
+2. **Local-first, registry-agnostic.** Runs on the user's machine; talks plain OCI so the default Harbor hub is swappable (architecture notes).
+3. **Composition is a typed DAG, not docker-compose.** Compose-familiar YAML, pipeline execution semantics (Req 5).
+4. **Trust is enforced at the seams.** Static type-check across stage edges before running (Req 6); provenance composes across stages or the build fails (Req 7).
+5. **The engine does the work; the runtime orchestrates.** No data transformation logic lives here — it delegates to `locus_engine`.
+
+## Architecture
+
+```mermaid
+flowchart TB
+    subgraph cli["locus CLI (Typer)"]
+        PULL[pull] 
+        RUN[run]
+        BUILD[build]
+        PUSH[push]
+        SEARCH[search]
+        INIT[init]
+    end
+
+    subgraph rt["Runtime"]
+        LF[LocusfileLoader\n+ validation]
+        DAG[PipelinePlanner\nDAG + static type-check]
+        EXEC[StageExecutor\nlocal-process | docker]
+        PROV[CrossStageProvenance\n+ run LineageStore]
+        WS[(Run_Workspace\nArrow artifacts)]
+    end
+
+    subgraph img["Image"]
+        MAN[Image_Manifest\naccepts/emits, engine modes, privacy]
+        CODE[engine-backed capability code]
+    end
+
+    subgraph dist["Distribution"]
+        CACHE[(local image cache)]
+        ORAS[OCI client / ORAS]
+        HUB[(Locus Hub / Harbor)]
+    end
+
+    subgraph serve["Serving"]
+        UI[Result UI\nFastAPI + static]
+        EXP[Exporter]
+    end
+
+    ENG[[locus_engine\nLayer 1]]
+
+    RUN --> LF --> DAG --> EXEC
+    EXEC --> img
+    img --> ENG
+    EXEC <--> WS
+    EXEC --> PROV
+    PULL --> ORAS <--> HUB
+    PULL --> CACHE
+    BUILD --> MAN
+    PUSH --> ORAS
+    SEARCH --> HUB
+    EXEC --> UI
+    EXEC --> EXP
+```
+
+The CLI dispatches subcommands. `run` is the core path: load + validate the Locusfile, plan the DAG (resolving and type-checking every stage edge), then execute stages in dependency order through the selected runtime backend, materializing typed `Locus_Artifact`s in the run workspace and threading provenance through a run-scoped lineage store. `pull`/`push`/`search` talk OCI to Harbor. `build` packages an image from its manifest and certifies provenance conformance.
+
+## Data Models
+
+All models are Pydantic v2. The `Locus_Artifact`, `Artifact_Type`, and provenance structures align with the engine's `ProvenancedTable`/`IntermediateRepresentation` so the contract is shared, not re-invented.
+
+```python
+from __future__ import annotations
+from enum import StrEnum
+from typing import Any
+from pydantic import BaseModel, Field
+
+
+class ArtifactKind(StrEnum):
+    IR = "ir"
+    TABLE = "table"
+    CHUNKS = "chunks"
+    EMBEDDINGS = "embeddings"
+    GRAPH = "graph"
+
+
+class ArtifactType(BaseModel):
+    """A versioned artifact type, e.g. kind=table version=(1,0) -> 'table/v1'."""
+    kind: ArtifactKind
+    major: int = 1
+    minor: int = 0
+
+    def tag(self) -> str:
+        return f"{self.kind.value}/v{self.major}"
+
+    def compatible_with(self, consumer: "ArtifactType") -> "Compat":
+        if self.kind != consumer.kind or self.major != consumer.major:
+            return Compat.INCOMPATIBLE
+        if self.minor != consumer.minor:
+            return Compat.MINOR_DIFF
+        return Compat.OK
+
+
+class Compat(StrEnum):
+    OK = "ok"
+    MINOR_DIFF = "minor_diff"   # compatible with warning (Req 6.6)
+    INCOMPATIBLE = "incompatible"
+
+
+class LocusArtifact(BaseModel):
+    """The only structure that crosses a stage boundary (Req 6.1)."""
+    type: ArtifactType
+    payload_path: str            # Arrow/Parquet file in the Run_Workspace (Req 6.2)
+    lineage_path: str | None = None   # serialized lineage graph for this artifact
+    produced_by: str             # stage id
+    engine_mode: str = "deterministic"  # "deterministic" | "llm"
+
+
+class PrivacyClass(StrEnum):
+    LOCAL_ONLY = "local_only"
+    CALLS_EXTERNAL = "calls_external"   # an LLM-backed image (Req 12.2)
+
+
+class ImageManifest(BaseModel):
+    """Build-time declaration of an image (Req 9.2)."""
+    name: str
+    version: str
+    description: str = ""
+    accepts: list[ArtifactType] = Field(default_factory=list)
+    emits: ArtifactType
+    engine_modes: list[str] = Field(default_factory=lambda: ["deterministic"])
+    privacy_class: PrivacyClass = PrivacyClass.LOCAL_ONLY
+    provenance_conformant: bool = False   # set by build-time certification (Req 9.4)
+    entrypoint: str                       # import path of the capability callable
+    dependencies: dict[str, str] = Field(default_factory=dict)  # pinned (Req 9.3)
+
+
+class StageSpec(BaseModel):
+    """One stage in a Locusfile pipeline."""
+    id: str
+    image: str                       # "name:version"
+    needs: list[str] = Field(default_factory=list)   # dependency ids (Req 5.4)
+    source: "SourceSpec | None" = None               # override pipeline source
+    config: dict[str, Any] = Field(default_factory=dict)
+
+
+class SourceSpec(BaseModel):
+    type: str                        # "files" | "url" | "api" | "sql"
+    path: str | None = None
+    uri: str | None = None
+
+
+class PortSpec(BaseModel):
+    ui: int | None = None
+    api: int | None = None
+
+
+class Locusfile(BaseModel):
+    """The run-configuration surface (Req 3)."""
+    image: str | None = None          # single-image shorthand
+    source: SourceSpec | None = None
+    pipeline: list[StageSpec] = Field(default_factory=list)
+    volumes: list[str] = Field(default_factory=list)
+    ports: PortSpec = Field(default_factory=PortSpec)
+    schema_mode: str = "infer"
+    schema_ref: str | None = None
+    llm: dict[str, Any] | None = None
+    env_file: str | None = None
+    export: dict[str, Any] | None = None
+    review: dict[str, Any] | None = None
+    mode: str = "strict"              # "strict" | "permissive" (Req 7.7/7.8)
+```
+
+## Components and Interfaces
+
+### CLI commands (Typer)
+
+```python
+# locus <command>
+def pull(image: str) -> None: ...                       # Req 2
+def run(locusfile: str = "locusfile.yaml",
+        runtime: str = "process") -> None: ...          # Req 1, 5, 8
+def build(manifest: str = "locus.image.yaml") -> None: ...  # Req 9
+def push(image: str, private: bool = False) -> None: ...    # Req 10
+def search(query: str = "") -> None: ...                # Req 11
+def login(registry: str | None = None) -> None: ...     # Req 10.1
+def init(path: str = ".") -> None: ...                  # Req 4.4 (.env + gitignore)
+def config_set_key(provider: str, key: str) -> None: ... # keyring (Req 4.1)
+```
+
+### LocusfileLoader
+
+Loads YAML, validates against the `Locusfile` model, performs credential safety checks (reject raw keys, enforce `.env` gitignore/untracked — Req 4.3/4.5), and resolves a single-image shorthand into a one-stage pipeline. Raises `ConfigError` with the offending setting (Req 3.7).
+
+### PipelinePlanner
+
+```python
+class PipelinePlanner:
+    def plan(self, lf: Locusfile, manifests: dict[str, ImageManifest]) -> PipelinePlan:
+        """Build the DAG, detect cycles (Req 5.6), topologically order stages, and
+        STATIC-CHECK every edge's artifact-type compatibility BEFORE execution
+        (Req 6.4/6.5). A major-version mismatch fails; a minor mismatch warns
+        (Req 6.6). Also verifies provenance-conformance per mode (Req 7.7/7.8)."""
+```
+
+`PipelinePlan` holds the ordered stages, parallelizable groups (waves of independent stages, Req 5.5), and the validated edge map. Planning fails fast: no stage runs if any edge is incompatible or a cycle exists.
+
+### StageExecutor and Runtime backends
+
+```python
+class RuntimeBackend(Protocol):
+    name: str
+    def run_stage(self, image: ResolvedImage, inputs: list[LocusArtifact],
+                  ctx: StageContext) -> LocusArtifact: ...
+
+class ProcessBackend:   # default (Req 1.3) — per-image venv, in-process engine call
+    ...
+class DockerBackend:    # optional (Req 1.4); errors clearly if Docker absent (Req 1.5)
+    ...
+```
+
+`StageExecutor` walks the plan wave by wave: gathers each stage's input artifacts (the pipeline source for root stages, dependency outputs otherwise — Req 5.3/5.4), invokes the backend, writes the output `Locus_Artifact` (Arrow file) to the run workspace, and records lineage. Stage caching (Req 5.8): if a stage's input artifact hashes and image version match a prior run, reuse the cached output.
+
+### CrossStageProvenance
+
+Wraps the engine's `ProvenanceComposer`/`LineageStore`. As artifacts flow between stages, cell ids and lineage edges are preserved in a run-scoped `Lineage_Store`; the terminal artifact's cells resolve to original `Source_Location`s (Req 7.5/7.6). Conformance is checked at build time (Req 9.4) and enforced at plan time per mode (Req 7.7/7.8).
+
+### Distribution: OCI client + cache
+
+```python
+class ImageStore:
+    def pull(self, ref: str) -> ResolvedImage: ...   # ORAS pull -> cache (Req 2.1)
+    def cached(self, ref: str) -> ResolvedImage | None: ...   # (Req 2.2)
+    def push(self, image: BuiltImage, *, private: bool) -> None: ...  # (Req 10)
+    def search(self, query: str) -> list[ImageSummary]: ...   # (Req 11)
+```
+
+Images are OCI artifacts: the manifest + a layer with the capability code + pinned deps. Private/public visibility maps to Harbor project visibility (Req 10.3/10.4); pulls authenticate via the `Registry_Credential`, which is wholly separate from LLM keys (Req 10.8).
+
+### ImageBuilder
+
+Packages an `ImageManifest` + entrypoint into a `BuiltImage`, pins dependency versions (Req 9.3), and runs the engine's `conformance` harness (`probe_extraction_engine`) against the capability to set `provenance_conformant` (Req 9.4/9.5).
+
+### Result serving and export
+
+`ResultServer` (FastAPI) serves the final artifact on the mapped UI port (Req 8.1): rows, per-cell faithfulness + source location, flagged rows, and the engine mode (Req 8.2/8.3), reading directly from the engine's emitted DataFrame + `_lineage` column. `Exporter` writes the configured format (Req 8.4). Both are local; no hosted service (Req 8.5).
+
+## Error Handling
+
+| Class | Examples | Behavior |
+|-------|----------|----------|
+| Plan-fatal, fail-fast | cycle in DAG, artifact-type mismatch, non-conformant stage in strict mode, missing image | raise before any stage runs (Req 5.6, 6.5, 7.7, 2.4) |
+| Config | invalid Locusfile, raw key in file, git-tracked `.env` | reject with offending setting (Req 3.7, 4.3, 4.5) |
+| Backend | Docker selected but absent | error identifying the backend; no silent fallback (Req 1.5) |
+| Permissive degradation | non-conformant stage in permissive mode | continue; mark downstream cells lineage-broken (Req 7.8) |
+| Runtime privacy | external-LLM stage about to send data | consent notice before egress (Req 12.1) |
+
+Errors derive from a `LocusRuntimeError` hierarchy, distinct from but mirroring the engine's `LocusError`.
+
+## Correctness Properties
+
+### Property 1: Fail-fast type safety
+
+For any composed pipeline, no stage executes unless every dependency edge is artifact-type compatible (major version equal, kind equal). An incompatible edge raises before execution.
+
+**Validates: Requirements 6.4, 6.5**
+
+### Property 2: Acyclic execution
+
+A pipeline graph containing a cycle never executes any stage and reports the cycle.
+
+**Validates: Requirements 5.6**
+
+### Property 3: Dependency ordering
+
+A stage executes only after all stages in its `needs` have completed, and its inputs are exactly those dependencies' outputs (or the pipeline source for root stages).
+
+**Validates: Requirements 5.3, 5.4**
+
+### Property 4: End-to-end provenance survival
+
+Every cell in the terminal artifact resolves, through the run lineage store, to at least one originating `Source_Location`, across all intermediate stages.
+
+**Validates: Requirements 7.1, 7.5, 7.6**
+
+### Property 5: Conformance gating
+
+In strict mode a non-conformant stage fails the run; in permissive mode it continues and the affected downstream cells are marked lineage-broken.
+
+**Validates: Requirements 7.7, 7.8**
+
+### Property 6: Credential locality
+
+No LLM credential or user data is transmitted to any hosted Locus service; credentials resolve only from local sources and a raw key in the Locusfile is rejected.
+
+**Validates: Requirements 4.1, 4.2, 4.3**
+
+### Property 7: Registry/credential separation
+
+The registry credential authorizes only image pull/push and never grants access to LLM provider credentials.
+
+**Validates: Requirements 10.8**
+
+### Property 8: Privacy disclosure honesty
+
+A run containing an external-LLM image never presents a blanket data-stays-local claim and surfaces a consent notice before data leaves.
+
+**Validates: Requirements 12.1, 12.3**
+
+## Testing strategy
+
+1. **Locusfile validation** — minimal (source + image) accepted; raw-key rejection; `.env` gitignore/untracked guardrails; single-image shorthand expands to one stage.
+2. **Planner unit tests** — DAG construction, cycle detection, topological waves, edge type-check (compatible / minor-warn / major-fail), conformance gating per mode.
+3. **Interchange round-trip** — a `Locus_Artifact` written by one stage (Arrow) reads back identically in the next; lineage path resolves.
+4. **Cross-stage provenance** — a two-stage pipeline (extract → redact) where the final masked cell still resolves to the original source; reuses the engine's conformance assertions.
+5. **Backend tests** — process backend runs a stub image end-to-end; docker backend errors cleanly when Docker is absent (mocked).
+6. **Distribution** — pull/cache hit-miss against a mocked OCI client; private-image auth; version resolution default.
+7. **Serving/export** — FastAPI test client returns rows + provenance + engine mode; exporter writes the configured format.
+8. **Privacy** — consent fires before an external-LLM stage; no blanket local claim when such a stage is present.
+9. **Golden two-stage pipeline** — using two real engine-backed stub images, assert the typed DAG, provenance survival, and final emitted output end-to-end.
+
+## Key design decisions and rationale
+
+- **Separate `locus` package depending on `locus_engine`.** Keeps the engine reusable and the runtime's heavier deps (Typer, FastAPI, oras, optional docker) out of the engine.
+- **Artifacts pass by file path in a run workspace, not in memory.** Enables the Docker backend (cross-process), stage caching (content-addressed), and large datasets — and reuses the engine's Arrow/Parquet emitters.
+- **Static type-check before any execution.** Composition is only trustworthy if mismatches fail fast; this is a pipeline-level type system over the fixed `ArtifactType` set.
+- **Provenance enforced at build (certification) and plan (gating).** Reuses the Layer 1 conformance harness so the guarantee is identical end to end; non-conformant images are visible, never silent.
+- **OCI/Harbor borrowed, CLI registry-agnostic.** No registry is built; the default hub is swappable to GHCR/ECR/self-hosted Harbor via config.
+```

locus_etl-0.0.1/.kiro/specs/locus-image-runtime/image-catalog.md

@@ -0,0 +1,120 @@
+# Locus Image Catalog (working list)
+
+> Candidate "capability images" to build and publish to Locus Hub. Each image contains
+> LOGIC (parser + extractor + validator + emitter + pinned config); the USER brings their
+> own data. Default engine = deterministic Python; LLM engine activates only when the user
+> supplies a key (see architecture-notes.md). Every image inherits the cell-level
+> provenance + faithfulness grounding contract from the Layer-1 engine.
+>
+> Status legend: [ ] not started | [~] in progress | [x] shipped
+> This catalog will likely move into the sibling `data-image-runtime` spec.
+
+## Guiding principle
+- ~80% of enterprise data is unstructured; extraction (not the model) is the bottleneck.
+- Highest recurring-spend jobs: invoices, receipts, bank statements, contracts, reports.
+- DO NOT build breadth-first. Ship one fully-grounded image at a time.
+- "Solve ALL data problems" is rejected as a goal — this list is the SUPERSET of what's
+  possible; discipline = ship incrementally starting with `doc-to-tables`.
+
+---
+
+## Tier 1 — Document -> Structured (highest, proven demand)
+1. [ ] `doc-to-tables` — tables from PDF/scan/Word -> rows. FIRST image to build.
+2. [ ] `invoice-extractor` — invoices -> fields + line items.
+3. [ ] `receipt-extractor` — receipts -> expense records.
+4. [ ] `bank-statement-extractor` — statements -> transaction tables.
+5. [ ] `resume-parser` — CVs -> candidate schema.
+6. [ ] `contract-extractor` — clauses, parties, dates, obligations, renewal terms.
+7. [ ] `form-extractor` — tax/customs/medical/court forms -> fields.
+8. [ ] `report-extractor` — annual/financial reports -> tables + KPIs.
+9. [ ] `scanned-ocr-to-table` — OCR-first for image-only scans (no text layer).
+
+## Tier 2 — Format Conversion & Reshaping (broad everyday demand)
+10. [ ] `any-to-json` — any source -> clean schema-stable JSON (dominant AI-ready format).
+11. [ ] `any-to-csv` / `any-to-parquet` — tabular export for analytics.
+12. [ ] `any-to-yaml` — config-shaped output.
+13. [ ] `any-to-markdown` — doc/web -> clean Markdown (standard RAG ingestion format).
+14. [ ] `json-reshaper` — JSON -> JSON remap/flatten/nest.
+15. [ ] `schema-mapper` — map source schema -> target schema (rename, type-align).
+
+## Tier 3 — Cleaning, Quality & Identity (the "data is messy" problems)
+16. [ ] `data-cleaner` — type coercion, normalization, missing-value handling.
+17. [ ] `deduplicator` / `entity-resolver` — match + merge same-real-world-entity records.
+18. [ ] `data-validator` — assert dataset vs rules/constraints, report violations.
+19. [ ] `data-enricher` — augment records with derived/looked-up fields.
+20. [ ] `normalizer` — standardize units, currencies, dates, addresses, phone numbers.
+
+## Tier 4 — RAG / AI-Prep (serves the "feed the LLM" goal directly)
+21. [ ] `chunker` — layout-aware chunking (make-or-break RAG step).
+22. [ ] `embedder` — chunk -> vector, output to Parquet / vector DB.
+23. [ ] `knowledge-graph-builder` — text -> entities + relationships (triples) for GraphRAG.
+24. [ ] `metadata-enricher` — LLM-generated metadata to improve retrieval.
+25. [ ] `qa-pair-generator` — corpus -> synthetic Q&A pairs for fine-tune/eval.
+
+## Tier 5 — Multimodal (next frontier; less crowded)
+26. [ ] `audio-to-table` — transcription + diarization -> structured transcript/records.
+27. [ ] `video-to-table` — frames + ASR -> structured events/timestamps.
+28. [ ] `image-to-data` — charts/diagrams/photos -> structured values.
+29. [ ] `email-extractor` — emails/threads -> structured fields.
+30. [ ] `log-parser` — app/system logs -> structured events.
+
+## Tier 6 — Unique / hard-problem images (Locus differentiation; lean on grounding engine)
+31. [ ] `pii-redactor` — detect + mask PII in structured + unstructured data.
+32. [ ] `provenance-tracker` — emit data with full cell-level lineage as the headline output.
+33. [ ] `fact-grounder` — score faithfulness of every cell in an EXISTING table vs sources
+       (validation-as-a-service). Flagship/marketing wedge.
+34. [ ] `synthetic-data-generator` — schema -> realistic synthetic rows, referential
+       integrity across multi-table schemas.
+35. [ ] `cross-doc-reconciler` — reconcile conflicting values across multiple docs, surface
+       the conflict with provenance.
+36. [ ] `table-joiner` — semantic/fuzzy join of tables lacking shared keys.
+37. [ ] `time-series-structurer` — messy event text -> clean time series (no peak flatten).
+38. [ ] `taxonomy-classifier` — auto-classify records into a user-defined taxonomy tree.
+39. [ ] `change-diff` — diff two versions of a doc/dataset -> structured changeset.
+40. [ ] `anomaly-flagger` — flag outlier/suspect rows with reasons (quality + fraud signal).
+
+## Tier 7 — Domain-specific verticals (additional; high willingness-to-pay)
+41. [ ] `medical-record-extractor` — clinical notes/EHR text -> structured codes (ICD/SNOMED-style).
+42. [ ] `legal-doc-extractor` — case law / filings -> citations, holdings, parties.
+43. [ ] `scientific-paper-extractor` — papers -> structured methods/results tables, citations.
+44. [ ] `financial-filing-extractor` — 10-K/earnings -> normalized financial line items.
+45. [ ] `real-estate-doc-extractor` — leases/deeds -> property + terms schema.
+46. [ ] `insurance-claim-extractor` — claims/policies -> structured claim records.
+47. [ ] `shipping-logistics-extractor` — BOL / customs / manifests -> structured shipment data.
+48. [ ] `survey-response-structurer` — free-text survey answers -> coded categorical data.
+49. [ ] `product-catalog-extractor` — supplier sheets/specs -> normalized product attributes.
+50. [ ] `menu-extractor` — restaurant menus (img/PDF) -> items + prices + modifiers.
+
+## Tier 8 — Web & API sources (additional)
+51. [ ] `web-to-table` — web pages/listings -> structured rows (price/spec scraping).
+52. [ ] `api-to-table` — paginated REST/GraphQL responses -> flattened tables.
+53. [ ] `sitemap-crawler-to-corpus` — crawl a site -> clean corpus for downstream images.
+54. [ ] `social-feed-structurer` — posts/threads -> structured engagement/content records.
+55. [ ] `spreadsheet-normalizer` — messy human Excel (merged cells, multi-header) -> tidy table.
+
+## Tier 9 — Operational / pipeline-utility images (additional; glue + governance)
+56. [ ] `schema-inferer` — sample data -> proposed Pydantic schema (bootstraps other images).
+57. [ ] `data-profiler` — dataset -> stats/quality report (nulls, cardinality, distributions).
+58. [ ] `data-masker` / `anonymizer` — reversible/irreversible masking beyond PII (k-anon).
+59. [ ] `format-detector` — sniff + classify unknown file/content types for routing.
+60. [ ] `language-detector-translator` — detect language, optionally normalize to one language.
+61. [ ] `unit-test-data-extractor` — code repos -> structured fixtures/test data.
+62. [ ] `csv-repair` — fix broken/ragged CSVs (bad quoting, mixed delimiters, encoding).
+63. [ ] `encoding-normalizer` — normalize text encodings / mojibake repair.
+64. [ ] `dataset-merger` — union/merge heterogeneous datasets into one schema.
+65. [ ] `incremental-sync` — detect new/changed source records since last run (CDC-style).
+
+## Build-order recommendation (LOCKED priority)
+1. `doc-to-tables` (Tier 1 #1) — proves full engine end-to-end; everything reuses its plumbing.
+2. Tier-1 specializations `invoice-extractor`, `receipt-extractor` — base + baked-in schema; highest revenue.
+3. 2-3 conversion images `any-to-json`, `any-to-markdown` — high pull volume, low build cost, drives adoption.
+4. One flagship unique image `fact-grounder` (or `provenance-tracker`) — the marketing wedge competitors lack.
+5. Tiers 4-9 follow once base engine + packaging are battle-tested.
+
+## Notes / open
+- Several Tier 6 images (fact-grounder, provenance-tracker, cross-doc-reconciler) are the
+  defensible differentiation — they ARE the grounding engine, packaged.
+- Each image should advertise: engine mode support (deterministic / LLM / hybrid),
+  privacy class (local-only vs calls-external-LLM), and required vs inferred schema.
+- Verticals (Tier 7) are mostly Tier-1 base + a baked-in domain schema + tuned prompts;
+  cheap to produce once base exists, high willingness-to-pay.