trainsieve 0.1.0__tar.gz

trainsieve-0.1.0/.gitignore

@@ -0,0 +1,18 @@
+.venv/
+__pycache__/
+*.pyc
+*.pyo
+*.egg-info/
+dist/
+build/
+.eggs/
+*.egg
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+*.db
+*.sqlite
+*.jsonl
+axolotl_config.yml
+output/
+export/

trainsieve-0.1.0/PKG-INFO

@@ -0,0 +1,11 @@
+Metadata-Version: 2.4
+Name: trainsieve
+Version: 0.1.0
+Summary: Production LLM traces → curated training datasets → fine-tune pipeline
+Requires-Python: >=3.11
+Requires-Dist: jsonlines>=4.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0
+Requires-Dist: sqlalchemy>=2.0
+Requires-Dist: typer>=0.12

trainsieve-0.1.0/README.md

@@ -0,0 +1,191 @@
+# sieve
+
+> Your model is only as good as the data you trained it on. Most teams never close that loop.
+
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org)
+[![Tests](https://img.shields.io/badge/tests-12%20passing-brightgreen)]()
+
+---
+
+Frontier labs (Anthropic, OpenAI, Google) continuously improve their models by mining production interactions for training signal. They call it a **data flywheel**. Every real user query that gets a great response becomes training data for the next version.
+
+Your team doesn't have that. You fine-tune once, ship, and hope.
+
+**sieve** is the missing piece: a CLI pipeline that turns your production LLM logs into a versioned, high-quality training dataset — and hands it off to Axolotl or LLaMA-Factory to retrain.
+
+```
+production logs ──► score ──► filter ──► versioned dataset ──► fine-tune
+```
+
+No API keys. No SaaS. Runs entirely on your machine.
+
+![sieve demo](demo.gif)
+
+---
+
+## The problem it solves
+
+You've fine-tuned a Llama model on your support chat data. It works well. Three months later, your product has changed, your users are asking different questions, and the model is drifting. You know the answers to those new questions exist somewhere in your logs.
+
+But finding them means:
+
+1. Manually exporting thousands of log rows
+2. Reading through them to find the good ones
+3. Formatting them for your training framework
+4. Remembering which examples you already used last time
+
+**sieve automates all of that.**
+
+---
+
+## Quickstart
+
+```bash
+git clone https://github.com/nidhisebastian008/sieve
+cd sieve
+python3 -m venv .venv && source .venv/bin/activate
+pip install -e .
+```
+
+```bash
+# ingest your logs (supports OpenAI, prompt/response, and conversations formats)
+sieve ingest production_logs.jsonl
+
+# score every interaction — no API key needed
+sieve score
+
+# see what you've got
+sieve stats
+```
+
+```
+     Sieve Pipeline Stats
+┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━┓
+┃ Metric             ┃ Value ┃
+┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━┩
+│ Total interactions │ 8420  │
+│ Scored             │ 8420  │
+│ Avg quality score  │ 0.614 │
+│ Dataset versions   │ 1     │
+│ Training runs      │ 0     │
+└────────────────────┴───────┘
+```
+
+```bash
+# curate only the good stuff
+sieve dataset create v1.0 --min-quality 0.7
+
+# export for fine-tuning
+sieve dataset export v1.0 --output train.jsonl
+
+# generate training config + command
+sieve train v1.0 --export-path train.jsonl --base-model meta-llama/Llama-3.2-3B-Instruct
+```
+
+```
+✓ Training run recorded (id: 32bd15fb)
+
+Run training with Axolotl:
+  pip install axolotl
+  axolotl train axolotl_config.yml
+```
+
+---
+
+## Dataset versioning
+
+The flywheel only works if you track what you've already trained on. sieve versions every dataset and supports incremental builds — next month's run only includes new interactions.
+
+```bash
+# next month: only new interactions not in v1.0
+sieve dataset create v2.0 --min-quality 0.7 --diff v1.0
+
+sieve dataset list
+```
+
+```
+┏━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ Name ┃ Interactions ┃ Min Quality ┃ Parent ┃ Created          ┃
+┡━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
+│ v1.0 │ 3102         │ 0.70        │ —      │ 2026-06-30 10:00 │
+│ v2.0 │ 891          │ 0.70        │ v1.0   │ 2026-07-30 09:00 │
+└──────┴──────────────┴─────────────┴────────┴──────────────────┘
+```
+
+Lineage is tracked in a local SQLite database at `~/.sieve/sieve.db`. You own your data.
+
+---
+
+## Input formats
+
+sieve normalises all common LLM log formats automatically:
+
+```jsonl
+{"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
+{"prompt": "...", "response": "..."}
+{"conversations": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}
+```
+
+---
+
+## Scorers
+
+| Scorer | Needs | What it checks |
+|---|---|---|
+| **Heuristic** (default) | Nothing | Response length, refusal patterns, empty messages |
+| **LLM-as-judge** (coming) | Ollama or API key | Helpfulness, factuality, relevance via model grading |
+
+---
+
+## Architecture
+
+sieve is built around pluggable interfaces. Swap out any stage.
+
+```
+sieve/
+├── ingest/      ← BaseIngester  (JSONL ✓ · Langfuse coming · OpenTelemetry coming)
+├── score/       ← BaseScorer   (Heuristic ✓ · LLM judge coming)
+├── curate/      ← versioning, lineage, diff, export
+└── trigger/     ← Axolotl ✓ · LLaMA-Factory coming · Modal coming
+```
+
+Implement `BaseIngester` or `BaseScorer`, drop it in — no core changes needed.
+
+---
+
+## Who this is for
+
+- Teams fine-tuning **open models** (Llama, Mistral, Qwen) on their own infrastructure
+- Companies with **data privacy requirements** that can't send logs to a third-party platform
+- Anyone doing **continuous fine-tuning** who is currently managing training data in spreadsheets
+
+If you're using OpenAI or Anthropic APIs and never touching open models, sieve is not for you — yet.
+
+---
+
+## Roadmap
+
+- [ ] Langfuse ingester — pull traces directly without manual export
+- [ ] OpenTelemetry ingester
+- [ ] LLM-as-judge scorer (Ollama + Anthropic/OpenAI)
+- [ ] LLaMA-Factory trigger
+- [ ] Modal cloud GPU trigger
+- [ ] HuggingFace dataset push
+- [ ] `sieve eval` — score base model vs fine-tuned on held-out examples
+
+---
+
+## Contributing
+
+PRs welcome. The most impactful areas right now: new ingesters and new scorers.
+
+```bash
+pytest tests/ -v   # 12 tests, all passing
+```
+
+---
+
+## License
+
+Apache 2.0

trainsieve-0.1.0/demo.tape

@@ -0,0 +1,57 @@
+Output demo.gif
+
+Set Shell "bash"
+Set FontSize 14
+Set Width 900
+Set Height 550
+Set Theme "Dracula"
+Set Padding 20
+Set TypingSpeed 40ms
+
+Hide
+Type "cd ~/sieve && source .venv/bin/activate"
+Enter
+Sleep 500ms
+Show
+
+Type "# sieve — production traces → training data pipeline"
+Sleep 1s
+Enter
+Sleep 300ms
+
+Type "sieve ingest /tmp/sample.jsonl"
+Sleep 500ms
+Enter
+Sleep 1.5s
+
+Type "sieve score"
+Sleep 500ms
+Enter
+Sleep 1.5s
+
+Type "sieve stats"
+Sleep 500ms
+Enter
+Sleep 2s
+
+Type "sieve dataset create v1.0 --min-quality 0.6"
+Sleep 500ms
+Enter
+Sleep 1.5s
+
+Type "sieve dataset list"
+Sleep 500ms
+Enter
+Sleep 2s
+
+Type "sieve dataset export v1.0 --output train.jsonl"
+Sleep 500ms
+Enter
+Sleep 1.5s
+
+Type "sieve train v1.0 --export-path train.jsonl"
+Sleep 500ms
+Enter
+Sleep 2s
+
+Sleep 2s

trainsieve-0.1.0/pyproject.toml

@@ -0,0 +1,23 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "trainsieve"
+version = "0.1.0"
+description = "Production LLM traces → curated training datasets → fine-tune pipeline"
+requires-python = ">=3.11"
+dependencies = [
+    "typer>=0.12",
+    "sqlalchemy>=2.0",
+    "pydantic>=2.0",
+    "rich>=13.0",
+    "jsonlines>=4.0",
+    "pyyaml>=6.0",
+]
+
+[project.scripts]
+sieve = "sieve.cli:app"
+
+[tool.hatch.build.targets.wheel]
+packages = ["sieve"]

trainsieve-0.1.0/sieve/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

trainsieve-0.1.0/sieve/cli.py

@@ -0,0 +1,175 @@
+from datetime import datetime
+from pathlib import Path
+from typing import Optional
+
+import typer
+from rich.console import Console
+from rich.table import Table
+
+app = typer.Typer(help="sieve — production traces → fine-tuning data pipeline", no_args_is_help=True)
+dataset_app = typer.Typer(no_args_is_help=True)
+app.add_typer(dataset_app, name="dataset", help="Dataset version management")
+
+console = Console()
+
+
+def _session():
+    from sieve.db import init_db, get_session
+    init_db()
+    return get_session()
+
+
+@app.command()
+def ingest(
+    path: Path = typer.Argument(..., help="JSONL file to ingest"),
+):
+    """Ingest interactions from a JSONL file."""
+    from sieve.ingest.jsonl import JSONLIngester
+
+    session = _session()
+    ingester = JSONLIngester(path)
+    count = 0
+    for interaction in ingester.ingest():
+        session.add(interaction)
+        count += 1
+    session.commit()
+    console.print(f"[green]✓[/green] Ingested [bold]{count}[/bold] interactions from {path}")
+
+
+@app.command()
+def score(
+    min_len: int = typer.Option(50, help="Min assistant response character length"),
+    max_len: int = typer.Option(4000, help="Max assistant response character length"),
+    rescore: bool = typer.Option(False, "--rescore", help="Re-score already scored interactions"),
+):
+    """Score interactions with heuristic scorer."""
+    from sieve.models import Interaction
+    from sieve.score.heuristic import HeuristicScorer
+
+    session = _session()
+    scorer = HeuristicScorer(min_len, max_len)
+
+    query = session.query(Interaction)
+    if not rescore:
+        query = query.filter(Interaction.quality_score.is_(None))
+
+    interactions = query.all()
+    for i in interactions:
+        i.quality_score = scorer.score(i)
+        i.scored_at = datetime.utcnow()
+        i.scorer = "heuristic"
+    session.commit()
+    console.print(f"[green]✓[/green] Scored [bold]{len(interactions)}[/bold] interactions")
+
+
+@app.command()
+def stats():
+    """Show pipeline stats."""
+    from sqlalchemy import func
+    from sieve.models import Interaction, DatasetVersion, TrainingRun
+
+    session = _session()
+
+    total = session.query(func.count(Interaction.id)).scalar()
+    scored = session.query(func.count(Interaction.id)).filter(Interaction.quality_score.isnot(None)).scalar()
+    avg = session.query(func.avg(Interaction.quality_score)).filter(Interaction.quality_score.isnot(None)).scalar()
+    versions = session.query(func.count(DatasetVersion.id)).scalar()
+    runs = session.query(func.count(TrainingRun.id)).scalar()
+
+    table = Table(title="Sieve Pipeline Stats")
+    table.add_column("Metric", style="bold")
+    table.add_column("Value")
+    table.add_row("Total interactions", str(total))
+    table.add_row("Scored", str(scored))
+    table.add_row("Avg quality score", f"{avg:.3f}" if avg else "—")
+    table.add_row("Dataset versions", str(versions))
+    table.add_row("Training runs", str(runs))
+    console.print(table)
+
+
+@dataset_app.command("create")
+def dataset_create(
+    name: str = typer.Argument(..., help="Version name, e.g. v1.0"),
+    min_quality: float = typer.Option(0.6, help="Minimum quality score filter"),
+    description: Optional[str] = typer.Option(None, help="Description"),
+    diff: Optional[str] = typer.Option(None, help="Parent version — only include new interactions"),
+):
+    """Create a dataset version from scored interactions."""
+    from sieve.curate.dataset import DatasetManager
+
+    session = _session()
+    mgr = DatasetManager(session)
+    version = mgr.create_version(name, min_quality, description, diff)
+    console.print(
+        f"[green]✓[/green] Dataset [bold]{name!r}[/bold] created "
+        f"with [bold]{len(version.interactions)}[/bold] interactions "
+        f"(min_quality={min_quality})"
+    )
+
+
+@dataset_app.command("export")
+def dataset_export(
+    name: str = typer.Argument(..., help="Version name to export"),
+    output: Path = typer.Option(Path("./export.jsonl"), help="Output JSONL path"),
+):
+    """Export a dataset version to JSONL for fine-tuning."""
+    from sieve.curate.dataset import DatasetManager
+
+    session = _session()
+    mgr = DatasetManager(session)
+    count = mgr.export_jsonl(name, output)
+    console.print(f"[green]✓[/green] Exported [bold]{count}[/bold] interactions to {output}")
+
+
+@dataset_app.command("list")
+def dataset_list():
+    """List all dataset versions."""
+    from sieve.models import DatasetVersion
+
+    session = _session()
+    versions = session.query(DatasetVersion).order_by(DatasetVersion.created_at).all()
+
+    if not versions:
+        console.print("[yellow]No dataset versions yet. Run: sieve dataset create[/yellow]")
+        return
+
+    table = Table(title="Dataset Versions")
+    table.add_column("Name", style="bold")
+    table.add_column("Interactions")
+    table.add_column("Min Quality")
+    table.add_column("Parent")
+    table.add_column("Created")
+    for v in versions:
+        table.add_row(
+            v.name,
+            str(len(v.interactions)),
+            f"{v.min_quality_score:.2f}" if v.min_quality_score is not None else "—",
+            v.parent_name or "—",
+            v.created_at.strftime("%Y-%m-%d %H:%M"),
+        )
+    console.print(table)
+
+
+@app.command()
+def train(
+    version: str = typer.Argument(..., help="Dataset version name"),
+    base_model: str = typer.Option("meta-llama/Llama-3.2-3B-Instruct", help="HuggingFace model ID"),
+    output_dir: Path = typer.Option(Path("./output"), help="Training output directory"),
+    export_path: Path = typer.Option(Path("./export.jsonl"), help="Path to exported JSONL dataset"),
+):
+    """Generate Axolotl config and show the training command."""
+    from sieve.trigger.axolotl import AxolotlTrigger
+
+    session = _session()
+    trigger = AxolotlTrigger(session)
+    run = trigger.trigger(version, export_path, base_model, output_dir)
+    config_path = run.config["config_path"]
+
+    console.print(f"[green]✓[/green] Training run recorded [dim](id: {run.id[:8]})[/dim]")
+    console.print(f"\n[bold]Run training with Axolotl:[/bold]")
+    console.print(f"  pip install axolotl")
+    console.print(f"  axolotl train {config_path}")
+
+
+if __name__ == "__main__":
+    app()

trainsieve-0.1.0/sieve/curate/dataset.py

@@ -0,0 +1,60 @@
+from pathlib import Path
+from typing import Optional
+
+import jsonlines
+from sqlalchemy.orm import Session
+
+from sieve.models import DatasetVersion, Interaction
+
+
+class DatasetManager:
+    def __init__(self, session: Session):
+        self.session = session
+
+    def create_version(
+        self,
+        name: str,
+        min_quality: float = 0.0,
+        description: Optional[str] = None,
+        parent_name: Optional[str] = None,
+    ) -> DatasetVersion:
+        query = self.session.query(Interaction)
+
+        if min_quality > 0:
+            query = query.filter(
+                Interaction.quality_score >= min_quality,
+                Interaction.quality_score.isnot(None),
+            )
+
+        if parent_name:
+            parent = self.session.query(DatasetVersion).filter_by(name=parent_name).first()
+            if parent:
+                parent_ids = [i.id for i in parent.interactions]
+                if parent_ids:
+                    query = query.filter(~Interaction.id.in_(parent_ids))
+
+        interactions = query.all()
+
+        version = DatasetVersion(
+            name=name,
+            description=description,
+            parent_name=parent_name,
+            min_quality_score=min_quality,
+        )
+        version.interactions = interactions
+        self.session.add(version)
+        self.session.commit()
+        return version
+
+    def export_jsonl(self, version_name: str, output_path: Path) -> int:
+        version = self.session.query(DatasetVersion).filter_by(name=version_name).first()
+        if not version:
+            raise ValueError(f"Version {version_name!r} not found")
+
+        output_path.parent.mkdir(parents=True, exist_ok=True)
+        count = 0
+        with jsonlines.open(output_path, mode="w") as writer:
+            for interaction in version.interactions:
+                writer.write({"messages": interaction.messages})
+                count += 1
+        return count

trainsieve-0.1.0/sieve/db.py

@@ -0,0 +1,24 @@
+from pathlib import Path
+
+from sqlalchemy import create_engine
+from sqlalchemy.orm import Session, sessionmaker
+
+from sieve.models import Base
+
+_DEFAULT_DB = Path.home() / ".sieve" / "sieve.db"
+
+
+def get_engine(db_path: Path = _DEFAULT_DB):
+    db_path.parent.mkdir(parents=True, exist_ok=True)
+    return create_engine(f"sqlite:///{db_path}", echo=False)
+
+
+def init_db(db_path: Path = _DEFAULT_DB):
+    engine = get_engine(db_path)
+    Base.metadata.create_all(engine)
+    return engine
+
+
+def get_session(db_path: Path = _DEFAULT_DB) -> Session:
+    engine = get_engine(db_path)
+    return sessionmaker(bind=engine)()

trainsieve-0.1.0/sieve/ingest/base.py

@@ -0,0 +1,10 @@
+from abc import ABC, abstractmethod
+from typing import Iterator
+
+from sieve.models import Interaction
+
+
+class BaseIngester(ABC):
+    @abstractmethod
+    def ingest(self) -> Iterator[Interaction]:
+        pass

trainsieve-0.1.0/sieve/ingest/jsonl.py

@@ -0,0 +1,35 @@
+from pathlib import Path
+from typing import Iterator
+
+import jsonlines
+
+from sieve.ingest.base import BaseIngester
+from sieve.models import Interaction
+
+
+class JSONLIngester(BaseIngester):
+    def __init__(self, path: Path):
+        self.path = path
+
+    def ingest(self) -> Iterator[Interaction]:
+        with jsonlines.open(self.path) as reader:
+            for obj in reader:
+                messages = obj.get("messages") or obj.get("conversations") or []
+
+                if not messages and "prompt" in obj and "response" in obj:
+                    messages = [
+                        {"role": "user", "content": obj["prompt"]},
+                        {"role": "assistant", "content": obj["response"]},
+                    ]
+
+                if not messages:
+                    continue
+
+                skip_keys = {"messages", "conversations", "prompt", "response"}
+                meta = {k: v for k, v in obj.items() if k not in skip_keys}
+
+                yield Interaction(
+                    source="jsonl",
+                    messages=messages,
+                    metadata_={"source_file": str(self.path), **meta},
+                )

trainsieve-0.1.0/sieve/models.py

@@ -0,0 +1,63 @@
+from datetime import datetime
+from uuid import uuid4
+
+from sqlalchemy import Column, DateTime, Float, ForeignKey, JSON, String, Table
+from sqlalchemy.orm import DeclarativeBase, relationship
+
+
+class Base(DeclarativeBase):
+    pass
+
+
+interaction_dataset = Table(
+    "interaction_dataset",
+    Base.metadata,
+    Column("interaction_id", String, ForeignKey("interactions.id")),
+    Column("dataset_version_id", String, ForeignKey("dataset_versions.id")),
+)
+
+
+class Interaction(Base):
+    __tablename__ = "interactions"
+
+    id = Column(String, primary_key=True, default=lambda: str(uuid4()))
+    source = Column(String, nullable=False)
+    messages = Column(JSON, nullable=False)  # OpenAI chat format
+    metadata_ = Column("metadata", JSON, default=dict)
+    ingested_at = Column(DateTime, default=datetime.utcnow)
+    quality_score = Column(Float, nullable=True)
+    scored_at = Column(DateTime, nullable=True)
+    scorer = Column(String, nullable=True)
+
+    dataset_versions = relationship(
+        "DatasetVersion", secondary=interaction_dataset, back_populates="interactions"
+    )
+
+
+class DatasetVersion(Base):
+    __tablename__ = "dataset_versions"
+
+    id = Column(String, primary_key=True, default=lambda: str(uuid4()))
+    name = Column(String, nullable=False, unique=True)
+    created_at = Column(DateTime, default=datetime.utcnow)
+    description = Column(String, nullable=True)
+    parent_name = Column(String, nullable=True)
+    min_quality_score = Column(Float, nullable=True)
+
+    interactions = relationship(
+        "Interaction", secondary=interaction_dataset, back_populates="dataset_versions"
+    )
+
+
+class TrainingRun(Base):
+    __tablename__ = "training_runs"
+
+    id = Column(String, primary_key=True, default=lambda: str(uuid4()))
+    dataset_version_name = Column(String, ForeignKey("dataset_versions.name"))
+    backend = Column(String, nullable=False)
+    config = Column(JSON, default=dict)
+    status = Column(String, default="triggered")
+    created_at = Column(DateTime, default=datetime.utcnow)
+    completed_at = Column(DateTime, nullable=True)
+    output_path = Column(String, nullable=True)
+    notes = Column(String, nullable=True)

trainsieve-0.1.0/sieve/score/base.py

@@ -0,0 +1,10 @@
+from abc import ABC, abstractmethod
+
+from sieve.models import Interaction
+
+
+class BaseScorer(ABC):
+    @abstractmethod
+    def score(self, interaction: Interaction) -> float:
+        """Return quality score 0.0–1.0."""
+        pass

trainsieve-0.1.0/sieve/score/heuristic.py

@@ -0,0 +1,46 @@
+from sieve.models import Interaction
+from sieve.score.base import BaseScorer
+
+_BAD_PATTERNS = [
+    "i cannot",
+    "i'm unable",
+    "i don't have access",
+    "as an ai",
+    "i apologize, but",
+    "i'm sorry, but i can't",
+]
+
+
+class HeuristicScorer(BaseScorer):
+    def __init__(self, min_response_len: int = 50, max_response_len: int = 4000):
+        self.min_response_len = min_response_len
+        self.max_response_len = max_response_len
+
+    def score(self, interaction: Interaction) -> float:
+        messages = interaction.messages or []
+        assistant_msgs = [m for m in messages if m.get("role") == "assistant"]
+        user_msgs = [m for m in messages if m.get("role") == "user"]
+
+        if not assistant_msgs or not user_msgs:
+            return 0.0
+
+        scores = []
+        for msg in assistant_msgs:
+            content = msg.get("content", "")
+            if not isinstance(content, str):
+                content = str(content)
+
+            length = len(content)
+            if length < self.min_response_len:
+                length_score = length / self.min_response_len
+            elif length > self.max_response_len:
+                overage = (length - self.max_response_len) / self.max_response_len
+                length_score = max(0.5, 1.0 - overage)
+            else:
+                length_score = 1.0
+
+            lowered = content.lower()
+            penalty = sum(0.15 for p in _BAD_PATTERNS if p in lowered)
+            scores.append(max(0.0, length_score - penalty))
+
+        return sum(scores) / len(scores)

trainsieve-0.1.0/sieve/trigger/axolotl.py

@@ -0,0 +1,61 @@
+from pathlib import Path
+from typing import Optional
+
+import yaml
+from sqlalchemy.orm import Session
+
+from sieve.models import TrainingRun
+
+
+class AxolotlTrigger:
+    def __init__(self, session: Session):
+        self.session = session
+
+    def generate_config(
+        self,
+        dataset_path: Path,
+        base_model: str,
+        output_dir: Path,
+        config_path: Optional[Path] = None,
+    ) -> Path:
+        config = {
+            "base_model": base_model,
+            "datasets": [{"path": str(dataset_path.resolve()), "type": "sharegpt"}],
+            "output_dir": str(output_dir.resolve()),
+            "sequence_len": 2048,
+            "micro_batch_size": 2,
+            "num_epochs": 3,
+            "learning_rate": 2e-4,
+            "lora_r": 16,
+            "lora_alpha": 32,
+            "lora_dropout": 0.05,
+            "lora_target_modules": ["q_proj", "v_proj"],
+            "bf16": True,
+            "load_in_4bit": True,
+            "val_set_size": 0.05,
+            "logging_steps": 10,
+            "save_steps": 100,
+        }
+
+        out = config_path or dataset_path.parent / "axolotl_config.yml"
+        out.write_text(yaml.dump(config, default_flow_style=False))
+        return out
+
+    def trigger(
+        self,
+        dataset_version_name: str,
+        dataset_path: Path,
+        base_model: str,
+        output_dir: Path,
+    ) -> TrainingRun:
+        config_path = self.generate_config(dataset_path, base_model, output_dir)
+
+        run = TrainingRun(
+            dataset_version_name=dataset_version_name,
+            backend="axolotl",
+            config={"config_path": str(config_path), "base_model": base_model},
+            status="triggered",
+        )
+        self.session.add(run)
+        self.session.commit()
+        return run

trainsieve-0.1.0/tests/test_dataset.py

@@ -0,0 +1,80 @@
+import tempfile
+from pathlib import Path
+
+import pytest
+from sqlalchemy import create_engine
+from sqlalchemy.orm import sessionmaker
+
+from sieve.curate.dataset import DatasetManager
+from sieve.models import Base, Interaction
+
+
+@pytest.fixture
+def session():
+    engine = create_engine("sqlite:///:memory:")
+    Base.metadata.create_all(engine)
+    Session = sessionmaker(bind=engine)
+    s = Session()
+    yield s
+    s.close()
+
+
+def _add_interaction(session, score: float, content: str = "test response " * 10):
+    i = Interaction(
+        source="test",
+        messages=[
+            {"role": "user", "content": "question"},
+            {"role": "assistant", "content": content},
+        ],
+        quality_score=score,
+    )
+    session.add(i)
+    session.commit()
+    return i
+
+
+def test_create_version_filters_by_quality(session):
+    _add_interaction(session, score=0.9)
+    _add_interaction(session, score=0.3)
+    _add_interaction(session, score=0.7)
+
+    mgr = DatasetManager(session)
+    v = mgr.create_version("v1.0", min_quality=0.6)
+    assert len(v.interactions) == 2
+
+
+def test_create_version_diff_excludes_parent(session):
+    i1 = _add_interaction(session, score=0.9)
+    i2 = _add_interaction(session, score=0.9)
+
+    mgr = DatasetManager(session)
+    v1 = mgr.create_version("v1.0", min_quality=0.5)
+    assert len(v1.interactions) == 2
+
+    i3 = _add_interaction(session, score=0.9)
+    v2 = mgr.create_version("v2.0", min_quality=0.5, parent_name="v1.0")
+    assert len(v2.interactions) == 1
+    assert v2.interactions[0].id == i3.id
+
+
+def test_export_jsonl(session, tmp_path):
+    _add_interaction(session, score=0.9)
+    mgr = DatasetManager(session)
+    mgr.create_version("v1.0", min_quality=0.5)
+
+    out = tmp_path / "export.jsonl"
+    count = mgr.export_jsonl("v1.0", out)
+    assert count == 1
+    assert out.exists()
+
+    import jsonlines
+    with jsonlines.open(out) as r:
+        rows = list(r)
+    assert len(rows) == 1
+    assert "messages" in rows[0]
+
+
+def test_export_unknown_version_raises(session, tmp_path):
+    mgr = DatasetManager(session)
+    with pytest.raises(ValueError, match="not found"):
+        mgr.export_jsonl("nonexistent", tmp_path / "out.jsonl")

trainsieve-0.1.0/tests/test_ingest.py

@@ -0,0 +1,49 @@
+import json
+import tempfile
+from pathlib import Path
+
+from sieve.ingest.jsonl import JSONLIngester
+
+
+def _write_jsonl(path: Path, rows: list) -> None:
+    with open(path, "w") as f:
+        for row in rows:
+            f.write(json.dumps(row) + "\n")
+
+
+def test_ingest_messages_format():
+    with tempfile.NamedTemporaryFile(suffix=".jsonl", mode="w", delete=False) as f:
+        path = Path(f.name)
+
+    _write_jsonl(path, [
+        {"messages": [{"role": "user", "content": "hi"}, {"role": "assistant", "content": "hello"}]},
+    ])
+    results = list(JSONLIngester(path).ingest())
+    assert len(results) == 1
+    assert results[0].messages[0]["role"] == "user"
+    assert results[0].source == "jsonl"
+
+
+def test_ingest_prompt_response_format():
+    with tempfile.NamedTemporaryFile(suffix=".jsonl", mode="w", delete=False) as f:
+        path = Path(f.name)
+
+    _write_jsonl(path, [
+        {"prompt": "What is Python?", "response": "A programming language."},
+    ])
+    results = list(JSONLIngester(path).ingest())
+    assert len(results) == 1
+    assert results[0].messages[0]["content"] == "What is Python?"
+    assert results[0].messages[1]["role"] == "assistant"
+
+
+def test_ingest_skips_empty_messages():
+    with tempfile.NamedTemporaryFile(suffix=".jsonl", mode="w", delete=False) as f:
+        path = Path(f.name)
+
+    _write_jsonl(path, [
+        {"no_messages": True},
+        {"messages": [{"role": "user", "content": "valid"}, {"role": "assistant", "content": "ok"}]},
+    ])
+    results = list(JSONLIngester(path).ingest())
+    assert len(results) == 1

trainsieve-0.1.0/tests/test_score.py

@@ -0,0 +1,40 @@
+from sieve.models import Interaction
+from sieve.score.heuristic import HeuristicScorer
+
+
+def _make(user: str, assistant: str) -> Interaction:
+    return Interaction(
+        source="test",
+        messages=[
+            {"role": "user", "content": user},
+            {"role": "assistant", "content": assistant},
+        ],
+    )
+
+
+scorer = HeuristicScorer()
+
+
+def test_good_response_scores_high():
+    i = _make("Explain recursion", "Recursion is when a function calls itself. " * 10)
+    assert scorer.score(i) >= 0.8
+
+
+def test_too_short_scores_low():
+    i = _make("What is 2+2?", "4")
+    assert scorer.score(i) < 0.5
+
+
+def test_bad_pattern_penalized():
+    i = _make("Help me", "As an AI, I cannot assist with that. " * 5)
+    assert scorer.score(i) <= 0.7
+
+
+def test_no_assistant_message_returns_zero():
+    i = Interaction(source="test", messages=[{"role": "user", "content": "hi"}])
+    assert scorer.score(i) == 0.0
+
+
+def test_empty_messages_returns_zero():
+    i = Interaction(source="test", messages=[])
+    assert scorer.score(i) == 0.0