runapprentice 0.1.0__tar.gz → 0.2.0__tar.gz

{runapprentice-0.1.0 → runapprentice-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: runapprentice
-Version: 0.1.0
+Version: 0.2.0
 Summary: Cut LLM cost without losing quality: capture traffic, build golden datasets, and optimize prompts.
 Project-URL: Homepage, https://runapprentice.com
 Project-URL: Documentation, https://docs.runapprentice.com
@@ -21,18 +21,44 @@ Requires-Dist: httpx>=0.27
 Requires-Dist: pydantic>=2.0
 Provides-Extra: langchain
 Requires-Dist: langchain-core>=0.3; extra == 'langchain'
+Provides-Extra: local
+Requires-Dist: mlx-lm<0.32,>=0.31.3; extra == 'local'
+Provides-Extra: optimize
+Requires-Dist: dspy<4,>=3.2.1; extra == 'optimize'
 Description-Content-Type: text/markdown
 
 # runapprentice
 
-Capture your LLM traffic, build a golden dataset, and optimize your prompt against it. Training a small model to take over from the frontier model is planned for a later release.
+The Python SDK for [Apprentice](https://runapprentice.com), a tool that cuts frontier-LLM cost
+on repeatable tasks. You give it a dataset of inputs and correct outputs for one task (uploaded
+as a CSV, or captured live from a LangChain app with one callback). Apprentice runs prompt
+optimization (DSPy GEPA) against the verified rows (gold plus silver), or fine-tunes a small open model on
+your Mac, and reports the score change on a held-out slice, so the gain is measured, never
+claimed. Gated, eval-verified takeover of production traffic is still in development.
 
-This README documents the **implemented surface** (v0.1). Anything not shown here does not exist yet.
+Docs: [docs.runapprentice.com](https://docs.runapprentice.com) · Reproducible benchmark:
+[apprentice-benchmark](https://github.com/singhabhishekkk/apprentice-benchmark)
+
+This README documents the **implemented surface**. Anything not shown here does not exist yet.
+
+## CLI (no code, no account required)
+
+```bash
+uv add 'runapprentice[optimize]'   # add [local] too for MLX fine-tuning on Apple silicon
+
+apprentice optimize <task> --local --data golden.csv              # free, your OPENAI_API_KEY
+apprentice train <task> --local --data golden.csv --effort high   # free, fine-tunes on your Mac
+```
+
+Both commands work on a CSV alone, no Apprentice account. No golden dataset yet? `apprentice
+login` also lets Apprentice generate a starter set and route rows to a subject-matter expert
+for review, and tracks every run in the console. `apprentice --help` (or any
+subcommand `--help`) documents every flag; `--json` gives clean machine-readable output.
 
 ## Install
 
 ```bash
-uv add runapprentice[langchain]    # [langchain] extra enables zero-code capture
+uv add 'runapprentice[langchain]'    # [langchain] extra enables zero-code capture
 ```
 
 ## Quickstart (the whole trial journey)
@@ -163,11 +189,25 @@ want grounding + refusal optimized together.
 
 ## Feedback (your end-users' signal)
 
+Capture records the call; feedback records whether it worked. That score is what the
+console's Drift tab charts, and what decides when a retrain is worth doing.
+
 ```python
-client.feedback(trace_id, good=True)                  # thumbs
-client.feedback(trace_id, score=0.5, note="partial")  # graded
+trace_id = client.capture(task="support-triage", input=question, output=answer)
+if trace_id:  # None means the capture was rejected locally; capture never raises
+    client.feedback(trace_id, good=True)                  # thumbs
+    client.feedback(trace_id, score=0.5, note="partial")  # graded
 ```
 
+In async mode (the default), `feedback()` first waits up to two seconds for pending
+captures to deliver, so it can never race its own trace; if the queue cannot drain in
+time it raises `ApprenticeError` instead of sending a request that would 404.
+
+Wire it to a signal you already have: a thumbs up or down, the user accepting or discarding
+the output, a downstream check that passed or failed. Do not add a second model to grade the
+first one, this score decides when to retrain. With no real signal, send nothing: captured
+rows still become gold when a human verifies them.
+
 ## Debugging
 
 ```python
@@ -195,7 +235,7 @@ count and the minimum).
 | `job.report()` | `OptimizationReport(baseline_score, optimized_score, optimized_prompt, ...)` | if job has no report |
 | `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score)` | 404 if never optimized |
 | `client.prompts.history(task)` | `list[PromptVersion]` | `ApprenticeError` on API failure |
-| `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace |
+| `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace; `ApprenticeError` if pending captures cannot drain in time, or on API failure |
 | `client.post_trace_failopen(record)` | trace_id or None | **never raises** |
 | `ApprenticeCallback(task, client, redact=None)` | LangChain callback | **never raises into your chain** |
 

{runapprentice-0.1.0 → runapprentice-0.2.0}/README.md

@@ -1,13 +1,35 @@
 # runapprentice
 
-Capture your LLM traffic, build a golden dataset, and optimize your prompt against it. Training a small model to take over from the frontier model is planned for a later release.
+The Python SDK for [Apprentice](https://runapprentice.com), a tool that cuts frontier-LLM cost
+on repeatable tasks. You give it a dataset of inputs and correct outputs for one task (uploaded
+as a CSV, or captured live from a LangChain app with one callback). Apprentice runs prompt
+optimization (DSPy GEPA) against the verified rows (gold plus silver), or fine-tunes a small open model on
+your Mac, and reports the score change on a held-out slice, so the gain is measured, never
+claimed. Gated, eval-verified takeover of production traffic is still in development.
 
-This README documents the **implemented surface** (v0.1). Anything not shown here does not exist yet.
+Docs: [docs.runapprentice.com](https://docs.runapprentice.com) · Reproducible benchmark:
+[apprentice-benchmark](https://github.com/singhabhishekkk/apprentice-benchmark)
+
+This README documents the **implemented surface**. Anything not shown here does not exist yet.
+
+## CLI (no code, no account required)
+
+```bash
+uv add 'runapprentice[optimize]'   # add [local] too for MLX fine-tuning on Apple silicon
+
+apprentice optimize <task> --local --data golden.csv              # free, your OPENAI_API_KEY
+apprentice train <task> --local --data golden.csv --effort high   # free, fine-tunes on your Mac
+```
+
+Both commands work on a CSV alone, no Apprentice account. No golden dataset yet? `apprentice
+login` also lets Apprentice generate a starter set and route rows to a subject-matter expert
+for review, and tracks every run in the console. `apprentice --help` (or any
+subcommand `--help`) documents every flag; `--json` gives clean machine-readable output.
 
 ## Install
 
 ```bash
-uv add runapprentice[langchain]    # [langchain] extra enables zero-code capture
+uv add 'runapprentice[langchain]'    # [langchain] extra enables zero-code capture
 ```
 
 ## Quickstart (the whole trial journey)
@@ -138,11 +160,25 @@ want grounding + refusal optimized together.
 
 ## Feedback (your end-users' signal)
 
+Capture records the call; feedback records whether it worked. That score is what the
+console's Drift tab charts, and what decides when a retrain is worth doing.
+
 ```python
-client.feedback(trace_id, good=True)                  # thumbs
-client.feedback(trace_id, score=0.5, note="partial")  # graded
+trace_id = client.capture(task="support-triage", input=question, output=answer)
+if trace_id:  # None means the capture was rejected locally; capture never raises
+    client.feedback(trace_id, good=True)                  # thumbs
+    client.feedback(trace_id, score=0.5, note="partial")  # graded
 ```
 
+In async mode (the default), `feedback()` first waits up to two seconds for pending
+captures to deliver, so it can never race its own trace; if the queue cannot drain in
+time it raises `ApprenticeError` instead of sending a request that would 404.
+
+Wire it to a signal you already have: a thumbs up or down, the user accepting or discarding
+the output, a downstream check that passed or failed. Do not add a second model to grade the
+first one, this score decides when to retrain. With no real signal, send nothing: captured
+rows still become gold when a human verifies them.
+
 ## Debugging
 
 ```python
@@ -170,7 +206,7 @@ count and the minimum).
 | `job.report()` | `OptimizationReport(baseline_score, optimized_score, optimized_prompt, ...)` | if job has no report |
 | `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score)` | 404 if never optimized |
 | `client.prompts.history(task)` | `list[PromptVersion]` | `ApprenticeError` on API failure |
-| `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace |
+| `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace; `ApprenticeError` if pending captures cannot drain in time, or on API failure |
 | `client.post_trace_failopen(record)` | trace_id or None | **never raises** |
 | `ApprenticeCallback(task, client, redact=None)` | LangChain callback | **never raises into your chain** |
 

{runapprentice-0.1.0 → runapprentice-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "runapprentice"
-version = "0.1.0"
+version = "0.2.0"
 description = "Cut LLM cost without losing quality: capture traffic, build golden datasets, and optimize prompts."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -23,12 +23,17 @@ dependencies = [
     "pydantic>=2.0",
 ]
 
+[project.scripts]
+apprentice = "runapprentice.cli:main"
+
 [project.urls]
 Homepage = "https://runapprentice.com"
 Documentation = "https://docs.runapprentice.com"
 
 [project.optional-dependencies]
 langchain = ["langchain-core>=0.3"]
+local = ["mlx-lm>=0.31.3,<0.32"]
+optimize = ["dspy>=3.2.1,<4"]
 
 [dependency-groups]
 dev = [

{runapprentice-0.1.0 → runapprentice-0.2.0}/src/runapprentice/__init__.py

@@ -5,6 +5,8 @@ from runapprentice.models import (
     DatasetRow,
     DatasetStatus,
     JobInfo,
+    JobSummary,
+    LocalRun,
     OptimizationReport,
     PromptVersion,
     TraceRecord,
@@ -18,6 +20,8 @@ __all__ = [
     "DatasetRow",
     "DatasetStatus",
     "JobInfo",
+    "JobSummary",
+    "LocalRun",
     "OptimizationReport",
     "PromptVersion",
     "TraceRecord",

{runapprentice-0.1.0 → runapprentice-0.2.0}/src/runapprentice/_capture.py

@@ -8,8 +8,8 @@ blip must not silently drop a trace on the first try). This dispatcher gives bot
   Apprentice's network).
 - A single daemon worker drains the queue and delivers each trace with bounded
   retry + exponential backoff.
-- The queue is bounded; if Apprentice's backend is down long enough to fill it, the
-  oldest pending traces are dropped (fail-open — the customer app is never affected).
+- The queue is bounded; if Apprentice's backend is down long enough to fill it, new
+  traces are rejected (fail-open; the customer app is never affected).
 - `flush()` (also run at interpreter exit) drains in-flight traces so short scripts
   don't lose their last captures.
 """
@@ -55,29 +55,20 @@ class CaptureDispatcher:
         self._worker.start()
         atexit.register(self.flush)
 
-    def submit(self, record: TraceRecord) -> None:
-        """Enqueue a trace without blocking. Drops the oldest if the queue is full."""
+    def submit(self, record: TraceRecord) -> bool:
+        """Enqueue without blocking. Return whether the trace was accepted locally."""
         try:
             self._queue.put_nowait(record)
+            return True
         except queue.Full:
-            # Backend is unreachable and the buffer is full — drop the oldest pending
-            # trace to make room for the newest, so we keep the freshest signal.
-            try:
-                self._queue.get_nowait()
-                self._queue.task_done()
-            except queue.Empty:
-                pass
             self._dropped += 1
             if self._dropped == 1 or self._dropped % 100 == 0:
                 logger.warning(
-                    "runapprentice: capture buffer full, dropped %d trace(s) "
+                    "runapprentice: capture buffer full, rejected %d trace(s) "
                     "(backend unreachable; app unaffected)",
                     self._dropped,
                 )
-            try:
-                self._queue.put_nowait(record)
-            except queue.Full:
-                pass
+            return False
 
     def _run(self) -> None:
         while True:
@@ -103,15 +94,17 @@ class CaptureDispatcher:
         if last_error is not None:
             self._on_exhausted(f"capture failed after {self._retries + 1} attempts: {last_error}")
 
-    def flush(self, timeout: float = 2.0) -> None:
+    def flush(self, timeout: float = 2.0) -> bool:
         """Wait up to `timeout` seconds for in-flight traces to finish delivering.
 
         Counts items as done only after the worker calls `task_done()` (i.e. after
         delivery + any retries complete), not merely when they leave the queue.
+        Returns whether every accepted trace completed within the timeout.
         """
         deadline = time.monotonic() + timeout
         while self._queue.unfinished_tasks > 0 and time.monotonic() < deadline:
             time.sleep(0.02)
+        return self._queue.unfinished_tasks == 0
 
     def close(self) -> None:
         self._queue.put(_SHUTDOWN)