runapprentice 0.1.0__py3-none-any.whl

runapprentice-0.1.0.dist-info/METADATA

@@ -0,0 +1,219 @@
+Metadata-Version: 2.4
+Name: runapprentice
+Version: 0.1.0
+Summary: Cut LLM cost without losing quality: capture traffic, build golden datasets, and optimize prompts.
+Project-URL: Homepage, https://runapprentice.com
+Project-URL: Documentation, https://docs.runapprentice.com
+Author: Abhishek Singh
+License-Expression: MIT
+License-File: LICENSE
+Keywords: dspy,evaluation,gepa,langchain,llm,prompt-optimization,rag
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.0
+Provides-Extra: langchain
+Requires-Dist: langchain-core>=0.3; extra == 'langchain'
+Description-Content-Type: text/markdown
+
+# runapprentice
+
+Capture your LLM traffic, build a golden dataset, and optimize your prompt against it. Training a small model to take over from the frontier model is planned for a later release.
+
+This README documents the **implemented surface** (v0.1). Anything not shown here does not exist yet.
+
+## Install
+
+```bash
+uv add runapprentice[langchain]    # [langchain] extra enables zero-code capture
+```
+
+## Quickstart (the whole trial journey)
+
+```python
+from runapprentice import Apprentice
+
+client = Apprentice(
+    api_key="ap_live_...",   # create once in the console, or env APPRENTICE_API_KEY
+)                            # talks to the hosted API; set base_url only to self-host
+
+# 1. Create a task (a "task" = one repeatable LLM job you want to make cheaper)
+client.tasks.create("ticket-triage")
+
+# 2. Point us at the data you already have (CSV with input/output columns)
+client.datasets.upload(
+    "ticket-triage",
+    path="golden.csv",
+    input_col="input",
+    output_col="output",
+    prompt="...your current system prompt...",   # the baseline to beat
+)
+
+# 3. Optimize the prompt against your data (GEPA, held-out eval)
+job = client.optimize("ticket-triage")
+report = job.wait().report()
+print(report.baseline_score, "->", report.optimized_score)
+
+# 4. Pull the optimized prompt back into your code (versioned)
+best = client.prompts.get("ticket-triage")
+print(best.version, best.text)
+```
+
+## Live capture: LangChain (one line, zero code changes)
+
+```python
+# init_chat_model comes from your LangChain model package (for example
+# langchain[openai]); runapprentice[langchain] only adds langchain-core for
+# the callback.
+from langchain.chat_models import init_chat_model
+from runapprentice.langchain import ApprenticeCallback
+
+model = init_chat_model(
+    "gpt-5.5",   # whatever YOU already use. We observe, we don't choose
+    callbacks=[ApprenticeCallback("ticket-triage", client)],
+)
+# Every call now logs input/output/model/latency/tokens to the task's dataset (raw tier).
+```
+
+**Fail-open guarantee:** the capture path can NEVER break your app. If the
+Apprentice backend is down, unreachable, or slow, your LLM calls proceed
+untouched. The only loss is the trace. First drop logs a WARNING, repeats log
+at DEBUG.
+
+**PII redaction** runs in your process, before anything is transmitted:
+
+```python
+ApprenticeCallback("ticket-triage", client, redact=lambda s: my_scrubber(s))
+```
+
+## RAG quickstart (grounding + refusal-aware)
+
+For retrieval-augmented tasks, each row carries the **exact context the model saw**
+(the retrieved passages), not just the question. Pick the `rag_composite` metric so
+optimization rewards grounding and correct refusals, not just answer overlap.
+
+```python
+from runapprentice import Apprentice
+client = Apprentice(api_key="ap_live_...")
+
+# rag_composite scores answer correctness + faithfulness to context + refusal
+# correctness (it should say "not enough information" when the context lacks it).
+client.tasks.create("support-rag", metric="rag_composite")
+
+client.datasets.upload("support-rag", rows=[
+    {"inputs": {"question": q, "context": retrieved_passages}, "output": gold_answer}
+    for q, retrieved_passages, gold_answer in your_golden_set
+])
+
+client.prompts.register("support-rag", {
+    "format": "f-string",
+    "messages": [
+        {"role": "system", "template":
+            "Answer using only the context. If the context does not contain the "
+            "answer, say you do not have enough information.\n{context}"},
+        {"role": "human", "template": "{question}"},
+    ],
+    "input_variables": ["context", "question"],
+})
+
+report = client.optimize("support-rag").wait().report()
+print(report.baseline_score, "->", report.optimized_score)
+```
+
+**RAG capture for simple chains.** For a standard LangChain RAG chain, the
+`ApprenticeCallback` records the retrieved context from `on_retriever_end`, capturing
+the `{question, context}` shape. For custom formatting, multiple retrievers, or
+non-standard chains, call `client.capture(..., inputs={"question": ..., "context": ...})`
+explicitly.
+
+### Register a LangChain prompt directly
+
+`prompts.register` also accepts a LangChain `ChatPromptTemplate`, no raw dict needed:
+
+```python
+from langchain_core.prompts import ChatPromptTemplate
+client.prompts.register("support-rag", ChatPromptTemplate.from_messages([
+    ("system", "Answer using only the context. If it lacks the answer, refuse.\n{context}"),
+    ("human", "{question}"),
+]))
+# round-trip the optimized prompt back into LangChain:
+optimized = client.prompts.to_langchain("support-rag")
+```
+
+### Metric menu
+
+| `metric=` | Use for | Scored by |
+|---|---|---|
+| `auto` (default) | let the backend infer from your rows | inferred |
+| `json_f1` | JSON / structured extraction | deterministic |
+| `text_f1` | short free-text answers | deterministic |
+| `semantic_f1` | free-text / RAG answers (default for RAG) | LLM judge |
+| `rag_faithfulness` | is every claim supported by the context | LLM judge |
+| `rag_composite` | RAG grounding **and** refusal correctness | LLM judge |
+
+RAG rows auto-route to `semantic_f1`; pass `metric="rag_composite"` explicitly when you
+want grounding + refusal optimized together.
+
+## Feedback (your end-users' signal)
+
+```python
+client.feedback(trace_id, good=True)                  # thumbs
+client.feedback(trace_id, score=0.5, note="partial")  # graded
+```
+
+## Debugging
+
+```python
+import runapprentice
+runapprentice.enable_debug_logging()   # or env APPRENTICE_DEBUG=1
+```
+
+Shows every API call with status + latency, every capture attempt, every job
+poll. Control-plane errors are designed to tell you what to do (e.g. unreachable
+backend includes the URL it tried; optimize with too few rows tells you the
+count and the minimum).
+
+## API reference (implemented surface)
+
+| Call | Returns | Raises |
+|---|---|---|
+| `Apprentice(api_key=, base_url=, timeout=)` | client | `ApprenticeError` on bad config |
+| `client.tasks.create(name, metric="auto")` | dict (`created` bool) | `ApprenticeError` on HTTP failure |
+| `client.datasets.upload(task, path= or rows=, input_col=, output_col=, prompt=)` | `DatasetStatus` | same; also if both/neither of path/rows |
+| `client.datasets.status(task)` | `DatasetStatus(gold, silver, raw, ready_for_optimization)` | same |
+| `client.prompts.register(task, template)` | dict | `ImportError` if a LangChain template is passed without the `[langchain]` extra |
+| `client.prompts.to_langchain(task= or artifact=)` | LangChain prompt | `ImportError` without the `[langchain]` extra |
+| `client.optimize(task)` | `Job` | 400 if fewer than the backend's configured verified-row minimum (20 by default) |
+| `client.job(job_id)` / `job.refresh()` / `job.wait(poll_seconds=, timeout_seconds=)` | `Job` | `ApprenticeError` on timeout |
+| `job.report()` | `OptimizationReport(baseline_score, optimized_score, optimized_prompt, ...)` | if job has no report |
+| `client.prompts.get(task, version=None)` | `PromptVersion(version, text, score)` | 404 if never optimized |
+| `client.prompts.history(task)` | `list[PromptVersion]` | `ApprenticeError` on API failure |
+| `client.feedback(trace_id, good=, score=, note=)` | None | 404 unknown trace |
+| `client.post_trace_failopen(record)` | trace_id or None | **never raises** |
+| `ApprenticeCallback(task, client, redact=None)` | LangChain callback | **never raises into your chain** |
+
+## Data tiers (how your rows are treated)
+
+- **raw**: captured from live traffic, unverified
+- **silver**: uploaded by you (curated) or passed deterministic checks
+- **gold**: human-verified
+- Optimization uses gold + silver; eval gates will use **gold only**.
+
+## For AI coding tools
+
+Module docstrings and this README are the source of truth. Key invariants an
+agent must preserve when editing this package:
+
+1. `post_trace_failopen` and everything in `runapprentice/langchain.py` must never
+   raise into the caller. Capture is fail-open by contract (see `tests/test_failopen.py`).
+2. Control-plane methods must raise `ApprenticeError` with actionable messages.
+3. Dependencies stay minimal: `httpx` + `pydantic`; LangChain only via the
+   `[langchain]` extra; never import from `apprentice-api`.
+4. Required tests for any new method: add a row to `../needed_test.md`.

runapprentice-0.1.0.dist-info/RECORD

@@ -0,0 +1,11 @@
+runapprentice/__init__.py,sha256=LtMfKOifoqwr0jQMB_uppWmoQ-3uO1SjcQq6oOj8MaE,560
+runapprentice/_capture.py,sha256=2XMi8lXSj6MSi9vUAUcard3z5eM-Kl5Wn3It5AKP-GU,4387
+runapprentice/client.py,sha256=AutaxNBLlI8gD22OxzLnhoN5LPvxcFhuqM5dgLveo1o,14607
+runapprentice/langchain.py,sha256=vQ27ZcvCfS2uUB09iJxqz0wAWymLei7Mpu2xtY7PaPg,15924
+runapprentice/models.py,sha256=55m8p6QRvxBU7mwGdqGy6gXrVVDLkoo7jFRL5Ds3y5A,1502
+runapprentice/openai.py,sha256=tAcLKlGVhcC-YRtKqQH0r3_W7uTLW-Ze1vIg9d3bM1Y,5021
+runapprentice/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+runapprentice-0.1.0.dist-info/METADATA,sha256=V1h6jumwvHNaaPJHE8OWDVFLEdxumsM9Y-qSyjAl21c,9263
+runapprentice-0.1.0.dist-info/WHEEL,sha256=mffPy8wBnZQn2VnJUU5jE99KsxaSfiyMHV9Yt0aLVxs,87
+runapprentice-0.1.0.dist-info/licenses/LICENSE,sha256=IY1vHVqwz9lYgq5UDL31X4dJWUIzbx07owwIdAZzd48,1071
+runapprentice-0.1.0.dist-info/RECORD,,

runapprentice-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.30.1
+Root-Is-Purelib: true
+Tag: py3-none-any

runapprentice-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Abhishek Singh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.