margin-meter 0.6.0__tar.gz → 0.6.2__tar.gz

{margin_meter-0.6.0 → margin_meter-0.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: margin-meter
-Version: 0.6.0
+Version: 0.6.2
 Summary: Tiny stdlib-only client that auto-instruments OpenAI/Anthropic/Gemini and emits LLM call + outcome economics to a Margin ingest API.
 Author: Margin
 License: MIT
@@ -12,6 +12,12 @@ Description-Content-Type: text/markdown
 
 # margin-meter (Python SDK)
 
+> **You probably do not need this to start.** Margin's first step installs nothing: sign in at
+> [trymargin.io/console](https://trymargin.io/console), connect the repository your agents live
+> in (or drop the folder), and press Execute. Each agent runs on the evals beside it and every
+> call is metered. This SDK is the **optional** route for metering your live production traffic,
+> once you want that as well ([`/start#production`](https://trymargin.io/start#production)).
+
 The tiny client a **Python** project imports to connect to Margin. It wraps your
 LLM calls and records their outcomes, emitting each one **over HTTP** to a
 Margin **ingest API** (`POST /api/ingest/calls` / `/api/ingest/outcomes`),
@@ -48,19 +54,30 @@ MAR-485.
 Two environment variables — the deployed API base and your project's key:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
-export MARGIN_INGEST_KEY="mgk_…"   # issued by the Margin owner (see below)
+export MARGIN_INGEST_URL="https://trymargin.io"
+export MARGIN_INGEST_KEY="mgk_…"   # mint your own at https://trymargin.io/start
 ```
 
-The Margin owner issues your project a key with the provisioning CLI in the
-Margin repo:
+## Get your key
 
-```bash
-python3 scripts/issue_ingest_key.py <your-project-slug>
-```
+Open **https://trymargin.io/start**, enter the email you want this project attached to, and
+the page shows you an `mgk_…` key. It takes about ten seconds and needs nobody's approval.
+
+The raw key is shown **once** — only its hash is stored. Keep it as `MARGIN_INGEST_KEY`.
 
-The raw `mgk_…` key is shown **once** — only its hash is stored. Give it to your
-project as `MARGIN_INGEST_KEY`.
+⭐ **Sign in at https://console.trymargin.io with that same email** to read your own
+cost-per-outcome. One email is one project, and the sign-in resolves through that address, so
+the console reads exactly the rows your key writes.
+
+⛔⛔ **THIS SECTION USED TO DESCRIBE A DIFFERENT, IMPOSSIBLE PATH, AND IT IS WORTH KNOWING
+WHY IT SURVIVED.** It said the key was *"issued by the Margin owner"* via
+`python3 scripts/issue_ingest_key.py <your-project-slug>` — a script inside the Margin
+repository, which is **private**. So the documented way to get started handed a reader a
+command they could not run, against code they could not read, on a path that needed a human
+to be awake. Measured 2026-09-21: the self-serve route had existed the whole time and this
+file never mentioned it once. The lesson is not "the README was stale" — it is that an
+instruction a reader cannot execute does not fail loudly. It reads as onboarding, and the
+reader concludes the product is not for them yet.
 
 ## Use
 
@@ -150,7 +167,7 @@ not have to hand-write a `record_outcome` call — run pytest with a workflow id
 and every test becomes an outcome:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 pytest --margin-workflow my-eval-suite
 ```
@@ -185,7 +202,7 @@ If your evals run under [promptfoo](https://promptfoo.dev), the JSON it already
 writes is an outcome file. Run your eval, then hand the output to `margin-meter`:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 promptfoo eval -o results.json
 margin-meter promptfoo --workflow my-eval-suite results.json
@@ -224,7 +241,7 @@ Add `--simulated` for a trial run whose rows must not seat real metrics.
 `$DEEPEVAL_RESULTS_FOLDER`). Point `margin-meter` at that file:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 deepeval test run test_eval.py
 margin-meter deepeval --workflow my-eval-suite .deepeval/.latest_run_full.json
@@ -247,7 +264,7 @@ per-case results to JSON (a list of results, or the `{results: […]}` /
 `margin-meter` at that file:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 margin-meter braintrust --workflow my-eval-suite --pass-threshold 0.7 results.json
 ```
@@ -272,7 +289,7 @@ If you evaluate RAG with [Ragas](https://docs.ragas.io), serialize the result an
 point `margin-meter` at it:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 python -c "import json; json.dump(result.to_pandas().to_dict(orient='records'), open('ragas.json','w'))"
 margin-meter ragas --workflow my-rag --pass-threshold 0.7 ragas.json
@@ -296,7 +313,7 @@ If you evaluate with [LangSmith](https://docs.smith.langchain.com), serialize th
 `evaluate()` result and point `margin-meter` at it:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 python -c "import json; json.dump(result.to_pandas().to_dict(orient='records'), open('ls.json','w'))"
 margin-meter langsmith --workflow my-app --pass-threshold 0.7 ls.json
@@ -315,6 +332,48 @@ evaluator keys** — LangChain's built-in judges (`correctness`, `criteria`,
 records `unknown` rather than an assumed grade. Put `margin_value` on a row to
 measure value-per-outcome; `--simulated` marks a trial run.
 
+## Report BOTH plugs from Langfuse
+
+[Langfuse](https://langfuse.com) is the one source that carries both halves of the two-plug
+shape in a single export, which is why it gets two commands rather than one.
+
+```bash
+export MARGIN_INGEST_URL="https://trymargin.io"
+export MARGIN_INGEST_KEY="mgk_…"
+
+# 1) Your AI in: every generation observation becomes a metered call.
+margin-meter langfuse-calls --workflow my-app observations.json
+
+# 2) Your evals in: every score becomes an outcome candidate.
+margin-meter langfuse --workflow my-app --pass-threshold 0.7 --min-confidence 0.7 scores.json
+```
+
+⚠️ **A numeric score needs `--pass-threshold`; a categorical needs `--pass-values A,B`; only
+a boolean maps to a verdict on its own.** Run `margin-meter langfuse --help` for the current
+set — a score that arrives without the flag its data type needs is an outcome with no verdict,
+which will not move your denominator.
+
+Serialise either the SDK's `fetch_observations` / `fetch_scores` result or a UI export to
+JSON. A bare list works, as do `{"data": […]}` / `{"scores": […]}` / `{"observations": […]}`
+envelopes. No live Langfuse API key is needed here, the same as every other adapter.
+
+⭐ **`langfuse-calls` is worth trying before you instrument anything.** A generation
+observation already carries `model`, token usage, cost, and the trace and parent ids, so the
+estate tree falls out of your trace hierarchy without a line of code in your agent. If your
+generations are in Langfuse, this is the shortest path to a real cost-per-outcome.
+
+⛔ **`--min-confidence` IS A REFUSAL, NOT A FILTER, AND THE DIFFERENCE MATTERS.** A score
+whose `metadata` or `comment` embeds a probability is counted only at or above the threshold;
+below it the row is **not an outcome at all** — neither a pass nor a fail. Forcing a
+low-confidence score into a binary "least wrong answer" would fabricate the denominator that
+cost-per-outcome divides by. A score with no stated probability is unaffected: the floor
+gates a confidence somebody stated, it never invents one.
+
+⚠️ **`substrate` is left empty, deliberately.** `substrate` names the ROUTE a call took
+(direct, openrouter, bedrock), which is a different thing from the model, and a Langfuse
+export carries no route evidence. `provider` is derived from the model string, overridable
+per batch with `--provider`, and left `unknown` rather than fabricated when neither names it.
+
 ## Multi-agent pipelines (name each stage)
 
 A crew of agents — a CrewAI pipeline, a planner→coder hand-off — runs in one

{margin_meter-0.6.0 → margin_meter-0.6.2}/README.md

@@ -1,5 +1,11 @@
 # margin-meter (Python SDK)
 
+> **You probably do not need this to start.** Margin's first step installs nothing: sign in at
+> [trymargin.io/console](https://trymargin.io/console), connect the repository your agents live
+> in (or drop the folder), and press Execute. Each agent runs on the evals beside it and every
+> call is metered. This SDK is the **optional** route for metering your live production traffic,
+> once you want that as well ([`/start#production`](https://trymargin.io/start#production)).
+
 The tiny client a **Python** project imports to connect to Margin. It wraps your
 LLM calls and records their outcomes, emitting each one **over HTTP** to a
 Margin **ingest API** (`POST /api/ingest/calls` / `/api/ingest/outcomes`),
@@ -36,19 +42,30 @@ MAR-485.
 Two environment variables — the deployed API base and your project's key:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
-export MARGIN_INGEST_KEY="mgk_…"   # issued by the Margin owner (see below)
+export MARGIN_INGEST_URL="https://trymargin.io"
+export MARGIN_INGEST_KEY="mgk_…"   # mint your own at https://trymargin.io/start
 ```
 
-The Margin owner issues your project a key with the provisioning CLI in the
-Margin repo:
+## Get your key
 
-```bash
-python3 scripts/issue_ingest_key.py <your-project-slug>
-```
+Open **https://trymargin.io/start**, enter the email you want this project attached to, and
+the page shows you an `mgk_…` key. It takes about ten seconds and needs nobody's approval.
+
+The raw key is shown **once** — only its hash is stored. Keep it as `MARGIN_INGEST_KEY`.
 
-The raw `mgk_…` key is shown **once** — only its hash is stored. Give it to your
-project as `MARGIN_INGEST_KEY`.
+⭐ **Sign in at https://console.trymargin.io with that same email** to read your own
+cost-per-outcome. One email is one project, and the sign-in resolves through that address, so
+the console reads exactly the rows your key writes.
+
+⛔⛔ **THIS SECTION USED TO DESCRIBE A DIFFERENT, IMPOSSIBLE PATH, AND IT IS WORTH KNOWING
+WHY IT SURVIVED.** It said the key was *"issued by the Margin owner"* via
+`python3 scripts/issue_ingest_key.py <your-project-slug>` — a script inside the Margin
+repository, which is **private**. So the documented way to get started handed a reader a
+command they could not run, against code they could not read, on a path that needed a human
+to be awake. Measured 2026-09-21: the self-serve route had existed the whole time and this
+file never mentioned it once. The lesson is not "the README was stale" — it is that an
+instruction a reader cannot execute does not fail loudly. It reads as onboarding, and the
+reader concludes the product is not for them yet.
 
 ## Use
 
@@ -138,7 +155,7 @@ not have to hand-write a `record_outcome` call — run pytest with a workflow id
 and every test becomes an outcome:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 pytest --margin-workflow my-eval-suite
 ```
@@ -173,7 +190,7 @@ If your evals run under [promptfoo](https://promptfoo.dev), the JSON it already
 writes is an outcome file. Run your eval, then hand the output to `margin-meter`:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 promptfoo eval -o results.json
 margin-meter promptfoo --workflow my-eval-suite results.json
@@ -212,7 +229,7 @@ Add `--simulated` for a trial run whose rows must not seat real metrics.
 `$DEEPEVAL_RESULTS_FOLDER`). Point `margin-meter` at that file:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 deepeval test run test_eval.py
 margin-meter deepeval --workflow my-eval-suite .deepeval/.latest_run_full.json
@@ -235,7 +252,7 @@ per-case results to JSON (a list of results, or the `{results: […]}` /
 `margin-meter` at that file:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 margin-meter braintrust --workflow my-eval-suite --pass-threshold 0.7 results.json
 ```
@@ -260,7 +277,7 @@ If you evaluate RAG with [Ragas](https://docs.ragas.io), serialize the result an
 point `margin-meter` at it:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 python -c "import json; json.dump(result.to_pandas().to_dict(orient='records'), open('ragas.json','w'))"
 margin-meter ragas --workflow my-rag --pass-threshold 0.7 ragas.json
@@ -284,7 +301,7 @@ If you evaluate with [LangSmith](https://docs.smith.langchain.com), serialize th
 `evaluate()` result and point `margin-meter` at it:
 
 ```bash
-export MARGIN_INGEST_URL="https://margin-ai-rho.vercel.app"
+export MARGIN_INGEST_URL="https://trymargin.io"
 export MARGIN_INGEST_KEY="mgk_…"
 python -c "import json; json.dump(result.to_pandas().to_dict(orient='records'), open('ls.json','w'))"
 margin-meter langsmith --workflow my-app --pass-threshold 0.7 ls.json
@@ -303,6 +320,48 @@ evaluator keys** — LangChain's built-in judges (`correctness`, `criteria`,
 records `unknown` rather than an assumed grade. Put `margin_value` on a row to
 measure value-per-outcome; `--simulated` marks a trial run.
 
+## Report BOTH plugs from Langfuse
+
+[Langfuse](https://langfuse.com) is the one source that carries both halves of the two-plug
+shape in a single export, which is why it gets two commands rather than one.
+
+```bash
+export MARGIN_INGEST_URL="https://trymargin.io"
+export MARGIN_INGEST_KEY="mgk_…"
+
+# 1) Your AI in: every generation observation becomes a metered call.
+margin-meter langfuse-calls --workflow my-app observations.json
+
+# 2) Your evals in: every score becomes an outcome candidate.
+margin-meter langfuse --workflow my-app --pass-threshold 0.7 --min-confidence 0.7 scores.json
+```
+
+⚠️ **A numeric score needs `--pass-threshold`; a categorical needs `--pass-values A,B`; only
+a boolean maps to a verdict on its own.** Run `margin-meter langfuse --help` for the current
+set — a score that arrives without the flag its data type needs is an outcome with no verdict,
+which will not move your denominator.
+
+Serialise either the SDK's `fetch_observations` / `fetch_scores` result or a UI export to
+JSON. A bare list works, as do `{"data": […]}` / `{"scores": […]}` / `{"observations": […]}`
+envelopes. No live Langfuse API key is needed here, the same as every other adapter.
+
+⭐ **`langfuse-calls` is worth trying before you instrument anything.** A generation
+observation already carries `model`, token usage, cost, and the trace and parent ids, so the
+estate tree falls out of your trace hierarchy without a line of code in your agent. If your
+generations are in Langfuse, this is the shortest path to a real cost-per-outcome.
+
+⛔ **`--min-confidence` IS A REFUSAL, NOT A FILTER, AND THE DIFFERENCE MATTERS.** A score
+whose `metadata` or `comment` embeds a probability is counted only at or above the threshold;
+below it the row is **not an outcome at all** — neither a pass nor a fail. Forcing a
+low-confidence score into a binary "least wrong answer" would fabricate the denominator that
+cost-per-outcome divides by. A score with no stated probability is unaffected: the floor
+gates a confidence somebody stated, it never invents one.
+
+⚠️ **`substrate` is left empty, deliberately.** `substrate` names the ROUTE a call took
+(direct, openrouter, bedrock), which is a different thing from the model, and a Langfuse
+export carries no route evidence. `provider` is derived from the model string, overridable
+per batch with `--provider`, and left `unknown` rather than fabricated when neither names it.
+
 ## Multi-agent pipelines (name each stage)
 
 A crew of agents — a CrewAI pipeline, a planner→coder hand-off — runs in one

{margin_meter-0.6.0 → margin_meter-0.6.2}/margin_meter/__init__.py

@@ -18,7 +18,9 @@ Two ways to instrument:
 from margin_meter.batching import BATCH_PATH
 from margin_meter.client import (
     CALLS_PATH,
+    ESTIMATE_PATH,
     OUTCOMES_PATH,
+    CallEstimate,
     DeliveryStats,
     IngestResult,
     MarginConfigError,
@@ -61,6 +63,8 @@ __all__ = [
     "MarginMeter",
     "IngestResult",
     "DeliveryStats",
+    "CallEstimate",
+    "ESTIMATE_PATH",
     "MarginMeterError",
     "MarginConfigError",
     "MarginIngestError",
@@ -102,4 +106,4 @@ __all__ = [
 # closing the invoice-reconcile coverage gap is "ship the fix, wait for the seed's
 # dependency refresh, re-run the reconcile", and a runtime that misreports which
 # SDK metered a call makes that unanswerable.
-__version__ = "0.6.0"
+__version__ = "0.6.2"

{margin_meter-0.6.0 → margin_meter-0.6.2}/margin_meter/__main__.py

@@ -10,6 +10,13 @@ Subcommands:
 * ``braintrust`` — report a Braintrust eval result file as outcomes (rung 2).
 * ``ragas``     — report a Ragas evaluation result file as outcomes (rung 2).
 * ``langsmith`` — report a LangSmith evaluate() result file as outcomes (rung 2).
+* ``langfuse``  — report Langfuse scores as outcomes (rung 2).
+* ``langfuse-calls`` — report Langfuse GENERATION observations as metered calls
+  (rung 2, the AI-in plug — Langfuse is the one source that carries both).
+* ``evals``     — draft eval criteria from the connected repo, list them, or open a PR
+  with the accepted ones (MAR-1167).
+* ``mcp``       — the same eval actions as an MCP server on stdio.
+* ``values``    — push what each outcome is worth from your own records (MAR-1496).
 
 Kept as a dispatcher so the surface is stable as more adapters/diagnostics arrive.
 """
@@ -19,7 +26,17 @@ from __future__ import annotations
 import sys
 from typing import Sequence
 
-from margin_meter import braintrust, deepeval, doctor, langsmith, promptfoo, ragas
+from margin_meter import (
+    braintrust,
+    deepeval,
+    doctor,
+    evals,
+    langfuse,
+    langsmith,
+    promptfoo,
+    ragas,
+    values,
+)
 
 _USAGE = """margin-meter — Margin's metering SDK
 
@@ -38,6 +55,17 @@ commands:
               (RAG eval; pass --pass-threshold to define a pass)
   langsmith   report a LangSmith evaluate() result file to Margin as outcomes
               (pass --pass-threshold to define a pass on continuous scores)
+  langfuse    report Langfuse scores to Margin as outcomes
+              (numeric needs --pass-threshold, categorical needs --pass-values;
+               a score below --min-confidence is not recorded)
+  langfuse-calls
+              report Langfuse GENERATION observations to Margin as metered calls
+              (cost from calculatedTotalCost, else priced from tokens)
+  evals       draft | list | pr: draft eval criteria from the connected repo,
+              list them, or open a PR with the ones a person accepted
+  mcp         serve the evals actions as MCP tools over stdio
+  values      push what each outcome is worth from your own records
+              (event_id,value_usd CSV on stdin or a file; --source, --field)
 
 run `margin-meter <command> --help` for a command's options.
 """
@@ -61,6 +89,16 @@ def main(argv: Sequence[str] | None = None) -> int:
         return ragas.main(rest)
     if command == "langsmith":
         return langsmith.main(rest)
+    if command == "langfuse":
+        return langfuse.main(rest)
+    if command == "langfuse-calls":
+        return langfuse.main_calls(rest)
+    if command == "evals":
+        return evals.main(rest)
+    if command == "mcp":
+        return evals.serve(rest)
+    if command == "values":
+        return values.main(rest)
     print(f"margin-meter: unknown command {command!r}\n", file=sys.stderr)
     print(_USAGE, file=sys.stderr)
     return 2

{margin_meter-0.6.0 → margin_meter-0.6.2}/margin_meter/braintrust.py

@@ -36,8 +36,8 @@ The mapping, one Braintrust result → one outcome:
   the grade is an LLM judge (rung 6 provenance); omitted otherwise. A bare
   ``model`` field is NOT used — in a Braintrust record it is usually the
   generation model under test, not the judge.
-* ``link``            ← the case ``id`` / ``span_id``, else a label from its
-  ``input``, else ``braintrust:<index>``.
+* ``link``            ← the case ``id`` / ``span_id``, else ``braintrust:<index>`` —
+  never its ``input`` (MAR-1548).
 * ``value_usd``       ← opt-in via ``metadata.margin_value`` — the only route to
   value-per-outcome (rung 8). Absent is the honest cost-per-outcome default,
   never defaulted to 0.
@@ -101,7 +101,7 @@ _JUDGE_MODEL_KEY = "margin_judge_model"
 # Mirror ``src/margin/ingest.MAX_VALUE_USD`` (kept as a literal so the SDK has no
 # dependency on the server package). A value above this is rejected by ingest and
 # would take the whole outcome row down with it, so the adapter drops the value
-# and keeps the row — the same treatment a negative value gets.
+# and keeps the row — the same treatment a malformed value gets.
 _MAX_VALUE_USD = 10_000_000.0
 
 # Model ids that name no real model — a custom/deterministic scorer leaves the
@@ -296,28 +296,22 @@ def value_from_result(result: dict) -> Optional[float]:
         raise ValueError(f"{_VALUE_KEY} must be a number, got {raw!r}")
     if not math.isfinite(val):
         raise ValueError(f"{_VALUE_KEY} must be finite, got {raw!r}")
-    if val < 0:
-        raise ValueError(f"{_VALUE_KEY} must be >= 0, got {val!r}")
-    if val > _MAX_VALUE_USD:
-        raise ValueError(f"{_VALUE_KEY} implausible (> {_MAX_VALUE_USD}), got {val!r}")
+    # A loss is a real value (MAR-1198): ingest takes any finite amount within +/-MAX.
+    if abs(val) > _MAX_VALUE_USD:
+        raise ValueError(f"{_VALUE_KEY} implausible (|value| > {_MAX_VALUE_USD}), got {val!r}")
     return val
 
 
 def link_from_result(result: dict, index: int) -> str:
-    """A traceable link back to the exact eval case: its id/span id, else a short
-    label built from the input, else the row index."""
+    """A traceable link back to the exact eval case: its id/span id, else the row
+    index.
+
+    ⛔ Never the case's ``input`` (MAR-1548): /security says no prompt reaches us.
+    """
     for key in ("id", "span_id", "root_span_id"):
         val = result.get(key)
         if isinstance(val, str) and val.strip():
             return val.strip()[:_MAX_LINK_LEN]
-    inp = result.get("input")
-    if isinstance(inp, str) and inp.strip():
-        return inp.strip()[:_MAX_LINK_LEN]
-    if isinstance(inp, (dict, list)):
-        try:
-            return json.dumps(inp, ensure_ascii=False, sort_keys=True)[:_MAX_LINK_LEN]
-        except (TypeError, ValueError):
-            pass
     return f"braintrust:{index}"