costplan 0.1.0__tar.gz

costplan-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Meenal Asesh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

costplan-0.1.0/PKG-INFO

@@ -0,0 +1,229 @@
+Metadata-Version: 2.4
+Name: costplan
+Version: 0.1.0
+Summary: How much will my agent cost? Plan AI agent budgets, caps and spend from a handful of pilot runs
+Author: Meenal Asesh
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ai-with-md/costplan
+Project-URL: Calculator, https://ai-with-md.github.io/costplan/
+Project-URL: Issues, https://github.com/ai-with-md/costplan/issues
+Keywords: llm,ai agents,cost,budget,token cost,llm cost,cost per task,statistics,forecasting
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# costplan
+
+**How much will my agent cost?** Plan AI agent budgets, caps and spend from a handful of pilot runs, with a [calculator that runs in your browser](https://ai-with-md.github.io/costplan/).
+
+[![PyPI](https://img.shields.io/pypi/v/costplan)](https://pypi.org/project/costplan/)
+[![tests](https://github.com/ai-with-md/costplan/actions/workflows/tests.yml/badge.svg)](https://github.com/ai-with-md/costplan/actions/workflows/tests.yml)
+![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)
+![License: MIT](https://img.shields.io/badge/license-MIT-green)
+
+[![The costplan calculator: run costs, a budget cap and the long tail of expensive runs](https://raw.githubusercontent.com/ai-with-md/costplan/main/docs/calculator.png)](https://ai-with-md.github.io/costplan/)
+
+Agent costs are lopsided. Most runs take a few tool calls and cost a few cents. A few get stuck in a loop, retry, or read a long document, and cost ten times more. In the example pilot above, the typical run costs $0.11 and the most expensive one $1.43. That makes averages slow to settle, and it makes "what budget cap should I set?" a question about the rare expensive runs, not the typical ones.
+
+costplan answers the planning questions from a pilot:
+
+1. What does a run really cost, and what does a **finished task** cost?
+2. What per-run budget cap should I set, so it almost never stops a normal run?
+3. How many pilot runs do I need before I can trust those numbers?
+4. What will it cost at 10,000 runs a day?
+5. Is the cheaper model actually cheaper per finished task?
+
+## The number most people get wrong
+
+To set a budget cap that stops at most 2% of normal runs, and be 95% sure of it, you need **149 pilot runs**. Fewer runs simply cannot rule out a rare expensive run that a tighter cap would cut off. Set the cap at the most expensive pilot run and this is the best you can promise:
+
+| Pilot runs | Cap stops at most |
+|---|---|
+| 20 | 13.9% of normal runs |
+| 30 | 9.5% |
+| 50 | 5.8% |
+| 100 | 3.0% |
+| 149 | 2.0% |
+| 300 | 1.0% |
+
+These numbers hold whatever the shape of your costs. costplan tells you what your current pilot can promise, and how many more runs the promise you want would take.
+
+## Try it
+
+**In the browser:** [ai-with-md.github.io/costplan](https://ai-with-md.github.io/costplan/). Paste your run costs, nothing to install.
+
+**In Python:**
+
+```bash
+pip install costplan
+```
+
+No dependencies beyond the standard library. Python 3.9 or newer.
+
+## Use
+
+### Reading your exports
+
+`load_runs` reads a CSV, JSON or JSON Lines export from your tracing tool or provider and finds the cost, token, success and run columns by their common names (`cost`, `total_cost`, `totalCost`, `calculatedTotalCost`, `prompt_tokens`, `input_tokens`, `usage.input`, `status`, `error`, `trace_id`, `traceId` and others).
+
+```python
+import costplan as cp
+
+runs = cp.load_runs("calls_export.jsonl", group_by="trace_id")
+print(runs.report())
+# Read 318 rows into 40 runs (cost from 'cost', status from 'status', runs from 'trace_id').
+
+costs, successes = runs.costs, runs.successes
+```
+
+- **One row per model call?** Pass `group_by` with the column that names the run (a trace or session id). Calls are added up into runs, and a run counts as failed if any of its calls failed.
+- **One row per run, already totalled?** Load it as it is. Grouping it again would count costs twice.
+- **No cost column, or calls your tool could not price?** Pass `input_price` and `output_price` in dollars per million tokens and costplan prices them from the token columns. costplan ships no price table, because prices change; take them from your provider's pricing page or a list such as [tokencost](https://github.com/AgentOps-AI/tokencost).
+- **Names not found?** Pass them yourself: `cost="spend"`, `success="passed"`, and dotted names for nested JSON such as `"usage.prompt_tokens"`.
+
+The column guesses are tested on files shaped like LangSmith run exports and Langfuse observation exports, using the field names in their documentation. Check `runs.report()` the first time you load a new kind of file.
+
+`run_cost` does the same pricing for a single call:
+
+```python
+cp.run_cost(input_tokens=12_000, output_tokens=800, input_price=3.0, output_price=15.0)
+# 0.048
+```
+
+### 1. What does a run cost?
+
+```python
+print(cp.summarize(costs, successes))
+# 60 runs.
+#   Average cost per run: $0.1577 (95% interval $0.1160 to $0.2167)
+#   Typical run (median): $0.1100 (95% interval $0.0920 to $0.1371)
+#   1 in 20 runs costs more than: $0.3272 (95% interval needs more runs)
+#   Most expensive run: $1.43
+#   Success rate: 85.0% (95% interval 73.9% to 91.9%)
+#   Cost per successful task: $0.1855 (95% interval $0.1308 to $0.2677)
+```
+
+The cost per successful task is the number that matters for a product: failed runs cost money too.
+
+### 2. What cap should I set?
+
+```python
+print(cp.recommend_cap(costs, stop_rate=0.02))
+# Goal: a cap that stops at most 2.0% of normal runs (95% confidence), from 60 pilot runs.
+#   Not enough runs to promise that yet: it takes 149.
+#   With 60 runs, a cap at the most expensive run ($1.43) can only promise to stop at most 4.9% of runs.
+#   Best single guess for the 98th percentile: $0.7737 (a guess, not a promise).
+```
+
+With enough runs, it returns a recommended cap. That cap is an upper confidence bound on the cost percentile, so it holds even if the pilot happened to miss some expensive runs. Enforce it with your agent framework, your own code, or a library such as [AgentBudget](https://github.com/AgentBudget/agentbudget).
+
+### 3. How many pilot runs do I need?
+
+```python
+cp.runs_needed_for_cap(stop_rate=0.02)       # 149
+cp.guaranteed_stop_rate(n_runs=30)           # 0.095
+cp.pilot_runs_for_average(spread=1.0, margin=0.10)   # 385 runs to pin the average within 10%
+```
+
+`spread` is the standard deviation divided by the average. Agent costs often have a spread near 1 or above because of the runaway runs, which is why averages need hundreds of runs to settle.
+
+### 4. What will it cost at scale?
+
+```python
+print(cp.project_spend(costs, runs=300_000))
+# 300,000 runs: about $47,301.75 (95% range $31,765.55 to $62,837.95).
+```
+
+At high volume the run to run variation averages out, and nearly all of the range comes from how well the pilot pinned down the average. A bigger pilot narrows it more than anything else.
+
+### 5. Is the cheaper model cheaper per finished task?
+
+```python
+print(cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model")))
+# Cost per successful task: large model $0.1855, small model $0.2197.
+#   small model costs 1.18 times as much as large model per success (95% interval 0.71 to 2.02, separate pilots). The difference could be noise.
+```
+
+In this example the small model is about 30% cheaper per run, but it fails and loops more often, so it probably costs more per finished task. With 60 runs each, costplan correctly says the difference is not yet certain.
+
+**Run both models on the same tasks if you can.** Hard tasks are expensive for every model, so comparing task by task removes that variation. With `paired=True` the same 60 tasks give an answer that separate pilots cannot:
+
+```python
+cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model"))
+#   small model costs 0.73 times as much as large model per success (95% interval 0.49 to 1.08, separate pilots). The difference could be noise.
+
+cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model"), paired=True)
+#   small model costs 0.73 times as much as large model per success (95% interval 0.61 to 0.88, same tasks, compared task by task). small model is reliably cheaper per success.
+```
+
+In simulation, pairing gives intervals about half as wide from the same number of runs.
+
+## Command line
+
+```bash
+costplan summary runs.csv
+costplan cap     runs.csv --stop-rate 0.02
+costplan project runs.csv --runs 300000
+costplan compare large.csv small.csv --names "large model" "small model"
+costplan compare large.csv small.csv --paired --task-column task_id
+costplan summary calls_export.jsonl --group-by trace_id -v
+costplan pilot   --stop-rate 0.02 --spread 1.0
+```
+
+Files can be CSV, JSON or JSON Lines, with the same automatic column finding as `load_runs`. Use `--cost-column`, `--success-column` and similar options when the guess is wrong, `--input-price` and `--output-price` for files with tokens but no costs, and `-v` to see which columns were used. Try it on the simulated files in [`examples/`](https://github.com/ai-with-md/costplan/tree/main/examples).
+
+## How the numbers are checked
+
+The test suite simulates thousands of pilots with lopsided costs where the truth is known, and checks each promise:
+
+| Promise | Held in simulation |
+|---|---|
+| The recommended cap stops at most 2% of runs | 98.5% of pilots (95% promised) |
+| A cap at the most expensive of 149 runs stops at most 2% | 94.8% |
+| The interval for cost per successful task covers the truth | 93% |
+| The spend projection covers the real bill | 92% |
+| The interval for the average cost covers the truth | 92% |
+| The paired comparison interval covers the true ratio | 96% |
+| Paired vs separate pilots, same number of runs | interval about half as wide |
+
+The caps and quantiles make no assumption about the shape of costs and keep their promise. The intervals for averages run slightly short of 95% on lopsided costs, which is typical for the bootstrap; treat them as a little optimistic, especially below about 30 runs. See [`tests/test_simulation.py`](https://github.com/ai-with-md/costplan/blob/main/tests/test_simulation.py).
+
+## How it compares
+
+- **[AgentBudget](https://github.com/AgentBudget/agentbudget)** and similar tools enforce a dollar limit while an agent runs. costplan tells you what limit to set, and what you will spend. They work well together.
+- **Price calculators and price lists** such as [tokencost](https://github.com/AgentOps-AI/tokencost) cost out a single request. costplan works on whole runs, including the expensive tail, retries and failures.
+- **Tracing tools** (Langfuse, LangSmith and others) record what runs cost. costplan turns those records into planning numbers with honest uncertainty.
+
+## Limits
+
+- The pilot must look like real traffic: the same prompts, tools, models and mix of requests. Change any of them and run a new pilot.
+- Caps and projections assume future runs behave like pilot runs. A new kind of request that triggers long loops will not show up in a pilot that never saw it.
+- The spend projection uses a normal approximation, which is a little narrow for very lopsided costs.
+
+## More tools
+
+Small, free tools for building and checking AI systems:
+
+- **[sturdytools](https://github.com/ai-with-md/sturdytools)**: retries, timeouts, rate limits and circuit breakers for the tools an AI agent calls.
+- **[evalsize](https://github.com/ai-with-md/evalsize)**: how many eval examples you need before a difference between prompts or models means anything.
+
+## Development
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+pytest
+```
+
+The calculator is a [marimo](https://marimo.io) notebook in [`app/calculator.py`](https://github.com/ai-with-md/costplan/blob/main/app/calculator.py), published to GitHub Pages as a WebAssembly app on every push to `main`.
+
+## License
+
+MIT

costplan-0.1.0/README.md

@@ -0,0 +1,209 @@
+# costplan
+
+**How much will my agent cost?** Plan AI agent budgets, caps and spend from a handful of pilot runs, with a [calculator that runs in your browser](https://ai-with-md.github.io/costplan/).
+
+[![PyPI](https://img.shields.io/pypi/v/costplan)](https://pypi.org/project/costplan/)
+[![tests](https://github.com/ai-with-md/costplan/actions/workflows/tests.yml/badge.svg)](https://github.com/ai-with-md/costplan/actions/workflows/tests.yml)
+![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)
+![License: MIT](https://img.shields.io/badge/license-MIT-green)
+
+[![The costplan calculator: run costs, a budget cap and the long tail of expensive runs](https://raw.githubusercontent.com/ai-with-md/costplan/main/docs/calculator.png)](https://ai-with-md.github.io/costplan/)
+
+Agent costs are lopsided. Most runs take a few tool calls and cost a few cents. A few get stuck in a loop, retry, or read a long document, and cost ten times more. In the example pilot above, the typical run costs $0.11 and the most expensive one $1.43. That makes averages slow to settle, and it makes "what budget cap should I set?" a question about the rare expensive runs, not the typical ones.
+
+costplan answers the planning questions from a pilot:
+
+1. What does a run really cost, and what does a **finished task** cost?
+2. What per-run budget cap should I set, so it almost never stops a normal run?
+3. How many pilot runs do I need before I can trust those numbers?
+4. What will it cost at 10,000 runs a day?
+5. Is the cheaper model actually cheaper per finished task?
+
+## The number most people get wrong
+
+To set a budget cap that stops at most 2% of normal runs, and be 95% sure of it, you need **149 pilot runs**. Fewer runs simply cannot rule out a rare expensive run that a tighter cap would cut off. Set the cap at the most expensive pilot run and this is the best you can promise:
+
+| Pilot runs | Cap stops at most |
+|---|---|
+| 20 | 13.9% of normal runs |
+| 30 | 9.5% |
+| 50 | 5.8% |
+| 100 | 3.0% |
+| 149 | 2.0% |
+| 300 | 1.0% |
+
+These numbers hold whatever the shape of your costs. costplan tells you what your current pilot can promise, and how many more runs the promise you want would take.
+
+## Try it
+
+**In the browser:** [ai-with-md.github.io/costplan](https://ai-with-md.github.io/costplan/). Paste your run costs, nothing to install.
+
+**In Python:**
+
+```bash
+pip install costplan
+```
+
+No dependencies beyond the standard library. Python 3.9 or newer.
+
+## Use
+
+### Reading your exports
+
+`load_runs` reads a CSV, JSON or JSON Lines export from your tracing tool or provider and finds the cost, token, success and run columns by their common names (`cost`, `total_cost`, `totalCost`, `calculatedTotalCost`, `prompt_tokens`, `input_tokens`, `usage.input`, `status`, `error`, `trace_id`, `traceId` and others).
+
+```python
+import costplan as cp
+
+runs = cp.load_runs("calls_export.jsonl", group_by="trace_id")
+print(runs.report())
+# Read 318 rows into 40 runs (cost from 'cost', status from 'status', runs from 'trace_id').
+
+costs, successes = runs.costs, runs.successes
+```
+
+- **One row per model call?** Pass `group_by` with the column that names the run (a trace or session id). Calls are added up into runs, and a run counts as failed if any of its calls failed.
+- **One row per run, already totalled?** Load it as it is. Grouping it again would count costs twice.
+- **No cost column, or calls your tool could not price?** Pass `input_price` and `output_price` in dollars per million tokens and costplan prices them from the token columns. costplan ships no price table, because prices change; take them from your provider's pricing page or a list such as [tokencost](https://github.com/AgentOps-AI/tokencost).
+- **Names not found?** Pass them yourself: `cost="spend"`, `success="passed"`, and dotted names for nested JSON such as `"usage.prompt_tokens"`.
+
+The column guesses are tested on files shaped like LangSmith run exports and Langfuse observation exports, using the field names in their documentation. Check `runs.report()` the first time you load a new kind of file.
+
+`run_cost` does the same pricing for a single call:
+
+```python
+cp.run_cost(input_tokens=12_000, output_tokens=800, input_price=3.0, output_price=15.0)
+# 0.048
+```
+
+### 1. What does a run cost?
+
+```python
+print(cp.summarize(costs, successes))
+# 60 runs.
+#   Average cost per run: $0.1577 (95% interval $0.1160 to $0.2167)
+#   Typical run (median): $0.1100 (95% interval $0.0920 to $0.1371)
+#   1 in 20 runs costs more than: $0.3272 (95% interval needs more runs)
+#   Most expensive run: $1.43
+#   Success rate: 85.0% (95% interval 73.9% to 91.9%)
+#   Cost per successful task: $0.1855 (95% interval $0.1308 to $0.2677)
+```
+
+The cost per successful task is the number that matters for a product: failed runs cost money too.
+
+### 2. What cap should I set?
+
+```python
+print(cp.recommend_cap(costs, stop_rate=0.02))
+# Goal: a cap that stops at most 2.0% of normal runs (95% confidence), from 60 pilot runs.
+#   Not enough runs to promise that yet: it takes 149.
+#   With 60 runs, a cap at the most expensive run ($1.43) can only promise to stop at most 4.9% of runs.
+#   Best single guess for the 98th percentile: $0.7737 (a guess, not a promise).
+```
+
+With enough runs, it returns a recommended cap. That cap is an upper confidence bound on the cost percentile, so it holds even if the pilot happened to miss some expensive runs. Enforce it with your agent framework, your own code, or a library such as [AgentBudget](https://github.com/AgentBudget/agentbudget).
+
+### 3. How many pilot runs do I need?
+
+```python
+cp.runs_needed_for_cap(stop_rate=0.02)       # 149
+cp.guaranteed_stop_rate(n_runs=30)           # 0.095
+cp.pilot_runs_for_average(spread=1.0, margin=0.10)   # 385 runs to pin the average within 10%
+```
+
+`spread` is the standard deviation divided by the average. Agent costs often have a spread near 1 or above because of the runaway runs, which is why averages need hundreds of runs to settle.
+
+### 4. What will it cost at scale?
+
+```python
+print(cp.project_spend(costs, runs=300_000))
+# 300,000 runs: about $47,301.75 (95% range $31,765.55 to $62,837.95).
+```
+
+At high volume the run to run variation averages out, and nearly all of the range comes from how well the pilot pinned down the average. A bigger pilot narrows it more than anything else.
+
+### 5. Is the cheaper model cheaper per finished task?
+
+```python
+print(cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model")))
+# Cost per successful task: large model $0.1855, small model $0.2197.
+#   small model costs 1.18 times as much as large model per success (95% interval 0.71 to 2.02, separate pilots). The difference could be noise.
+```
+
+In this example the small model is about 30% cheaper per run, but it fails and loops more often, so it probably costs more per finished task. With 60 runs each, costplan correctly says the difference is not yet certain.
+
+**Run both models on the same tasks if you can.** Hard tasks are expensive for every model, so comparing task by task removes that variation. With `paired=True` the same 60 tasks give an answer that separate pilots cannot:
+
+```python
+cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model"))
+#   small model costs 0.73 times as much as large model per success (95% interval 0.49 to 1.08, separate pilots). The difference could be noise.
+
+cp.compare_cost_per_success(large_costs, large_ok, small_costs, small_ok, names=("large model", "small model"), paired=True)
+#   small model costs 0.73 times as much as large model per success (95% interval 0.61 to 0.88, same tasks, compared task by task). small model is reliably cheaper per success.
+```
+
+In simulation, pairing gives intervals about half as wide from the same number of runs.
+
+## Command line
+
+```bash
+costplan summary runs.csv
+costplan cap     runs.csv --stop-rate 0.02
+costplan project runs.csv --runs 300000
+costplan compare large.csv small.csv --names "large model" "small model"
+costplan compare large.csv small.csv --paired --task-column task_id
+costplan summary calls_export.jsonl --group-by trace_id -v
+costplan pilot   --stop-rate 0.02 --spread 1.0
+```
+
+Files can be CSV, JSON or JSON Lines, with the same automatic column finding as `load_runs`. Use `--cost-column`, `--success-column` and similar options when the guess is wrong, `--input-price` and `--output-price` for files with tokens but no costs, and `-v` to see which columns were used. Try it on the simulated files in [`examples/`](https://github.com/ai-with-md/costplan/tree/main/examples).
+
+## How the numbers are checked
+
+The test suite simulates thousands of pilots with lopsided costs where the truth is known, and checks each promise:
+
+| Promise | Held in simulation |
+|---|---|
+| The recommended cap stops at most 2% of runs | 98.5% of pilots (95% promised) |
+| A cap at the most expensive of 149 runs stops at most 2% | 94.8% |
+| The interval for cost per successful task covers the truth | 93% |
+| The spend projection covers the real bill | 92% |
+| The interval for the average cost covers the truth | 92% |
+| The paired comparison interval covers the true ratio | 96% |
+| Paired vs separate pilots, same number of runs | interval about half as wide |
+
+The caps and quantiles make no assumption about the shape of costs and keep their promise. The intervals for averages run slightly short of 95% on lopsided costs, which is typical for the bootstrap; treat them as a little optimistic, especially below about 30 runs. See [`tests/test_simulation.py`](https://github.com/ai-with-md/costplan/blob/main/tests/test_simulation.py).
+
+## How it compares
+
+- **[AgentBudget](https://github.com/AgentBudget/agentbudget)** and similar tools enforce a dollar limit while an agent runs. costplan tells you what limit to set, and what you will spend. They work well together.
+- **Price calculators and price lists** such as [tokencost](https://github.com/AgentOps-AI/tokencost) cost out a single request. costplan works on whole runs, including the expensive tail, retries and failures.
+- **Tracing tools** (Langfuse, LangSmith and others) record what runs cost. costplan turns those records into planning numbers with honest uncertainty.
+
+## Limits
+
+- The pilot must look like real traffic: the same prompts, tools, models and mix of requests. Change any of them and run a new pilot.
+- Caps and projections assume future runs behave like pilot runs. A new kind of request that triggers long loops will not show up in a pilot that never saw it.
+- The spend projection uses a normal approximation, which is a little narrow for very lopsided costs.
+
+## More tools
+
+Small, free tools for building and checking AI systems:
+
+- **[sturdytools](https://github.com/ai-with-md/sturdytools)**: retries, timeouts, rate limits and circuit breakers for the tools an AI agent calls.
+- **[evalsize](https://github.com/ai-with-md/evalsize)**: how many eval examples you need before a difference between prompts or models means anything.
+
+## Development
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"
+pytest
+```
+
+The calculator is a [marimo](https://marimo.io) notebook in [`app/calculator.py`](https://github.com/ai-with-md/costplan/blob/main/app/calculator.py), published to GitHub Pages as a WebAssembly app on every push to `main`.
+
+## License
+
+MIT

costplan-0.1.0/pyproject.toml

@@ -0,0 +1,37 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "costplan"
+version = "0.1.0"
+description = "How much will my agent cost? Plan AI agent budgets, caps and spend from a handful of pilot runs"
+readme = "README.md"
+license = "MIT"
+authors = [{name = "Meenal Asesh"}]
+requires-python = ">=3.9"
+dependencies = []
+keywords = ["llm", "ai agents", "cost", "budget", "token cost", "llm cost", "cost per task", "statistics", "forecasting"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+
+[project.optional-dependencies]
+dev = ["pytest"]
+
+[project.scripts]
+costplan = "costplan.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/ai-with-md/costplan"
+Calculator = "https://ai-with-md.github.io/costplan/"
+Issues = "https://github.com/ai-with-md/costplan/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests", "src"]
+addopts = "--doctest-modules"

costplan-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

costplan-0.1.0/src/costplan/__init__.py

@@ -0,0 +1,36 @@
+"""How much will my agent cost? Plan agent budgets from a handful of pilot runs."""
+
+from .loading import LoadedRuns, load_runs
+from .planning import (
+    CapAdvice,
+    CostComparison,
+    CostSummary,
+    SpendProjection,
+    compare_cost_per_success,
+    guaranteed_stop_rate,
+    pilot_runs_for_average,
+    project_spend,
+    recommend_cap,
+    run_cost,
+    runs_needed_for_cap,
+    summarize,
+)
+
+__all__ = [
+    "load_runs",
+    "LoadedRuns",
+    "run_cost",
+    "summarize",
+    "recommend_cap",
+    "runs_needed_for_cap",
+    "guaranteed_stop_rate",
+    "pilot_runs_for_average",
+    "project_spend",
+    "compare_cost_per_success",
+    "CostSummary",
+    "CapAdvice",
+    "SpendProjection",
+    "CostComparison",
+]
+
+__version__ = "0.1.0"

costplan-0.1.0/src/costplan/_stats.py

@@ -0,0 +1,109 @@
+"""Small statistics helpers built on the standard library only."""
+
+from __future__ import annotations
+
+import math
+import random
+from statistics import NormalDist
+from typing import Callable, List, Optional, Sequence, Tuple
+
+_NORMAL = NormalDist()
+
+
+def check_level(value: float, name: str) -> None:
+    if not 0 < value < 1:
+        raise ValueError(f"{name} must be strictly between 0 and 1, got {value!r}")
+
+
+def z_two_sided(confidence: float) -> float:
+    return _NORMAL.inv_cdf(1 - (1 - confidence) / 2)
+
+
+def wilson(successes: int, n: int, confidence: float) -> Tuple[float, float]:
+    z = z_two_sided(confidence)
+    p = successes / n
+    denom = 1 + z * z / n
+    centre = (p + z * z / (2 * n)) / denom
+    half = z / denom * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
+    return max(0.0, centre - half), min(1.0, centre + half)
+
+
+def binom_cdf_table(n: int, p: float) -> List[float]:
+    """[P(X <= k) for k = 0..n] for X ~ Binomial(n, p), computed in log space."""
+    if p <= 0:
+        return [1.0] * (n + 1)
+    if p >= 1:
+        return [0.0] * n + [1.0]
+    log_p, log_q = math.log(p), math.log1p(-p)
+    log_n = math.lgamma(n + 1)
+    table, total = [], 0.0
+    for i in range(n + 1):
+        total += math.exp(log_n - math.lgamma(i + 1) - math.lgamma(n - i + 1) + i * log_p + (n - i) * log_q)
+        table.append(min(1.0, total))
+    return table
+
+
+def quantile(sorted_values: Sequence[float], p: float) -> float:
+    """Linear interpolation quantile (the usual default in numpy and pandas)."""
+    n = len(sorted_values)
+    if n == 1:
+        return sorted_values[0]
+    pos = p * (n - 1)
+    lo = math.floor(pos)
+    hi = min(lo + 1, n - 1)
+    return sorted_values[lo] + (sorted_values[hi] - sorted_values[lo]) * (pos - lo)
+
+
+def quantile_interval(
+    sorted_values: Sequence[float], p: float, confidence: float
+) -> Tuple[Optional[float], Optional[float]]:
+    """Distribution-free interval for the p-th quantile from order statistics.
+
+    Makes no assumption about the shape of the cost distribution. Either end
+    is None when there are too few runs to bound it at this confidence.
+    """
+    n = len(sorted_values)
+    cdf = binom_cdf_table(n, p)
+    tail = (1 - confidence) / 2
+    low = None
+    for k in range(n, 0, -1):  # largest k with P(X_(k) > q) at most the tail
+        if cdf[k - 1] <= tail:
+            low = sorted_values[k - 1]
+            break
+    high = upper_quantile_bound(sorted_values, p, 1 - tail, cdf)
+    return low, high
+
+
+def upper_quantile_bound(
+    sorted_values: Sequence[float], p: float, confidence: float, cdf: Optional[List[float]] = None
+) -> Optional[float]:
+    """Smallest order statistic that is above the p-th quantile with the given confidence."""
+    n = len(sorted_values)
+    cdf = cdf or binom_cdf_table(n, p)
+    for k in range(1, n + 1):
+        if cdf[k - 1] >= confidence:
+            return sorted_values[k - 1]
+    return None
+
+
+def bootstrap_interval(
+    columns: Sequence[Sequence],
+    statistic: Callable[..., Optional[float]],
+    confidence: float,
+    resamples: int,
+    seed: int,
+) -> Tuple[float, float]:
+    """Percentile bootstrap, resampling rows of the given columns together."""
+    rng = random.Random(seed)
+    n = len(columns[0])
+    values: List[float] = []
+    for _ in range(resamples):
+        idx = [rng.randrange(n) for _ in range(n)]
+        value = statistic(*[[col[i] for i in idx] for col in columns])
+        if value is not None and math.isfinite(value):
+            values.append(value)
+    if not values:
+        return float("nan"), float("nan")
+    values.sort()
+    tail = (1 - confidence) / 2
+    return quantile(values, tail), quantile(values, 1 - tail)