llmwarden 0.1.0__tar.gz

llmwarden-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: ci
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: uv sync --group dev
+      - run: uv run ruff check . && uv run ruff format --check .
+      - run: uv run mypy
+      - run: uv run pytest

llmwarden-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,48 @@
+name: publish
+
+# Publishes to PyPI when a version tag is pushed, for example `git tag v0.1.0 && git push --tags`.
+# Uses PyPI trusted publishing (OIDC): no API token is stored anywhere. Set it up once on
+# PyPI as described in docs/releasing.md.
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          python-version: "3.12"
+      - run: uv sync --group dev
+      - run: uv run ruff check . && uv run ruff format --check .
+      - run: uv run mypy
+      - run: uv run pytest
+      - name: Check the tag matches the package version
+        run: |
+          tag="${GITHUB_REF_NAME#v}"
+          version="$(uv version --short)"
+          test "$tag" = "$version" || { echo "tag v$tag != pyproject version $version"; exit 1; }
+      - run: uv build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/project/llmwarden/
+    permissions:
+      id-token: write   # required for trusted publishing
+      contents: read
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

llmwarden-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+# Planning artifacts stay local
+docs/superpowers/
+.remember/
+.claude/
+.superpowers/
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# Secrets never live in the repo
+.env
+.env.*

llmwarden-0.1.0/.python-version

@@ -0,0 +1 @@
+3.13

llmwarden-0.1.0/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## 0.1.0 (2026-10-15)
+
+First version. Everything here is described in `docs/architecture.md`.
+
+- `lw.model(name, api_key=...)` creates a model with its key from a string, a function or a secret reference, and `Model.complete` makes one call with messages or `system` and `user`.
+- Three backends behind one contract: `OpenAIBackend` (OpenAI, OpenRouter, Ollama, vLLM, any OpenAI-compatible server), `AnthropicBackend` (beta features pass through), `LiteLLMBackend`. The provider in the model name picks the backend; `backend=` overrides it. `FakeBackend` is public for applications' own tests.
+- Secret stores: `env` built in; AWS Secrets Manager, Azure Key Vault, Google Secret Manager and HashiCorp Vault as extras. Values are cached by reference, fetched once under concurrency, re-fetched once on an auth error, and never written or logged.
+- Endpoint guard for `public_only` models: https only, public addresses only, the checked address pinned in the transport, no redirects, built from the HTTP library the backend's SDK uses.
+- Key hygiene: every error reaches the caller scrubbed with the original dropped; masked `repr`; a logging filter.
+- Retries run in llmwarden with every library retry turned off, so every attempt is recorded. Optional fallback model with a shared call id.
+- Structured output into a Pydantic class on every backend, natively where the provider supports it and through the prompt where it does not, always validated by llmwarden.
+- Observability: tags through a context variable, one `AttemptRecord` per attempt, an in-memory sink and `lw.report`, a function sink, and an OpenTelemetry sink using the GenAI attribute names.
+- Prices: the model's own, then the backend's, then a small dated table, else unknown. Unknown is never zero.
+
+Not in this version: streaming, async, embeddings, tool calling.

llmwarden-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Nadeem Khan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llmwarden-0.1.0/PKG-INFO

@@ -0,0 +1,198 @@
+Metadata-Version: 2.5
+Name: llmwarden
+Version: 0.1.0
+Summary: Call any model with any key, fetched safely from wherever you keep secrets, and see exactly where your money goes
+Project-URL: Homepage, https://github.com/nadeem4/llmwarden
+Project-URL: Repository, https://github.com/nadeem4/llmwarden
+Project-URL: Issues, https://github.com/nadeem4/llmwarden/issues
+Author: Nadeem Khan
+License-Expression: MIT
+License-File: LICENSE
+Keywords: anthropic,cost,litellm,llm,observability,openai,opentelemetry,secrets
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Security
+Requires-Python: >=3.12
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic>=2.9
+Provides-Extra: all
+Requires-Dist: anthropic>=1.7.0; extra == 'all'
+Requires-Dist: azure-identity>=1.17; extra == 'all'
+Requires-Dist: azure-keyvault-secrets>=4.8; extra == 'all'
+Requires-Dist: boto3>=1.35; extra == 'all'
+Requires-Dist: google-cloud-secret-manager>=2.20; extra == 'all'
+Requires-Dist: hvac>=2.3; extra == 'all'
+Requires-Dist: litellm>=1.105.0; extra == 'all'
+Requires-Dist: openai>=3.17.0; extra == 'all'
+Requires-Dist: opentelemetry-api>=1.27; extra == 'all'
+Requires-Dist: python-dotenv>=1.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=1.7.0; extra == 'anthropic'
+Provides-Extra: aws
+Requires-Dist: boto3>=1.35; extra == 'aws'
+Provides-Extra: azure
+Requires-Dist: azure-identity>=1.17; extra == 'azure'
+Requires-Dist: azure-keyvault-secrets>=4.8; extra == 'azure'
+Provides-Extra: dotenv
+Requires-Dist: python-dotenv>=1.0; extra == 'dotenv'
+Provides-Extra: gcp
+Requires-Dist: google-cloud-secret-manager>=2.20; extra == 'gcp'
+Provides-Extra: litellm
+Requires-Dist: litellm>=1.105.0; extra == 'litellm'
+Provides-Extra: openai
+Requires-Dist: openai>=3.17.0; extra == 'openai'
+Provides-Extra: otel
+Requires-Dist: opentelemetry-api>=1.27; extra == 'otel'
+Provides-Extra: vault
+Requires-Dist: hvac>=2.3; extra == 'vault'
+Description-Content-Type: text/markdown
+
+# llmwarden
+
+Call any model with any key, fetched safely from wherever you keep secrets, and see exactly where your money goes and which requests fail.
+
+> **Status: 0.1.0 released.** The core, the three backends, the five secret stores, the endpoint guard, key hygiene, recorded retries, tags, reports and the OpenTelemetry sink are implemented and covered by a contract test suite that runs the same scenarios against every backend. Install it with `pip install "llmwarden[openai]"`. The design is in [docs/architecture.md](docs/architecture.md), the working rules in [docs/contributing.md](docs/contributing.md), and what shipped in [CHANGELOG.md](CHANGELOG.md).
+
+## The problem
+
+An AI application is rarely one model call. A retrieval pipeline rewrites the question, reranks results and writes the answer. A multi-agent system plans, writes, checks and summarizes. Each part may use a different provider, a different model and a different key.
+
+Calling many providers is a solved problem. What is not solved is everything around the call:
+
+- **Secrets live in many places.** One team keeps keys in environment variables, another in AWS Secrets Manager, Azure Key Vault or HashiCorp Vault. Keys get rotated. Every application writes its own code to fetch them.
+- **Custom endpoints are a hole.** If users can type a base URL, a hosted app can be made to send requests, with a key attached, to an internal address or a cloud metadata service.
+- **Keys leak.** Provider SDK errors keep the request, the request holds the key, and the traceback ends up in a log.
+- **Retries hide cost.** Libraries retry quietly, and the failed attempts never show up in any report.
+- **Nobody knows where the money went.** Usage comes back per call, in a different shape for each provider, with nothing linking it to the user, request or feature that caused it.
+
+Every project ends up rewriting the same layer. llmwarden is that layer, written once.
+
+## What it provides
+
+### 1. A model with its key, in one line
+
+```python
+import llmwarden as lw
+
+fast = lw.model(
+    "groq/llama-3.3-70b-versatile", api_key=lw.secret("aws", "prod/llm", field="groq_key")
+)
+strong = lw.model("anthropic/claude-sonnet-5-5", api_key=lw.secret("env", "ANTHROPIC_API_KEY"))
+
+reply = fast.complete(system="Rewrite the question.", user=question)
+```
+
+No config file. Create a model in code, give it a key, call it. A model is cheap to create, so a hosted app can build one per request with the end user's own key.
+
+### 2. Secrets from any store, and rotation that just works
+
+- The key can be a plain string, a function that returns one, or a reference: `lw.secret(store, name, field=None)`.
+- `env` is built in. AWS Secrets Manager, Azure Key Vault, HashiCorp Vault and Google Secret Manager are optional extras. Register a store once at startup, with its region or address, then refer to secrets by name.
+- Write your own store with one method.
+- When a key is rotated, `model.refresh()` fetches it again. If a call fails because the key was refused and the key came from a store, llmwarden fetches it again and retries once, so a rotated key works without a restart.
+- Fetched keys stay in memory only. They are never written to disk, logs or errors.
+
+### 3. A guard for custom endpoints
+
+With `public_only=True`, meant for hosted apps that strangers use:
+
+- A custom base URL must be `https`.
+- Its host must resolve to public addresses only. Private, loopback, link-local and cloud metadata addresses are refused.
+- The request goes to the exact address that was checked, so a DNS change between check and call cannot reach a private address.
+- Redirects are not followed.
+- URLs with credentials, fragments or query tricks are refused.
+
+### 4. Key hygiene
+
+- A key is handed to the client that needs it and to nothing else.
+- Every provider error is re-raised as a fresh error with the key scrubbed and the original dropped, so the key cannot appear in a traceback.
+- A model prints with its key masked.
+
+### 5. Observability and cost attribution, out of the box
+
+```python
+with lw.tags(request="q-1842", customer="acme"):
+    plan = planner.complete(..., tags={"agent": "planner"})
+    draft = writer.complete(..., tags={"agent": "writer"})
+
+lw.report(group_by=["customer", "agent"])
+```
+
+- **Tags** set once where a request comes in apply to every call inside it, however deep. Add more on any call.
+- **Every attempt is recorded,** retries included: model, outcome, error kind, tokens, cost, time and tags. Retries happen inside llmwarden, never hidden inside a provider library, so none go uncounted.
+- **Reports** add up cost, tokens, failures and time by any tag: per customer, per feature, per agent, per request.
+- **OpenTelemetry** spans are emitted for every attempt, using the standard GenAI attribute names, so Grafana, Jaeger, Langfuse, Phoenix, Datadog or any other OpenTelemetry tool can show them with no extra code.
+- Unknown prices and token counts are reported as unknown, never as zero.
+- Prompts and replies are never recorded. Keys never are.
+
+### 6. Structured output, the same everywhere
+
+Pass a Pydantic class and get back a validated object, on every provider, because llmwarden validates the reply itself. A reply that fails validation is retried like any other failure, and recorded.
+
+### 7. Swappable backends
+
+Three backends ship in the first version, and the same contract test suite runs against each:
+
+| Backend | Reaches | Install |
+|---|---|---|
+| `OpenAIBackend` | OpenAI, OpenRouter, Ollama, vLLM, any OpenAI-compatible server | `llmwarden[openai]` |
+| `AnthropicBackend` | Anthropic, with beta features passed through | `llmwarden[anthropic]` |
+| `LiteLLMBackend` | everything LiteLLM lists | `llmwarden[litellm]` |
+
+The provider in the model name picks the backend. Pass `backend=` to override it, or write your own adapter in one file. Retries, records, scrubbing, schema validation and pricing live in llmwarden, so every backend behaves the same.
+
+## What it is not
+
+- **Not another provider client.** Calls go through an existing library behind one small interface. Wherever that library already does something well, llmwarden uses it, and fills in only what is missing.
+- **Not a framework.** It does not know about your stages, agents or pipelines. Your code decides which model each part uses.
+- **Not a proxy or gateway server.** It is a library inside your application.
+- **Not a dashboard.** It emits OpenTelemetry, which the tools you already use can show.
+
+## Not yet
+
+Streaming, async calls, embeddings and tool calling come after the first release, as additions to the same interface.
+
+## Install
+
+```
+pip install "llmwarden[openai]"          # one backend
+pip install "llmwarden[anthropic,aws]"   # a backend and a secret store
+pip install "llmwarden[all]"             # everything
+```
+
+Python 3.12 or newer.
+
+## Examples
+
+Runnable scripts in [examples/](examples/), each a few lines with the setup it needs at the top:
+
+| File | Shows |
+|---|---|
+| [01_quickstart.py](examples/01_quickstart.py) | one model, one call, usage and cost on the reply |
+| [02_structured_output.py](examples/02_structured_output.py) | a Pydantic schema validated on every backend, retried on bad output |
+| [03_secret_stores.py](examples/03_secret_stores.py) | a cloud store, a field inside a JSON secret, rotation, a store of your own |
+| [04_hosted_app_custom_endpoint.py](examples/04_hosted_app_custom_endpoint.py) | one model per request with the user's key, a guarded user-typed base URL, scrubbed errors |
+| [05_cost_report.py](examples/05_cost_report.py) | tags, every attempt recorded, a report by customer and agent, your own sink, OpenTelemetry |
+| [06_testing_your_app.py](examples/06_testing_your_app.py) | testing your code offline with `FakeBackend` |
+
+The last three run without a key or a network; the test suite runs them.
+
+## Who it is for
+
+Anyone shipping an application with more than one model call, especially one that is hosted for other people: retrieval pipelines, multi-agent systems, and tools where users bring their own key or point at their own server.
+
+## Developing
+
+```
+uv sync --group dev
+uv run pytest
+```
+
+Every test runs without a network connection. See [docs/contributing.md](docs/contributing.md) for the working rules and [docs/releasing.md](docs/releasing.md) for how a version reaches PyPI.
+
+## Next steps
+
+Streaming, async and tool calling, as additions to the same interface.

llmwarden-0.1.0/README.md

@@ -0,0 +1,146 @@
+# llmwarden
+
+Call any model with any key, fetched safely from wherever you keep secrets, and see exactly where your money goes and which requests fail.
+
+> **Status: 0.1.0 released.** The core, the three backends, the five secret stores, the endpoint guard, key hygiene, recorded retries, tags, reports and the OpenTelemetry sink are implemented and covered by a contract test suite that runs the same scenarios against every backend. Install it with `pip install "llmwarden[openai]"`. The design is in [docs/architecture.md](docs/architecture.md), the working rules in [docs/contributing.md](docs/contributing.md), and what shipped in [CHANGELOG.md](CHANGELOG.md).
+
+## The problem
+
+An AI application is rarely one model call. A retrieval pipeline rewrites the question, reranks results and writes the answer. A multi-agent system plans, writes, checks and summarizes. Each part may use a different provider, a different model and a different key.
+
+Calling many providers is a solved problem. What is not solved is everything around the call:
+
+- **Secrets live in many places.** One team keeps keys in environment variables, another in AWS Secrets Manager, Azure Key Vault or HashiCorp Vault. Keys get rotated. Every application writes its own code to fetch them.
+- **Custom endpoints are a hole.** If users can type a base URL, a hosted app can be made to send requests, with a key attached, to an internal address or a cloud metadata service.
+- **Keys leak.** Provider SDK errors keep the request, the request holds the key, and the traceback ends up in a log.
+- **Retries hide cost.** Libraries retry quietly, and the failed attempts never show up in any report.
+- **Nobody knows where the money went.** Usage comes back per call, in a different shape for each provider, with nothing linking it to the user, request or feature that caused it.
+
+Every project ends up rewriting the same layer. llmwarden is that layer, written once.
+
+## What it provides
+
+### 1. A model with its key, in one line
+
+```python
+import llmwarden as lw
+
+fast = lw.model(
+    "groq/llama-3.3-70b-versatile", api_key=lw.secret("aws", "prod/llm", field="groq_key")
+)
+strong = lw.model("anthropic/claude-sonnet-5-5", api_key=lw.secret("env", "ANTHROPIC_API_KEY"))
+
+reply = fast.complete(system="Rewrite the question.", user=question)
+```
+
+No config file. Create a model in code, give it a key, call it. A model is cheap to create, so a hosted app can build one per request with the end user's own key.
+
+### 2. Secrets from any store, and rotation that just works
+
+- The key can be a plain string, a function that returns one, or a reference: `lw.secret(store, name, field=None)`.
+- `env` is built in. AWS Secrets Manager, Azure Key Vault, HashiCorp Vault and Google Secret Manager are optional extras. Register a store once at startup, with its region or address, then refer to secrets by name.
+- Write your own store with one method.
+- When a key is rotated, `model.refresh()` fetches it again. If a call fails because the key was refused and the key came from a store, llmwarden fetches it again and retries once, so a rotated key works without a restart.
+- Fetched keys stay in memory only. They are never written to disk, logs or errors.
+
+### 3. A guard for custom endpoints
+
+With `public_only=True`, meant for hosted apps that strangers use:
+
+- A custom base URL must be `https`.
+- Its host must resolve to public addresses only. Private, loopback, link-local and cloud metadata addresses are refused.
+- The request goes to the exact address that was checked, so a DNS change between check and call cannot reach a private address.
+- Redirects are not followed.
+- URLs with credentials, fragments or query tricks are refused.
+
+### 4. Key hygiene
+
+- A key is handed to the client that needs it and to nothing else.
+- Every provider error is re-raised as a fresh error with the key scrubbed and the original dropped, so the key cannot appear in a traceback.
+- A model prints with its key masked.
+
+### 5. Observability and cost attribution, out of the box
+
+```python
+with lw.tags(request="q-1842", customer="acme"):
+    plan = planner.complete(..., tags={"agent": "planner"})
+    draft = writer.complete(..., tags={"agent": "writer"})
+
+lw.report(group_by=["customer", "agent"])
+```
+
+- **Tags** set once where a request comes in apply to every call inside it, however deep. Add more on any call.
+- **Every attempt is recorded,** retries included: model, outcome, error kind, tokens, cost, time and tags. Retries happen inside llmwarden, never hidden inside a provider library, so none go uncounted.
+- **Reports** add up cost, tokens, failures and time by any tag: per customer, per feature, per agent, per request.
+- **OpenTelemetry** spans are emitted for every attempt, using the standard GenAI attribute names, so Grafana, Jaeger, Langfuse, Phoenix, Datadog or any other OpenTelemetry tool can show them with no extra code.
+- Unknown prices and token counts are reported as unknown, never as zero.
+- Prompts and replies are never recorded. Keys never are.
+
+### 6. Structured output, the same everywhere
+
+Pass a Pydantic class and get back a validated object, on every provider, because llmwarden validates the reply itself. A reply that fails validation is retried like any other failure, and recorded.
+
+### 7. Swappable backends
+
+Three backends ship in the first version, and the same contract test suite runs against each:
+
+| Backend | Reaches | Install |
+|---|---|---|
+| `OpenAIBackend` | OpenAI, OpenRouter, Ollama, vLLM, any OpenAI-compatible server | `llmwarden[openai]` |
+| `AnthropicBackend` | Anthropic, with beta features passed through | `llmwarden[anthropic]` |
+| `LiteLLMBackend` | everything LiteLLM lists | `llmwarden[litellm]` |
+
+The provider in the model name picks the backend. Pass `backend=` to override it, or write your own adapter in one file. Retries, records, scrubbing, schema validation and pricing live in llmwarden, so every backend behaves the same.
+
+## What it is not
+
+- **Not another provider client.** Calls go through an existing library behind one small interface. Wherever that library already does something well, llmwarden uses it, and fills in only what is missing.
+- **Not a framework.** It does not know about your stages, agents or pipelines. Your code decides which model each part uses.
+- **Not a proxy or gateway server.** It is a library inside your application.
+- **Not a dashboard.** It emits OpenTelemetry, which the tools you already use can show.
+
+## Not yet
+
+Streaming, async calls, embeddings and tool calling come after the first release, as additions to the same interface.
+
+## Install
+
+```
+pip install "llmwarden[openai]"          # one backend
+pip install "llmwarden[anthropic,aws]"   # a backend and a secret store
+pip install "llmwarden[all]"             # everything
+```
+
+Python 3.12 or newer.
+
+## Examples
+
+Runnable scripts in [examples/](examples/), each a few lines with the setup it needs at the top:
+
+| File | Shows |
+|---|---|
+| [01_quickstart.py](examples/01_quickstart.py) | one model, one call, usage and cost on the reply |
+| [02_structured_output.py](examples/02_structured_output.py) | a Pydantic schema validated on every backend, retried on bad output |
+| [03_secret_stores.py](examples/03_secret_stores.py) | a cloud store, a field inside a JSON secret, rotation, a store of your own |
+| [04_hosted_app_custom_endpoint.py](examples/04_hosted_app_custom_endpoint.py) | one model per request with the user's key, a guarded user-typed base URL, scrubbed errors |
+| [05_cost_report.py](examples/05_cost_report.py) | tags, every attempt recorded, a report by customer and agent, your own sink, OpenTelemetry |
+| [06_testing_your_app.py](examples/06_testing_your_app.py) | testing your code offline with `FakeBackend` |
+
+The last three run without a key or a network; the test suite runs them.
+
+## Who it is for
+
+Anyone shipping an application with more than one model call, especially one that is hosted for other people: retrieval pipelines, multi-agent systems, and tools where users bring their own key or point at their own server.
+
+## Developing
+
+```
+uv sync --group dev
+uv run pytest
+```
+
+Every test runs without a network connection. See [docs/contributing.md](docs/contributing.md) for the working rules and [docs/releasing.md](docs/releasing.md) for how a version reaches PyPI.
+
+## Next steps
+
+Streaming, async and tool calling, as additions to the same interface.