llmbuffet 0.2.0__tar.gz

llmbuffet-0.2.0/.env.example

@@ -0,0 +1,61 @@
+# llmbuffet — API keys for free-tier providers.
+#
+# Copy this file to `.env` (which is gitignored) and fill in the keys for
+# whichever providers you want to use. You do NOT need all of them — llmbuffet
+# uses whatever you have configured and skips the rest. Every key below is for
+# a FREE tier; no credit card is required for any of these.
+#
+# llmbuffet reads these from the environment. Either `export` them in your
+# shell, or use a tool like `direnv` / `python-dotenv`, or pass them inline.
+# llmbuffet itself does not parse this file — it only reads the environment.
+
+# Groq — https://console.groq.com/keys  (free, very fast)
+GROQ_API_KEY=
+
+# Cerebras — https://cloud.cerebras.ai  (free, very fast)
+CEREBRAS_API_KEY=
+
+# OpenRouter — https://openrouter.ai/keys  (many ':free' models)
+OPENROUTER_API_KEY=
+
+# Google Gemini — https://aistudio.google.com/apikey  (generous free tier)
+GEMINI_API_KEY=
+
+# GitHub Models — https://github.com/settings/tokens  (any PAT works; free)
+GITHUB_TOKEN=
+
+# Cloudflare Workers AI — https://dash.cloudflare.com/profile/api-tokens
+# Needs both the token and your account id.
+CLOUDFLARE_API_TOKEN=
+CLOUDFLARE_ACCOUNT_ID=
+
+# Mistral — https://console.mistral.ai/api-keys  (free tier)
+MISTRAL_API_KEY=
+
+# Cohere — https://dashboard.cohere.com/api-keys  (free trial keys)
+COHERE_API_KEY=
+
+# SambaNova — https://cloud.sambanova.ai/apis  (free tier)
+SAMBANOVA_API_KEY=
+
+# NVIDIA NIM — https://build.nvidia.com  (free credits, huge model catalog)
+NVIDIA_API_KEY=
+
+# Z.ai / Zhipu GLM — https://z.ai  (free GLM flash models)
+ZHIPU_API_KEY=
+
+# Ollama Cloud — https://ollama.com/settings/keys
+OLLAMA_API_KEY=
+
+# LongCat (Meituan) — https://longcat.chat
+LONGCAT_API_KEY=
+
+# ---------------------------------------------------------------------------
+# Zero-setup providers (no key needed — included automatically):
+#   • OVHcloud AI Endpoints  (anonymous/keyless)
+#   • LLM7                    (works without a key; set LLM7_API_KEY for more)
+# So llmbuffet works out of the box even with this whole file empty.
+# ---------------------------------------------------------------------------
+
+# LLM7 — optional key for higher limits: https://token.llm7.io
+LLM7_API_KEY=

llmbuffet-0.2.0/.github/ISSUE_TEMPLATE/add-provider.md

@@ -0,0 +1,23 @@
+---
+name: Add a free provider
+about: Propose adding a new free-tier LLM provider to the catalog
+title: "Add provider: <name>"
+labels: ["good first issue", "provider"]
+---
+
+**Provider:** <name + homepage>
+
+**Free tier?** (link to the free-tier docs — must be usable without a credit card)
+
+**OpenAI-compatible?** (does it expose `/v1/chat/completions`? If yes, this is a
+one-block PR to `src/llmbuffet/providers.toml`. If no, it needs a small adapter.)
+
+**Base URL:**
+
+**Models to include** (name + free daily request limit if known):
+-
+
+**Env var for the API key:** `<PROVIDER>_API_KEY`
+
+See [CONTRIBUTING.md](../../CONTRIBUTING.md) for the (small) steps. New providers
+are the most valuable contribution to llmbuffet — thank you!

llmbuffet-0.2.0/.github/workflows/ci.yml

@@ -0,0 +1,29 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+      - name: Lint
+        run: ruff check src tests
+      - name: Test
+        run: pytest

llmbuffet-0.2.0/.gitignore

@@ -0,0 +1,31 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+env/
+
+# Tooling
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+# Secrets & local state — never commit these
+.env
+.env.*
+!.env.example
+*.key
+**/api_key
+.llmbuffet/
+llmbuffet-state/
+
+# Editor / OS
+.idea/
+.vscode/
+.DS_Store

llmbuffet-0.2.0/CHANGELOG.md

@@ -0,0 +1,57 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/), and the project aims to follow
+[Semantic Versioning](https://semver.org/).
+
+## [0.2.0] — 2026-06-03
+
+### Added
+- **Six more providers** (15 total / 53 models): NVIDIA NIM, OVHcloud AI
+  Endpoints, LLM7, Ollama Cloud, Z.ai/Zhipu GLM, LongCat; expanded model lists
+  for Groq, Cerebras, OpenRouter, GitHub Models, SambaNova, Mistral, Gemini.
+- **Keyless / zero-setup providers.** OVHcloud works with no API key
+  (anonymous); LLM7's key is optional. `pip install llmbuffet && llmbuffet ask`
+  now works with no signup at all. Catalog gains `auth` and `key_optional`.
+- **Model selection.** New `llmbuffet models` lists every `provider/model` id;
+  `ask -m provider/model` pins an exact model on an exact provider.
+- **Streaming proxy.** The proxy honors `stream: true` with a buffered
+  OpenAI-style SSE stream, so stream-only clients (chat UIs, agents) work.
+- **429 cooldown.** A rate-limited provider is deprioritized for a cooldown
+  window instead of being retried immediately.
+- **Reasoning-model handling.** Thinking models get a `max_tokens` floor and
+  `<think>…</think>` blocks are stripped from output.
+- `llmbuffet ask --json` requests JSON and strips code fences.
+
+### Hardening (post-review)
+- Proxy now validates all request fields and returns OpenAI-style `400`s for
+  malformed input; a catch-all ensures no request can kill a server thread.
+- Optional proxy auth: `--api-key` / `LLMBUFFET_PROXY_KEY` requires a Bearer
+  token; a warning fires when binding to a non-loopback host without one.
+- Quota store is now thread-safe (lock + unique temp file) and best-effort, so
+  a persistence hiccup can't abort a successful completion.
+- A provider that returns `429` has its remaining models skipped for that
+  request; cooldowns update under a lock with `max()`.
+- Verified live against 11 providers + the OpenAI SDK (non-streaming & SSE).
+  Fixed the LongCat model id (`LongCat-2.0-Preview`); LLM7 leads the keyless
+  pool (most reliable zero-key provider).
+
+## [0.1.0] — 2026-06-02
+
+Initial release.
+
+### Added
+- Provider catalog (`providers.toml`) covering 9 free-tier providers and 24
+  models: Groq, Cerebras, OpenRouter, Google Gemini, GitHub Models, Cloudflare
+  Workers AI, Mistral, Cohere, SambaNova.
+- Quota-aware, least-used-first router with automatic failover across providers.
+- Persistent per-provider/day quota tracking (`~/.config/llmbuffet/quota.json`,
+  resets at UTC midnight).
+- OpenAI-compatible proxy server (`llmbuffet proxy`) exposing
+  `/v1/chat/completions` and `/v1/models` — a drop-in `OPENAI_BASE_URL`.
+- CLI: `ask`, `providers`, `quota`, `proxy`.
+- Python API: `from llmbuffet import Buffet`.
+- Three request/response adapters (openai, gemini, cloudflare) and per-user
+  catalog overrides via `~/.config/llmbuffet/providers.toml`.
+- Full unit-test suite with a faked transport (no network) and CI on Python
+  3.11–3.13.

llmbuffet-0.2.0/CONTRIBUTING.md

@@ -0,0 +1,59 @@
+# Contributing to llmbuffet
+
+Thanks for helping! The two highest-value contributions are **adding free
+providers** and **keeping the existing catalog accurate** as free tiers drift.
+
+## Dev setup
+
+```bash
+git clone https://github.com/0xzr/llmbuffet
+cd llmbuffet
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+pytest          # 0 network calls — everything is faked
+ruff check src tests
+```
+
+## Adding a provider
+
+The whole catalog is [`src/llmbuffet/providers.toml`](src/llmbuffet/providers.toml).
+Most providers are OpenAI-compatible, so adding one is just a TOML block:
+
+```toml
+[[provider]]
+id = "myprovider"
+label = "My Provider"
+adapter = "openai"                       # "openai" | "gemini" | "cloudflare"
+base_url = "https://api.myprovider.ai/v1"
+key_env = "MYPROVIDER_API_KEY"           # env var the user sets; never a key
+models = [
+    { name = "some-model", rpd = 0 },    # rpd = free daily request hint, 0 = unknown
+]
+```
+
+Rules of thumb:
+
+- **Free tier only.** llmbuffet is about free pools. If a provider needs a card
+  on file to use the tier, it doesn't belong in the default catalog.
+- **Never commit a key.** Only the *name* of the env var goes in the catalog.
+- If the provider isn't OpenAI-compatible, it needs a small adapter in
+  [`src/llmbuffet/client.py`](src/llmbuffet/client.py) (see the `gemini` one for
+  a ~30-line template) and a unit test in `tests/`.
+- Add the env var to [`.env.example`](.env.example) and the signup steps to
+  [`docs/ACCOUNTS.md`](docs/ACCOUNTS.md).
+
+## Fixing a stale limit or endpoint
+
+Free tiers change constantly. If a model name, base URL, or daily limit is
+wrong, a one-line PR to `providers.toml` is perfect and very welcome.
+
+## Tests
+
+Every code path is unit-tested without touching the network via an injected
+fake transport (`tests/helpers.py`). Please keep it that way — new behavior
+should come with a fake-backed test. Run `pytest` and `ruff check` before
+opening a PR.
+
+## Code of conduct
+
+Be kind. Assume good faith. We're all here to make free LLMs easier to use.

llmbuffet-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 0xzr (github.com/0xzr)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llmbuffet-0.2.0/PKG-INFO

@@ -0,0 +1,240 @@
+Metadata-Version: 2.4
+Name: llmbuffet
+Version: 0.2.0
+Summary: An all-you-can-eat buffet of free-tier LLM APIs behind one OpenAI-compatible endpoint, with automatic failover and quota tracking.
+Project-URL: Homepage, https://github.com/0xzr/llmbuffet
+Project-URL: Repository, https://github.com/0xzr/llmbuffet
+Project-URL: Issues, https://github.com/0xzr/llmbuffet/issues
+Author: 0xzr
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cerebras,failover,free,gateway,gemini,groq,llm,load-balancer,openai,openrouter,proxy,router
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Utilities
+Requires-Python: >=3.11
+Requires-Dist: httpx>=0.27
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# 🍽️ llmbuffet — one free LLM API gateway for every free tier
+
+**A free, OpenAI-compatible LLM gateway that pools 15 free-tier providers (Groq, Cerebras, NVIDIA NIM, Gemini, OpenRouter, GitHub Models, Cloudflare & more) behind one endpoint — with automatic failover and quota tracking. Works out of the box with zero API keys.**
+
+[![CI](https://github.com/0xzr/llmbuffet/actions/workflows/ci.yml/badge.svg)](https://github.com/0xzr/llmbuffet/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org)
+
+> Stop juggling a dozen free LLM SDKs and rate limits. Point your OpenAI client at `llmbuffet` and never pay for a hobby project's inference again.
+
+Groq, Cerebras, Google Gemini, OpenRouter, GitHub Models, Cloudflare Workers AI, Mistral, Cohere, SambaNova — each hands out a generous **free tier**, but each has its own SDK, its own rate limits, and its own daily cap. `llmbuffet` puts all of them into one pool:
+
+- 🔌 **One OpenAI-compatible endpoint.** Point any existing OpenAI SDK / tool at `llmbuffet` and it just works — no code changes.
+- 🔁 **Automatic failover.** Hit a rate limit or a 5xx on one provider? `llmbuffet` transparently moves to the next.
+- 📊 **Quota-aware routing.** Spreads load least-used-first and respects each provider's free daily limit, so you squeeze the most out of every tier.
+- 🧩 **One catalog, your keys.** Drop in the keys you have; `llmbuffet` skips the rest. No key is ever stored in the repo.
+- 🪶 **Tiny.** Pure-Python, one dependency (`httpx`). The proxy runs on the standard library.
+
+> Why it exists: stitching together a dozen free LLM tiers by hand is fiddly and breaks constantly. `llmbuffet` makes "never pay for a hobby project's LLM calls again" a one-command setup.
+
+---
+
+## Install
+
+```bash
+pip install llmbuffet      # or: pipx install llmbuffet
+```
+
+## Zero-config: it works with no keys at all
+
+Two providers in the catalog need **no signup** (OVHcloud is keyless; LLM7's key is optional), so this works the moment you install:
+
+```bash
+pip install llmbuffet
+llmbuffet ask "Explain the CAP theorem in one sentence."
+```
+
+Add provider keys (below) to unlock more models, higher limits, and better failover.
+
+## 60-second quickstart (with keys)
+
+1. Grab one or more free API keys — **all free, no credit card**. You only need
+   **one** to start (Groq and Cerebras are the fastest to sign up for).
+   👉 **[docs/ACCOUNTS.md](docs/ACCOUNTS.md) has 1-minute, click-by-click steps for every provider.**
+
+   | Provider | Get a key |
+   |---|---|
+   | Groq | <https://console.groq.com/keys> |
+   | Cerebras | <https://cloud.cerebras.ai> |
+   | OpenRouter | <https://openrouter.ai/keys> |
+   | Google Gemini | <https://aistudio.google.com/apikey> |
+   | GitHub Models | any GitHub PAT |
+
+2. Export the ones you have (see [`.env.example`](.env.example) for all of them):
+
+   ```bash
+   export GROQ_API_KEY=gsk_...
+   export CEREBRAS_API_KEY=csk-...
+   ```
+
+3. Ask something:
+
+   ```bash
+   llmbuffet ask "Explain the CAP theorem in one sentence."
+   ```
+
+   or pipe context in:
+
+   ```bash
+   cat error.log | llmbuffet ask "What's the root cause here?"
+   ```
+
+Check what's wired up:
+
+```bash
+llmbuffet providers
+```
+
+```
+llmbuffet catalog: 15 providers, 53 models
+
+  ✓ ovh          OVHcloud AI Endpoints (keyless)  5 models   [configured]
+  ✓ llm7         LLM7 (key optional)           1 models   [configured]
+  · groq         Groq                          6 models   [set GROQ_API_KEY]
+  · cerebras     Cerebras                      4 models   [set CEREBRAS_API_KEY]
+  · nvidia       NVIDIA NIM                    5 models   [set NVIDIA_API_KEY]
+  ...
+```
+
+## Choosing a model or provider
+
+By default llmbuffet auto-picks the least-used provider you have. To pin a choice:
+
+```bash
+llmbuffet models                       # list every provider/model id
+llmbuffet ask -m groq/llama-3.3-70b-versatile "hi"   # exact provider + model
+llmbuffet ask -m llama-3.3-70b-versatile "hi"        # that model on any provider
+llmbuffet ask -p cerebras,groq "hi"                  # restrict to these providers
+```
+
+Same idea through the proxy via the OpenAI `model` field: `"auto"`, `"groq"`, or `"groq/llama-3.3-70b-versatile"`.
+
+### Providers in the box
+
+| Provider | Key env | Notes |
+|---|---|---|
+| OVHcloud AI Endpoints | — | **keyless**, works out of the box |
+| LLM7 | `LLM7_API_KEY` | key optional |
+| Groq | `GROQ_API_KEY` | very fast |
+| Cerebras | `CEREBRAS_API_KEY` | very fast, large daily cap |
+| NVIDIA NIM | `NVIDIA_API_KEY` | big model catalog (build.nvidia.com) |
+| OpenRouter | `OPENROUTER_API_KEY` | many `:free` models |
+| Google Gemini | `GEMINI_API_KEY` | generous free tier |
+| GitHub Models | `GITHUB_TOKEN` | any PAT works |
+| Cloudflare Workers AI | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` | |
+| Mistral | `MISTRAL_API_KEY` | |
+| Cohere | `COHERE_API_KEY` | |
+| SambaNova | `SAMBANOVA_API_KEY` | |
+| Z.ai / Zhipu GLM | `ZHIPU_API_KEY` | |
+| Ollama Cloud | `OLLAMA_API_KEY` | |
+| LongCat (Meituan) | `LONGCAT_API_KEY` | |
+
+Full signup steps for each: **[docs/ACCOUNTS.md](docs/ACCOUNTS.md)**.
+
+## The killer feature: a drop-in OpenAI proxy
+
+Run the gateway:
+
+```bash
+llmbuffet proxy --port 8080
+```
+
+Now point **any** OpenAI-compatible app or SDK at it — no other changes:
+
+```bash
+export OPENAI_BASE_URL=http://localhost:8080/v1
+export OPENAI_API_KEY=anything        # llmbuffet ignores it
+```
+
+```python
+from openai import OpenAI
+
+client = OpenAI()  # picks up OPENAI_BASE_URL
+resp = client.chat.completions.create(
+    model="auto",                      # or "groq", or "groq/llama-3.3-70b-versatile"
+    messages=[{"role": "user", "content": "Say hi in French."}],
+)
+print(resp.choices[0].message.content)
+```
+
+The `model` field controls routing:
+
+| `model` value | Routes to |
+|---|---|
+| `auto` (or omitted) | any configured provider, least-used first |
+| `groq` | any model on Groq |
+| `groq/llama-3.3-70b-versatile` | that exact model |
+| `llama-3.3-70b-versatile` | that model on any provider that has it |
+
+## Use it as the free LLM backend for your AI agent
+
+Coding agents and agent frameworks (aider, Continue, Cline, the OpenAI Agents SDK, LangChain, ...) almost all speak the OpenAI API — so they can run on pooled free inference through `llmbuffet`, with **failover when one provider rate-limits you mid-run** (exactly when long agent loops tend to die):
+
+```bash
+llmbuffet proxy --port 8080
+export OPENAI_BASE_URL=http://localhost:8080/v1 OPENAI_API_KEY=anything
+aider --model openai/auto          # or point any OpenAI-compatible tool here
+```
+
+The proxy supports `stream: true` (Server-Sent Events), so streaming chat UIs and agent loops work too. Full integration snippets (aider, LangChain, Continue/Cline, OpenAI Agents SDK) are in **[docs/AGENTS.md](docs/AGENTS.md)**.
+
+## Use it as a library
+
+```python
+from llmbuffet import Buffet
+
+buffet = Buffet.from_default_config()
+reply = buffet.ask("Summarize the plot of Hamlet in 20 words.")
+print(reply.text)
+print(f"served by {reply.provider_id}/{reply.model}")
+```
+
+## How routing works
+
+For each request `llmbuffet` builds the list of `(provider, model)` candidates you have keys for, orders them **least-used-today first** (providers already over their free daily hint sink to the bottom), then tries them in order until one returns a non-empty completion. Every success is recorded to a small per-day counter at `~/.config/llmbuffet/quota.json` (reset at UTC midnight). See [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the full picture.
+
+## Adding or overriding providers
+
+The built-in catalog lives in [`src/llmbuffet/providers.toml`](src/llmbuffet/providers.toml). To add a provider or override a model list without forking, drop a `providers.toml` at `~/.config/llmbuffet/providers.toml` (or point `LLMBUFFET_CONFIG` at one). Same-`id` entries override the built-ins; new ids are appended. See [CONTRIBUTING.md](CONTRIBUTING.md) for the (small) anatomy of a provider.
+
+## Comparison
+
+| | llmbuffet | Calling each SDK by hand | A paid gateway |
+|---|---|---|---|
+| Free tiers pooled | ✅ 15 providers | ⚠️ you wire each one | ❌ |
+| Automatic failover | ✅ | ❌ | ✅ |
+| Quota tracking | ✅ per-day | ❌ | varies |
+| Drop-in OpenAI proxy | ✅ | ❌ | ✅ |
+| Cost | $0 | $0 | 💸 |
+| Dependencies | 1 (`httpx`) | many | a service |
+
+## Status
+
+`llmbuffet` is `0.1` and moving fast. Provider endpoints and free-tier limits drift — if something breaks, please [open an issue](https://github.com/0xzr/llmbuffet/issues) or send a one-line PR to `providers.toml`. Contributions of new free providers are especially welcome.
+
+## Found this useful?
+
+⭐ **Star the repo** — it's the single biggest thing that helps others discover llmbuffet, and it keeps the free-provider catalog maintained. New free providers and one-line limit fixes are always welcome ([CONTRIBUTING.md](CONTRIBUTING.md)).
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+