tokenbiryani 0.2.0__py3-none-any.whl

tokenbiryani-0.2.0.dist-info/METADATA

@@ -0,0 +1,697 @@
+Metadata-Version: 2.5
+Name: tokenbiryani
+Version: 0.2.0
+Summary: A pooling gateway for Claude accounts: routes on live rate-limit headers, keeps prompt caches warm, fails over without dropping a stream.
+Project-URL: Homepage, https://github.com/Abhi-shekes/tokenbiryani
+Project-URL: Documentation, https://abhi-shekes.github.io/tokenbiryani/
+Project-URL: Repository, https://github.com/Abhi-shekes/tokenbiryani
+Project-URL: Issues, https://github.com/Abhi-shekes/tokenbiryani/issues
+Project-URL: Changelog, https://github.com/Abhi-shekes/tokenbiryani/blob/main/CHANGELOG.md
+License: 
+                                         Apache License
+                                   Version 2.0, January 2004
+                                http://www.apache.org/licenses/
+        
+           TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+        
+           1. Definitions.
+        
+              "License" shall mean the terms and conditions for use, reproduction,
+              and distribution as defined by Sections 1 through 9 of this document.
+        
+              "Licensor" shall mean the copyright owner or entity authorized by
+              the copyright owner that is granting the License.
+        
+              "Legal Entity" shall mean the union of the acting entity and all
+              other entities that control, are controlled by, or are under common
+              control with that entity. For the purposes of this definition,
+              "control" means (i) the power, direct or indirect, to cause the
+              direction or management of such entity, whether by contract or
+              otherwise, or (ii) ownership of fifty percent (50%) or more of the
+              outstanding shares, or (iii) beneficial ownership of such entity.
+        
+              "You" (or "Your") shall mean an individual or Legal Entity
+              exercising permissions granted by this License.
+        
+              "Source" form shall mean the preferred form for making modifications,
+              including but not limited to software source code, documentation
+              source, and configuration files.
+        
+              "Object" form shall mean any form resulting from mechanical
+              transformation or translation of a Source form, including but
+              not limited to compiled object code, generated documentation,
+              and conversions to other media types.
+        
+              "Work" shall mean the work of authorship, whether in Source or
+              Object form, made available under the License, as indicated by a
+              copyright notice that is included in or attached to the work
+              (an example is provided in the Appendix below).
+        
+              "Derivative Works" shall mean any work, whether in Source or Object
+              form, that is based on (or derived from) the Work and for which the
+              editorial revisions, annotations, elaborations, or other modifications
+              represent, as a whole, an original work of authorship. For the purposes
+              of this License, Derivative Works shall not include works that remain
+              separable from, or merely link (or bind by name) to the interfaces of,
+              the Work and Derivative Works thereof.
+        
+              "Contribution" shall mean any work of authorship, including
+              the original version of the Work and any modifications or additions
+              to that Work or Derivative Works thereof, that is intentionally
+              submitted to Licensor for inclusion in the Work by the copyright owner
+              or by an individual or Legal Entity authorized to submit on behalf of
+              the copyright owner. For the purposes of this definition, "submitted"
+              means any form of electronic, verbal, or written communication sent
+              to the Licensor or its representatives, including but not limited to
+              communication on electronic mailing lists, source code control systems,
+              and issue tracking systems that are managed by, or on behalf of, the
+              Licensor for the purpose of discussing and improving the Work, but
+              excluding communication that is conspicuously marked or otherwise
+              designated in writing by the copyright owner as "Not a Contribution."
+        
+              "Contributor" shall mean Licensor and any individual or Legal Entity
+              on behalf of whom a Contribution has been received by Licensor and
+              subsequently incorporated within the Work.
+        
+           2. Grant of Copyright License. Subject to the terms and conditions of
+              this License, each Contributor hereby grants to You a perpetual,
+              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+              copyright license to reproduce, prepare Derivative Works of,
+              publicly display, publicly perform, sublicense, and distribute the
+              Work and such Derivative Works in Source or Object form.
+        
+           3. Grant of Patent License. Subject to the terms and conditions of
+              this License, each Contributor hereby grants to You a perpetual,
+              worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+              (except as stated in this section) patent license to make, have made,
+              use, offer to sell, sell, import, and otherwise transfer the Work,
+              where such license applies only to those patent claims licensable
+              by such Contributor that are necessarily infringed by their
+              Contribution(s) alone or by combination of their Contribution(s)
+              with the Work to which such Contribution(s) was submitted. If You
+              institute patent litigation against any entity (including a
+              cross-claim or counterclaim in a lawsuit) alleging that the Work
+              or a Contribution incorporated within the Work constitutes direct
+              or contributory patent infringement, then any patent licenses
+              granted to You under this License for that Work shall terminate
+              as of the date such litigation is filed.
+        
+           4. Redistribution. You may reproduce and distribute copies of the
+              Work or Derivative Works thereof in any medium, with or without
+              modifications, and in Source or Object form, provided that You
+              meet the following conditions:
+        
+              (a) You must give any other recipients of the Work or
+                  Derivative Works a copy of this License; and
+        
+              (b) You must cause any modified files to carry prominent notices
+                  stating that You changed the files; and
+        
+              (c) You must retain, in the Source form of any Derivative Works
+                  that You distribute, all copyright, patent, trademark, and
+                  attribution notices from the Source form of the Work,
+                  excluding those notices that do not pertain to any part of
+                  the Derivative Works; and
+        
+              (d) If the Work includes a "NOTICE" text file as part of its
+                  distribution, then any Derivative Works that You distribute must
+                  include a readable copy of the attribution notices contained
+                  within such NOTICE file, excluding those notices that do not
+                  pertain to any part of the Derivative Works, in at least one
+                  of the following places: within a NOTICE text file distributed
+                  as part of the Derivative Works; within the Source form or
+                  documentation, if provided along with the Derivative Works; or,
+                  within a display generated by the Derivative Works, if and
+                  wherever such third-party notices normally appear. The contents
+                  of the NOTICE file are for informational purposes only and
+                  do not modify the License. You may add Your own attribution
+                  notices within Derivative Works that You distribute, alongside
+                  or as an addendum to the NOTICE text from the Work, provided
+                  that such additional attribution notices cannot be construed
+                  as modifying the License.
+        
+              You may add Your own copyright statement to Your modifications and
+              may provide additional or different license terms and conditions
+              for use, reproduction, or distribution of Your modifications, or
+              for any such Derivative Works as a whole, provided Your use,
+              reproduction, and distribution of the Work otherwise complies with
+              the conditions stated in this License.
+        
+           5. Submission of Contributions. Unless You explicitly state otherwise,
+              any Contribution intentionally submitted for inclusion in the Work
+              by You to the Licensor shall be under the terms and conditions of
+              this License, without any additional terms or conditions.
+              Notwithstanding the above, nothing herein shall supersede or modify
+              the terms of any separate license agreement you may have executed
+              with Licensor regarding such Contributions.
+        
+           6. Trademarks. This License does not grant permission to use the trade
+              names, trademarks, service marks, or product names of the Licensor,
+              except as required for reasonable and customary use in describing the
+              origin of the Work and reproducing the content of the NOTICE file.
+        
+           7. Disclaimer of Warranty. Unless required by applicable law or
+              agreed to in writing, Licensor provides the Work (and each
+              Contributor provides its Contributions) on an "AS IS" BASIS,
+              WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+              implied, including, without limitation, any warranties or conditions
+              of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+              PARTICULAR PURPOSE. You are solely responsible for determining the
+              appropriateness of using or redistributing the Work and assume any
+              risks associated with Your exercise of permissions under this License.
+        
+           8. Limitation of Liability. In no event and under no legal theory,
+              whether in tort (including negligence), contract, or otherwise,
+              unless required by applicable law (such as deliberate and grossly
+              negligent acts) or agreed to in writing, shall any Contributor be
+              liable to You for damages, including any direct, indirect, special,
+              incidental, or consequential damages of any character arising as a
+              result of this License or out of the use or inability to use the
+              Work (including but not limited to damages for loss of goodwill,
+              work stoppage, computer failure or malfunction, or any and all
+              other commercial damages or losses), even if such Contributor
+              has been advised of the possibility of such damages.
+        
+           9. Accepting Warranty or Additional Liability. While redistributing
+              the Work or Derivative Works thereof, You may choose to offer,
+              and charge a fee for, acceptance of support, warranty, indemnity,
+              or other liability obligations and/or rights consistent with this
+              License. However, in accepting such obligations, You may act only
+              on Your own behalf and on Your sole responsibility, not on behalf
+              of any other Contributor, and only if You agree to indemnify,
+              defend, and hold each Contributor harmless for any liability
+              incurred by, or claims asserted against, such Contributor by reason
+              of your accepting any such warranty or additional liability.
+        
+           END OF TERMS AND CONDITIONS
+        
+           APPENDIX: How to apply the Apache License to your work.
+        
+              To apply the Apache License to your work, attach the following
+              boilerplate notice, with the fields enclosed by brackets "[]"
+              replaced with your own identifying information. (Don't include
+              the brackets!)  The text should be enclosed in the appropriate
+              comment syntax for the file format. We also recommend that a
+              file or class name and description of purpose be included on the
+              same "printed page" as the copyright notice for easier
+              identification within third-party archives.
+        
+           Copyright [yyyy] [name of copyright owner]
+        
+           Licensed under the Apache License, Version 2.0 (the "License");
+           you may not use this file except in compliance with the License.
+           You may obtain a copy of the License at
+        
+               http://www.apache.org/licenses/LICENSE-2.0
+        
+           Unless required by applicable law or agreed to in writing, software
+           distributed under the License is distributed on an "AS IS" BASIS,
+           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+           See the License for the specific language governing permissions and
+           limitations under the License.
+License-File: LICENSE
+Keywords: anthropic,claude,claude-code,gateway,llm-gateway,load-balancing,prompt-caching,proxy,rate-limit
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: Proxy Servers
+Requires-Python: >=3.8
+Requires-Dist: cryptography>=41.0
+Requires-Dist: fastapi>=0.100
+Requires-Dist: httpx>=0.24
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: uvicorn>=0.23
+Provides-Extra: all
+Requires-Dist: botocore>=1.31; extra == 'all'
+Requires-Dist: google-auth>=2.20; extra == 'all'
+Requires-Dist: redis>=5.0; extra == 'all'
+Provides-Extra: bedrock
+Requires-Dist: botocore>=1.31; extra == 'bedrock'
+Provides-Extra: dev
+Requires-Dist: botocore>=1.31; extra == 'dev'
+Requires-Dist: fakeredis>=2.20; extra == 'dev'
+Requires-Dist: google-auth>=2.20; extra == 'dev'
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: playwright>=1.40; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1; extra == 'dev'
+Requires-Dist: pytest>=7.4; extra == 'dev'
+Requires-Dist: redis>=5.0; extra == 'dev'
+Requires-Dist: ruff>=0.1; extra == 'dev'
+Requires-Dist: types-pyyaml; extra == 'dev'
+Requires-Dist: watchfiles>=0.21; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-material<10,>=9.5; extra == 'docs'
+Provides-Extra: redis
+Requires-Dist: redis>=5.0; extra == 'redis'
+Provides-Extra: secrets
+Provides-Extra: vertex
+Requires-Dist: google-auth>=2.20; extra == 'vertex'
+Description-Content-Type: text/markdown
+
+# tokenbiryani
+
+**A pooling gateway for Claude accounts.** One Anthropic-compatible endpoint in front of
+every credential you own. It routes on live rate-limit headers, keeps a conversation on
+the account holding its prompt cache, fails over without dropping a stream, and queues
+honestly when the whole pool is dry.
+
+```bash
+pip install tokenbiryani          # or: pipx install tokenbiryani
+tokenbiryani init                 # writes tokenbiryani.yaml + a virtual key
+tokenbiryani serve                # starts with an empty pool
+tokenbiryani console              # opens the browser, already signed in
+```
+
+![The console's Overview screen: readiness, next reset, queue depth, cache hit rate and
+spend across the top, a capacity horizon below it, and every account in the pool with
+its headroom meters, p95, cache rate and spend](docs/images/console-overview.png)
+
+Nothing needs exporting first. The console's wizard takes your first credential,
+**verifies it** — the same rate-limit-header check `tokenbiryani doctor` performs — and
+hands you the two lines that use it, with a working key already in them:
+
+```bash
+export ANTHROPIC_BASE_URL=http://localhost:8787
+export ANTHROPIC_AUTH_TOKEN=bir_...
+claude                            # Claude Code now runs through the pool
+```
+
+Prefer the terminal:
+
+```bash
+tokenbiryani accounts add work --api-key sk-ant-...   # probed before it is stored
+tokenbiryani accounts test                            # all of them, headers included
+```
+
+No client changes. The gateway speaks the Messages API verbatim — it swaps the auth
+header and picks an upstream, and touches nothing else in the request.
+
+> The one exception is the Bedrock and Vertex adapters. Those platforms address the
+> model in the URL and stamp their own `anthropic_version`, so exactly two fields are
+> translated, in one file (`providers/translate.py`), and nowhere else. Everything else
+> the caller sent — including parameters this gateway has never heard of — travels
+> through untouched.
+
+---
+
+## Why not just use a generic proxy
+
+LiteLLM, Portkey and friends are breadth plays: many providers, lowest-common-denominator
+routing. This is a depth play on one provider, and four things fall out of that.
+
+| | tokenbiryani | Generic proxy |
+|---|---|---|
+| **Routing signal** | Live mirror of `anthropic-ratelimit-*` headers — real remaining request and token budget, per account, per reset window | Round-robin, or reactive backoff after a 429 lands |
+| **Prompt cache** | Session affinity keeps a conversation on the account holding its cache; cache-break is a tracked metric | Round-robin shreds the cache silently |
+| **Claude Code** | First-class client — long streaming turns, huge cached prefixes, tool loops | Treated as generic chat completion |
+| **Failover** | Transparent up to the first streamed token, with an explicit documented boundary | Usually all-or-nothing |
+
+### The benchmark
+
+The same workload — 24 concurrent conversations, 8 turns each, across 4 accounts —
+under each strategy, against a mock upstream that models Anthropic's per-credential
+prompt cache. Prices are illustrative ratios, not a price list.
+
+| Strategy | Cache hit | Cache breaks | Cost | vs sticky | Billed input |
+|---|---|---|---|---|---|
+| sticky_headroom | 79.6% | 0 | $0.4774 | — | 433,152 |
+| round_robin | 47.8% | 144 | $0.9535 | 2.00x | 433,152 |
+| least_loaded | 47.8% | 144 | $0.9535 | 2.00x | 433,152 |
+| headroom | 47.8% | 144 | $0.9535 | 2.00x | 433,152 |
+
+**Cache-blind routing costs 2.00x here.** And note that round-robin, least-loaded and
+most-headroom all pay exactly the same penalty: any strategy that ignores affinity
+visits every account once per conversation, so they all take the same number of cache
+misses. The penalty is inherent to cache-blindness, not a quirk of round-robin.
+
+Reproduce it with `python benchmarks/cache_affinity.py`. `tests/test_benchmark.py` fails if sticky ever stops winning.
+
+### The prompt cache is the constraint
+
+Anthropic's cache is scoped per credential. A Claude Code turn resends a large stable
+prefix each time; warm, it bills at roughly a tenth of the input rate. **A gateway that
+balances load without cache awareness can cost several times more than no gateway at
+all.** So the default strategy is sticky, not balanced: affinity first, rebalance only
+when the owner genuinely cannot serve. Every forced break is counted.
+
+---
+
+## How a request flows
+
+```
+auth → admit → route → lease → proxy → reconcile → recover
+```
+
+1. **Auth** — a virtual key (`bir_…`); real credentials never leave the process.
+2. **Admit** — estimate cost; if no account could *ever* serve it, fail fast rather than
+   queue forever.
+3. **Route** — filter to eligible accounts, score, pick.
+4. **Lease** — atomically reserve the estimate so concurrent requests can't collectively
+   overshoot one account into a 429.
+5. **Proxy** — stream bytes through untouched.
+6. **Reconcile** — parse limit headers and usage; update the mirror, release the lease.
+7. **Recover** — classify the failure, transition the account, decide whether to retry.
+
+### Routing strategies
+
+Set `routing.strategy`:
+
+| Strategy | Behaviour |
+|---|---|
+| `sticky_headroom` | **Default.** Affinity, then most headroom. |
+| `headroom` | `sticky_headroom` with affinity off, and nothing else changed. Identical to the default for traffic that has no cache owner, so it never routes better. |
+| `cost_tiered` | Drain cheap accounts first, spill upward. |
+| `priority` | Strict ordered failover: primary, then backup. |
+| `least_loaded` | Baseline. |
+| `round_robin` | Baseline. Ignores every signal, on purpose — it's there to benchmark against. |
+
+Weights are config, not code. See `routing.weights` in `tokenbiryani.example.yaml`.
+`tokenbiryani strategies` lists what this install has, including any installed plugins:
+a strategy can ship in its own package under the `tokenbiryani.strategies` entry point,
+supplying either weights or its own scoring.
+
+### Failure handling
+
+| Upstream | Account | Request |
+|---|---|---|
+| `429` | cooldown until `retry-after` | retry elsewhere |
+| `529` | short cooldown | retry |
+| `500`/`502`/`503` | error tick, breaker may trip | retry with jitter |
+| `401`/`403` auth | **disabled**, operator alerted | retry elsewhere |
+| `403` model denied | model marked unsupported here | retry elsewhere |
+| `400` / `413` | none — not the account's fault | **returned as-is, never retried** |
+
+Retrying a `400` across the whole pool is the classic amplification bug in gateways like
+this. The taxonomy exists to prevent exactly that.
+
+### The one failure that can't be hidden
+
+Once the first SSE byte reaches the client, transparent failover is impossible — the
+client is already rendering. So the gateway buffers the upstream stream until the first
+`content_block_delta` (or `retry.first_token_grace_seconds`). A failure **before** that
+point is retried silently on another account. A failure **after** it arrives as an SSE
+`error` frame and the request ends.
+
+---
+
+## Docker
+
+```bash
+docker compose up -d      # gateway + Redis + a mock Anthropic
+docker compose down       # stop it
+```
+
+Nothing to set first. The stack comes up with an empty pool, which is what the
+onboarding wizard is for: open `http://localhost:8787/console` and add your first
+credential there. Until you do, `/healthz` answers 503 and the container reports
+itself unhealthy — an honest answer, since a pool with no accounts can serve
+nothing.
+
+`./src` is bind-mounted and watched, so editing a file on the host restarts the
+gateway in about a second; the console's HTML and CSS need only a browser refresh;
+and `./tests` is mounted too, so `docker compose exec gateway pytest -q` runs the
+suite against the running stack.
+
+The image runs as a non-root user, carries a healthcheck wired to `/healthz`, and
+publishes to `127.0.0.1` rather than your LAN. `docker compose` builds the Dockerfile's
+`dev` target; the default `runtime` target is what ships — wheel only, no source, no
+test dependencies.
+
+> The keys in `docker-compose.yml` are development values in a public repository.
+> Copy `.env.example` to `.env` and replace them before this touches anything real —
+> in particular `TOKENBIRYANI_SECRET_KEY`, which encrypts stored account credentials
+> and must outlive the container. [docs/deployment.md](docs/deployment.md) explains both.
+
+## Operating it
+
+Open **`http://localhost:8787/console`**. Everything the gateway can do, it can do from
+there:
+
+- **Accounts** — add, name, test, rename, rotate, enable, disable and delete
+  credentials without editing a file. Anthropic API keys, Bedrock, Vertex, and Claude
+  subscriptions via a browser login. Credentials are encrypted at rest; accounts
+  declared in `tokenbiryani.yaml` render locked, because the file is yours.
+- **Usage** — token, cost and cache-hit-rate charts over 1h / 24h / 7d / 30d, grouped
+  by account, model or virtual key, with a totals table. This history is persisted, so
+  it survives a restart. See [docs/usage.md](docs/usage.md).
+- **Overview** — capacity horizon, live request feed, per-account meters.
+- **Efficiency** — the four questions the meters cannot answer: will the quota last
+  the week, is the prompt cache even switched on, is one conversation running away,
+  and is the pool reserving output it never uses. See
+  [docs/pacing.md](docs/pacing.md).
+- **Requests** — the routing inspector: why each request went where it did.
+- **Connect a client** — the exact export lines for this gateway's address.
+- **Keys** — mint and revoke virtual keys.
+
+![The Accounts screen: the pool, each account's state and where it was declared, its
+cost tier, priority and spend, with Test and Edit on every
+row](docs/images/console-accounts.png)
+
+Click an account and it opens: its limits, what its meters can and cannot tell you,
+where its credential came from, an error breakdown by class, and its own recent
+requests.
+
+![An account detail panel: a subscription account explaining that it reports
+rolling-window utilisation rather than per-window budgets, where its token file lives
+and when it expires, then state, requests, failures, p95, cache hit and spend, over its
+recent requests](docs/images/console-account-detail.png)
+
+Settings shows what the gateway is *currently running* — strategy and price table, each
+marked with where its value came from, and a Reload that re-reads the file.
+
+![The Settings screen: routing strategy and price table, both marked "from the file",
+with the bundled table's date beside it and live counts of accounts, queue depth and
+priced models](docs/images/console-settings.png)
+
+It is one server-rendered page plus a stylesheet, inside the package — no build step, no
+Node toolchain added to a `pipx install`. The shell carries no data and needs no key; it
+asks for an admin key on first load and keeps it in that browser only.
+
+### Does it actually work against the real API?
+
+The one thing no mock can tell you is whether Anthropic spells its rate-limit headers
+the way the router expects. If it doesn't, the mirror stays empty, every account reads
+as full, and routing quietly degrades to round-robin — shredding the prompt cache while
+looking healthy.
+
+```bash
+tokenbiryani doctor --api-key sk-ant-...
+```
+
+One real request, `max_tokens=1`. It prints the headers the upstream actually returned
+next to the nine the limit mirror looks for, and what the mirror parsed out of them.
+Non-zero exit if anything is missing. Run it once after you first point this at
+production.
+
+Or stay in the terminal:
+
+```bash
+tokenbiryani status          # the pool, in the terminal you're already in
+tokenbiryani status --json   # same data, for scripts
+```
+
+```
+  POOL  702k tok ready · next reset 00:12 · queue 0 · cache 94%
+
+  ACCOUNT   STATE            REQ    INPUT    OUTPUT   RESET   CACHE
+  acct-01   ● ready          98%      82%       79%   00:41    97%
+  acct-02   ● cooling         4%       0%        6%   00:27     —
+
+  1h  412 requests · 3 failovers · 1 cache break · 0 errors · $18.40
+```
+
+| Endpoint | |
+|---|---|
+| `GET /console` | the operator console |
+| `POST /v1/messages` | Messages API, streaming and not |
+| `POST /v1/messages/count_tokens`, `GET /v1/models` | passthrough |
+| `GET /healthz` | 200 while any account is ready |
+| `GET /admin/status` | pool snapshot |
+| `GET /admin/usage` | bucketed usage history for the charts |
+| `POST /admin/accounts` · `PATCH` · `DELETE` · `POST /admin/accounts/{id}/test` | manage credentials at runtime |
+| `POST /admin/keys` · `DELETE /admin/keys/{name}` | mint and revoke keys at runtime |
+| `GET /admin/accounts/{id}` | one account: limits, error breakdown by class, its own recent requests |
+| `POST /admin/reload` | re-read the config file |
+| `GET /admin/requests/{id}` | **why that request went where it did** — attempt chain, per-candidate scores, verdicts |
+| `GET /admin/horizon` | projected capacity for the next hour |
+| `GET /admin/estimation` | what the output estimator has learned, per model |
+| `GET /admin/pacing` | **is this pool on course to spend its week** — or run dry early, or strand quota |
+| `GET /admin/sessions` | the most expensive conversations in the window, runaways flagged |
+| `GET /admin/cache-advice` | **why the cache hit rate is what it is** — whether the client ever sent a breakpoint, per key and model |
+| `GET /admin/events` | live SSE feed |
+
+The request inspector is the point. `filtered — cooling, 27s remaining` is a complete
+answer; "load balanced" is not.
+
+**`/admin/*` requires a key with `admin: true`.** It exposes account ids, spend and key
+management, so a tenant key must not reach it. A gateway with no keys configured at all
+is loopback development mode and stays fully open.
+
+### Managing keys at runtime
+
+```bash
+curl -sX POST localhost:8787/admin/keys -H "x-api-key: $ADMIN_KEY" \
+  -d '{"name":"tenant-1","pool":["acct-02"],"rpm":60,"spend_cap_usd":5}'
+# -> {"key": "bir_...", "record": {...}}   the plaintext appears exactly once
+
+curl -sX DELETE localhost:8787/admin/keys/tenant-1 -H "x-api-key: $ADMIN_KEY"
+```
+
+![The Keys screen: a form minting a key with pool, rpm, spend cap, priority and admin
+flag, above a table of existing keys marked "in config" with their pools and
+limits](docs/images/console-keys.png)
+
+Minted keys are stored **hashed**, so a leaked state store is not a leaked key, and they
+live in the shared store — one instance honours a key another minted. Keys declared in
+the config file belong to the file: the API will not revoke them.
+
+### The spill lane
+
+With `batch.enabled: true`, a `batch`-priority request that finds the pool saturated goes
+to the **Message Batches API** instead of waiting. The gateway holds the connection while
+it polls, bounded by that request's own wait budget. A batch that outlives the budget is
+**cancelled upstream** and its id returned in `x-tokenbiryani-batch-id`, so nothing is
+silently abandoned. If submission fails the request falls back to the normal queue — the
+spill lane is an optimisation, never a dependency. Streaming requests never spill.
+
+### Account types
+
+| `type` | Notes |
+|---|---|
+| `anthropic_api` | Anthropic API keys. The default. |
+| `bedrock` | AWS Bedrock. SigV4-signed; its binary event-stream is decoded back to SSE so the rest of the gateway sees ordinary streaming. Needs `pip install "tokenbiryani[bedrock]"`. |
+| `vertex` | Google Vertex AI. Bearer token from application-default credentials; returns real SSE already. Needs `pip install "tokenbiryani[vertex]"`. |
+| `oauth` | A Claude subscription (Max/Pro). Three token sources, offered in the console's Add-account dialog: this machine's Claude Code login (nothing to configure), a long-lived token from `claude setup-token`, or an OAuth login the gateway runs itself — that last one stays disabled until you supply the provider endpoints, which this project will not guess at. Read [docs/oauth.md](docs/oauth.md) first: a subscription reports rolling-window utilisation rather than per-window budgets, so it keeps headroom routing and failover but has no leases, no admission control and no capacity horizon. |
+
+![The Add-account dialog for a Claude subscription: three token sources — this
+machine's Claude Code login, a long-lived token, or an OAuth login — with the machine
+scan listing each credentials file it found, its plan, and whether it is current or
+stale](docs/images/console-subscription-session.png)
+
+All four sit in one pool, so a request can fail over from an API key to Bedrock. Use
+`options.model_map` to translate your callers' model names into each platform's ids.
+
+```yaml
+accounts:
+  - id: acct-01
+    type: anthropic_api
+    api_key: ${ANTHROPIC_API_KEY}
+  - id: acct-bedrock
+    type: bedrock
+    cost_tier: 1.2
+    options:
+      region: us-east-1
+      model_map:
+        claude-test-1: anthropic.claude-3-5-sonnet-20241022-v2:0
+  - id: acct-vertex
+    type: vertex
+    options:
+      project: my-project
+      region: us-central1
+```
+
+### Request headers
+
+| Header | |
+|---|---|
+| `X-TokenBiryani-Session` | pin a conversation to one affinity key instead of the fingerprint |
+| `X-TokenBiryani-Priority` | `interactive` (default) or `batch`. Batch traffic yields the queue to interactive traffic when the pool is saturated |
+| `X-TokenBiryani-Max-Wait` | seconds this request will wait for capacity. A client can shorten its own budget but never extend it past the operator's ceiling |
+
+Per-key defaults for the last two live under `keys:` as `priority` and `max_wait_seconds`.
+
+### Spend caps
+
+Caps come at two scopes. `keys[].spend_cap_usd` bounds everything a key does;
+`session_cap_usd` and `session_max_turns` bound one *conversation* inside that
+allowance. The second exists because the first cannot see a runaway: one agent loop
+resending a large prefix a few hundred times is a whole key's cap with that key's
+name on it, and until it trips there is nothing to look at. `GET /admin/sessions`
+ranks conversations by cost and flags the ones past `sessions.runaway_turns`.
+
+Caps are **windowed, not lifetime** (`spend.window_hours`, default 24). A lifetime cap on
+a persistent store would eventually wedge the gateway shut and stay that way.
+
+They only survive a restart if the store does. `store.backend: memory` (the default)
+loses affinity and the spend ledger when the process dies — meaning every cap silently
+resets. Use `sqlite` for a real deployment:
+
+```yaml
+store:
+  backend: sqlite
+  path: tokenbiryani.db
+```
+
+### Running more than one instance
+
+`store.backend: redis` shares affinity, the spend ledger and per-key rate counters
+between processes. Without it two instances each keep their own affinity map — so a
+conversation ping-pongs between them and loses its cache — and each enforces its own
+half of every spend cap.
+
+```yaml
+store:
+  backend: redis
+  url: redis://127.0.0.1:6379/0
+  namespace: tokenbiryani
+```
+
+Needs the optional dependency: `pip install "tokenbiryani[redis]"`.
+
+### Costs
+
+Costs are reported, and spend caps enforced, only for models that have a price.
+`pricing: builtin` takes the dated table that ships with this release — the console
+shows its date beside every cost — and naming a model under `pricing:` overrides it.
+
+Prices live in a dated data file rather than in code, because the gateway must never
+bill you against a number nobody can attribute. A dated file whose date is on screen
+can be attributed; a dict compiled into a release cannot.
+
+### Privacy and security
+
+- Prompt bodies are **never logged**. Only accounting: ids, accounts, tokens, latency.
+- Credentials are never logged and are masked in every admin response.
+- Binds to loopback. Refuses to start on a public interface without both
+  `server.allow_remote: true` and at least one configured key.
+
+---
+
+## Documentation
+
+Full docs build from `docs/` with `mkdocs serve`. Start with
+[Why is my bill higher?](docs/caching.md) — it is the page that changes how you
+configure this thing.
+
+## Status
+
+All eight milestones in `PLAN.md` are implemented and tested; see `TODO.md` for the
+table and for what was deliberately left out.
+
+Working today: passthrough and streaming, multi-account pooling, the error taxonomy,
+retry and failover, the rate-limit mirror, token estimation and leases, headroom scoring,
+circuit breakers, session affinity and cache accounting, admission control and a bounded
+priority queue, virtual keys with model/pool/rpm/spend scoping,
+structured logs, config hot reload, the admin API, and the CLI. Request priority with a
+per-request wait budget, a batch spill lane, and SQLite-backed persistence for affinity
+and windowed spend, a Redis store for multi-instance deployments, and runtime key
+management behind an admin boundary, plus Bedrock and Vertex adapters and pluggable
+routing strategies, and the operator console. 379 tests including a lease-concurrency
+suite, a reproducible benchmark, and an end-to-end smoke test over real sockets
+(`scripts/smoke.sh`).
+
+All eight milestones in `PLAN.md` are built. `docs/UI-DESIGN.md` is the console's design
+brief, and the console follows it.
+
+**Credential types.** Anthropic API keys are the supported path. A *subscription*
+account works — routing your own subscription through your own local gateway is the
+ordinary case, and it reports enough (rolling-window utilisation) to route on — but
+pooling several so their limits add up runs against Anthropic's consumer terms. Leases,
+admission control and the capacity horizon need absolute token counts and stay dark for
+those accounts. See [docs/oauth.md](docs/oauth.md).
+
+## License
+
+Apache-2.0.