overlaat 0.0.1__tar.gz

overlaat-0.0.1/.github/workflows/ci.yml

@@ -0,0 +1,96 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+    tags: ['v*']
+  pull_request:
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install linter
+        run: pip install ruff
+      - name: Lint with ruff
+        run: ruff check .
+
+  test:
+    needs: [lint]
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install package + dev deps
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+      - name: Run tests
+        run: pytest
+
+  build:
+    needs: [lint, test]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Verify tag matches package version
+        if: startsWith(github.ref, 'refs/tags/v')
+        run: |
+          PKG=$(python -c "import re; print(re.search(r'__version__ = \"([^\"]+)\"', open('overlaat/__init__.py').read()).group(1))")
+          TAG="${GITHUB_REF_NAME#v}"
+          echo "tag=$TAG package=$PKG"
+          test "$PKG" = "$TAG"
+      - name: Build distributions
+        run: |
+          python -m pip install --upgrade pip build twine
+          python -m build
+          twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/*
+
+  release:
+    name: Release
+    if: startsWith(github.ref, 'refs/tags/') || github.event_name == 'workflow_dispatch'
+    needs: [lint, test, build]
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write       # trusted publishing to PyPI (OIDC)
+      contents: write       # create/update the GitHub Release
+      attestations: write   # build provenance attestation
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist
+      - name: Generate artifact attestation
+        uses: actions/attest-build-provenance@v2
+        with:
+          subject-path: 'dist/*'
+      - name: Publish to PyPI
+        if: startsWith(github.ref, 'refs/tags/')
+        uses: pypa/gh-action-pypi-publish@release/v1
+      - name: Upload to GitHub Release
+        uses: softprops/action-gh-release@v2
+        with:
+          files: dist/*
+          fail_on_unmatched_files: false
+          prerelease: ${{ contains(github.ref, 'alpha') || contains(github.ref, 'beta') }}

overlaat-0.0.1/.gitignore

@@ -0,0 +1,19 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+*.env
+!*.env.example
+tmp/
+.DS_Store
+*.log
+dist/
+build/
+*.egg-info/
+CLAUDE.md
+PUBLISH.md
+# Diagram build inputs — we publish only the .excalidraw.svg (it embeds the
+# editable scene). The PNG screenshot and the .excalidraw source stay local.
+docs/overlaat-llm-stack.png
+docs/dashboard.png
+docs/*.excalidraw

overlaat-0.0.1/CHANGELOG.md

@@ -0,0 +1,59 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to
+follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html) — though while the
+status is **experimental**, the APIs and the Postgres schema may change between
+versions without a compatibility guarantee.
+
+## [0.0.1] — 2026-06-19
+
+First public release. Overlaat is a sidecar that sits in *front* of a self-hosted,
+multi-backend LiteLLM gateway and adds the two things the gateway does not do: a fair
+waiting-queue instead of a 429-cliff, and one honest usage event per request. This
+version implements:
+
+### queue-proxy (`:4000`) — the single network entry point
+- **Per-model FIFO wait-queue.** Every `/v1/chat/completions`, `/v1/completions`,
+  `/v1/embeddings`, and `/rerank` call passes through a per-model `asyncio.Semaphore`;
+  overflow **waits in FIFO order** rather than being rejected with HTTP 429.
+- **Caps derived from the gateway config**, not tuned separately — the slot size for
+  each model is read from `max_parallel_requests` in `litellm-config.yaml`. Models with
+  no cap (and all non-LLM paths) pass through without a queue.
+- **Streaming-compatible** (SSE and plain JSON), forwarding the body unbuffered and
+  headers 1:1 except hop-by-hop. Injects `stream_options.include_usage=true` on
+  streaming chat so token counts arrive reliably.
+- **One lifecycle event per request** written to Postgres `request_events`, *including
+  queued and client-abandoned calls* that insert-on-completion logging structurally
+  misses. Captures `t_enqueue` / `t_acquire` / `t_first_token` / `t_done`, outcome,
+  model, key fingerprint, and token counts (NULL, never zero, when unreported). The
+  writer is non-blocking (bounded queue + background batch insert); on overflow or DB
+  error the event is dropped and counted — the hot path is never slowed.
+- **Control + status endpoints:** `/__queue/health`, `/__queue/status`,
+  `/__queue/cancel/{req_id}`, `/__queue/cancel-all`. Cancellation affects **queued
+  requests only** (in-flight calls are deliberately not cancellable).
+
+### usage-api (`:4100`) — read-only dashboard
+- FastAPI service that **only ever reads** the event/host tables, serving an HTML
+  dashboard plus `/now`, `/timeline`, `/models`, `/perf`, `/consumers`, `/healthz`.
+- Derives the three honest concurrency curves (**offered / active / queued**),
+  throughput bucketed by time-weighted measured concurrency (min-sample guarded), and a
+  **solo decode tok/s** backend-health signal that isolates engine degradation from load.
+
+### host sampler (optional, macOS)
+- `host_logger` samples GPU% / RAM and per-backend RSS into `host_samples` every few
+  seconds, and logs swap-slot cold loads into `model_loads`. Memory is attributed
+  per-backend by RSS; GPU% is kept host-wide (per-process GPU is not measurable for
+  Metal/MLX).
+
+### schema & packaging
+- `schema.sql` — idempotent DDL for the three tables (`request_events`, `host_samples`,
+  `model_loads`); all timestamps are epoch seconds (UTC).
+- Pure-Python package (`hatchling`, Python ≥ 3.11) with example config, env, and run
+  scripts under `examples/`.
+
+### Not yet implemented
+- **Cost-weighted admission** is design-only (see `docs/COST-SCHEDULER.md`); the queue
+  is plain per-model FIFO in this version.
+
+[0.0.1]: https://github.com/tdamsma/overlaat/releases/tag/v0.0.1

overlaat-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thijs Damsma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

overlaat-0.0.1/PKG-INFO

@@ -0,0 +1,270 @@
+Metadata-Version: 2.4
+Name: overlaat
+Version: 0.0.1
+Summary: Fair waiting-queue + honest usage accounting for a Mac Studio shared as a small, trusted team's personal LLM compute server — a sidecar in front of a multi-backend LiteLLM gateway.
+Project-URL: Homepage, https://github.com/tdamsma/overlaat
+Project-URL: Repository, https://github.com/tdamsma/overlaat
+Project-URL: Issues, https://github.com/tdamsma/overlaat/issues
+Project-URL: Changelog, https://github.com/tdamsma/overlaat/blob/main/CHANGELOG.md
+Author: Thijs Damsma
+License-Expression: MIT
+License-File: LICENSE
+Keywords: apple-silicon,gateway,litellm,llm,mlx,observability,queue,self-hosted
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: Proxy Servers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Requires-Python: >=3.11
+Requires-Dist: fastapi
+Requires-Dist: httpx
+Requires-Dist: psycopg[binary]
+Requires-Dist: pyyaml
+Requires-Dist: uvicorn[standard]
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Overlaat
+
+Overlaat puts a **fair waiting-queue** and **honest usage accounting** in front of a
+self-hosted, multi-backend LLM gateway ([LiteLLM](https://github.com/BerriAI/litellm)).
+It is two small services and one Postgres schema. That is the whole thing.
+
+It is built for one specific situation, and we wrote it for ours: a **Mac Studio (or a
+similar big Apple-Silicon box) running as a small, trusted team's personal compute
+server** — a handful of models behind one gateway, a mix of interactive chat and bursty
+batch jobs, on one trusted network where API keys are *attribution*, not secrets.
+
+It is deliberately narrow. It is **not** a load balancer, **not** multi-tenant billing,
+**not** an enterprise analytics stack, and **not** trying to be. It does two things and
+tries to do them without lying to you.
+
+*Overlaat* is Dutch for a controlled spillway in a dike: it sheds overflow by design
+instead of breaching. That is the posture toward load — when requests outrun your
+backends, the excess pools in a fair FIFO queue and drains in order, rather than a
+`429`-cascade tearing through every caller's retry loop.
+
+![Where Overlaat sits in a self-hosted LLM stack](docs/overlaat-llm-stack.excalidraw.svg)
+
+## Why it exists
+
+Share one box between a few people and a few agents and two things start to hurt.
+
+1. **Overflow is a cliff, not a queue.** LiteLLM's `max_parallel_requests` (and any
+   swap-layer concurrency limit) *reject* on overflow — they return `429` rather than
+   making the caller wait. A burst of parallel jobs against a single-slot model becomes
+   a `429`-cascade, and every caller has to grow its own backoff logic. Nobody wants to
+   write that backoff loop. Several people writing it independently is worse.
+2. **Usage accounting lies by omission.** Insert-on-completion spend logging only ever
+   writes a row for a call that *ran to completion*. Calls that sat queued, calls the
+   client abandoned mid-stream, long-running calls still in flight — all invisible. You
+   cannot answer "what was actually happening on the box at 14:03" from rows that only
+   appear after the fact.
+
+Overlaat sits in the gap between "Ollama on my laptop" (no queueing, no accounting, fine
+for one person) and "enterprise gateway with a full analytics stack" (more machinery
+than a personal compute server should have to run). Fair queueing and truthful usage
+attribution, with as little machinery as we could get away with.
+
+## What it is
+
+A **sidecar in *front* of LiteLLM**, plus a **read-only dashboard**:
+
+- **queue-proxy** (`:4000`) — the single network entry point. Every request flows
+  through here and is FIFO-queued behind a **per-model semaphore**; the slot size for
+  each model is *derived* from your backend config, not tuned separately. Because it is
+  the one component on the full call path, it is also the one instrumentation site: it
+  emits **exactly one lifecycle event per request** to Postgres — *including* queued and
+  client-abandoned calls that insert-on-completion logging structurally misses.
+- **usage-api** (`:4100`) — a read-only FastAPI dashboard over those events. It never
+  writes; it only reads. Restart it whenever you like, independently of the proxy.
+
+The one principle the whole thing is built on: **instrument the call path once, derive
+everything else.** The proxy writes one honest row per request; the host sampler writes
+host facts every few seconds; the dashboard is pure query. No second source of truth to
+reconcile, no survivor bias, no two endpoints that compute "latency" three different ways.
+
+## Architecture
+
+```mermaid
+flowchart TD
+    clients["Clients<br/>chat · batch jobs · agents"]
+    proxy["overlaat queue-proxy :4000<br/>per-model FIFO semaphore<br/>wait, not 429"]
+    gw["LiteLLM gateway :4002<br/>loopback-only · routes to every backend"]
+
+    subgraph be ["GPU backends — behind LiteLLM"]
+        direction LR
+        b1["DeepSeek<br/>llama.cpp"]
+        b2["Qwen 3<br/>MLX"]
+        b3["Whisper<br/>MLX · STT"]
+        b4["Ollama<br/>embeddings"]
+    end
+
+    pg[("Postgres<br/>request_events<br/>host_samples · model_loads")]
+    usage["overlaat usage-api :4100<br/>read-only dashboard<br/>/ /now /timeline /models /consumers"]
+    host["host sampler<br/>GPU% · RAM · per-backend RSS"]
+
+    clients -->|single network entry| proxy
+    proxy -->|forwards admitted calls · loopback| gw
+    gw --> b1 & b2 & b3 & b4
+    proxy -. one lifecycle event per request<br/>including queued and abandoned .-> pg
+    host -. host facts every few seconds .-> pg
+    pg -->|read only — never writes| usage
+
+    classDef overlaat fill:#3b82f6,stroke:#1e3a5f,color:#ffffff;
+    class proxy,usage overlaat;
+```
+
+Keep LiteLLM bound to loopback. That is the trick that makes the proxy the *only* entry
+point — and therefore the single, complete instrumentation site. If there is a second
+door into the gateway, your accounting has a hole in it.
+
+## Quickstart
+
+You need a reachable Postgres (the same one LiteLLM uses is fine) and a configured
+LiteLLM gateway on loopback.
+
+```bash
+# 1. Install
+pip install -e .        # or: uv pip install -e .
+
+# 2. Apply the schema (idempotent)
+psql "$DATABASE_URL" -f schema.sql
+
+# 3. Configure
+cp examples/overlaat.env.example overlaat.env          # fill in DATABASE_URL etc.
+chmod 600 overlaat.env                                  # it holds DB credentials
+cp examples/litellm-config.example.yaml litellm-config.yaml   # your model list
+cp examples/run-queue-proxy.sh examples/run-usage-api.sh .    # the two run scripts
+
+# 4. Run the two services (behind a supervisor of your choice)
+OVERLAAT_ENV=./overlaat.env ./run-queue-proxy.sh        # :4000 entry, in front of LiteLLM
+OVERLAAT_ENV=./overlaat.env ./run-usage-api.sh          # :4100 read-only dashboard
+```
+
+Point your clients at `:4000` instead of LiteLLM directly. Open
+`http://your-host:4100/` for the dashboard. The queue-proxy derives one semaphore per
+model from `litellm-config.yaml`, so that file is the single source of truth for
+concurrency.
+
+> The proxy runs a **single uvicorn worker on purpose**: the in-memory per-model
+> semaphores and the instrumentation live in that one process, so FIFO ordering and
+> event emission must not be sharded across workers. This is a feature, not a TODO.
+
+## Honest concurrency: three curves
+
+The dashboard never invents a concurrency number. From `request_events` it derives, at
+any time *t* and per model, exactly three time series: **offered** (`t_enqueue ≤ t <
+t_done` — everything in the system, including still-queued), **active** (`t_acquire ≤ t
+< t_done` — actually occupying a backend slot, bounded by the cap *by definition*), and
+**queued** = offered − active. Throughput-vs-concurrency buckets each completed call on
+the time-weighted average `active(t)` over its own `[acquire, done]` interval, and cells
+with too few samples are marked insufficient and never shown as a trend. If we don't have
+the data, the dashboard says so instead of drawing a confident line.
+
+See [`docs/OBSERVABILITY.md`](docs/OBSERVABILITY.md) for the curves and their caveats,
+and [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) for the call-path and instrumentation
+design.
+
+## Roadmap
+
+- **Capacity-aware priority scheduler** — *not yet implemented.* Today's code runs
+  independent **per-model FIFO semaphores**: each model admits up to its own cap, and
+  the caps sum freely. That is a fine v1, but it lets two models be individually
+  "under cap" while collectively oversubscribing the single GPU.
+
+  The planned next step replaces those independent semaphores with **one global
+  priority queue + cost-weighted admission against a single shared GPU budget**
+  (`B = 1.0`). Each run costs its fraction of the GPU (`cost = 1 / cap`, so a `cap=4`
+  model costs `0.25`); the scheduler admits the highest-priority request that *fits*
+  the remaining budget and releases that cost on completion — so multiple models run
+  **in parallel up to real capacity** instead of up to the sum of their caps. Packing
+  is **work-conserving** (leftover budget keeps serving cheap jobs) with a
+  **reservation + aging** guard so a drip of cheap high-priority jobs can't starve an
+  expensive one. Backend hard caps still bind (`model_in_flight < cap` **and**
+  `used + cost ≤ B`), and **large-model switching** falls out for free: a swap-slot
+  ("fat-slot") group where only one big model is resident at a time is modeled as
+  `cost = 1.0`, so admitting one fills the budget and blocks the rest until it
+  completes. Optional **per-key priority** is un-gameable (`effective_priority =
+  min(requested, key_max)`, batch keys capped low). There is **no preemption** —
+  Metal can't reorder dispatched GPU kernels, so the only lever is *admission*.
+
+  The trade is deliberate: a shared budget is **lower peak concurrency** than summed
+  caps but **honest about the one GPU and free of thrash**, and it keeps the same
+  "wait, don't reject" spillway posture. Full design (packing policy, starvation
+  proof, the scalar-cost VRAM-vs-compute caveat):
+  [`docs/COST-SCHEDULER.md`](docs/COST-SCHEDULER.md).
+
+  > Note: today, large-model switching is performed by the underlying swap layer
+  > (e.g. llama-swap); Overlaat only **observes and logs** it (`model_loads`). The
+  > scheduler above folds that switching into its own budget arithmetic.
+
+- **Storage-backend agnostic (at least Postgres + SQLite)** — *not yet implemented.*
+  Today both writers and the dashboard talk to Postgres directly (`psycopg`). The plan
+  is a thin storage abstraction over the three tables so a single-box deployment can
+  run on **SQLite** with zero extra services, while a shared/multi-host setup keeps
+  **Postgres**. The event schema is intentionally simple (epoch-second timestamps, no
+  DB-specific types), so this is mostly an insert/query adapter plus dialect-aware DDL.
+
+## Built with LLMs, said openly
+
+This software was developed with **strong assistance from large language models** —
+Claude, and the very local models it queues — with humans leading the ideas, the
+architecture, the testing, and the debugging. We say this openly because it shaped how
+the project was built: a lot of the code, the docs, and this README were drafted by a
+model and then dogfooded against the real gateway it sits in front of. If you are not
+happy with AI-assisted code, this software is not for you.
+
+The flip side of saying it openly: the design decisions are human-owned and it runs in
+real use, but it is experimental — read the code before you rely on it.
+
+## Acknowledgements
+
+Overlaat is a thin layer, and it would not exist without the work it sits on top of:
+
+- [LiteLLM](https://github.com/BerriAI/litellm) — the gateway it stands in front of.
+- [FastAPI](https://fastapi.tiangolo.com/) / [Starlette](https://www.starlette.io/) /
+  [uvicorn](https://www.uvicorn.org/) — the two services.
+- [httpx](https://www.python-httpx.org/) — the streaming pass-through.
+- [psycopg](https://www.psycopg.org/) and [PostgreSQL](https://www.postgresql.org/) —
+  the one honest event store.
+- and the local-inference ecosystem it exists to queue:
+  [MLX](https://github.com/ml-explore/mlx),
+  [llama.cpp](https://github.com/ggml-org/llama.cpp),
+  [Ollama](https://github.com/ollama/ollama),
+  [vLLM](https://github.com/vllm-project/vllm),
+  [llama-swap](https://github.com/mostlygeek/llama-swap).
+
+## Status and caveats
+
+- **Experimental.** Shared as-is. **No support promise**, no compatibility guarantee
+  between versions.
+- **MIT licensed.** See [`LICENSE`](LICENSE).
+- Built and dogfooded on an **Apple-Silicon multi-backend** setup, but it is
+  **backend-agnostic**: all it needs is an OpenAI-compatible LiteLLM gateway in front of
+  whatever engines you run, and a Postgres to write events to.
+
+Known caveats, stated up front because that is the whole point of the project:
+
+- **Per-process GPU is not reliably measurable** on all platforms — notably Metal/MLX
+  workloads on macOS report 0. GPU% is therefore kept host-wide; **memory is attributed
+  per-backend via RSS**.
+- **Token counts are NULL** when a backend reports no `usage`. The proxy injects
+  `stream_options.include_usage=true` on streaming chat to minimize this; NULL is never
+  counted as zero.
+- **Engine tail after client-abandon.** On disconnect the slot releases at `t_done`,
+  but a single-stream engine may keep decoding briefly. The "active" curve measures
+  *slot occupancy*, not literal GPU-busy after release. In-flight requests are therefore
+  *not* safely cancellable; only still-queued requests are.