taskiq-llm-queue 0.1.0a1__tar.gz

taskiq_llm_queue-0.1.0a1/.gitignore

@@ -0,0 +1,35 @@
+# Environments and secrets (never commit tokens: GitHub, PyPI, AWS)
+.env
+.env.*
+!.env.example
+.pypirc
+*.pem
+*.key
+.venv/
+venv/
+
+# Python build and cache
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+coverage.xml
+htmlcov/
+
+# Test and tool output
+.hypothesis/
+.locust/
+*.log
+reports/
+
+# Editors and OS
+.idea/
+.vscode/
+*.swp
+.DS_Store
+Thumbs.db

taskiq_llm_queue-0.1.0a1/CHANGELOG.md

@@ -0,0 +1,85 @@
+# Changelog
+
+All notable changes are listed here. The project uses semantic versioning; while it is 0.x,
+a minor release may break the public API, with a deprecation announced one minor release ahead
+(see STABILITY.md).
+
+## [Unreleased]
+
+## [0.1.0a1] - 2026-10-29
+
+First pre-release. Everything listed below is new.
+
+### Changed before 0.1 (breaking against the earlier dev snapshots)
+- Every call returns a result object with `.meta`: `ConverseResult`, `ChatResult`, `EmbedResult`, `InvokeResult`.
+- `timeout=` / `deadline=` became `timeout_secs=` / `deadline_at=`; `request_id=` is accepted on every call.
+- `invoke_model(model, body, *, content_type=, accept=)` replaces guessing from a `"body"` key.
+- Cancellation raises `LLMQueueCancelled`.
+- `__all__` lists only the client surface; wire and plumbing names import from their submodules.
+- `aws_region` defaults to `None`; `redis_password` is a `SecretStr`; `_SECS` env names (old `_S` names warn once).
+- Redis keys are hash-tagged for cluster mode (key schema 2); ids are hashed with keyed BLAKE2b (`LLMQ_ID_HASH_SALT`), and xxhash is no longer a dependency.
+- `QueuedChatBedrockConverse` takes `model_alias=`; an alias passed as `model=` raises.
+- The empty `s3` extra is gone.
+
+### Added
+- Package skeleton (src layout, hatchling, uv, ruff, mypy strict, pytest).
+- Frozen contracts for 0.x:
+  - `QueueConfig` / `ModelConfig` / `Target`, with `alias[@target]` resolution and `LLMQ_*` env loading
+  - envelope v1 and `LLMReply`, with the version rule and hashed-id labels
+  - `request_context`
+  - `KeySpace` (Redis key schema)
+  - error tree and `ErrorInfo`
+  - protocols for providers, limiter, claims, results and receive gates
+  - metric names
+- Contract test suite.
+- Docker test stack (Valkey 8.1 with a password, ElasticMQ 1.7.1 with all queues and DLQs).
+- SQS capability check script: ElasticMQ 1.7.1 passes 20/20, moto 5.2.5 server passes 19/20 (no `StartMessageMoveTask`).
+- `HybridSQSBroker`: strict priority across queues with reserved slots, capacity-aware receive, receive gates, real long polls, batched acks and heartbeats, release on shutdown, byte-aware batched send, delays, FIFO, fair queues, expiry, message metadata; unknown tasks and newer envelopes are released, never dropped.
+- Reliability:
+  - adaptive concurrency limiter in Lua (baseline start, ramp, AIMD, pause gate, lane share, leases), with a sync twin and a local fallback limiter
+  - Lua claim store for exactly-once execution, circuit breaker and fallback resolver
+  - live config store with an audit stream
+- Results: JSON result backend (no pickle; `GETDEL`; shared-owner safe) and a per-pod dispatcher. Measured: 1,000 ready results take 1 `MGET`.
+- Providers:
+  - Bedrock with SDK retries off (`total_max_attempts=1`), and OpenAI-compatible with `max_retries=0`
+  - error-family classification and `Retry-After` parsing
+  - provider registry for built-in, import-path and entry-point providers, with allowed and denied hosts
+- `fakellm`: a fake Bedrock + OpenAI server with real error shapes, SigV4 region detection, scenarios, timed phases and an admin API, plus a Docker service.
+- `docs/DESIGN.md`: the delivery-safety rules (DS1–DS19) and broker features (P1–P17, A1–A9) that the code refers to.
+- Makefile and `tools/llmq_dev.py` task runner.
+- S3 offload for large messages (P9) and an S3 result backend (P10).
+- LangChain adapters for Bedrock and OpenAI-compatible chat and embeddings; LangChain 0.3 and 1.x; LangGraph compatible.
+- Ops: canary router, failover to the direct path, `llm-queue` CLI (routing, fallback, limiter, breaker, queues, jobs), metrics hooks.
+- `LLMQueue` client with jobs, cancellation, sync calls and an ordered drain on `stop()`.
+- Performance: orjson for JSON and the SQS wire format, one parse per received message, results encoded once, 6 Valkey round trips per worker call, `speedups` extra (hiredis).
+- Logging: structlog key-value events over stdlib `logging`; the library never configures logging.
+- Reliability fixes from a multi-pod review: infrastructure errors release messages instead of deleting them, owner-checked claims, a pod heartbeat for direct and sync paths, the limiter grows only when busy, throttles never trip the breaker, S3 payloads deleted only after ack.
+- Runnable `samples/` (13), including a replay of a real app's flows against real Bedrock.
+- 619 tests; CI on Python 3.11–3.14, oldest-dependency and LangChain 1.x jobs, integration on the Docker stack; trusted publishing to PyPI.
+
+### Known gaps
+- Valkey restarted without persistence wipes all claims; in-flight calls are then treated as lost and may run a second time (S50: 0–16 re-runs). Keep Valkey persistence on.
+- A reply that can't be stored is cached per pod; a redelivery to a different pod calls the provider again.
+- Exhausted jobs reach a dead-letter queue only if their queue has one configured.
+- Cluster-mode Valkey is covered by key-slot tests, not yet by a real cluster run.
+- A call queued during a Valkey outage longer than `valkey_wait_failover_secs` can run twice if a worker takes it before its cancel flag is replayed.
+
+### Fixed before release (found by the scenario harness, `tools/scenarios`)
+- Limiter collapse under sporadic throttles: cuts only on clustered throttles (≥2 and ≥5 % in a 1 s window), at most one cut per window, pause only when half the window throttles, growth scales with the limit. 10K burst with 2 % throttles: 22.5 → 76.1 req/s, 6,721 → 10,000 ok (S35).
+- Limit overshoot: growth slows near the last cut point; settles at 57–63 instead of swinging 44–64 (S07).
+- Spurious claim takeovers under load: the heartbeat treated the worker's own just-finished claim as lost and cancelled the result publish (S35c, S27: 19 and 4 losses → 0). A real claim loss now ends the task quietly with a warning, no ERROR traceback.
+- Valkey pools are sized to worker slots + 16; a failed lease release is retried on the next limiter call instead of holding the slot for 240 s.
+- Throttle storms: a cut that does not lower the throttle ratio stops at the start value; a throttled step gets `throttle_retry_attempts` (new, default 6) tries; an rt call walks the fallback chain up to 3 times before `no_capacity`; leases live 60 s (was 240 s, renewed every 10 s) so a killed pod's slots return within a minute; a send queued behind a burst no longer counts as SQS down while other sends complete. 10K burst with 30 % random throttles and a worker kill: 16.3 → 49.7-59.3 req/s, 17.5 % → 99.2 % ok (S35b); S35 76.1 → 90.4 req/s.
+- SQS down: the send is bounded by `sqs_send_timeout_secs` and half the remaining budget, then the call runs direct; first call served in 5.7 s instead of timing out (S20).
+- Valkey down on a send-only pod: result-read errors now trip failover; waiting callers run direct; cancels are replayed on recovery (S21).
+- Target-specific errors (ResourceNotFound, AccessDenied, model not in region, 404) move to the next fallback target instead of ending the call (S38).
+- Per-attempt timeout `attempt_timeout_secs`; configurable provider read/connect timeouts (S39).
+- A worker missing a model alias releases the message unclaimed (jittered delay) instead of acking an `internal_error`; it answers `model_not_configured` only near the delivery budget or deadline (S48).
+
+- Valkey blip: a worker releasing work after a Valkey error is no longer cancelled by the claim heartbeat mid-release; the SQS message is released before the claim, so no call is lost (S50: 32/40 → 40/40 ok).
+- Result dispatcher: a new same-lane waiter no longer forces a full read of every pending result (float rounding woke the loop); 1K waiters 43 → 26 MGET/s (S31).
+
+### Added before release
+- Scenario harness: 54 scenarios (work, retry, pause, cancel, purge, region change, timeouts, chaos, scale, 10K burst) on an isolated compose profile; `make scenarios`, `make load`, `make chaos`.
+- Scenario harness real-Bedrock mode (`SCN_PROVIDER=aws`): queues stay local, provider calls go to Bedrock; only AWS-safe scenarios run. S01, S03, S16, S17, S29, S40, S41 pass against real Bedrock (~108 calls).
+- Settings: `attempt_timeout_secs`, `provider_read_timeout_secs`, `provider_connect_timeout_secs`, `sqs_send_timeout_secs`, `valkey_wait_failover_secs`, limiter window settings.

taskiq_llm_queue-0.1.0a1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 The taskiq-llm-queue authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

taskiq_llm_queue-0.1.0a1/NOTICE

@@ -0,0 +1,15 @@
+taskiq-llm-queue
+Copyright (c) 2026 The taskiq-llm-queue authors. Licensed under the MIT License.
+
+The SQS broker's design and feature set build on two MIT-licensed projects. This package is
+an independent implementation; any code adapted from them keeps its original notice below.
+
+taskiq-aio-sqs — https://github.com/vonsteer/taskiq-aio-sqs
+  Copyright (c) 2022-2024 Jesse Constante. MIT License.
+  Ideas used: aiobotocore-based broker, visibility heartbeat, message metadata,
+  extended (S3) messages, batching, fair queues.
+
+taskiq-sqs — https://github.com/taskiq-python/taskiq-sqs
+  Copyright (c) 2026 Mike Shultz and Taskiq team. MIT License.
+  Ideas used: multi-queue structure, queue declaration, message expiry,
+  batch flush on shutdown, S3 offload middleware and result backend.

taskiq_llm_queue-0.1.0a1/PKG-INFO

@@ -0,0 +1,296 @@
+Metadata-Version: 2.5
+Name: taskiq-llm-queue
+Version: 0.1.0a1
+Summary: An LLM request client with its own durable queue: taskiq + Amazon SQS, adaptive limits, retry and fallback.
+Project-URL: Homepage, https://github.com/techiewonk/taskiq-llm-queue
+Project-URL: Documentation, https://github.com/techiewonk/taskiq-llm-queue#readme
+Project-URL: Source, https://github.com/techiewonk/taskiq-llm-queue
+Project-URL: Issues, https://github.com/techiewonk/taskiq-llm-queue/issues
+Project-URL: Changelog, https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md
+Author: The taskiq-llm-queue authors
+License-Expression: MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: bedrock,langchain,langgraph,llm,openai,queue,rate-limit,sqs,taskiq
+Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: aiobotocore>=2.23
+Requires-Dist: orjson<4,>=3.10
+Requires-Dist: pydantic<3,>=2.10
+Requires-Dist: redis>=5.0.1
+Requires-Dist: structlog>=24.1
+Requires-Dist: taskiq<0.14,>=0.13
+Provides-Extra: langchain-aws
+Requires-Dist: langchain-aws<2,>=0.2.24; extra == 'langchain-aws'
+Requires-Dist: langchain-core<2,>=0.3.68; extra == 'langchain-aws'
+Provides-Extra: langchain-openai
+Requires-Dist: langchain-core<2,>=0.3.68; extra == 'langchain-openai'
+Requires-Dist: langchain-openai<2,>=0.3.26; extra == 'langchain-openai'
+Provides-Extra: metrics
+Requires-Dist: prometheus-client>=0.20; extra == 'metrics'
+Provides-Extra: openai
+Requires-Dist: httpx<1,>=0.27; extra == 'openai'
+Requires-Dist: openai<4,>=1.40; extra == 'openai'
+Provides-Extra: speedups
+Requires-Dist: hiredis>=3; extra == 'speedups'
+Description-Content-Type: text/markdown
+
+# taskiq-llm-queue
+
+> **Status:** pre-alpha (`0.1.0a1`). Built and tested: the `LLMQueue` client, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 619 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see [CHANGELOG](https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md)).
+
+An LLM request client with its own durable queue, built on [taskiq](https://github.com/taskiq-python/taskiq) and Amazon SQS.
+
+Your code keeps calling LangChain models (`ainvoke`, `aembed_query`) or a small client API. Each call becomes a task on an SQS queue. Workers in your own app's processes run it under a limit learned from the provider's throttling, retry and fall back when needed, and return the result through Redis/Valkey.
+
+```
+your code ──> client / LangChain adapter ──> SQS (per model: -rt queue first, base, bulk)
+                                               │
+              result <── Redis/Valkey <── worker in your pods ──> Bedrock / OpenAI-compatible
+            (per-pod dispatcher)          (adaptive limiter, retry, fallback)
+```
+
+## Why
+
+- **Spikes wait in a queue instead of failing.** Throttled calls aren't lost or retried in a storm.
+- **No quota numbers to maintain.** The limit for each model and target is learned from throttle errors and shared by all pods.
+- **Chat first.** Separate `-rt` queues and reserved capacity keep live traffic ahead of background and bulk work.
+- **No extra servers.** The same app pods send and execute; SQS is only the broker.
+- **Drop-in.** LangChain/LangGraph code doesn't change: tool calling, callbacks (Langfuse, LangSmith) and response objects are identical.
+- **Safe rollout.** `direct` mode uses the same limiter, and a live canary moves traffic per model and lane without restarts.
+
+## Install
+
+```bash
+uv add "taskiq-llm-queue[langchain-aws]"                     # Bedrock + LangChain
+uv add "taskiq-llm-queue[openai,langchain-openai]"           # OpenAI-compatible + LangChain
+uv add "taskiq-llm-queue[speedups]"                          # hiredis parser for redis-py
+```
+
+| | Packages |
+|---|---|
+| Core | `taskiq`, `aiobotocore`, `redis>=5.0.1`, `pydantic>=2.10`, `orjson`, `structlog` |
+| `openai` | `openai`, `httpx` |
+| `langchain-aws` / `langchain-openai` | LangChain **0.3 and 1.x** are both supported |
+| `metrics` | `prometheus-client` |
+| `speedups` | `hiredis` |
+
+The package declares version **ranges**; pin exact versions in your app's lockfile. `aiobotocore` pins a narrow `botocore` window, so let your resolver pick an `aiobotocore` that matches your `boto3`.
+
+## Quickstart
+
+```python
+from taskiq_llm_queue import LLMQueue, ModelConfig, QueueConfig, Target, request_context
+from taskiq_llm_queue.langchain import QueuedChatBedrockConverse
+
+llmq = LLMQueue(QueueConfig(
+    key_prefix="llmq:local",
+    redis_host="localhost", redis_port=6379, redis_password="localdev", redis_ssl=False,
+    sqs_endpoint_url="http://localhost:9324",                 # ElasticMQ locally; omit on AWS
+    models={
+        "gpt120b": ModelConfig(
+            provider="bedrock", model_id="openai.gpt-oss-120b-1:0",
+            targets={"primary": Target(region="ap-south-1"),
+                     "apac": Target(profile_id="apac.openai.gpt-oss-120b-1:0")},
+            queues={"rt": "llm-gpt120b-rt", "default": "llm-gpt120b"}),
+    },
+))                                                            # or QueueConfig.from_env()  (LLMQ_* variables)
+
+async def main():
+    await llmq.start(mode="both")                             # "send" | "work" | "both"
+    model = QueuedChatBedrockConverse(llmq=llmq, model_alias="gpt120b")
+    with request_context(lane="rt", session_id="s-1", user_id="u-1"):
+        reply = await model.ainvoke("Summarise today's market in one line.")
+    print(reply.content, reply.response_metadata["served_model"])
+    await llmq.stop()                                         # drains in-flight work
+```
+
+In a FastAPI app, call `start()` in the lifespan startup and `stop()` on shutdown. Send-only processes, such as CLIs and scripts, use `mode="send"`.
+
+## Client API
+
+Every call returns a result object with `.meta`: `served_model`, `served_target`, `fallback_used`, `attempts`, `queue_wait_ms`, `provider_ms`, `worker`.
+
+```python
+res = await llmq.converse("gpt120b", {"messages": [...], "system": [...], "toolConfig": ...},
+                          lane="rt", fallback="chain", timeout_secs=90, request_id="req-1")
+res.response                    # exact Bedrock Converse dict
+
+emb = await llmq.embed("titan", ["a", "b"], lane="bulk")
+emb.vectors                     # list[list[float]]; `emb` itself also works as a list
+
+chat = await llmq.chat("llama", {"messages": [{"role": "user", "content": "hi"}]})
+chat.response                   # OpenAI-compatible chat completion dict
+
+raw = await llmq.invoke_model("titan", {"inputText": "a"}, content_type="application/json")
+raw.json(), raw.body, raw.content_type
+
+await llmq.cancel("req-1")      # not run if not started; the caller gets LLMQueueCancelled
+```
+
+- **Timing:** `timeout_secs` (relative) or `deadline_at` (absolute epoch seconds) on every call.
+- **Model ids** are aliases, optionally with a target: `gpt120b`, `gpt120b@apac`. Unknown aliases are rejected when the config is built.
+- **Errors:** everything derives from `LLMQueueError`: `LLMQueueTimeout`, `LLMQueueCancelled`, `ProviderError(retryable, retry_after_secs)`, `QueueUnavailable`, `ConfigError`.
+- **Sync callers:** `converse_sync`, `chat_sync`, `embed_sync`, `invoke_model_sync` run `direct` under the shared limiter (counted as sync fallbacks). Set `strict_async=True` to raise instead. `close_sync()` (or `stop()`) closes the sync runner.
+- **Jobs:** `register_job(name, fn)` on workers, `send_job(name, payload, timeout_secs=...)` anywhere; jobs run on the job queues with the same claims and DLQ rules.
+
+## LangChain and LangGraph
+
+| Class | Based on | How |
+|---|---|---|
+| `QueuedChatBedrockConverse` | `langchain_aws.ChatBedrockConverse` | Builds the request and parses the reply with the installed langchain-aws's own code, so `bind_tools`, structured output, cache usage and service tier match exactly |
+| `QueuedBedrockEmbeddings` | `langchain_aws.BedrockEmbeddings` | `aembed_query` / `aembed_documents` go through the queue |
+| `QueuedChatOpenAI` | `langchain_openai.ChatOpenAI` | `http_async_client` with a queued httpx transport: chat completions and embeddings go through the queue; queue meta lands in `response_metadata`. The Responses API (`/responses`) is not supported yet |
+| `QueuedOpenAIEmbeddings` | `langchain_openai.OpenAIEmbeddings` | Same transport; `check_embedding_ctx_length=False` by default for non-OpenAI servers |
+
+- **Model names:** pass the queue alias as `model_alias=` (plus `model_id=` or `llmq=`). Passing an alias as `model=` raises a clear error.
+- **Streaming:** all queued classes force `disable_streaming=True` / `streaming=False`. `astream()` and LangGraph `stream_mode="messages"` return one complete message, so nothing bypasses the queue.
+- **Laziness:** queued classes find the running `LLMQueue` on first use and check routing and the kill switch on every call, so you can create them at import time.
+
+## Configuration
+
+`QueueConfig.from_env()` reads `LLMQ_*` variables (durations end in `_SECS`; older `_S` names still work with a one-time warning). Notable settings:
+
+| Setting | Default | Note |
+|---|---|---|
+| `aws_region` | `None` | Left to the AWS credential chain / IRSA |
+| `redis_url` or `redis_host`/`port`/`username`/`password`/`ssl` | localhost | `redis_password` is a `SecretStr` |
+| `redis_socket_timeout_secs` / `connect_timeout` / `health_check_interval` | 5 / 2 / 30 | Connection errors retried 3× with backoff |
+| `id_hash_salt` (`LLMQ_ID_HASH_SALT`) | library constant | **Set your own in production**; keys the BLAKE2b hash of session/user ids |
+| `timeout_secs` per lane | rt 90, bg 300, bulk 900 | |
+| `attempt_timeout_secs` | 180 | One provider attempt; a hung target moves to the next fallback step |
+| `provider_read_timeout_secs` / `provider_connect_timeout_secs` | 300 / 10 | Bedrock botocore and OpenAI httpx timeouts |
+| `sqs_send_timeout_secs` | 5 | SQS counts as down only when no send completed for this long; then the call runs direct |
+| `throttle_retry_attempts` | 6 | Tries per fallback step for throttles (backoff holds no slot) |
+| `valkey_wait_failover_secs` | 10 | Result reads failing this long make waiting callers run direct |
+| `redis_max_connections` | 32 | Raised automatically to worker slots + 16 |
+
+Redis keys are cluster-safe (hash-tagged per request and per model+target), so cluster-mode Valkey (e.g. ElastiCache Serverless) works.
+
+## Providers
+
+```python
+ModelConfig(provider="openai", model_id="llama-3.1-70b",
+            targets={"primary": Target(base_url="http://vllm:8000/v1")}, api_key_env="LOCAL_LLM_KEY")
+QueueConfig(allowed_providers={"bedrock"}, denied_hosts=["*.openai.azure.com"])   # enforced on sender and worker
+QueueConfig(providers={"myprov": "mypkg.providers:MyProvider"})                    # custom provider, loaded by every pod
+```
+
+A custom provider implements `async call(op, params, target) -> dict` and `classify(exc) -> ErrorInfo(kind="throttle" | "retryable" | "fatal", retry_after_secs)`. It can also be registered through the `taskiq_llm_queue.providers` entry point. SDK-level retries are off inside workers (botocore `total_max_attempts=1`, openai `max_retries=0`), so throttles reach the limiter.
+
+## Lanes and queues
+
+| Lane | Queue | Use |
+|---|---|---|
+| `rt` | `llm-<model>-rt` | Live user requests; polled first; may use the whole limit |
+| `bg` | `llm-<model>` | Background work; at most 80 % of the limit |
+| `bulk` | `llm-<model>-bulk` if configured (e.g. embeddings), else base | Ingestion/batch; lowest priority |
+
+A queued `bg`/`bulk` call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.
+
+## Results
+
+Results are stored as JSON (never pickle) for 5 minutes and read once (`GETDEL`). Each pod runs one result dispatcher: a pub/sub wake reads only the announced results, and a periodic batched sweep catches the rest, so 10K waiting calls don't mean 10K polling loops.
+
+## Fallbacks: how to check and update
+
+A fallback chain is tried **inside the worker**. Each step is retried with backoff before moving on:
+
+1. same model, primary target
+2. same model, another target (e.g. a cross-region profile `gpt120b@apac`)
+3. another model (only when the call's policy is `chain`)
+
+Embeddings never switch model. Throttles shrink the limiter but never trip the circuit breaker; only 5xx and timeouts do.
+
+```bash
+uv run llm-queue fallback get                       # whole config (JSON); starts from your app's defaults
+uv run llm-queue fallback get --model gpt120b       # resolved chain for one model
+uv run llm-queue fallback test --model gpt120b      # dry run: which target would serve now
+uv run llm-queue fallback set --model gpt120b --chain "gpt20b"
+uv run llm-queue fallback set --file fallback.json  # merges per model; --replace to overwrite all
+uv run llm-queue fallback reset                     # back to the startup default
+```
+
+Chains are validated and reloaded by every pod within 30 s with no restart. An invalid config is rejected and the last good one stays. Every change is written to an audit stream.
+
+```json
+{ "chains": {
+    "gpt120b": ["gpt20b"],
+    "gpt20b": [{"model": "gpt120b", "require_headroom": true}]
+  },
+  "skip_targets": { "gpt120b": ["apac"] } }
+```
+
+**To see fallbacks happen:** `res.meta` / `response_metadata` show `served_model`, `served_target`, `fallback_used` and `attempts`, and the `llmq_fallback_total{model,step}` metric counts them. `samples/05_fallback_on_throttle.py` shows it end to end.
+
+## Operating it
+
+```bash
+uv run llm-queue routing set --model gpt20b --lane rt --pct 10   # canary: 10 % of sessions via SQS
+uv run llm-queue routing kill on                                 # everything direct within 30 s
+uv run llm-queue routing reset
+uv run llm-queue limiter show --model gpt120b                    # learned limit, in-flight, pause
+uv run llm-queue limiter reset --model gpt120b [--target apac]   # forget the learned limit
+uv run llm-queue breaker show | reset
+uv run llm-queue queues                                          # messages per queue and DLQ
+uv run llm-queue job send ingest --payload-file payload.json --queue jobs-ingest
+```
+
+## Failure behaviour
+
+| Failure | What happens |
+|---|---|
+| Provider throttles | Limiter shrinks and pauses that model + target; brokers stop receiving it unless another target in the chain has headroom; work waits in SQS |
+| Provider down (5xx, timeouts) | Circuit breaker opens → next fallback step |
+| Worker killed mid-task | Message reappears after the visibility timeout; the claim prevents a second provider call |
+| Rolling deploy | Pods stop receiving at once, release unstarted work, finish running calls, then exit (90 s budget in the reference deployment) |
+| **SQS unreachable** | That model switches to `direct` (+ shared limiter) for 300 s on transient send errors only. An ambiguous send is claimed first, so it can't run twice |
+| **Redis/Valkey unreachable** | Workers release messages instead of losing them; senders run `direct` with a local per-pod cap (last known limit ÷ live pods) |
+
+## Logging
+
+The library logs with structlog's key-value API on top of stdlib `logging` and **never configures logging itself**. Records are named `taskiq_llm_queue.*`; the message reads `event key=value ...` and the fields are also on `record.llmq` (a dict). Prompts, responses and params are never logged.
+
+```python
+import logging
+logging.getLogger("taskiq_llm_queue").setLevel(logging.INFO)   # stdlib apps: that's all
+
+# loguru apps: the usual InterceptHandler picks these up like any stdlib logger.
+# structlog apps: use structlog.stdlib.ProcessorFormatter on your handler to render them.
+```
+
+The `llm-queue` CLI logs warnings and errors to stderr, so its stdout is always plain JSON.
+
+## Security and data
+
+- Encryption is server-side on the queues (SSE-SQS or SSE-KMS); the library needs no setting for either.
+- Prompts and responses are never logged. `user_id` / `session_id` are hashed (keyed BLAKE2b) in task labels; raw ids stay inside the message body.
+- Nothing secret goes into messages; credentials come from the AWS chain (IRSA / Pod Identity) or env.
+- Unknown task names or newer message versions are released for another pod, never silently dropped.
+
+## Run it locally (no AWS needed)
+
+```bash
+make up              # Valkey (password) + ElasticMQ + fakellm
+make test            # unit + integration against the stack
+make down
+# Windows without make:  uv run python tools/llmq_dev.py up | test | down
+```
+
+Runnable examples are in [`samples/`](https://github.com/techiewonk/taskiq-llm-queue/tree/main/samples): quickstart, chat, embeddings, lanes, fallback on throttle, send-only and worker pods, LangChain, LangGraph, jobs, cancel and timeouts, sync calls, CLI, and `13_research_assistant_flow.py`, which replays a real app's flows locally or against real Bedrock (`--target aws`, settings from `.env` or AWS Secrets Manager). The stack, capability checks and test layers are described in [docs/TESTING.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/TESTING.md); delivery-safety rules in [docs/DESIGN.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/DESIGN.md).
+
+## Credits
+
+Broker design and feature set build on [taskiq-aio-sqs](https://github.com/vonsteer/taskiq-aio-sqs) (MIT, © 2022-2024 Jesse Constante) and [taskiq-sqs](https://github.com/taskiq-python/taskiq-sqs) (MIT, © 2026 Mike Shultz and the Taskiq team). Licence: MIT.