taskiq-llm-queue 0.1.0a1__tar.gz → 0.1.0a2__tar.gz

{taskiq_llm_queue-0.1.0a1 → taskiq_llm_queue-0.1.0a2}/CHANGELOG.md

@@ -1,85 +1,120 @@
-# Changelog
-
-All notable changes are listed here. The project uses semantic versioning; while it is 0.x,
-a minor release may break the public API, with a deprecation announced one minor release ahead
-(see STABILITY.md).
-
-## [Unreleased]
-
-## [0.1.0a1] - 2026-10-29
-
-First pre-release. Everything listed below is new.
-
-### Changed before 0.1 (breaking against the earlier dev snapshots)
-- Every call returns a result object with `.meta`: `ConverseResult`, `ChatResult`, `EmbedResult`, `InvokeResult`.
-- `timeout=` / `deadline=` became `timeout_secs=` / `deadline_at=`; `request_id=` is accepted on every call.
-- `invoke_model(model, body, *, content_type=, accept=)` replaces guessing from a `"body"` key.
-- Cancellation raises `LLMQueueCancelled`.
-- `__all__` lists only the client surface; wire and plumbing names import from their submodules.
-- `aws_region` defaults to `None`; `redis_password` is a `SecretStr`; `_SECS` env names (old `_S` names warn once).
-- Redis keys are hash-tagged for cluster mode (key schema 2); ids are hashed with keyed BLAKE2b (`LLMQ_ID_HASH_SALT`), and xxhash is no longer a dependency.
-- `QueuedChatBedrockConverse` takes `model_alias=`; an alias passed as `model=` raises.
-- The empty `s3` extra is gone.
-
-### Added
-- Package skeleton (src layout, hatchling, uv, ruff, mypy strict, pytest).
-- Frozen contracts for 0.x:
-  - `QueueConfig` / `ModelConfig` / `Target`, with `alias[@target]` resolution and `LLMQ_*` env loading
-  - envelope v1 and `LLMReply`, with the version rule and hashed-id labels
-  - `request_context`
-  - `KeySpace` (Redis key schema)
-  - error tree and `ErrorInfo`
-  - protocols for providers, limiter, claims, results and receive gates
-  - metric names
-- Contract test suite.
-- Docker test stack (Valkey 8.1 with a password, ElasticMQ 1.7.1 with all queues and DLQs).
-- SQS capability check script: ElasticMQ 1.7.1 passes 20/20, moto 5.2.5 server passes 19/20 (no `StartMessageMoveTask`).
-- `HybridSQSBroker`: strict priority across queues with reserved slots, capacity-aware receive, receive gates, real long polls, batched acks and heartbeats, release on shutdown, byte-aware batched send, delays, FIFO, fair queues, expiry, message metadata; unknown tasks and newer envelopes are released, never dropped.
-- Reliability:
-  - adaptive concurrency limiter in Lua (baseline start, ramp, AIMD, pause gate, lane share, leases), with a sync twin and a local fallback limiter
-  - Lua claim store for exactly-once execution, circuit breaker and fallback resolver
-  - live config store with an audit stream
-- Results: JSON result backend (no pickle; `GETDEL`; shared-owner safe) and a per-pod dispatcher. Measured: 1,000 ready results take 1 `MGET`.
-- Providers:
-  - Bedrock with SDK retries off (`total_max_attempts=1`), and OpenAI-compatible with `max_retries=0`
-  - error-family classification and `Retry-After` parsing
-  - provider registry for built-in, import-path and entry-point providers, with allowed and denied hosts
-- `fakellm`: a fake Bedrock + OpenAI server with real error shapes, SigV4 region detection, scenarios, timed phases and an admin API, plus a Docker service.
-- `docs/DESIGN.md`: the delivery-safety rules (DS1–DS19) and broker features (P1–P17, A1–A9) that the code refers to.
-- Makefile and `tools/llmq_dev.py` task runner.
-- S3 offload for large messages (P9) and an S3 result backend (P10).
-- LangChain adapters for Bedrock and OpenAI-compatible chat and embeddings; LangChain 0.3 and 1.x; LangGraph compatible.
-- Ops: canary router, failover to the direct path, `llm-queue` CLI (routing, fallback, limiter, breaker, queues, jobs), metrics hooks.
-- `LLMQueue` client with jobs, cancellation, sync calls and an ordered drain on `stop()`.
-- Performance: orjson for JSON and the SQS wire format, one parse per received message, results encoded once, 6 Valkey round trips per worker call, `speedups` extra (hiredis).
-- Logging: structlog key-value events over stdlib `logging`; the library never configures logging.
-- Reliability fixes from a multi-pod review: infrastructure errors release messages instead of deleting them, owner-checked claims, a pod heartbeat for direct and sync paths, the limiter grows only when busy, throttles never trip the breaker, S3 payloads deleted only after ack.
-- Runnable `samples/` (13), including a replay of a real app's flows against real Bedrock.
-- 619 tests; CI on Python 3.11–3.14, oldest-dependency and LangChain 1.x jobs, integration on the Docker stack; trusted publishing to PyPI.
-
-### Known gaps
-- Valkey restarted without persistence wipes all claims; in-flight calls are then treated as lost and may run a second time (S50: 0–16 re-runs). Keep Valkey persistence on.
-- A reply that can't be stored is cached per pod; a redelivery to a different pod calls the provider again.
-- Exhausted jobs reach a dead-letter queue only if their queue has one configured.
-- Cluster-mode Valkey is covered by key-slot tests, not yet by a real cluster run.
-- A call queued during a Valkey outage longer than `valkey_wait_failover_secs` can run twice if a worker takes it before its cancel flag is replayed.
-
-### Fixed before release (found by the scenario harness, `tools/scenarios`)
-- Limiter collapse under sporadic throttles: cuts only on clustered throttles (≥2 and ≥5 % in a 1 s window), at most one cut per window, pause only when half the window throttles, growth scales with the limit. 10K burst with 2 % throttles: 22.5 → 76.1 req/s, 6,721 → 10,000 ok (S35).
-- Limit overshoot: growth slows near the last cut point; settles at 57–63 instead of swinging 44–64 (S07).
-- Spurious claim takeovers under load: the heartbeat treated the worker's own just-finished claim as lost and cancelled the result publish (S35c, S27: 19 and 4 losses → 0). A real claim loss now ends the task quietly with a warning, no ERROR traceback.
-- Valkey pools are sized to worker slots + 16; a failed lease release is retried on the next limiter call instead of holding the slot for 240 s.
-- Throttle storms: a cut that does not lower the throttle ratio stops at the start value; a throttled step gets `throttle_retry_attempts` (new, default 6) tries; an rt call walks the fallback chain up to 3 times before `no_capacity`; leases live 60 s (was 240 s, renewed every 10 s) so a killed pod's slots return within a minute; a send queued behind a burst no longer counts as SQS down while other sends complete. 10K burst with 30 % random throttles and a worker kill: 16.3 → 49.7-59.3 req/s, 17.5 % → 99.2 % ok (S35b); S35 76.1 → 90.4 req/s.
-- SQS down: the send is bounded by `sqs_send_timeout_secs` and half the remaining budget, then the call runs direct; first call served in 5.7 s instead of timing out (S20).
-- Valkey down on a send-only pod: result-read errors now trip failover; waiting callers run direct; cancels are replayed on recovery (S21).
-- Target-specific errors (ResourceNotFound, AccessDenied, model not in region, 404) move to the next fallback target instead of ending the call (S38).
-- Per-attempt timeout `attempt_timeout_secs`; configurable provider read/connect timeouts (S39).
-- A worker missing a model alias releases the message unclaimed (jittered delay) instead of acking an `internal_error`; it answers `model_not_configured` only near the delivery budget or deadline (S48).
-
-- Valkey blip: a worker releasing work after a Valkey error is no longer cancelled by the claim heartbeat mid-release; the SQS message is released before the claim, so no call is lost (S50: 32/40 → 40/40 ok).
-- Result dispatcher: a new same-lane waiter no longer forces a full read of every pending result (float rounding woke the loop); 1K waiters 43 → 26 MGET/s (S31).
-
-### Added before release
-- Scenario harness: 54 scenarios (work, retry, pause, cancel, purge, region change, timeouts, chaos, scale, 10K burst) on an isolated compose profile; `make scenarios`, `make load`, `make chaos`.
-- Scenario harness real-Bedrock mode (`SCN_PROVIDER=aws`): queues stay local, provider calls go to Bedrock; only AWS-safe scenarios run. S01, S03, S16, S17, S29, S40, S41 pass against real Bedrock (~108 calls).
-- Settings: `attempt_timeout_secs`, `provider_read_timeout_secs`, `provider_connect_timeout_secs`, `sqs_send_timeout_secs`, `valkey_wait_failover_secs`, limiter window settings.
+# Changelog
+
+All notable changes are listed here. The project uses semantic versioning; while it is 0.x,
+a minor release may break the public API, with a deprecation announced one minor release ahead
+(see STABILITY.md).
+
+## [Unreleased]
+
+## [0.1.0a2] - 2026-10-30
+
+### Added
+- `ModelConfig.max_concurrency` (env `LLMQ_AC_MAX_BY_MODEL`, JSON `{"alias": N}`): a hard cluster-wide cap on calls in flight per model alias and target. The adaptive limiter still learns and cuts below it but never grows past it. Start limits above the cap are clamped (logged once), and a limit learned before the cap was set is clamped on next use. The cap also bounds the sync `direct` path (same Lua) and the per-pod local caps used while Valkey is down. Scenario S53 checks it: 2,000 queued, peak provider concurrency exactly 10.
+- The receive budget subtracts the pod's own callers already waiting for a limiter slot, so a full bucket leaves the backlog in SQS.
+- Settings per provider and per model, live on every pod without a restart (`model_settings.py`, README "Settings per provider and model"): `max_concurrency`, `limiter_initial`, `bg_share`, lane timeouts, `attempt_timeout_secs`, `throttle_retry_attempts`, the retry curve and breaker threshold/cooldown resolve through six layers (live model > live provider > env model > env provider > env global > default). Env layers come from the new `LLMQ_SETTINGS` JSON; live layers from Valkey `P:{cfg}:models`, validated, audited and reloaded every `cfg_reload_secs`.
+- `shared_max_concurrency` on a provider: one bucket across all its models and pods (limiter key `{m:@<provider>:shared}`); each call holds its model slot and the shared slot. Scenario S55: shared cap 8 over two models, combined peak exactly 8, 600/600 OK.
+- `llm-queue settings get [--model M | --provider P] | set | unset | reset`; `get` shows every value with its source. Scenario S54: a live `max_concurrency` cut from 20 to 5 mid-run reached all 3 workers within one reload, 900/900 OK.
+- Retry curve knobs `LLMQ_RETRY_MULTIPLIER` (4), `LLMQ_RETRY_SPREAD` (2.5), `LLMQ_RETRY_MAX_SECS` (30) and `LLMQ_THROTTLE_RETRY_MAX_SECS` (10).
+- DEBUG trace per call on `taskiq_llm_queue.core`: `call_started`, `slot_acquired` / `slot_unavailable`, `attempt_done`, `backoff`, `rt_capacity_wait`, `call_requeued`, `call_done`, each with the `request_id`.
+- `sqs_timeouts_to_failover` (`LLMQ_SQS_TIMEOUTS_TO_FAILOVER`, 3).
+- Rate budgets per model target and per provider: `rpm`, `tpm`, `rate_window` (`smooth` token bucket or `fixed` calendar minute), `rate_burst`, `rt_rate_share` (env `LLMQ_RATE_WINDOW`, `LLMQ_RT_RATE_SHARE`), `shared_rpm` / `shared_tpm`; settable in `LLMQ_SETTINGS` and live. Out of budget, rt moves to the next target and then fails `rate_limited` with `retry_after_secs`; queued bg/bulk wait in SQS for their own target. TPM is corrected from the response `usage`. Scenarios R01-R12.
+- `lane_max_concurrency.<lane>`: a bucket per lane inside a model's bucket (per target, all pods, env and live). Scenario S57: bulk 3 and bg 5 held exactly under 600 parallel calls.
+- `llm-queue status`: waiting / working / started this minute / budget left per model and target; gauges `llmq_rate_used`, `llmq_rate_remaining`.
+- Limiter knobs as `LLMQ_AC_*` variables: `UTILISATION`, `WINDOW_SECS`, `LOSS_TOLERANCE`, `MIN_THROTTLES`, `PAUSE_RATIO`, `SETTLE_RATIO`, `PROBE_SECS`, `HOLD_RATIO`, `PEAK_HALFLIFE_SECS`.
+- `llmq_inflight` and `llmq_breaker_open` gauges (per model target used by the pod) and SQS counters from every broker hook: `llmq_sqs_received_total`, `_sent_total`, `_acked_total`, `_released_total` (by reason), `_expired_total`, `_errors_total`, and `llmq_sqs_wait_seconds`.
+- The Valkey state schema key `<prefix>:schema` is written on start; state from a newer schema refuses to start.
+- `cancel()` (and a caller timeout) also stops a provider attempt already running: workers subscribe to `<prefix>:cancel` and cancel that attempt (scenario S56: slot freed 0.03–0.05 s after the cancel).
+- Send batching on bg (`default`) and bulk queues (`sqs_batch_wait_secs`, 0.02 s; 0 turns it off); rt still sends at once.
+- `s3_offload_bucket` / `s3_offload_prefix` / `s3_endpoint_url`: oversized messages carry their params in S3 (`S3OffloadMiddleware` on every broker) instead of a Valkey `params_ref`.
+
+### Changed
+- A received rt call keeps looking for a free slot for `rt_capacity_wait_secs` (30 s, env `LLMQ_RT_CAPACITY_WAIT_SECS`, live per provider/model, never past its deadline) instead of 3 passes; `core.RT_CAPACITY_PASSES` is gone. S35b: 36 of 6,043 rt calls had given up with 33–157 s of deadline left.
+- Failed calls raise with `.meta` (attempts, queue wait, last target tried) on every `LLMQueueError`; error replies fill `queue_wait_ms` and the last target.
+- Retry backoff: retry n sleeps a random time in `[base * multiplier^n, that * spread]`, i.e. 1-2.5 s, 4-10 s, 16-30 s, then 30 s (was `base * 2^n + uniform(0, base)`, capped at 20 s). After a throttle one sleep is capped at 10 s. `core.MAX_BACKOFF_SECS` is gone; `backoff_secs` takes a `Backoff` curve.
+
+- A model fails over to direct after `sqs_timeouts_to_failover` SQS send timeouts within 60 s, not after one (a refused send still fails over at once); the timed-out call itself still runs direct. One timeout in a 10K burst used to cost the model 300 s of SQS.
+- A 5xx that opens a target's breaker ends the call's retries on that target; in-flight calls kept retrying the open target (S12).
+- `QueuedChatBedrockConverse` on langchain-aws 0.2.24: the queue-only client placeholder went into `model_kwargs` (no `bedrock_client` field there), reached the Converse request and failed envelope serialization. Only fields the installed class has are filled.
+- A pod that lacks a model waits a random fraction of a second before handing the message back, so the message no longer comes back in phase with that pod's own polls (S48: the old pod had won 8 redeliveries in a row).
+- `limiter_initial_clamped` warns only for a start you set, not for the default start (20) above a small `max_concurrency`.
+
+### Removed
+- The old `_S` / suffix-less env spellings (`LLMQ_CONSUMER_DRAIN_S`, `LLMQ_RETRY_BASE_S`, `LLMQ_SQS_WAIT_TIME_S`, `LLMQ_TIMEOUT_RT`, ...) are no longer read; the README lists each with its `_SECS` name.
+
+## [0.1.0a1] - 2026-10-29
+
+First pre-release. Everything listed below is new.
+
+### Changed before 0.1 (breaking against the earlier dev snapshots)
+- Every call returns a result object with `.meta`: `ConverseResult`, `ChatResult`, `EmbedResult`, `InvokeResult`.
+- `timeout=` / `deadline=` became `timeout_secs=` / `deadline_at=`; `request_id=` is accepted on every call.
+- `invoke_model(model, body, *, content_type=, accept=)` replaces guessing from a `"body"` key.
+- Cancellation raises `LLMQueueCancelled`.
+- `__all__` lists only the client surface; wire and plumbing names import from their submodules.
+- `aws_region` defaults to `None`; `redis_password` is a `SecretStr`; `_SECS` env names (old `_S` names warn once).
+- Redis keys are hash-tagged for cluster mode (key schema 2); ids are hashed with keyed BLAKE2b (`LLMQ_ID_HASH_SALT`), and xxhash is no longer a dependency.
+- `QueuedChatBedrockConverse` takes `model_alias=`; an alias passed as `model=` raises.
+- The empty `s3` extra is gone.
+
+### Added
+- Package skeleton (src layout, hatchling, uv, ruff, mypy strict, pytest).
+- Frozen contracts for 0.x:
+  - `QueueConfig` / `ModelConfig` / `Target`, with `alias[@target]` resolution and `LLMQ_*` env loading
+  - envelope v1 and `LLMReply`, with the version rule and hashed-id labels
+  - `request_context`
+  - `KeySpace` (Redis key schema)
+  - error tree and `ErrorInfo`
+  - protocols for providers, limiter, claims, results and receive gates
+  - metric names
+- Contract test suite.
+- Docker test stack (Valkey 8.1 with a password, ElasticMQ 1.7.1 with all queues and DLQs).
+- SQS capability check script: ElasticMQ 1.7.1 passes 20/20, moto 5.2.5 server passes 19/20 (no `StartMessageMoveTask`).
+- `HybridSQSBroker`: strict priority across queues with reserved slots, capacity-aware receive, receive gates, real long polls, batched acks and heartbeats, release on shutdown, byte-aware batched send, delays, FIFO, fair queues, expiry, message metadata; unknown tasks and newer envelopes are released, never dropped.
+- Reliability:
+  - adaptive concurrency limiter in Lua (baseline start, ramp, AIMD, pause gate, lane share, leases), with a sync twin and a local fallback limiter
+  - Lua claim store for exactly-once execution, circuit breaker and fallback resolver
+  - live config store with an audit stream
+- Results: JSON result backend (no pickle; `GETDEL`; shared-owner safe) and a per-pod dispatcher. Measured: 1,000 ready results take 1 `MGET`.
+- Providers:
+  - Bedrock with SDK retries off (`total_max_attempts=1`), and OpenAI-compatible with `max_retries=0`
+  - error-family classification and `Retry-After` parsing
+  - provider registry for built-in, import-path and entry-point providers, with allowed and denied hosts
+- `fakellm`: a fake Bedrock + OpenAI server with real error shapes, SigV4 region detection, scenarios, timed phases and an admin API, plus a Docker service.
+- `docs/DESIGN.md`: the delivery-safety rules (DS1–DS19) and broker features (P1–P17, A1–A9) that the code refers to.
+- Makefile and `tools/llmq_dev.py` task runner.
+- S3 offload for large messages (P9) and an S3 result backend (P10).
+- LangChain adapters for Bedrock and OpenAI-compatible chat and embeddings; LangChain 0.3 and 1.x; LangGraph compatible.
+- Ops: canary router, failover to the direct path, `llm-queue` CLI (routing, fallback, limiter, breaker, queues, jobs), metrics hooks.
+- `LLMQueue` client with jobs, cancellation, sync calls and an ordered drain on `stop()`.
+- Performance: orjson for JSON and the SQS wire format, one parse per received message, results encoded once, 6 Valkey round trips per worker call, `speedups` extra (hiredis).
+- Logging: structlog key-value events over stdlib `logging`; the library never configures logging.
+- Reliability fixes from a multi-pod review: infrastructure errors release messages instead of deleting them, owner-checked claims, a pod heartbeat for direct and sync paths, the limiter grows only when busy, throttles never trip the breaker, S3 payloads deleted only after ack.
+- Runnable `samples/` (13), including a replay of a real app's flows against real Bedrock.
+- 619 tests; CI on Python 3.11–3.14, oldest-dependency and LangChain 1.x jobs, integration on the Docker stack; trusted publishing to PyPI.
+
+### Known gaps
+- Valkey restarted without persistence wipes all claims; in-flight calls are then treated as lost and may run a second time (S50: 0–16 re-runs). Keep Valkey persistence on.
+- A reply that can't be stored is cached per pod; a redelivery to a different pod calls the provider again.
+- Exhausted jobs reach a dead-letter queue only if their queue has one configured.
+- Cluster-mode Valkey is covered by key-slot tests, not yet by a real cluster run.
+- A call queued during a Valkey outage longer than `valkey_wait_failover_secs` can run twice if a worker takes it before its cancel flag is replayed.
+
+### Fixed before release (found by the scenario harness, `tools/scenarios`)
+- Limiter collapse under sporadic throttles: cuts only on clustered throttles (≥2 and ≥5 % in a 1 s window), at most one cut per window, pause only when half the window throttles, growth scales with the limit. 10K burst with 2 % throttles: 22.5 → 76.1 req/s, 6,721 → 10,000 ok (S35).
+- Limit overshoot: growth slows near the last cut point; settles at 57–63 instead of swinging 44–64 (S07).
+- Spurious claim takeovers under load: the heartbeat treated the worker's own just-finished claim as lost and cancelled the result publish (S35c, S27: 19 and 4 losses → 0). A real claim loss now ends the task quietly with a warning, no ERROR traceback.
+- Valkey pools are sized to worker slots + 16; a failed lease release is retried on the next limiter call instead of holding the slot for 240 s.
+- Throttle storms: a cut that does not lower the throttle ratio stops at the start value; a throttled step gets `throttle_retry_attempts` (new, default 6) tries; an rt call walks the fallback chain up to 3 times before `no_capacity`; leases live 60 s (was 240 s, renewed every 10 s) so a killed pod's slots return within a minute; a send queued behind a burst no longer counts as SQS down while other sends complete. 10K burst with 30 % random throttles and a worker kill: 16.3 → 49.7-59.3 req/s, 17.5 % → 99.2 % ok (S35b); S35 76.1 → 90.4 req/s.
+- SQS down: the send is bounded by `sqs_send_timeout_secs` and half the remaining budget, then the call runs direct; first call served in 5.7 s instead of timing out (S20).
+- Valkey down on a send-only pod: result-read errors now trip failover; waiting callers run direct; cancels are replayed on recovery (S21).
+- Target-specific errors (ResourceNotFound, AccessDenied, model not in region, 404) move to the next fallback target instead of ending the call (S38).
+- Per-attempt timeout `attempt_timeout_secs`; configurable provider read/connect timeouts (S39).
+- A worker missing a model alias releases the message unclaimed (jittered delay) instead of acking an `internal_error`; it answers `model_not_configured` only near the delivery budget or deadline (S48).
+
+- Valkey blip: a worker releasing work after a Valkey error is no longer cancelled by the claim heartbeat mid-release; the SQS message is released before the claim, so no call is lost (S50: 32/40 → 40/40 ok).
+- Result dispatcher: a new same-lane waiter no longer forces a full read of every pending result (float rounding woke the loop); 1K waiters 43 → 26 MGET/s (S31).
+
+### Added before release
+- Scenario harness: 54 scenarios (work, retry, pause, cancel, purge, region change, timeouts, chaos, scale, 10K burst) on an isolated compose profile; `make scenarios`, `make load`, `make chaos`.
+- Scenario harness real-Bedrock mode (`SCN_PROVIDER=aws`): queues stay local, provider calls go to Bedrock; only AWS-safe scenarios run. S01, S03, S16, S17, S29, S40, S41 pass against real Bedrock (~108 calls).
+- Settings: `attempt_timeout_secs`, `provider_read_timeout_secs`, `provider_connect_timeout_secs`, `sqs_send_timeout_secs`, `valkey_wait_failover_secs`, limiter window settings.

{taskiq_llm_queue-0.1.0a1 → taskiq_llm_queue-0.1.0a2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: taskiq-llm-queue
-Version: 0.1.0a1
+Version: 0.1.0a2
 Summary: An LLM request client with its own durable queue: taskiq + Amazon SQS, adaptive limits, retry and fallback.
 Project-URL: Homepage, https://github.com/techiewonk/taskiq-llm-queue
 Project-URL: Documentation, https://github.com/techiewonk/taskiq-llm-queue#readme
@@ -49,7 +49,7 @@ Description-Content-Type: text/markdown
 
 # taskiq-llm-queue
 
-> **Status:** pre-alpha (`0.1.0a1`). Built and tested: the `LLMQueue` client, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 619 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see [CHANGELOG](https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md)).
+> **Status:** pre-alpha (`0.1.0a2`). Built and tested: the `LLMQueue` client, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 761 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see [CHANGELOG](https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md)).
 
 An LLM request client with its own durable queue, built on [taskiq](https://github.com/taskiq-python/taskiq) and Amazon SQS.
 
@@ -62,6 +62,8 @@ your code ──> client / LangChain adapter ──> SQS (per model: -rt queue f
             (per-pod dispatcher)          (adaptive limiter, retry, fallback)
 ```
 
+**How it works:** [docs/ARCHITECTURE.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/ARCHITECTURE.md) walks through the components, every data flow (happy path, fallback, failover, jobs, cancellation, crash recovery), the adaptive limiter, the Valkey key schema and the full configuration reference.
+
 ## Why
 
 - **Spikes wait in a queue instead of failing.** Throttled calls aren't lost or retried in a storm.
@@ -137,7 +139,7 @@ chat.response                   # OpenAI-compatible chat completion dict
 raw = await llmq.invoke_model("titan", {"inputText": "a"}, content_type="application/json")
 raw.json(), raw.body, raw.content_type
 
-await llmq.cancel("req-1")      # not run if not started; the caller gets LLMQueueCancelled
+await llmq.cancel("req-1")      # not run if queued, stopped if running; caller gets LLMQueueCancelled
 ```
 
 - **Timing:** `timeout_secs` (relative) or `deadline_at` (absolute epoch seconds) on every call.
@@ -161,7 +163,7 @@ await llmq.cancel("req-1")      # not run if not started; the caller gets LLMQue
 
 ## Configuration
 
-`QueueConfig.from_env()` reads `LLMQ_*` variables (durations end in `_SECS`; older `_S` names still work with a one-time warning). Notable settings:
+`QueueConfig.from_env()` reads `LLMQ_*` variables (durations end in `_SECS`; older `_S` spellings are not read). Notable settings:
 
 | Setting | Default | Note |
 |---|---|---|
@@ -173,12 +175,100 @@ await llmq.cancel("req-1")      # not run if not started; the caller gets LLMQue
 | `attempt_timeout_secs` | 180 | One provider attempt; a hung target moves to the next fallback step |
 | `provider_read_timeout_secs` / `provider_connect_timeout_secs` | 300 / 10 | Bedrock botocore and OpenAI httpx timeouts |
 | `sqs_send_timeout_secs` | 5 | SQS counts as down only when no send completed for this long; then the call runs direct |
+| `sqs_timeouts_to_failover` | 3 | Send timeouts within 60 s before the model's later calls skip SQS for `sqs_failover_secs` (a refused send: at once) |
+| `sqs_batch_wait_secs` | 0.02 | bg and bulk sends within this window share one `SendMessageBatch` (0 = off); rt never batches |
+| `s3_offload_bucket` (`LLMQ_S3_OFFLOAD_BUCKET`) | none | Messages over `inline_max_bytes` (200 KB) carry their params in S3 under `s3_offload_prefix` instead of Valkey; add a lifecycle rule to expire leftovers |
 | `throttle_retry_attempts` | 6 | Tries per fallback step for throttles (backoff holds no slot) |
+| `retry_base_secs` / `retry_multiplier` / `retry_spread` / `retry_max_secs` | 1 / 4 / 2.5 / 30 | Retry n sleeps a random time in [base·4ⁿ, base·4ⁿ·2.5]: **1–2.5 s, 4–10 s, 16–30 s**, then 30 s; see [Retry backoff](#retry-backoff) |
+| `throttle_retry_max_secs` | 10 | Cap of one sleep after a throttle, so 6 throttle tries fit an rt deadline |
+| `rt_capacity_wait_secs` | 30 | How long a received rt call keeps looking for a free slot before `no_capacity` (never past its deadline) |
 | `valkey_wait_failover_secs` | 10 | Result reads failing this long make waiting callers run direct |
 | `redis_max_connections` | 32 | Raised automatically to worker slots + 16 |
+| `worker_slots` (`LLMQ_WORKER_SLOTS=gpt120b:50,jobs:4`) | 50 per model alias (4 for `jobs`) | Tasks one pod runs at once for that alias's queues |
+| `ModelConfig.limiter_initial` (`LLMQ_AC_INITIAL='{"gpt120b": 40}'`) | 20 | Start of the adaptive limit before anything was learned |
+| `ModelConfig.max_concurrency` (`LLMQ_AC_MAX_BY_MODEL='{"gpt120b": 30}'`) | `None` (adaptive up to `LLMQ_AC_MAX`) | Hard cap on calls in flight across all pods, per alias **and target**; see below |
+| `limiter.max_limit` (`LLMQ_AC_MAX`) | 2000 | Global ceiling of every adaptive limit |
+| `limiter.bg_share` (`LLMQ_AC_BG_SHARE`) | 0.8 | Fraction of a limit `bg`/`bulk` may use; `rt` may use all of it |
+| `settings_default` (`LLMQ_SETTINGS`, JSON) | `{}` | Per-provider and per-model values of the settings below; see [Settings per provider and model](#settings-per-provider-and-model) |
 
 Redis keys are cluster-safe (hash-tagged per request and per model+target), so cluster-mode Valkey (e.g. ElastiCache Serverless) works.
 
+### Settings per provider and model
+
+A provider (`bedrock`, `openai`, or a custom one) serves several models. These settings can be given once for a provider, overridden for one model, and changed live on every pod without a restart:
+
+`max_concurrency`, `limiter_initial`, `bg_share`, `timeout_secs.<lane>`, `embed_timeout_secs.<lane>`, `attempt_timeout_secs`, `throttle_retry_attempts`, `retry_base_secs`, `retry_multiplier`, `retry_spread`, `retry_max_secs`, `throttle_retry_max_secs`, `breaker_threshold`, `breaker_cooldown_secs`, `lane_max_concurrency.<lane>`, the rate budget `rpm`, `tpm`, `rate_window`, `rate_burst`, `rt_rate_share` ([Rate limits](#rate-limits-rpm--tpm)), and for a provider only `shared_max_concurrency`, `shared_rpm`, `shared_tpm`.
+
+Each one resolves through six layers; the highest that has a value wins:
+
+| # | Layer | Where it comes from |
+|---|---|---|
+| 6 | Live model | Valkey, `llm-queue settings set --model M ...` |
+| 5 | Live provider | Valkey, `llm-queue settings set --provider P ...` |
+| 4 | Env model | `LLMQ_SETTINGS` `models`, plus `ModelConfig.max_concurrency` / `limiter_initial` and `LLMQ_AC_MAX_BY_MODEL` / `LLMQ_AC_INITIAL` |
+| 3 | Env provider | `LLMQ_SETTINGS` `providers` |
+| 2 | Env global | `LLMQ_ATTEMPT_TIMEOUT_SECS`, `LLMQ_TIMEOUT_RT_SECS`, `LLMQ_RETRY_*`, `LLMQ_CB_*`, `LLMQ_AC_BG_SHARE`, ... (code, `.env` or Secrets Manager) |
+| 1 | Built-in default | The defaults in the tables above |
+
+```bash
+# .env: provider defaults and one model override (layers 3-4), validated at boot
+LLMQ_SETTINGS='{"providers": {"openai": {"shared_max_concurrency": 40, "attempt_timeout_secs": 120}},
+                "models": {"gpt120b": {"max_concurrency": 30, "timeout_secs": {"rt": 60}}}}'
+
+# live, reaches every pod within LLMQ_CFG_RELOAD_SECS (30 s); every change is audited
+llm-queue settings set --model gpt120b max_concurrency=20 timeout_secs.rt=45
+llm-queue settings set --provider bedrock attempt_timeout_secs=90 retry_multiplier=3
+llm-queue settings get --model gpt120b      # every setting with its value and source
+llm-queue settings unset --model gpt120b max_concurrency
+llm-queue settings reset                    # drop all live overrides
+```
+
+`settings get` shows each value's source (`live:model`, `live:provider`, `env:model`, `env:provider`, `env`, `default`), so a value that differs from your `.env` is easy to explain. A live value wins over `.env` until it is unset. An unknown alias or provider, an unknown setting or a bad value is refused before anything is written. If the stored value is unreadable or Valkey is down, pods keep the last good one.
+
+**Provider-wide cap.** `shared_max_concurrency` on a provider bounds the calls in flight across *all* its models and pods, e.g. one vLLM server hosting three models. Each call then holds two slots, its model's and the provider's. Measured (scenario S55): with a shared cap of 8 on `bedrock` and gpt20b and gpt120b allowed 20 each, 600 calls peaked at exactly 8 at the provider, all OK, none duplicated.
+
+**Live bucket change.** Measured (scenario S54): gpt20b ran at `max_concurrency` 20 on 3 workers; `settings set --model gpt20b max_concurrency=5` mid-run brought the provider peak from 20 to 5 after one reload, with 900/900 calls OK.
+
+**Models with a single target.** A model with only one target (no APAC profile, no `chain` fallback) has nowhere to go when it is throttled: after `throttle_retry_attempts` (6) throttles in a row the call fails. Under a 30 % throttle storm that is 0.3⁶ ≈ 0.07 % of calls (S35b: 4 of 3,020 gpt20b rt calls). Give such a model a fallback target or model, or raise its `throttle_retry_attempts` (10 tries: 0.3¹⁰ ≈ 0.0006 %).
+
+**Errors carry meta.** A failed call raises `ProviderError` / `LLMQueueTimeout` with `.meta` (attempts, queue wait, last target tried), the same fields a successful result has.
+
+Not live (needs a restart): connection settings, the model list (targets, queues, provider), credentials, provider HTTP timeouts, `worker_slots` and `retry_attempts`.
+
+### Retry backoff
+
+Each fallback step retries a failed attempt `retry_attempts` times (3), or `throttle_retry_attempts` times (6) once the provider throttled it. Retry *n* (0-based) sleeps a random time in `[base · multiplierⁿ, base · multiplierⁿ · spread]`, capped at `retry_max_secs`:
+
+| Retry | Defaults (base 1, multiplier 4, spread 2.5, cap 30) | After a throttle (cap 10) |
+|---|---|---|
+| 1st | 1–2.5 s | 1–2.5 s |
+| 2nd | 4–10 s | 4–10 s |
+| 3rd | 16–30 s | 10 s |
+| 4th and later | 30 s | 10 s |
+
+A provider `Retry-After` larger than the drawn delay wins, up to `retry_max_secs`. A sleep that would pass the call's deadline is skipped and the next fallback step is tried. No limiter slot is held while sleeping. All five knobs are `LLMQ_*` variables and can be set per provider or model like the settings above.
+
+A sleeping call frees its limiter slot but keeps its worker task (one of `worker_slots`). Under a long throttle storm a longer curve therefore lowers per-pod throughput; if queued callers start to time out, raise `worker_slots` or lower `retry_multiplier` / `throttle_retry_max_secs` for that provider or model. The scenario harness scales the whole curve by 0.2 (base 0.2 s, caps 6 s and 2 s) to keep runs short.
+
+### Setting names
+
+Every duration ends in `_SECS` (milliseconds in `_MS`). Old spellings from pre-release builds, such as `LLMQ_CONSUMER_DRAIN_S`, `LLMQ_RETRY_BASE_S`, `LLMQ_SQS_WAIT_TIME_S` or `LLMQ_TIMEOUT_RT`, are **not read**: a variable with one of them is silently ignored and the default applies. Rename them in your `.env`, Helm values and Secrets Manager entries:
+
+| Old spelling (ignored) | Use |
+|---|---|
+| `LLMQ_CFG_RELOAD_S` | `LLMQ_CFG_RELOAD_SECS` |
+| `LLMQ_CONSUMER_DRAIN_S` | `LLMQ_CONSUMER_DRAIN_SECS` |
+| `LLMQ_RETRY_BASE_S` | `LLMQ_RETRY_BASE_SECS` |
+| `LLMQ_SQS_WAIT_TIME_S` | `LLMQ_SQS_WAIT_TIME_SECS` |
+| `LLMQ_SQS_FAILOVER_S` | `LLMQ_SQS_FAILOVER_SECS` |
+| `LLMQ_STATE_TTL_S` | `LLMQ_STATE_TTL_SECS` |
+| `LLMQ_AC_PAUSE_S` | `LLMQ_AC_PAUSE_SECS` |
+| `LLMQ_AC_LEASE_TTL_S` | `LLMQ_AC_LEASE_TTL_SECS` |
+| `LLMQ_CB_WINDOW_S` | `LLMQ_CB_WINDOW_SECS` |
+| `LLMQ_CB_COOLDOWN_S` | `LLMQ_CB_COOLDOWN_SECS` |
+| `LLMQ_TIMEOUT_RT` / `_BG` / `_BULK` | `LLMQ_TIMEOUT_RT_SECS` / `_BG_SECS` / `_BULK_SECS` |
+| `LLMQ_TIMEOUT_EMBED_RT` / `_BG` / `_BULK` | `LLMQ_TIMEOUT_EMBED_RT_SECS` / `_BG_SECS` / `_BULK_SECS` |
+
 ## Providers
 
 ```python
@@ -200,6 +290,59 @@ A custom provider implements `async call(op, params, target) -> dict` and `class
 
 A queued `bg`/`bulk` call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.
 
+### Concurrency: queue depth vs bucket size
+
+The queue depth is unbounded: SQS holds every request that is not running. How many run at once is set by three knobs:
+
+- **`worker_slots`** (per pod, per model alias, default 50): tasks one pod runs at once. Total pod capacity is `pods × worker_slots`.
+- **The limiter** (cluster-wide, per model alias and target, in Valkey): the number of provider calls in flight across all pods. It starts at `limiter_initial` (20), grows on busy successes, is cut on clustered throttles, and never exceeds `LLMQ_AC_MAX` (2000).
+- **`max_concurrency`** (cluster-wide, per model alias **and target**, default unset): a hard cap on that limit. With `max_concurrency=10`, at most 10 calls of that alias run against each of its targets at once, whatever the pod count. A model with two targets can therefore have 2 × 10 in flight. The limiter still adapts below the cap. Set `limiter_initial == max_concurrency` for a fixed bucket that only shrinks under real throttling; a `limiter_initial` above the cap is clamped to it (logged once as `limiter_initial_clamped`). Lowering the cap also clamps a limit learned earlier, on its next use.
+
+`bg` and `bulk` may use `bg_share` (80 %) of the limit, so with a cap of 10, background work runs at most 8 at once and keeps 2 slots for `rt`. Set `LLMQ_AC_BG_SHARE=1` to give all lanes the whole bucket.
+
+**A bucket per lane.** `lane_max_concurrency.<lane>` gives one lane of a model its own cap inside the model's bucket, per target and across all pods, e.g. bulk embeddings at most 5 while bg may use 15 of 20:
+
+```bash
+LLMQ_SETTINGS='{"models": {"titan": {"max_concurrency": 20, "lane_max_concurrency": {"bulk": 5, "bg": 15}}}}'
+llm-queue settings set --model titan lane_max_concurrency.bulk=3      # live, every pod
+```
+
+A call in a capped lane holds a slot of the model's bucket and one of the lane's (`{m:<alias>#<lane>:<target>}`, a fixed bucket that never learns). A full lane leaves its own backlog in SQS while the other lanes keep receiving. Measured (scenario S57): gpt20b bucket 20, bulk 3, bg 5, 300 bulk + 300 bg calls at once: bulk peaked at exactly 3, bg at exactly 5, an rt call in the middle started in 0.23 s, all 600 OK.
+
+A pod receives at most min(free worker slots, limiter headroom − its own calls already waiting for a slot), so a full bucket leaves the backlog in SQS rather than in pod memory. The rest wait in SQS until their lane deadline (`timeout_secs`: rt 90 s, bg 300 s, bulk 900 s). `bg`/`bulk` calls that do get received but find no slot within 15 s go back to SQS with a delay. An `rt` call that is received keeps walking its fallback chain for a free slot (2 s per step) for up to `rt_capacity_wait_secs` (30 s, never past its deadline), then fails with `no_capacity`. Size the bucket so that `cap × calls per second per slot × deadline` covers the backlog you expect. `max_concurrency` and `bg_share` can also be set per provider and changed live ([Settings per provider and model](#settings-per-provider-and-model)), and a provider can have one bucket shared by all its models. While Valkey is down, each pod runs under a local cap of `last limit / live pods`, never above `max_concurrency`. This is best effort: with the pod count unknown, each pod may run up to the cap.
+
+Measured (scenario S53, fakellm at 200 ms p50, 3 workers × 50 slots, `max_concurrency=10`, `bg_share=1`): 2,000 bg calls queued (2,000 visible in SQS at peak), peak provider concurrency exactly 10 (from the call log and fakellm's own counter), limiter never above 10, 2,000/2,000 done, 0 lost, 0 duplicates, 61 s wall time. SQS in-flight peaked at 40, which is 10 running plus finished messages waiting for the 0.5 s batched delete.
+
+### Rate limits (RPM / TPM)
+
+Concurrency caps how many calls run at once; a provider quota is usually per minute. Set `rpm` (calls that may **start** per minute) and / or `tpm` (tokens per minute) per model, and `shared_rpm` / `shared_tpm` per provider for an account quota. Each budget is per model **target** (a model with a primary and an APAC profile has one budget on each), shared by every pod in Valkey. Unset means no rate limit.
+
+```bash
+LLMQ_SETTINGS='{"models": {"gpt120b": {"rpm": 500, "tpm": 200000}},
+                "providers": {"bedrock": {"shared_rpm": 800}}}'
+llm-queue settings set --model gpt120b rpm=300 rate_window=fixed    # live, every pod
+llm-queue status --model gpt120b       # waiting / working / started this minute / left
+```
+
+A call starts only with a rate token **and** a concurrency slot. How the budget behaves:
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `rate_window` | `smooth` | `smooth`: a token bucket, `rpm / 60` per second with `rate_burst` back to back. `fixed`: the whole `rpm` from each calendar minute (500 at 10:00:00, the next 500 from 10:01:00) |
+| `rate_burst` | 10 % of `rpm` | Calls that may start back to back in `smooth` mode; TPM uses the same fraction |
+| `rt_rate_share` | 0.2 | Part of each budget only `rt` may use: `bg`/`bulk` get at most 80 % of the rate, so a backlog never starves live calls |
+
+When a budget is used up:
+
+- **`bg` / `bulk`** wait for **their own target's** budget: the message goes back to SQS with a delay until the next token, and that pod stops receiving the model's `bg`/`bulk` work until then (the backlog stays in SQS). They never spend a fallback target's budget.
+- **`rt`** moves to the next target in its chain (each target has its own budget), then keeps trying for `rt_capacity_wait_secs` (30 s, never past its deadline), then fails with `ProviderError("rate_limited ...")`, `retryable=True`, `retry_after_secs` = time to the next token (HTTP 429 through `http_transport`).
+- **TPM** reserves `input characters / 4 + max output tokens` (1,000 when the request sets none) before the call and is corrected with the response's `usage` after it, so a generous `max_tokens` does not hold the budget.
+- **Valkey down**: each pod applies `limit / live pods` in process (best effort; the total is right while the pod count is right).
+
+`llm-queue status` prints per model: SQS `waiting` / `delayed` / `received` per lane queue, and per target `working` (calls in flight, all pods), `limit`, `started` this minute, `rpm_left`, `tokens`, `tpm_left`, plus the provider's shared budget. Metrics: `llmq_rate_used` and `llmq_rate_remaining` per model target; waiting messages are SQS's own `ApproximateNumberOfMessagesVisible` in CloudWatch.
+
+The adaptive limiter stays on underneath: if the real quota is lower than the `rpm` you set, throttles still cut concurrency.
+
 ## Results
 
 Results are stored as JSON (never pickle) for 5 minutes and read once (`GETDEL`). Each pod runs one result dispatcher: a pub/sub wake reads only the announced results, and a periodic batched sweep catches the rest, so 10K waiting calls don't mean 10K polling loops.
@@ -244,6 +387,9 @@ uv run llm-queue routing reset
 uv run llm-queue limiter show --model gpt120b                    # learned limit, in-flight, pause
 uv run llm-queue limiter reset --model gpt120b [--target apac]   # forget the learned limit
 uv run llm-queue breaker show | reset
+uv run llm-queue settings get --model gpt120b                   # effective settings + source
+uv run llm-queue settings set --model gpt120b max_concurrency=20 # live bucket size, all pods
+uv run llm-queue settings set --provider openai shared_max_concurrency=40
 uv run llm-queue queues                                          # messages per queue and DLQ
 uv run llm-queue job send ingest --payload-file payload.json --queue jobs-ingest
 ```
@@ -271,6 +417,8 @@ logging.getLogger("taskiq_llm_queue").setLevel(logging.INFO)   # stdlib apps: th
 # structlog apps: use structlog.stdlib.ProcessorFormatter on your handler to render them.
 ```
 
+To trace one call on a worker, set `taskiq_llm_queue.core` to DEBUG. Each call then logs `call_started` (with `queue_wait_ms`), `slot_acquired` / `slot_unavailable` (`waited_ms`), `attempt_done` (target, outcome, `provider_ms`), `backoff` (`sleep_secs`), `rt_capacity_wait`, `call_requeued` and `call_done` (ok or error code, attempts, `total_ms`), all with the `request_id`. Below DEBUG these cost one level check per call.
+
 The `llm-queue` CLI logs warnings and errors to stderr, so its stdout is always plain JSON.
 
 ## Security and data