taskiq-llm-queue 0.1.0a2__tar.gz → 0.2.0__tar.gz

{taskiq_llm_queue-0.1.0a2 → taskiq_llm_queue-0.2.0}/CHANGELOG.md

@@ -6,6 +6,26 @@ a minor release may break the public API, with a deprecation announced one minor
 
 ## [Unreleased]
 
+## [0.2.0] - 2026-11-02
+
+### Changed
+- Development status Alpha. `llm-queue queues` takes subcommands (`list` default, `create`, `redrive`, `purge`); `queues`, `status` and `job send` use the configured queue backend. Job messages carry their name in the `llmq_job` label (`send_job` and `llm-queue job send`); older messages still match by args.
+- Scenario harness: `SCN_QUEUE_BACKEND=valkey` runs every scenario on Valkey Streams against `valkey-scn-persist` (AOF `always`, `noeviction`); S20 and S33 are SQS-only, S33v Valkey-only.
+
+### Verified
+- 869 tests (unit, integration and every scenario on SQS): 866 pass, 2 skip by design; S27 at its full 450K-call default exceeds what one laptop load generator offers (196K in 30 min) and passes at the gate load (6K/min × 3 min, rt queue wait p95 1.1 s). All scenarios pass on Valkey; samples 01-17 exit 0 on both backends; real Bedrock: S01, S03, S04, S05, S16, S17, S19, S22, S26, S28, S29, S40, S41 pass.
+
+### Fixed
+- Jobs whose payload was offloaded to S3 were never run: the job filter read the job name from the args, which offload empties, so the message was released every 30 s until the DLQ. Jobs now carry the name in the `llmq_job` label (older messages still match by args). Scenario S24.
+- The first sync call stalled the caller's event loop 0.2-0.6 s: every new `ProviderRegistry` re-scanned all installed packages' entry points, holding the GIL. The scan now runs once per process. The sync runner's session also reuses the main session's parsed botocore data. A first burst of sync calls on real AWS still stalls the loop 0.1-1.8 s once per pod while clients and HTTPS connections are built (S22 reports it as `cold_burst_lag_ms`); warm bursts stay under 60 ms.
+- Tracing: `QueuedChatBedrockConverse` and `QueuedChatOpenAI` now default `name` to the parent class, so Langfuse / LangSmith generations keep the name `ChatBedrockConverse` / `ChatOpenAI`. After a fallback, `llm_output["model_name"]` is the Bedrock id that served the call, so tracers show and price the right model.
+
+### Added
+- Valkey Streams queue backend: `queue_backend="valkey"` (`LLMQ_QUEUE_BACKEND`), optional `queue_redis_url` (`LLMQ_QUEUE_REDIS_URL`). `ValkeyQueueClient` (`broker/valkey_queue.py`) serves the SQS calls `HybridSQSBroker` makes (visibility, receipt-handle generations, delay, DLQ after `maxReceiveCount`, long polls woken by pub/sub), so the broker and everything above it are unchanged; `HybridSQSBroker(client_factory=...)` is the hook. CLI `queues create | list | redrive | purge`; gauges `llmq_queue_depth{queue,kind}` and `llmq_queue_oldest_seconds{queue}`. FIFO queues are refused. A client re-creates a queue it knew when its keys vanish (restart without persistence); release deadlines and idle re-checks are jittered so the releasing pod does not always win the message back; with the queues on the state Valkey, Valkey recovering also ends the queue failover windows (`FailoverSettings.queues_on_state_valkey`). All scenarios pass on Valkey except the SQS-only S20 / S33 (skipped; S33v measures Valkey). Plan and results: docs/VALKEY_QUEUE_PLAN.md.
+- Scenarios S04 (LangChain / LangGraph drop-in: `ainvoke`, `abatch`, lanes from `request_context`, `bind_tools`, structured output, a LangGraph tool loop, sync `invoke`, queued embeddings), S24 (S3 offload through an in-process moto S3) and S33 (SQS call counts through a counting proxy: bg sends and acks batched, rt single, idle long polls bounded). They replace the three skipped placeholders.
+- `SCN_AWS_LOW_VOLUME=1` lets `SCN_PROVIDER=aws` run scenarios marked `low-volume`. S10, S11, S18, S23, S37, S47 and S49 are now marked not AWS-safe, because they inject faults or read the fakellm call log.
+- Sample `17_langgraph_rag_agent.py`: an app-style graph with queued embeddings for retrieval, a structured-output router, a tool loop with `fallback_policy="chain"`, a bg-lane summary node, `stream_mode="updates"` and a sync `.invoke()`.
+
 ## [0.1.0a2] - 2026-10-30
 
 ### Added

{taskiq_llm_queue-0.1.0a2 → taskiq_llm_queue-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: taskiq-llm-queue
-Version: 0.1.0a2
+Version: 0.2.0
 Summary: An LLM request client with its own durable queue: taskiq + Amazon SQS, adaptive limits, retry and fallback.
 Project-URL: Homepage, https://github.com/techiewonk/taskiq-llm-queue
 Project-URL: Documentation, https://github.com/techiewonk/taskiq-llm-queue#readme
@@ -12,7 +12,7 @@ License-Expression: MIT
 License-File: LICENSE
 License-File: NOTICE
 Keywords: bedrock,langchain,langgraph,llm,openai,queue,rate-limit,sqs,taskiq
-Classifier: Development Status :: 2 - Pre-Alpha
+Classifier: Development Status :: 3 - Alpha
 Classifier: Framework :: AsyncIO
 Classifier: Intended Audience :: Developers
 Classifier: Operating System :: OS Independent
@@ -49,7 +49,7 @@ Description-Content-Type: text/markdown
 
 # taskiq-llm-queue
 
-> **Status:** pre-alpha (`0.1.0a2`). Built and tested: the `LLMQueue` client, the SQS broker, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 761 tests on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see [CHANGELOG](https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md)).
+> **Status:** alpha (`0.2.0`). Built and tested: the `LLMQueue` client, the queue broker on Amazon SQS or Valkey Streams, adaptive limiter, claims, circuit breaker, retry and fallback, JSON and S3 result backends, Bedrock and OpenAI-compatible providers, LangChain/LangGraph adapters, canary routing, failover and the CLI. 869 tests (unit, integration and the scenario harness) on Valkey 8.1, ElasticMQ 1.7.1 and a fake LLM server; CI on Python 3.11–3.14. Breaking changes are still possible before 0.1 (see [CHANGELOG](https://github.com/techiewonk/taskiq-llm-queue/blob/main/CHANGELOG.md)).
 
 An LLM request client with its own durable queue, built on [taskiq](https://github.com/taskiq-python/taskiq) and Amazon SQS.
 
@@ -168,6 +168,8 @@ await llmq.cancel("req-1")      # not run if queued, stopped if running; caller
 | Setting | Default | Note |
 |---|---|---|
 | `aws_region` | `None` | Left to the AWS credential chain / IRSA |
+| `queue_backend` (`LLMQ_QUEUE_BACKEND`) | `sqs` | `valkey` puts every queue on Valkey Streams; see [Queue backend](#queue-backend-sqs-or-valkey-streams) |
+| `queue_redis_url` (`LLMQ_QUEUE_REDIS_URL`) | `None` | With `valkey`: a separate Valkey for the queues (else the state Valkey) |
 | `redis_url` or `redis_host`/`port`/`username`/`password`/`ssl` | localhost | `redis_password` is a `SecretStr` |
 | `redis_socket_timeout_secs` / `connect_timeout` / `health_check_interval` | 5 / 2 / 30 | Connection errors retried 3× with backoff |
 | `id_hash_salt` (`LLMQ_ID_HASH_SALT`) | library constant | **Set your own in production**; keys the BLAKE2b hash of session/user ids |
@@ -290,6 +292,22 @@ A custom provider implements `async call(op, params, target) -> dict` and `class
 
 A queued `bg`/`bulk` call waits at most 15 s for a slot, then goes back to the queue with a delay instead of holding a worker.
 
+### Queue backend: SQS or Valkey Streams
+
+Queues live on Amazon SQS by default. `LLMQ_QUEUE_BACKEND=valkey` puts them on Valkey Streams instead, with the same queue names and the same delivery rules: visibility timeout with heartbeat, release with a delay, redelivery after a pod dies, a DLQ after `maxReceiveCount`, delays, expiry, strict lane priority and batched send / ack. The broker is unchanged: `ValkeyQueueClient` answers the SQS calls it makes ([docs/VALKEY_QUEUE_PLAN.md](docs/VALKEY_QUEUE_PLAN.md)).
+
+| | SQS | Valkey Streams |
+|---|---|---|
+| Create queues | your infrastructure code | `llm-queue queues create [--visibility-secs 60 --max-receives 10 --retention-secs 345600]` |
+| Where | AWS, multi-AZ, managed | `LLMQ_QUEUE_REDIS_URL`, or the state Valkey when unset |
+| Valkey down | calls run direct; queued work stays in SQS | queue, limiter and results are down together; calls run direct |
+| Backlog | cheap, days | Valkey memory: size it (bodies ≤ 200 KB; larger params by reference) |
+| Metrics | CloudWatch | `llmq_queue_depth{queue,kind}`, `llmq_queue_oldest_seconds{queue}` from the pods |
+| DLQ redrive | console / `StartMessageMoveTask` | `llm-queue queues redrive --queue <name>` |
+| FIFO queues | yes | no (refused at boot) |
+
+Use a dedicated Valkey with AOF (`everysec`) and Multi-AZ for queues in production: a failover can lose or repeat the last messages (handlers are idempotent, a loss shows as a caller timeout).
+
 ### Concurrency: queue depth vs bucket size
 
 The queue depth is unbounded: SQS holds every request that is not running. How many run at once is set by three knobs:
@@ -437,7 +455,7 @@ make down
 # Windows without make:  uv run python tools/llmq_dev.py up | test | down
 ```
 
-Runnable examples are in [`samples/`](https://github.com/techiewonk/taskiq-llm-queue/tree/main/samples): quickstart, chat, embeddings, lanes, fallback on throttle, send-only and worker pods, LangChain, LangGraph, jobs, cancel and timeouts, sync calls, CLI, and `13_research_assistant_flow.py`, which replays a real app's flows locally or against real Bedrock (`--target aws`, settings from `.env` or AWS Secrets Manager). The stack, capability checks and test layers are described in [docs/TESTING.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/TESTING.md); delivery-safety rules in [docs/DESIGN.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/DESIGN.md).
+Runnable examples are in [`samples/`](https://github.com/techiewonk/taskiq-llm-queue/tree/main/samples): quickstart, chat, embeddings, lanes, fallback on throttle, send-only and worker pods, LangChain, LangGraph, jobs, cancel and timeouts, sync calls, CLI, `13_research_assistant_flow.py` (a real app's flows locally or against real Bedrock: `--target aws`, settings from `.env` or AWS Secrets Manager), settings and buckets (14), retries and errors (15), rate limits and lane caps (16) and an app-style LangChain + LangGraph graph (17). Every sample reads `LLMQ_*` overrides, so `LLMQ_QUEUE_BACKEND=valkey` runs all of them on Valkey Streams ([samples/README.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/samples/README.md) "Settings"). The stack, capability checks and test layers are described in [docs/TESTING.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/TESTING.md); delivery-safety rules in [docs/DESIGN.md](https://github.com/techiewonk/taskiq-llm-queue/blob/main/docs/DESIGN.md).
 
 ## Credits