@maestro-js/agent-skills 1.0.0-alpha.18 → 1.0.0-alpha.2

package/{skills → dist}/choosing-an-npm-package/SKILL.md

@@ -46,20 +46,19 @@ Before installing any dependency, check this list for the user's preferred packa
 - **@maestro-js/form** — Headless React form primitives with client and server validation, hidden inputs for custom components, controlled/uncontrolled state management, and Standard Schema (Zod) FormData parsing with automatic type coercion. Use when building custom form components that need native constraint validation, server error integration, file upload handling, or type-safe FormData parsing. Provides HeadlessForm root component, useValidation/useWarningState/useControlledState/useReset hooks, HiddenInput/HiddenFileInput/HiddenSelect components, parseFormData/parseFormDataClientSide utilities, and composeValidators.
 - **@maestro-js/hash** — Pluggable hashing and verification with driver-based backends for @maestro-js/hash. Use when working with @maestro-js/hash, hashing values, verifying hashes, detecting when a rehash is needed, or rotating signing keys. Built-in drivers: SHA256 (HMAC-SHA256 with key rotation and timing-safe comparison).
 - **@maestro-js/helpers** — Standalone utility functions for async control flow and file handling in the @maestro-js/helpers package. Use when working with @maestro-js/helpers. Key capabilities: retry with configurable attempts/delay/backoff, sleep, withMaxConcurrency for limiting parallel executions, withRateLimit with fixed or sliding window, raceWithFallback for promise racing with default values, AsyncIterableCollection for lazy async iteration (map, filter, reduce, forEach, toArray), sanitizeFilename for safe cross-platform filenames, and withRetry for wrapping functions with retry behavior.
-- **@maestro-js/init** — Project scaffolding for the Maestro framework. Generates a complete monorepo with config, schemas, services, web app, queue processor, and infrastructure.
-- **@maestro-js/iso-zod** — Zod custom schemas for ISO 8601 temporal types (Instant, DateTime, ZonedDateTime, Date, Time, YearMonth, MonthDay, Duration) backed by iso-fns validation. Use when defining Zod schemas that include ISO temporal fields. Provides isoZ factory functions that return typed Zod schemas and an isoZRegistry for introspecting registered ISO type metadata.
 - **@maestro-js/log** — Structured logging with pluggable transports and channel-based routing for the Maestro framework. Use when working with @maestro-js/log, adding or modifying log transports, configuring log channels, filtering log messages by level or channel, integrating logging into other packages, or writing tests that capture log output. Key capabilities: create loggers with pluggable transports (console, file, Sentry, stub), route messages through named channels, filter by level and channel, override Node.js console, custom formatters, and the silence transport for suppressing output.
 - **@maestro-js/mail** — Driver-based email sending with queued delivery, scheduled sending, and mailable templates for the @maestro-js/mail package. Use when working with @maestro-js/mail. Covers creating mail services with SendGrid/SES/MailTrap/Stub drivers, sending emails immediately or via queue, scheduling future delivery, building reusable mailable templates, processing webhook events, generating ICS calendar attachments, and testing with the stub driver.
 - **@maestro-js/metrics** — Use when working with @maestro-js/metrics. Unified metrics collection piped through the Log transport system. Covers four metric types (counters, distributions, sets, gauges), provider setup, logger integration, Sentry transport, and custom transport authoring. Trigger on metrics emission, observability instrumentation, Sentry metrics integration, or application telemetry tasks.
 - **@maestro-js/password** — 
-- **@maestro-js/permissions** — Subject-action authorization with type-safe action maps and profile-based rule resolution. Use when you need role-based access control, permission checks (can/cannot), or authorization gates that throw on denial. Provides a static driver for declarative allow/forbid rule definitions. Built on CASL.
-- **@maestro-js/queue** — Database-backed job queue with middleware, configurable retries, and backoff strategies. Use when dispatching background jobs, processing queues with concurrency control, scheduling delayed work, handling retries and permanent failures, or integrating with mail/recurring-jobs. Provides db (MySQL) and local (in-memory) drivers.
+- **@maestro-js/queue** — Database-backed job queue for deferring work beyond the current request with scheduled execution, concurrency control, and distributed tracing. Use when working with @maestro-js/queue, including dispatching jobs, processing queues, configuring the db driver, writing queue consumers, handling retries/stuck jobs, integrating with mail or recurring-jobs, and understanding the queue database schema.
+- **@maestro-js/queue2** — Database-backed job queue for deferring work beyond the current request with scheduled execution, concurrency control, and distributed tracing. Use when working with @maestro-js/queue, including dispatching jobs, processing queues, configuring the db driver, writing queue consumers, handling retries/stuck jobs, integrating with mail or recurring-jobs, and understanding the queue database schema.
 - **@maestro-js/rate-limiting** — Per-key rate limiting with configurable decay windows. Use when enforcing attempt limits on login, API endpoints, or any action that needs throttling. Provides increment/check/clear workflow, guarded callback execution via attempt(), and ISO 8601 duration reporting. Requires a cache driver (e.g. from @maestro-js/cache).
 - **@maestro-js/react-router-file-routes** — 
-- **@maestro-js/recurring-jobs** — Use when working with @maestro-js/recurring-jobs. Cron-based recurring job scheduling with distributed lock-based deduplication. Use for cron scheduling, periodic background tasks, and recurring jobs that must run exactly once across processes. Drivers: Db (MySQL), Cache (Redis via @maestro-js/cache), Local (in-memory). Follows the Provider pattern with ServiceRegistry. Key capabilities include cron expression parsing, 60-second polling loop, per-job distributed locking, optional middleware wrapping, eager cron validation, structured lifecycle log events (jobDue, jobLockAcquired, jobFailed, jobProcessed), and graceful shutdown via abort/stopAllWork.
+- **@maestro-js/recurring-jobs** — Use when working with @maestro-js/recurring-jobs. Cron-based job scheduling backed by the Queue package. Covers creating recurring jobs, managing schedules, start/stop processing, rescheduling, listing jobs, and inspecting upcoming runs. Follows the Provider pattern with ServiceRegistry. Key capabilities include cron expression parsing, deduplication across process restarts, per-job dedicated queues, structured log events, and paginated execution history.
 - **@maestro-js/session** — 
 - **@maestro-js/signed-url** — URL signing and verification with pluggable hash drivers for @maestro-js/signed-url. Use when generating tamper-proof signed URLs with expiration (e.g. file download links, email verification) or validating signed URL parameters. Accepts any hash driver compatible with @maestro-js/hash.
 - **@maestro-js/sql** — Composable SQL query builders for dynamic WHERE clauses and Common Table Expressions (CTEs). Use when working with @maestro-js/sql, building dynamic SQL filters, constructing parameterized WHERE clauses, wrapping Db services with CTEs, or generating safe parameterized SQL fragments. Key capabilities: WhereClauseBuilder for incremental AND/OR conditions with nested sub-groups, CommonTableExpressionBuilder for wrapping a Db service so all queries auto-prepend WITH clauses, cloning builders, recursive CTEs, and dynamic parameter resolution.
+- **@maestro-js/tracing** — Distributed tracing package built on OpenTelemetry with pluggable providers (Sentry, Maestro). Use when working with @maestro-js/tracing. Key capabilities: wrap functions for automatic span tracking, create active spans with duration measurement, proxy entire objects for tracing, serialize/deserialize trace context across process boundaries (dehydrate/hydrate), add custom log transports for span events, integrate Sentry as a tracing provider.
 
 ## General Guidance
 

package/{skills → dist}/maestro-packages/SKILL.md

@@ -34,9 +34,9 @@ Each package has a linked detail page with its full API, driver options, and usa
 - **log** — Structured logging with pluggable transports and channel-based routing for the Maestro framework. Use when working with @maestro-js/log, adding or modifying log transports, configuring log channels, filtering log messages by level or channel, integrating logging into other packages, or writing tests that capture log output. Key capabilities: create loggers with pluggable transports (console, file, Sentry, stub), route messages through named channels, filter by level and channel, override Node.js console, custom formatters, and the silence transport for suppressing output. [View details](./log.md)
 - **mail** — Driver-based email sending with queued delivery, scheduled sending, and mailable templates for the @maestro-js/mail package. Use when working with @maestro-js/mail. Covers creating mail services with SendGrid/SES/MailTrap/Stub drivers, sending emails immediately or via queue, scheduling future delivery, building reusable mailable templates, processing webhook events, generating ICS calendar attachments, and testing with the stub driver. [View details](./mail.md)
 - **metrics** — Use when working with @maestro-js/metrics. Unified metrics collection piped through the Log transport system. Covers four metric types (counters, distributions, sets, gauges), provider setup, logger integration, Sentry transport, and custom transport authoring. Trigger on metrics emission, observability instrumentation, Sentry metrics integration, or application telemetry tasks. [View details](./metrics.md)
-- **permissions** — Subject-action authorization with type-safe action maps and profile-based rule resolution. Use when you need role-based access control, permission checks (can/cannot), or authorization gates that throw on denial. Provides a static driver for declarative allow/forbid rule definitions. Built on CASL. [View details](./permissions.md)
-- **queue** — Database-backed job queue with middleware, configurable retries, and backoff strategies. Use when dispatching background jobs, processing queues with concurrency control, scheduling delayed work, handling retries and permanent failures, or integrating with mail/recurring-jobs. Provides db (MySQL) and local (in-memory) drivers. [View details](./queue.md)
+- **queue** — Database-backed job queue for deferring work beyond the current request with scheduled execution, concurrency control, and distributed tracing. Use when working with @maestro-js/queue, including dispatching jobs, processing queues, configuring the db driver, writing queue consumers, handling retries/stuck jobs, integrating with mail or recurring-jobs, and understanding the queue database schema. [View details](./queue.md)
 - **rate-limiting** — Per-key rate limiting with configurable decay windows. Use when enforcing attempt limits on login, API endpoints, or any action that needs throttling. Provides increment/check/clear workflow, guarded callback execution via attempt(), and ISO 8601 duration reporting. Requires a cache driver (e.g. from @maestro-js/cache). [View details](./rate-limiting.md)
 - **react-router-file-routes** —  [View details](./react-router-file-routes.md)
-- **recurring-jobs** — Use when working with @maestro-js/recurring-jobs. Cron-based recurring job scheduling with distributed lock-based deduplication. Use for cron scheduling, periodic background tasks, and recurring jobs that must run exactly once across processes. Drivers: Db (MySQL), Cache (Redis via @maestro-js/cache), Local (in-memory). Follows the Provider pattern with ServiceRegistry. Key capabilities include cron expression parsing, 60-second polling loop, per-job distributed locking, optional middleware wrapping, eager cron validation, structured lifecycle log events (jobDue, jobLockAcquired, jobFailed, jobProcessed), and graceful shutdown via abort/stopAllWork. [View details](./recurring-jobs.md)
+- **recurring-jobs** — Use when working with @maestro-js/recurring-jobs. Cron-based job scheduling backed by the Queue package. Covers creating recurring jobs, managing schedules, start/stop processing, rescheduling, listing jobs, and inspecting upcoming runs. Follows the Provider pattern with ServiceRegistry. Key capabilities include cron expression parsing, deduplication across process restarts, per-job dedicated queues, structured log events, and paginated execution history. [View details](./recurring-jobs.md)
 - **sql** — Composable SQL query builders for dynamic WHERE clauses and Common Table Expressions (CTEs). Use when working with @maestro-js/sql, building dynamic SQL filters, constructing parameterized WHERE clauses, wrapping Db services with CTEs, or generating safe parameterized SQL fragments. Key capabilities: WhereClauseBuilder for incremental AND/OR conditions with nested sub-groups, CommonTableExpressionBuilder for wrapping a Db service so all queries auto-prepend WITH clauses, cloning builders, recursive CTEs, and dynamic parameter resolution. [View details](./sql.md)
+- **tracing** — Distributed tracing package built on OpenTelemetry with pluggable providers (Sentry, Maestro). Use when working with @maestro-js/tracing. Key capabilities: wrap functions for automatic span tracking, create active spans with duration measurement, proxy entire objects for tracing, serialize/deserialize trace context across process boundaries (dehydrate/hydrate), add custom log transports for span events, integrate Sentry as a tracing provider. [View details](./tracing.md)

package/dist/maestro-packages/queue.md

@@ -0,0 +1,284 @@
+---
+name: queue
+description: "Database-backed job queue for deferring work beyond the current request with scheduled execution, concurrency control, and distributed tracing. Use when working with @maestro-js/queue, including dispatching jobs, processing queues, configuring the db driver, writing queue consumers, handling retries/stuck jobs, integrating with mail or recurring-jobs, and understanding the queue database schema."
+---
+
+# @maestro-js/queue
+
+Database-backed job queue that stores messages in MySQL via a pluggable driver and processes them asynchronously. Follows the Provider pattern from `@maestro-js/service-registry` with a producer/consumer split supporting scheduled execution, concurrency, and distributed trace propagation.
+
+## Quick Setup
+
+```ts
+import { Queue } from '@maestro-js/queue'
+import { Db } from '@maestro-js/db'
+import { Log } from '@maestro-js/log'
+
+// Register the queue service with the db driver
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({
+      db: Db.provider('default'),
+      databaseTable: 'jobs'
+    }),
+    logger: Log
+  })
+)
+
+// Create a named queue with a work function
+const orderQueue = Queue.create({
+  name: 'fulfill-orders',
+  workFn: async (body) => {
+    await fulfillOrder(body.orderId)
+    return { fulfilled: true }
+  },
+  concurrency: 3
+})
+
+// Enqueue a message
+await orderQueue.addMessage({ orderId: 'ord_123' }, { scheduledDate: null })
+
+// Start processing
+orderQueue.startProcessing()
+```
+
+## Database Schema
+
+The db driver expects a MySQL table (name configurable via `databaseTable`). The column-to-field mapping used internally:
+
+| Column          | Type          | Description                                              |
+| --------------- | ------------- | -------------------------------------------------------- |
+| `id`            | int (PK, AI)  | Message ID                                               |
+| `batch_id`      | int / NULL    | Lock batch for concurrent dequeue                        |
+| `job_type`      | varchar       | Queue name string                                        |
+| `body`          | text (JSON)   | Serialized message body                                  |
+| `trace`         | text (JSON)   | Serialized `DehydratedTrace` for distributed tracing     |
+| `status`        | enum          | `waiting`, `processing`, `success`, `error`, `stuck`     |
+| `scheduledDate` | datetime      | Earliest time the message can be processed               |
+| `enqueuedDate`  | datetime      | When the message was inserted                            |
+| `started_ts`    | datetime      | When processing began                                    |
+| `result`        | text          | JSON result on success, error stack on failure           |
+| `recovered`     | tinyint       | 1 if the message was recovered from stuck state          |
+| `running_time`  | int / NULL    | Seconds elapsed during processing                        |
+| `unique_key`    | int / NULL    | MD5-derived dedup key (nullable)                         |
+| `priority`      | bigint / NULL | Lower value = higher priority (defaults to `Date.now()`) |
+
+The driver requires a `FORCE INDEX (locking_update)` index on `(batch_id, status, job_type, scheduledDate)` for efficient locking queries.
+
+## API Reference
+
+### Provider Setup
+
+```ts
+Queue.Provider.create(config: {
+  driver: Queue.Driver
+  logger: Log.Logger<[Queue.LogMessage]>
+  tracing?: Queue.Tracing  // optional distributed tracing adapter
+}): Queue.QueueService
+```
+
+```ts
+Queue.Provider.register(name: Queue.Provider.Key, service: Queue.QueueService): void
+```
+
+### Drivers
+
+```ts
+Queue.drivers.db(config: {
+  db: { update, select, insert }  // Db provider interface
+  databaseTable: string
+  extraFields?: string[]           // additional columns to insert
+  fastestPollingRateMs?: number    // default 100
+  slowestPollingRateMs?: number    // default 10_000
+  pollingBackoffRate?: number      // default 1.1
+}): Queue.Driver
+```
+
+### Queue Instance Methods
+
+Create a queue:
+
+```ts
+const queue = Queue.create<MessageBody, Result>({
+  name: string,
+  workFn: (body: MessageBody) => Promise<Result>,
+  logger?: Log.LogFunctions<[Queue.LogMessage]>,  // additional per-queue logger
+  concurrency?: number  // default 1
+})
+```
+
+Dispatch a message:
+
+```ts
+await queue.addMessage(body, {
+  scheduledDate: Iso.Instant | null,   // null = process immediately
+  tracingContext?: DehydratedTrace | null
+})
+```
+
+Update a waiting message:
+
+```ts
+await queue.updateMessage({
+  messageId: number,
+  scheduledDate: Iso.Instant,
+  body: MessageBody
+})
+```
+
+List messages (paginated):
+
+```ts
+const messages = await queue.listMessages({ pageSize: 50, pageNumber: 0 })
+```
+
+Processing control:
+
+```ts
+queue.startProcessing()     // begin polling and processing
+queue.stopProcessing()      // stop polling
+await queue.processUntilEmpty({ pollingMs?: number })  // process then resolve when empty
+```
+
+### Service-Level Methods
+
+```ts
+Queue.list() // names of all registered queues
+Queue.get<Body, Result>(name) // retrieve a queue by name
+Queue.startProcessing() // start all queues
+Queue.stopProcessing() // stop all queues
+```
+
+### Message Statuses
+
+| Status       | Meaning                                                        |
+| ------------ | -------------------------------------------------------------- |
+| `waiting`    | Queued and eligible for processing when scheduled time arrives |
+| `processing` | Locked by a worker and currently executing                     |
+| `success`    | Work function completed without error                          |
+| `error`      | Work function threw or timed out                               |
+| `stuck`      | Processing exceeded timeout; recovered or marked stuck         |
+
+### Log Events
+
+The queue emits structured log messages with these `event` values:
+
+- `messageEnqueued` -- message added to the queue
+- `messageDequeued` -- message picked up by a worker
+- `messageProcessed` -- work function completed successfully
+- `messageFailed` -- work function threw an error
+- `queueStarted` -- queue processing started
+- `queueStopped` -- queue processing stopped
+
+## Common Patterns
+
+### Schedule a Future Job
+
+```ts
+import { instantFns } from 'iso-fns2'
+
+// Schedule 1 hour from now
+const oneHourLater = instantFns.add(instantFns.now(), { hours: 1 })
+await queue.addMessage({ task: 'reminder' }, { scheduledDate: oneHourLater })
+```
+
+### Deduplication with Unique Keys
+
+The db driver supports unique keys via the `addMessage` options on the driver level. When a duplicate `uniqueKey` is inserted, the row is not duplicated -- instead the priority is updated if the new priority is higher (lower number).
+
+### Process Until Empty (Worker Scripts)
+
+```ts
+// Useful for one-off batch processing or tests
+await queue.processUntilEmpty({ pollingMs: 1000 })
+// Resolves when no waiting/processing messages remain
+```
+
+### Concurrency Control
+
+```ts
+// High-concurrency queue for lightweight jobs
+const emailQueue = Queue.create({
+  name: 'send-emails',
+  workFn: async (body) => sendEmail(body),
+  concurrency: 10
+})
+```
+
+### Stuck Job Recovery
+
+The db driver automatically recovers stuck jobs every 60 seconds. Jobs in `processing` status that exceed the timeout (default 45 minutes) are reset to `waiting` and reprocessed. Set `shouldRecoverStuckJobs: false` in the driver `start` options to mark them as `stuck` instead of recovering.
+
+### Distributed Tracing
+
+Pass a `tracing` adapter when creating the service to propagate trace context across producer/consumer boundaries:
+
+```ts
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'jobs' }),
+    logger: Log,
+    tracing: {
+      hydrate: (trace, callback) => {
+        /* restore trace context */ return callback()
+      },
+      dehydrate: () => {
+        /* return current trace context */ return null
+      },
+      startActiveSpan: (name, fn) => fn(),
+      getActiveSpan: () => undefined
+    }
+  })
+)
+```
+
+The trace data is serialized as `DehydratedTrace` (traceId, spanId, traceFlags, traceState) and stored in the `trace` column.
+
+## Testing
+
+This package has no tests yet (`pnpm --filter @maestro-js/queue test` prints "No tests yet"). To test queue behavior in consuming packages, mock the driver interface:
+
+```ts
+const mockDriver: Queue.Driver = {
+  start: () => {},
+  stop: () => {},
+  listMessages: async () => [],
+  addMessage: async () => ({ id: 1 }),
+  updateMessage: async () => {},
+  isQueueEmpty: async () => true
+}
+
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: mockDriver,
+    logger: Log
+  })
+)
+```
+
+## Cross-Package Integration
+
+### Mail (async email sending)
+
+The `@maestro-js/mail` package uses Queue to dispatch email-sending jobs asynchronously. Mail enqueues messages; a queue consumer processes them via the configured mail driver.
+
+### Recurring Jobs (cron scheduling)
+
+The `@maestro-js/recurring-jobs` package uses Queue to execute scheduled/cron-based jobs. It dispatches queue messages on a schedule and relies on the queue's processing infrastructure.
+
+### Db (database driver)
+
+The db driver requires a `Db` provider that exposes `select`, `update`, and `insert` methods. Pass `Db.provider('default')` as the `db` config option.
+
+### Tracing
+
+Integrate with `@maestro-js/tracing` by implementing the `Queue.Tracing` interface to propagate OpenTelemetry-compatible trace context across the producer/consumer boundary.
+
+## Source Files
+
+- `packages/queue/src/index.ts` -- Provider pattern, facade, queue creation and lifecycle
+- `packages/queue/src/queue-types.ts` -- TypeScript interfaces for QueueMessage, QueueDriver, QueueTracing, log events
+- `packages/queue/src/db-queue-driver.ts` -- MySQL-backed driver with polling, batched locking, stuck job recovery