@maestro-js/agent-skills 1.0.0-alpha.0

package/dist/maestro-packages/metrics.md

@@ -0,0 +1,236 @@
+---
+name: metrics
+description: "Use when working with @maestro-js/metrics. Unified metrics collection piped through the Log transport system. Covers four metric types (counters, distributions, sets, gauges), provider setup, logger integration, Sentry transport, and custom transport authoring. Trigger on metrics emission, observability instrumentation, Sentry metrics integration, or application telemetry tasks."
+---
+
+# Metrics
+
+## Overview
+
+The Metrics package provides unified application metrics collection that routes through the Log transport system. Emit counters, distributions, sets, and gauges as structured log messages, then attach transports to forward metrics to external systems like Sentry.
+
+## Quick Setup
+
+```ts
+import { Metrics } from '@maestro-js/metrics'
+import { Log } from '@maestro-js/log'
+import * as Sentry from '@sentry/node'
+
+// 1. Create a typed logger for metrics messages
+const metricsLogger = Log.create<[Metrics.LogMessage]>()
+
+// 2. Register the metrics service
+Metrics.Provider.register('default', Metrics.Provider.create({ logger: metricsLogger }))
+
+// 3. Attach the Sentry transport (or any custom transport)
+metricsLogger.addTransport(Metrics.transports.sentryMetrics(Sentry.metrics))
+
+// 4. Emit metrics
+Metrics.increment('order_placed', 1, { tags: { region: 'us-east' } })
+```
+
+## API Reference
+
+### Provider
+
+#### `Metrics.Provider.create(config)`
+
+Create a metrics service instance.
+
+```ts
+Metrics.Provider.create({
+  logger: Log.Logger<[Metrics.LogMessage]>
+})
+```
+
+The `logger` parameter is a typed `Log.Logger` that accepts `Metrics.LogMessage` entries. All metric emissions flow through this logger, so output destination is controlled entirely by the transports attached to it.
+
+#### `Metrics.Provider.register(name, service)`
+
+Register a metrics service instance in the named registry.
+
+```ts
+Metrics.Provider.register('default', Metrics.Provider.create({ logger: metricsLogger }))
+```
+
+### Metric Methods
+
+All four methods are available directly on the `Metrics` facade (resolving the `'default'` provider) or on a named provider instance.
+
+#### `Metrics.increment(name, value?, data?)`
+
+Emit a counter metric, incrementing by `value` (defaults to 1 when omitted).
+
+```ts
+Metrics.increment('user_signup')                                    // increment by 1
+Metrics.increment('items_purchased', 3, { tags: { store: 'web' } }) // increment by 3
+```
+
+#### `Metrics.distribution(name, value, data?)`
+
+Emit a distribution metric for statistical aggregation (p50, p95, p99).
+
+```ts
+Metrics.distribution('api_response_time', 142, { unit: 'millisecond' })
+Metrics.distribution('payload_size', 2048, { unit: 'byte', tags: { endpoint: '/upload' } })
+```
+
+#### `Metrics.set(name, value, data?)`
+
+Emit a set metric tracking unique string or integer values.
+
+```ts
+Metrics.set('unique_customers', 'cust_123')
+Metrics.set('unique_customers', 'cust_456', { tags: { channel: 'mobile' } })
+```
+
+#### `Metrics.gauge(name, value, data?)`
+
+Emit a gauge metric capturing a point-in-time value that can increase or decrease.
+
+```ts
+Metrics.gauge('active_connections', 47)
+Metrics.gauge('queue_depth', 312, { tags: { queue: 'email' } })
+```
+
+### MetricsAdditionalData
+
+Optional metadata passed as the last argument to any metric method.
+
+```ts
+interface MetricsAdditionalData {
+  unit?: MeasurementUnit   // 'millisecond', 'byte', 'percent', 'none', etc.
+  tags?: Record<string, Primitive>  // arbitrary key-value pairs for grouping
+  timestamp?: number       // custom Unix timestamp
+}
+```
+
+**Supported unit categories:**
+
+- **Duration**: `'nanosecond'`, `'microsecond'`, `'millisecond'`, `'second'`, `'minute'`, `'hour'`, `'day'`, `'week'`
+- **Information**: `'bit'`, `'byte'`, `'kilobyte'`, `'megabyte'`, `'gigabyte'`, `'terabyte'`, etc.
+- **Fraction**: `'ratio'`, `'percent'`
+- **None**: `''`, `'none'`
+
+### MetricsLogMessage
+
+Discriminated union of the four metric operations, keyed by `op`. Each variant carries `name`, `value`, and `data` fields. Used to type the logger and transports.
+
+```ts
+type MetricsLogMessage =
+  | { op: 'increment'; name: string; value: number | undefined; data: MetricsAdditionalData | undefined }
+  | { op: 'distribution'; name: string; value: number; data: MetricsAdditionalData | undefined }
+  | { op: 'set'; name: string; value: number | string; data: MetricsAdditionalData | undefined }
+  | { op: 'gauge'; name: string; value: number; data: MetricsAdditionalData | undefined }
+```
+
+## Transports
+
+### Built-in: Sentry Metrics
+
+Route all metrics to Sentry via `Metrics.transports.sentryMetrics()`.
+
+```ts
+import * as Sentry from '@sentry/node'
+
+const metricsLogger = Log.create<[Metrics.LogMessage]>()
+metricsLogger.addTransport(Metrics.transports.sentryMetrics(Sentry.metrics))
+```
+
+The transport accepts any object matching the `Metrics.MetricFunctions` interface (the four metric methods), so it works with any Sentry-compatible metrics API.
+
+### Custom Transports
+
+Write a custom transport by implementing `Log.Transport<[Metrics.LogMessage]>`. The `write` method receives a log message whose `data[0]` is the `MetricsLogMessage`.
+
+```ts
+function datadogTransport(client: DatadogClient): Log.Transport<[Metrics.LogMessage]> {
+  return {
+    write(message) {
+      const metric = message.data[0]
+      if (metric.op === 'increment') {
+        client.increment(metric.name, metric.value ?? 1, metric.data?.tags)
+      } else if (metric.op === 'distribution') {
+        client.distribution(metric.name, metric.value, metric.data?.tags)
+      } else if (metric.op === 'gauge') {
+        client.gauge(metric.name, metric.value, metric.data?.tags)
+      }
+      // handle 'set' as needed
+    }
+  }
+}
+
+metricsLogger.addTransport(datadogTransport(myDatadogClient))
+```
+
+## Common Patterns
+
+### Tagging metrics by request context
+
+```ts
+Metrics.increment('api_request', 1, {
+  tags: { method: 'POST', path: '/orders', status: 201 }
+})
+
+Metrics.distribution('db_query_time', 23, {
+  unit: 'millisecond',
+  tags: { table: 'orders', operation: 'insert' }
+})
+```
+
+### Tracking unique visitors
+
+```ts
+Metrics.set('unique_visitors', sessionId, {
+  tags: { page: '/checkout' }
+})
+```
+
+### Monitoring resource usage
+
+```ts
+Metrics.gauge('memory_usage_mb', process.memoryUsage().heapUsed / 1024 / 1024, {
+  unit: 'megabyte'
+})
+
+Metrics.gauge('active_websocket_connections', connectionPool.size)
+```
+
+### Multiple metric providers
+
+Register named providers for different metric pipelines.
+
+```ts
+const appLogger = Log.create<[Metrics.LogMessage]>()
+const perfLogger = Log.create<[Metrics.LogMessage]>()
+
+appLogger.addTransport(Metrics.transports.sentryMetrics(Sentry.metrics))
+perfLogger.addTransport(datadogTransport(myDatadogClient))
+
+Metrics.Provider.register('default', Metrics.Provider.create({ logger: appLogger }))
+Metrics.Provider.register('perf', Metrics.Provider.create({ logger: perfLogger }))
+
+// Facade calls use 'default'
+Metrics.increment('order_placed')
+```
+
+## Cross-Package Integration
+
+### Depends on
+
+- **@maestro-js/service-registry** -- Provider pattern, named registry, deferred Proxy resolution.
+- **@maestro-js/log** -- Typed logger and transport system. Metrics emissions are structured log messages routed through Log transports.
+
+### Works with
+
+- **@maestro-js/log** -- Metrics piggybacks on the Log transport architecture. Create a dedicated `Log.Logger<[Metrics.LogMessage]>` for metrics, then attach transports to route emissions to Sentry, Datadog, or any custom destination. The same logger can have multiple transports to fan out metrics to several backends simultaneously.
+
+## Package Info
+
+- **Package name**: `@maestro-js/metrics`
+- **Source**: `packages/metrics/src/index.ts`
+- **Types**: `packages/metrics/src/metrics-types.ts`
+- **Sentry transport**: `packages/metrics/src/sentry-metrics-transport.ts`
+- **Tests**: None yet
+- **Build**: `pnpm --filter @maestro-js/metrics build`
+- **Typecheck**: `pnpm --filter @maestro-js/metrics typecheck`

package/dist/maestro-packages/queue.md

@@ -0,0 +1,284 @@
+---
+name: queue
+description: "Database-backed job queue for deferring work beyond the current request with scheduled execution, concurrency control, and distributed tracing. Use when working with @maestro-js/queue, including dispatching jobs, processing queues, configuring the db driver, writing queue consumers, handling retries/stuck jobs, integrating with mail or recurring-jobs, and understanding the queue database schema."
+---
+
+# @maestro-js/queue
+
+Database-backed job queue that stores messages in MySQL via a pluggable driver and processes them asynchronously. Follows the Provider pattern from `@maestro-js/service-registry` with a producer/consumer split supporting scheduled execution, concurrency, and distributed trace propagation.
+
+## Quick Setup
+
+```ts
+import { Queue } from '@maestro-js/queue'
+import { Db } from '@maestro-js/db'
+import { Log } from '@maestro-js/log'
+
+// Register the queue service with the db driver
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({
+      db: Db.provider('default'),
+      databaseTable: 'jobs'
+    }),
+    logger: Log
+  })
+)
+
+// Create a named queue with a work function
+const orderQueue = Queue.create({
+  name: 'fulfill-orders',
+  workFn: async (body) => {
+    await fulfillOrder(body.orderId)
+    return { fulfilled: true }
+  },
+  concurrency: 3
+})
+
+// Enqueue a message
+await orderQueue.addMessage({ orderId: 'ord_123' }, { scheduledDate: null })
+
+// Start processing
+orderQueue.startProcessing()
+```
+
+## Database Schema
+
+The db driver expects a MySQL table (name configurable via `databaseTable`). The column-to-field mapping used internally:
+
+| Column          | Type          | Description                                              |
+| --------------- | ------------- | -------------------------------------------------------- |
+| `id`            | int (PK, AI)  | Message ID                                               |
+| `batch_id`      | int / NULL    | Lock batch for concurrent dequeue                        |
+| `job_type`      | varchar       | Queue name string                                        |
+| `body`          | text (JSON)   | Serialized message body                                  |
+| `trace`         | text (JSON)   | Serialized `DehydratedTrace` for distributed tracing     |
+| `status`        | enum          | `waiting`, `processing`, `success`, `error`, `stuck`     |
+| `scheduledDate` | datetime      | Earliest time the message can be processed               |
+| `enqueuedDate`  | datetime      | When the message was inserted                            |
+| `started_ts`    | datetime      | When processing began                                    |
+| `result`        | text          | JSON result on success, error stack on failure           |
+| `recovered`     | tinyint       | 1 if the message was recovered from stuck state          |
+| `running_time`  | int / NULL    | Seconds elapsed during processing                        |
+| `unique_key`    | int / NULL    | MD5-derived dedup key (nullable)                         |
+| `priority`      | bigint / NULL | Lower value = higher priority (defaults to `Date.now()`) |
+
+The driver requires a `FORCE INDEX (locking_update)` index on `(batch_id, status, job_type, scheduledDate)` for efficient locking queries.
+
+## API Reference
+
+### Provider Setup
+
+```ts
+Queue.Provider.create(config: {
+  driver: Queue.Driver
+  logger: Log.Logger<[Queue.LogMessage]>
+  tracing?: Queue.Tracing  // optional distributed tracing adapter
+}): Queue.QueueService
+```
+
+```ts
+Queue.Provider.register(name: Queue.Provider.Key, service: Queue.QueueService): void
+```
+
+### Drivers
+
+```ts
+Queue.drivers.db(config: {
+  db: { update, select, insert }  // Db provider interface
+  databaseTable: string
+  extraFields?: string[]           // additional columns to insert
+  fastestPollingRateMs?: number    // default 100
+  slowestPollingRateMs?: number    // default 10_000
+  pollingBackoffRate?: number      // default 1.1
+}): Queue.Driver
+```
+
+### Queue Instance Methods
+
+Create a queue:
+
+```ts
+const queue = Queue.create<MessageBody, Result>({
+  name: string,
+  workFn: (body: MessageBody) => Promise<Result>,
+  logger?: Log.LogFunctions<[Queue.LogMessage]>,  // additional per-queue logger
+  concurrency?: number  // default 1
+})
+```
+
+Dispatch a message:
+
+```ts
+await queue.addMessage(body, {
+  scheduledDate: Iso.Instant | null,   // null = process immediately
+  tracingContext?: DehydratedTrace | null
+})
+```
+
+Update a waiting message:
+
+```ts
+await queue.updateMessage({
+  messageId: number,
+  scheduledDate: Iso.Instant,
+  body: MessageBody
+})
+```
+
+List messages (paginated):
+
+```ts
+const messages = await queue.listMessages({ pageSize: 50, pageNumber: 0 })
+```
+
+Processing control:
+
+```ts
+queue.startProcessing()     // begin polling and processing
+queue.stopProcessing()      // stop polling
+await queue.processUntilEmpty({ pollingMs?: number })  // process then resolve when empty
+```
+
+### Service-Level Methods
+
+```ts
+Queue.list() // names of all registered queues
+Queue.get<Body, Result>(name) // retrieve a queue by name
+Queue.startProcessing() // start all queues
+Queue.stopProcessing() // stop all queues
+```
+
+### Message Statuses
+
+| Status       | Meaning                                                        |
+| ------------ | -------------------------------------------------------------- |
+| `waiting`    | Queued and eligible for processing when scheduled time arrives |
+| `processing` | Locked by a worker and currently executing                     |
+| `success`    | Work function completed without error                          |
+| `error`      | Work function threw or timed out                               |
+| `stuck`      | Processing exceeded timeout; recovered or marked stuck         |
+
+### Log Events
+
+The queue emits structured log messages with these `event` values:
+
+- `messageEnqueued` -- message added to the queue
+- `messageDequeued` -- message picked up by a worker
+- `messageProcessed` -- work function completed successfully
+- `messageFailed` -- work function threw an error
+- `queueStarted` -- queue processing started
+- `queueStopped` -- queue processing stopped
+
+## Common Patterns
+
+### Schedule a Future Job
+
+```ts
+import { instantFns } from 'iso-fns2'
+
+// Schedule 1 hour from now
+const oneHourLater = instantFns.add(instantFns.now(), { hours: 1 })
+await queue.addMessage({ task: 'reminder' }, { scheduledDate: oneHourLater })
+```
+
+### Deduplication with Unique Keys
+
+The db driver supports unique keys via the `addMessage` options on the driver level. When a duplicate `uniqueKey` is inserted, the row is not duplicated -- instead the priority is updated if the new priority is higher (lower number).
+
+### Process Until Empty (Worker Scripts)
+
+```ts
+// Useful for one-off batch processing or tests
+await queue.processUntilEmpty({ pollingMs: 1000 })
+// Resolves when no waiting/processing messages remain
+```
+
+### Concurrency Control
+
+```ts
+// High-concurrency queue for lightweight jobs
+const emailQueue = Queue.create({
+  name: 'send-emails',
+  workFn: async (body) => sendEmail(body),
+  concurrency: 10
+})
+```
+
+### Stuck Job Recovery
+
+The db driver automatically recovers stuck jobs every 60 seconds. Jobs in `processing` status that exceed the timeout (default 45 minutes) are reset to `waiting` and reprocessed. Set `shouldRecoverStuckJobs: false` in the driver `start` options to mark them as `stuck` instead of recovering.
+
+### Distributed Tracing
+
+Pass a `tracing` adapter when creating the service to propagate trace context across producer/consumer boundaries:
+
+```ts
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: Queue.drivers.db({ db: Db.provider('default'), databaseTable: 'jobs' }),
+    logger: Log,
+    tracing: {
+      hydrate: (trace, callback) => {
+        /* restore trace context */ return callback()
+      },
+      dehydrate: () => {
+        /* return current trace context */ return null
+      },
+      startActiveSpan: (name, fn) => fn(),
+      getActiveSpan: () => undefined
+    }
+  })
+)
+```
+
+The trace data is serialized as `DehydratedTrace` (traceId, spanId, traceFlags, traceState) and stored in the `trace` column.
+
+## Testing
+
+This package has no tests yet (`pnpm --filter @maestro-js/queue test` prints "No tests yet"). To test queue behavior in consuming packages, mock the driver interface:
+
+```ts
+const mockDriver: Queue.Driver = {
+  start: () => {},
+  stop: () => {},
+  listMessages: async () => [],
+  addMessage: async () => ({ id: 1 }),
+  updateMessage: async () => {},
+  isQueueEmpty: async () => true
+}
+
+Queue.Provider.register(
+  'default',
+  Queue.Provider.create({
+    driver: mockDriver,
+    logger: Log
+  })
+)
+```
+
+## Cross-Package Integration
+
+### Mail (async email sending)
+
+The `@maestro-js/mail` package uses Queue to dispatch email-sending jobs asynchronously. Mail enqueues messages; a queue consumer processes them via the configured mail driver.
+
+### Recurring Jobs (cron scheduling)
+
+The `@maestro-js/recurring-jobs` package uses Queue to execute scheduled/cron-based jobs. It dispatches queue messages on a schedule and relies on the queue's processing infrastructure.
+
+### Db (database driver)
+
+The db driver requires a `Db` provider that exposes `select`, `update`, and `insert` methods. Pass `Db.provider('default')` as the `db` config option.
+
+### Tracing
+
+Integrate with `@maestro-js/tracing` by implementing the `Queue.Tracing` interface to propagate OpenTelemetry-compatible trace context across the producer/consumer boundary.
+
+## Source Files
+
+- `packages/queue/src/index.ts` -- Provider pattern, facade, queue creation and lifecycle
+- `packages/queue/src/queue-types.ts` -- TypeScript interfaces for QueueMessage, QueueDriver, QueueTracing, log events
+- `packages/queue/src/db-queue-driver.ts` -- MySQL-backed driver with polling, batched locking, stuck job recovery