@gravito/stream 2.0.1 → 2.0.2

package/README.md

@@ -1,351 +1,167 @@
 # @gravito/stream
 
-Lightweight, high-performance queueing for Gravito. Supports multiple storage drivers, embedded and standalone workers, and flexible job serialization.
+> Lightweight, high-performance queue and background job system for Galaxy Architecture.
 
-**Status**: v0.1.0 - core features complete with Memory, Database, Redis, Kafka, and SQS drivers.
+[![npm version](https://img.shields.io/npm/v/@gravito/stream.svg)](https://www.npmjs.com/package/@gravito/stream)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
+[![Bun](https://img.shields.io/badge/Bun-1.0+-black.svg)](https://bun.sh/)
 
-## Features
+**@gravito/stream** is the standard background processing unit for Gravito applications. Built on the **Orbit** pattern, it provides a unified abstraction for various message brokers and queue systems, allowing you to scale from simple in-memory tasks to distributed event-driven architectures with zero friction.
 
-- **Zero runtime overhead**: Thin wrappers that delegate to drivers
-- **Multi-driver support**: Memory, Database, Redis, Kafka, SQS, RabbitMQ
-- **Modular**: Install only the driver you need (core < 50KB)
-- **Embedded or standalone workers**: Run in-process during development or standalone in production
-- **AI-friendly**: Strong typing, clear JSDoc, and predictable APIs
-- **Custom Retry Strategies**: Built-in exponential backoff with per-job overrides
-- **Dead Letter Queue (DLQ)**: Automatic handling of permanently failed jobs, with retry and clear operations
-- **Priority Queues**: Assign priority (critical, high, low) to any job
-- **Rate Limiting**: Control job consumption rate per queue (requires Redis)
+## ✨ Features
 
-## Installation
+- 🪐 **Orbit Integration** - Native integration with PlanetCore micro-kernel and dependency injection.
+- 🔌 **Multi-Broker Support** - Built-in drivers for **Redis**, **SQS**, **Kafka**, **RabbitMQ**, **Database** (SQL), and **Memory**.
+- 🛠️ **Job-Based API** - Clean, class-based job definitions with built-in serialization and failure handling.
+- 🚀 **High Throughput** - Optimized for **Bun**, supporting batch consumption, concurrent processing, and adaptive polling.
+- 🛡️ **Reliability** - Built-in exponential backoff retries, Dead Letter Queues (DLQ), and sequential job grouping.
+- 📝 **Audit & Persistence** - Optional SQL-based persistence layer for archiving job history and providing complete audit trails.
+- 🕒 **Scheduler** - Built-in CRON-based task scheduling for recurring jobs.
+- 🏢 **Worker Modes** - Run embedded workers during development or standalone worker processes in production.
+
+## 📦 Installation
 
 ```bash
 bun add @gravito/stream
 ```
 
-## Quick Start
+## 🚀 Quick Start
+
+### 1. Define a Job
 
-### 1. Define a job
+Create a class extending `Job` and implement the `handle` logic:
 
 ```typescript
-import { Job } from '@gravito/stream'
+import { Job } from '@gravito/stream';
 
-export class SendWelcomeEmail extends Job {
-  constructor(private userId: string) {
-    super()
+export class ProcessOrder extends Job {
+  constructor(private orderId: string) {
+    super();
   }
 
   async handle(): Promise<void> {
-    const user = await User.find(this.userId)
-    await mail.send(new WelcomeEmail(user))
+    // Business logic: process the order
+    console.log(`Processing order: ${this.orderId}`);
   }
-}
-```
-
-### 3. Rate Limit & Priority (Optional)
-
-```typescript
-const queue = c.get('queue')
-
-// High priority job
-await queue.push(new SendWelcomeEmail(user.id))
-  .onQueue('emails')
-  .withPriority('high') // 'critical' | 'high' | 'default' | 'low'
 
-// Configure rate limits in Consumer
-const consumer = new Consumer(manager, {
-  rateLimits: {
-    emails: { limit: 10, window: 60 } // Max 10 jobs per minute
+  async failed(error: Error): Promise<void> {
+    // Optional: cleanup or notify on permanent failure
+    console.error(`Order ${this.orderId} failed: ${error.message}`);
   }
-})
-```
-
-### 4. Concurrency & Groups
-
-```typescript
-const consumer = new Consumer(manager, {
-  queues: ['default'],
-  concurrency: 5,           // Process up to 5 jobs concurrently
-  groupJobsSequential: true // Ensure jobs with same groupId run sequentially (default: true)
-})
+}
 ```
 
-Jobs with the same `groupId` will always be processed in order, even with high concurrency. Jobs from different groups (or no group) will run in parallel.
+### 2. Initialize OrbitStream
 
-### 5. Polling & Batching Optimization
+Register the orbit in your application bootstrap:
 
 ```typescript
-const consumer = new Consumer(manager, {
-  queues: ['default'],
-  // Polling Strategy
-  pollInterval: 1000,       // Initial poll interval
-  minPollInterval: 100,     // Adaptive: reduce to 100ms when jobs found
-  maxPollInterval: 5000,    // Adaptive: backoff up to 5s when idle
-  backoffMultiplier: 1.5,   // Exponential backoff factor
-  
-  // Batch Consumption
-  batchSize: 10,            // Fetch 10 jobs at once (requires concurrency > 1 for parallel processing)
-  concurrency: 10,
-  
-  // Blocking Pop (Redis/SQS)
-  useBlocking: true,        // Use BLPOP when batchSize=1 (reduces CPU usage)
-  blockingTimeout: 5        // Block for 5 seconds
-})
-```
+import { PlanetCore } from '@gravito/core';
+import { OrbitStream } from '@gravito/stream';
 
-### 6. Monitoring & Stats
+const core = new PlanetCore();
 
-```typescript
-const stats = consumer.getStats()
-console.log(`Processed: ${stats.processed}, Failed: ${stats.failed}`)
+core.addOrbit(OrbitStream.configure({
+  default: 'redis',
+  connections: {
+    redis: {
+      driver: 'redis',
+      host: 'localhost',
+      port: 6379
+    }
+  },
+  autoStartWorker: process.env.NODE_ENV === 'development',
+  workerOptions: { queues: ['default'] }
+}));
 
-// Metrics are also included in the heartbeat if monitor is enabled
+await core.bootstrap();
 ```
 
-### 2. Enqueue a job
-
-```typescript
-const queue = c.get('queue')
-
-await queue.push(new SendWelcomeEmail(user.id))
-  .onQueue('emails')
-  .delay(60)
-```
+### 3. Enqueue Jobs
 
-### 3. Configure OrbitStream (Memory driver)
+Access the `queue` service from the request context or container:
 
 ```typescript
-import { OrbitStream } from '@gravito/stream'
-
-const core = await PlanetCore.boot({
-  orbits: [
-    OrbitStream.configure({
-      default: 'memory',
-      connections: {
-        memory: { driver: 'memory' }
-      },
-      autoStartWorker: true,
-      workerOptions: {
-        queues: ['default', 'emails']
-      }
-    })
-  ]
-})
-```
+core.app.post('/orders', async (c) => {
+  const { id } = await c.req.json();
+  const queue = c.get('queue');
 
-### 5. Polling & Batching Optimization
+  // Push with fluent configuration
+  await queue.push(new ProcessOrder(id))
+    .onQueue('high-priority')
+    .delay(30)
+    .backoff(5, 2); // Start with 5s delay, then double for each retry
 
-```typescript
-const consumer = new Consumer(manager, {
-  queues: ['default'],
-  // Polling Strategy
-  pollInterval: 1000,       // Initial poll interval
-  minPollInterval: 100,     // Adaptive: reduce to 100ms when jobs found
-  maxPollInterval: 5000,    // Adaptive: backoff up to 5s when idle
-  backoffMultiplier: 1.5,   // Exponential backoff factor
-  
-  // Batch Consumption
-  batchSize: 10,            // Fetch 10 jobs at once (requires concurrency > 1 for parallel processing)
-  concurrency: 10,
-  
-  // Blocking Pop (Redis/SQS)
-  useBlocking: true,        // Use BLPOP when batchSize=1 (reduces CPU usage)
-  blockingTimeout: 5        // Block for 5 seconds
-})
+  return c.json({ success: true });
+});
 ```
 
-## Database Driver Example
-
-```typescript
-import { OrbitStream } from '@gravito/stream'
-
-// Create a database service adapter that implements DatabaseService interface
-const dbService = {
-  execute: async (sql, bindings) => yourDbClient.query(sql, bindings),
-  transaction: async (callback) => yourDbClient.transaction(callback),
-}
+## 🔧 Advanced Configuration
 
-const core = await PlanetCore.boot({
-  orbits: [
-    OrbitStream.configure({
-      default: 'database',
-      connections: {
-        database: {
-          driver: 'database',
-          table: 'jobs',
-          dbService: dbService // Pass your database service
-        }
-      }
-    })
-  ]
-})
-```
+### Multi-Queue & Concurrency
 
-## RabbitMQ Driver Example
+Configure the consumer to handle multiple queues with different priorities and concurrency levels:
 
 ```typescript
-import { OrbitStream } from '@gravito/stream'
-import amqp from 'amqplib'
-
-const connection = await amqp.connect('amqp://localhost')
-
-const core = await PlanetCore.boot({
-  orbits: [
-    OrbitStream.configure({
-      default: 'rabbitmq',
-      connections: {
-        rabbitmq: {
-          driver: 'rabbitmq',
-          client: connection,
-          exchange: 'gravito.events',
-          exchangeType: 'fanout'
-        }
-      }
-    })
-  ]
-})
-```
-
-## Database Schema
-
-```sql
-CREATE TABLE jobs (
-  id BIGSERIAL PRIMARY KEY,
-  queue VARCHAR(255) NOT NULL,
-  payload TEXT NOT NULL,
-  attempts INT DEFAULT 0,
-  reserved_at TIMESTAMP,
-  available_at TIMESTAMP NOT NULL,
-  created_at TIMESTAMP NOT NULL DEFAULT NOW()
-);
-
--- Optimized index for batch popping with SKIP LOCKED
-CREATE INDEX idx_jobs_queue_available_reserved ON jobs(queue, available_at, reserved_at);
-CREATE INDEX idx_jobs_reserved ON jobs(reserved_at);
+const consumer = new Consumer(manager, {
+  queues: ['critical', 'default', 'low'],
+  concurrency: 10,           // Max 10 concurrent jobs
+  groupJobsSequential: true, // Process jobs with same groupId in strict order
+  batchSize: 5,              // Fetch 5 jobs per poll
+});
 ```
 
-## Persistence and Audit Mode
-
-The `@gravito/stream` package supports an optional persistence layer (using SQLite or MySQL) for archiving job history and providing an audit trail.
+### Persistence & Audit Trail
 
-### Configuration
+Keep a history of all jobs (completed, failed, or enqueued):
 
 ```typescript
 OrbitStream.configure({
-  // ... other config
+  // ... connections
   persistence: {
-    adapter: new SQLitePersistence(DB), // or MySQLPersistence
-    archiveCompleted: true, // Archive jobs when they complete successfully
-    archiveFailed: true,    // Archive jobs when they fail permanently
-    archiveEnqueued: true,  // (Audit Mode) Archive jobs immediately when pushed
-    bufferSize: 100,        // (Optional) Batch size for buffered writes. Recommended: 50-200. Default: 0 (disabled)
-    flushInterval: 1000     // (Optional) Max time (ms) to wait before flushing buffer. Default: 0
+    adapter: new SQLitePersistence(db),
+    archiveCompleted: true,
+    archiveFailed: true,
+    archiveEnqueued: true, // Audit Mode: Log immediately when pushed
+    bufferSize: 100        // Batch writes for performance
   }
-})
-```
-
-### Audit Mode (`archiveEnqueued: true`)
-
-When Audit Mode is enabled, every job pushed to the queue is immediately written to the SQL archive with a `waiting` status. This happens in parallel with the main queue operation (Fire-and-Forget).
-
-- **Benefit**: Provides a complete audit trail. Even if the queue driver (e.g., Redis) crashes and loses data, the SQL archive will contain the record of the job being enqueued.
-- **Performance**: Designed to be non-blocking. The SQL write happens asynchronously and does not delay the `push()` operation.
-
-## Standalone Worker
-
-```bash
-bun run packages/stream/cli/queue-worker.ts \
-  --connection=database \
-  --queues=default,emails \
-  --workers=4
+});
 ```
 
-## Debugging & Monitoring
-
-### Enable Debug Mode
+## 📖 API Reference
 
-Enable verbose logging to see detailed information about job lifecycle events (enqueue, process, complete, fail).
+### `QueueManager`
 
-```typescript
-OrbitStream.configure({
-  debug: true, // Enable debug logging
-  // ...
-})
-```
+Accessed via `c.get('queue')` or `core.container.make('queue')`.
 
-```typescript
-const consumer = new Consumer(manager, {
-  debug: true, // Enable consumer debug logging
-  // ...
-})
-```
+- **`push(job)`**: Dispatch a job to the queue.
+- **`pushMany(jobs)`**: Dispatch multiple jobs efficiently.
+- **`size(queue?)`**: Get the number of jobs in a queue.
+- **`clear(queue?)`**: Remove all jobs from a queue.
 
-### Monitoring
+### `Job` Fluent Methods
 
-The Consumer emits events that you can listen to for custom monitoring:
-
-```typescript
-consumer.on('job:started', ({ job, queue }) => {
-  console.log(`Job ${job.id} started on ${queue}`)
-})
-
-consumer.on('job:failed', ({ job, error }) => {
-  console.error(`Job ${job.id} failed: ${error.message}`)
-})
-```
-
-## API Reference
-
-### Job
-
-```typescript
-abstract class Job implements Queueable {
-  abstract handle(): Promise<void>
-  async failed(error: Error): Promise<void>
-
-  onQueue(queue: string): this
-  onConnection(connection: string): this
-  delay(seconds: number): this
-  
-  /**
-   * Set retry backoff strategy.
-   * @param seconds - Initial delay in seconds
-   * @param multiplier - Multiplier for each subsequent attempt (default: 2)
-   */
-  backoff(seconds: number, multiplier = 2): this
-}
-```
-
-### QueueManager
-
-```typescript
-class QueueManager {
-  async push<T extends Job>(job: T): Promise<T>
-  async pushMany<T extends Job>(jobs: T[]): Promise<void>
-  async pop(queue?: string, connection?: string): Promise<Job | null>
-  async size(queue?: string, connection?: string): Promise<number>
-  async clear(queue?: string, connection?: string): Promise<void>
-  async complete(job: Job): Promise<void>
-  async fail(job: Job, error: Error): Promise<void>
-  registerJobClasses(jobClasses: Array<new (...args: unknown[]) => Job>): void
-}
-```
+- **`onQueue(name)`**: Specify target queue.
+- **`onConnection(name)`**: Use a specific broker connection.
+- **`delay(seconds)`**: Set initial delay.
+- **`backoff(seconds, multiplier?)`**: Configure retry strategy.
+- **`withPriority(priority)`**: Set job priority.
 
-## Implemented Drivers
+## 🔌 Supported Drivers
 
-- **MemoryDriver** - in-memory (development)
-- **DatabaseDriver** - PostgreSQL/MySQL/SQLite
-- **RedisDriver** - delayed jobs, priority queues, rate limiting, and DLQ support
-- **KafkaDriver** - topics and consumer groups
-- **SQSDriver** - standard/FIFO queues and long polling
-- **RabbitMQDriver** - exchanges, queues, and advanced confirm mode
+- **Redis** - Feature-rich (DLQ, Rate limiting, Priorities).
+- **SQS** - AWS managed queue (Standard/FIFO).
+- **Kafka** - High-throughput distributed streams.
+- **RabbitMQ** - Traditional AMQP broker.
+- **Database** - Simple SQL-based persistence (PostgreSQL, MySQL, SQLite).
+- **Memory** - Fast, zero-config for local development/testing.
 
-## Best Practices
+## 🤝 Contributing
 
-1.  **Idempotency**: Ensure your jobs are idempotent. Jobs may be retried if they fail or if the worker crashes.
-2.  **Granularity**: Keep jobs small and focused. Large jobs can block workers and increase memory usage.
-3.  **Timeouts**: Set appropriate timeouts for your jobs to prevent them from hanging indefinitely.
-4.  **Error Handling**: Use the `failed` method or throw errors to trigger retries. Avoid swallowing errors unless you want to suppress retries.
+Contributions, issues and feature requests are welcome!
+Feel free to check the [issues page](https://github.com/gravito-framework/gravito/issues).
 
-## License
+## 📝 License
 
-MIT
+MIT © [Carl Lee](https://github.com/gravito-framework/gravito)

package/README.zh-TW.md

@@ -1,34 +1,167 @@
 # @gravito/stream
 
-> Gravito 的佇列模組，支援多種驅動與獨立 Worker。
+> Galaxy 架構的高效能輕量化隊列與背景任務系統。
 
-## 安裝
+[![npm version](https://img.shields.io/npm/v/@gravito/stream.svg)](https://www.npmjs.com/package/@gravito/stream)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-blue.svg)](https://www.typescriptlang.org/)
+[![Bun](https://img.shields.io/badge/Bun-1.0+-black.svg)](https://bun.sh/)
+
+**@gravito/stream** 是 Gravito 應用程式的標準背景處理單元。基於 **Orbit** 模式構建，它為各種訊息代理（Broker）和隊列系統提供了統一的抽象層，讓您能夠從簡單的記憶體任務無縫擴展到分佈式事件驅動架構。
+
+## ✨ 特性
+
+- 🪐 **Orbit 整合** - 與 PlanetCore 微核心及依賴注入系統原生整合。
+- 🔌 **多 Broker 支援** - 內建支援 **Redis**、**SQS**、**Kafka**、**RabbitMQ**、**資料庫** (SQL) 與 **記憶體** (Memory) 驅動。
+- 🛠️ **基於 Job 的 API** - 簡潔的類別式（Class-based）任務定義，內建序列化與錯誤處理。
+- 🚀 **高吞吐量** - 針對 **Bun** 進行優化，支援批量消費（Batching）、並發處理與自適應輪詢（Polling）。
+- 🛡️ **可靠性** - 內建指數退避重試（Exponential Backoff）、死信隊列（DLQ）與順序任務分組。
+- 📝 **審計與持久化** - 可選的 SQL 持久化層，用於存檔任務歷史並提供完整的審計追蹤（Audit Trail）。
+- 🕒 **排程器** - 內建基於 CRON 的排程功能，支援週期性任務。
+- 🏢 **Worker 模式** - 開發環境可運行嵌入式 Worker，生產環境可運行獨立 Worker 進程。
+
+## 📦 安裝
 
 ```bash
 bun add @gravito/stream
 ```
 
-## 快速開始
+## 🚀 快速上手
+
+### 1. 定義任務 (Job)
+
+建立一個繼承自 `Job` 的類別並實作 `handle` 邏輯：
 
 ```typescript
-import { Job } from '@gravito/stream'
+import { Job } from '@gravito/stream';
 
-export class SendWelcomeEmail extends Job {
-  constructor(private userId: string) {
-    super()
+export class ProcessOrder extends Job {
+  constructor(private orderId: string) {
+    super();
   }
 
   async handle(): Promise<void> {
-    const user = await User.find(this.userId)
-    await mail.send(new WelcomeEmail(user))
+    // 業務邏輯：處理訂單
+    console.log(`正在處理訂單: ${this.orderId}`);
+  }
+
+  async failed(error: Error): Promise<void> {
+    // 選配：在永久失敗時進行清理或通知
+    console.error(`訂單 ${this.orderId} 失敗: ${error.message}`);
   }
 }
 ```
 
+### 2. 初始化 OrbitStream
+
+在應用程式啟動時註冊 Orbit：
+
+```typescript
+import { PlanetCore } from '@gravito/core';
+import { OrbitStream } from '@gravito/stream';
+
+const core = new PlanetCore();
+
+core.addOrbit(OrbitStream.configure({
+  default: 'redis',
+  connections: {
+    redis: {
+      driver: 'redis',
+      host: 'localhost',
+      port: 6379
+    }
+  },
+  autoStartWorker: process.env.NODE_ENV === 'development',
+  workerOptions: { queues: ['default'] }
+}));
+
+await core.bootstrap();
+```
+
+### 3. 將任務推入隊列
+
+從請求上下文或容器中獲取 `queue` 服務：
+
 ```typescript
-const queue = c.get('queue')
+core.app.post('/orders', async (c) => {
+  const { id } = await c.req.json();
+  const queue = c.get('queue');
+
+  // 使用流暢介面進行配置
+  await queue.push(new ProcessOrder(id))
+    .onQueue('high-priority') // 指定隊列
+    .delay(30)                // 延遲 30 秒執行
+    .backoff(5, 2);           // 重試策略：初始延遲 5s，之後每次翻倍
 
-await queue.push(new SendWelcomeEmail(user.id))
-  .onQueue('emails')
-  .delay(60)
+  return c.json({ success: true });
+});
 ```
+
+## 🔧 進階配置
+
+### 多隊列與並發處理
+
+配置消費者以處理不同優先級的隊列與並發等級：
+
+```typescript
+const consumer = new Consumer(manager, {
+  queues: ['critical', 'default', 'low'],
+  concurrency: 10,           // 最大同時執行 10 個任務
+  groupJobsSequential: true, // 相同 groupId 的任務將嚴格依序執行
+  batchSize: 5,              // 每次輪詢獲取 5 個任務
+});
+```
+
+### 持久化與審計追蹤
+
+保留所有任務（成功、失敗或排隊中）的歷史記錄：
+
+```typescript
+OrbitStream.configure({
+  // ... 連線配置
+  persistence: {
+    adapter: new SQLitePersistence(db),
+    archiveCompleted: true,
+    archiveFailed: true,
+    archiveEnqueued: true, // 審計模式：推入隊列時立即記錄
+    bufferSize: 100        // 批量寫入以提升效能
+  }
+});
+```
+
+## 📖 API 參考
+
+### `QueueManager`
+
+透過 `c.get('queue')` 或 `core.container.make('queue')` 獲取。
+
+- **`push(job)`**: 將任務派發至隊列。
+- **`pushMany(jobs)`**: 高效派發多個任務。
+- **`size(queue?)`**: 獲取隊列中的任務數量。
+- **`clear(queue?)`**: 清空隊列中的所有任務。
+
+### `Job` 流暢方法
+
+- **`onQueue(name)`**: 指定目標隊列。
+- **`onConnection(name)`**: 使用特定連線。
+- **`delay(seconds)`**: 設置初始延遲。
+- **`backoff(seconds, multiplier?)`**: 配置重試策略。
+- **`withPriority(priority)`**: 設置任務優先級。
+
+## 🔌 支援的驅動 (Drivers)
+
+- **Redis** - 功能豐富（支援 DLQ、限流、優先級）。
+- **SQS** - AWS 託管隊列（Standard/FIFO）。
+- **Kafka** - 高吞吐量分佈式串流。
+- **RabbitMQ** - 傳統 AMQP 代理。
+- **Database** - 簡單的 SQL 持久化方案（PostgreSQL, MySQL, SQLite）。
+- **Memory** - 快速、開發/測試環境零配置。
+
+## 🤝 貢獻
+
+歡迎提交貢獻、問題與功能請求！
+請隨時查看 [Issues 頁面](https://github.com/gravito-framework/gravito/issues)。
+
+## 📝 授權
+
+MIT © [Carl Lee](https://github.com/gravito-framework/gravito)