@siglo/sqs 0.0.0-stage → 1.0.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2017 Jerico de guzman
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,163 @@
-# Temporary Holding Version
+# @siglo/sqs Adapter Guide
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+`@siglo/sqs` provides an AWS SQS batch consumer adapter for executing `Siglo` workflows. It is optimized for AWS Lambda event source mappings and standalone long-polling worker processes, implementing AWS partial batch failure protocols (`ReportBatchItemFailures`), message deduplication, and tenant namespace extraction.
+
+---
+
+## 1. Overview & Architecture
+
+When AWS Lambda is triggered by an SQS queue, it passes a batch of records (`SQSEvent`). If one message in the batch fails, returning an uncaught exception causes the entire batch to be retried by AWS, resulting in duplicate processing of previously successful messages.
+
+`@siglo/sqs` solves this using AWS SQS Partial Batch Responses:
+
+```mermaid
+flowchart TD
+    SQS["AWS SQS"] -->|Batch of N records| Lambda["AWS Lambda Trigger"]
+    Lambda --> Handler["createSqsBatchHandler<br/>• Iterates event.Records<br/>• Checks IdempotencyStore<br/>• Parses JSON Record Body<br/>• Executes Siglo"]
+    Handler --> Split{Record Result}
+    Split -->|Success Message| SSuccess["• Complete Idemp Key<br/>• Increment Count"]
+    Split -->|Failed Message| SFail["• Fail Idemp Key<br/>• Append to batchItemFailures"]
+    SSuccess --> BatchRes["Return Batch Result<br/>{ batchItemFailures: [...] }"]
+    SFail --> BatchRes
+    BatchRes --> Final["SQS deletes succeeded records<br/>SQS retries only failed items"]
+```
+
+---
+
+## 2. Installation & Requirements
+
+Install `@siglo/sqs`:
+
+```bash
+npm install @siglo/core @siglo/sqs
+npm install --save-dev @types/aws-lambda
+```
+
+Peer dependency matrix:
+- `Siglo`: `*` (v1.0.0+)
+- `@aws-sdk/client-sqs`: `^3.0.0` (optional if running in AWS Lambda runtime)
+
+---
+
+## 3. API Reference
+
+### `createSqsBatchHandler(workflow, options)`
+
+Creates an AWS Lambda-compliant batch processor function:
+
+```typescript
+function createSqsBatchHandler<TInput = any, TOutput = any>(
+    workflow: Flow<TInput, TOutput>,
+    options?: SigloSqsOptions<TInput, TOutput>
+): (event: SigloSqsEvent) => Promise<SqsBatchResponse>;
+```
+
+### TypeScript Interfaces
+
+```typescript
+export interface SigloSqsRecord {
+    messageId: string;
+    receiptHandle: string;
+    body: string;
+    attributes?: Record<string, string>;
+    messageAttributes?: Record<string, any>;
+}
+
+export interface SigloSqsEvent {
+    Records: SigloSqsRecord[];
+}
+
+export interface SqsBatchResponse {
+    batchItemFailures: Array<{ itemIdentifier: string }>;
+    processedCount: number;
+}
+```
+
+---
+
+## 4. Configuration Options (`SigloSqsOptions`)
+
+| Option | Type | Default | Description |
+| :--- | :--- | :--- | :--- |
+| `input` | `(record: SigloSqsRecord) => Promise<TInput> \| TInput` | `JSON.parse(record.body)` | Custom mapper transforming an SQS record into workflow parameters. |
+| `output` | `(result: any, record: SigloSqsRecord) => Promise<TOutput> \| TOutput` | `result` | Custom post-processor for step result. |
+| `idempotencyStore` | `IdempotencyStore` | `undefined` | Distributed store to deduplicate across re-deliveries. |
+| `idempotencyTtlMs` | `number` | `86400000` (24h) | Time-to-live for idempotency lock. |
+| `timeout` | `number` | `undefined` | SLA execution timeout per record in milliseconds. |
+| `namespace` | `string \| ((record: SigloSqsRecord) => string \| undefined)` | `record.messageAttributes.namespace` | Namespace extractor for multi-tenant isolation. |
+
+---
+
+## 5. AWS Lambda Configuration: `ReportBatchItemFailures`
+
+To enable partial batch responses in AWS Lambda:
+1. In the AWS Lambda Console, navigate to your SQS Trigger configuration.
+2. Under **Additional settings**, check the option: **Report batch item failures**.
+3. In AWS SAM / CloudFormation / Serverless Framework / CDK:
+   ```yaml
+   FunctionEventSource:
+     Type: SQS
+     Properties:
+       Queue: !GetAtt MyQueue.Arn
+       FunctionResponseTypes:
+         - ReportBatchItemFailures
+   ```
+
+When this is enabled, returning `{ batchItemFailures: [{ itemIdentifier: 'msg-01' }] }` tells AWS SQS to delete all successful messages in the batch from the queue, while leaving only `msg-01` in the queue to be retried or routed to the Dead Letter Queue (DLQ).
+
+---
+
+## 6. Complete AWS Lambda Production Example
+
+```typescript
+import { SQSEvent, SQSBatchResponse, Context } from 'aws-lambda';
+import { Step, flow } from '@siglo/core';
+import { createSqsBatchHandler } from '@siglo/sqs';
+import { PostgresIdempotencyStore } from '@siglo/postgres';
+import { Pool } from 'pg';
+
+// 1. Initialize PostgreSQL Connection Pool outside handler
+const pool = new Pool({
+    connectionString: process.env.DATABASE_URL,
+    max: 5,
+});
+
+const idempotencyStore = new PostgresIdempotencyStore({ client: pool });
+
+// 2. Define Notification Workflow
+interface NotificationPayload {
+    userId: string;
+    template: string;
+    destination: string;
+}
+
+const formatNotification = Step.create('formatNotification', async (param: NotificationPayload) => {
+    return { formattedBody: `Hello user ${param.userId}, template ${param.template}` };
+});
+
+const dispatchEmail = Step.create('dispatchEmail', async (param: any) => {
+    // Send email via AWS SES or external provider
+    return { dispatched: true, messageRef: `ses_${Date.now()}` };
+});
+
+const notificationFlow = flow(formatNotification, dispatchEmail);
+
+// 3. Create SQS Batch Handler
+const sqsHandler = createSqsBatchHandler(notificationFlow, {
+    idempotencyStore,
+    timeout: 10000, // 10-second deadline per message
+    namespace: (record) => record.messageAttributes?.tenantId?.stringValue,
+});
+
+// 4. Export AWS Lambda Handler
+export const handler = async (event: SQSEvent, context: Context): Promise<SQSBatchResponse> => {
+    context.callbackWaitsForEmptyEventLoop = false;
+
+    const response = await sqsHandler(event);
+    console.log(`Processed ${response.processedCount} messages. Failures: ${response.batchItemFailures.length}`);
+
+    return {
+        batchItemFailures: response.batchItemFailures,
+    };
+};
+```

package/dist/index.d.ts

@@ -0,0 +1,33 @@
+import { Flow, IdempotencyStore } from '@siglo/core';
+export interface SigloSqsRecord {
+    messageId: string;
+    receiptHandle: string;
+    body: string;
+    attributes?: Record<string, string>;
+    messageAttributes?: Record<string, any>;
+}
+export interface SigloSqsEvent {
+    Records: SigloSqsRecord[];
+}
+export interface SigloSqsOptions<TInput = any, TOutput = any> {
+    input?: (record: SigloSqsRecord) => Promise<TInput> | TInput;
+    output?: (result: any, record: SigloSqsRecord) => Promise<TOutput> | TOutput;
+    idempotencyStore?: IdempotencyStore;
+    idempotencyTtlMs?: number;
+    timeout?: number;
+    namespace?: string | ((record: SigloSqsRecord) => string | undefined);
+}
+export interface SqsBatchResponse {
+    batchItemFailures: Array<{
+        itemIdentifier: string;
+    }>;
+    processedCount: number;
+}
+/**
+ * Creates an AWS Lambda compatible SQS batch message handler for rivulet workflows.
+ */
+export declare function createSqsBatchHandler<TInput = any, TOutput = any>(workflow: Flow<TInput, TOutput>, options?: SigloSqsOptions<TInput, TOutput>): (event: SigloSqsEvent) => Promise<SqsBatchResponse>;
+export type RivuletSqsRecord = SigloSqsRecord;
+export type RivuletSqsEvent = SigloSqsEvent;
+export type RivuletSqsOptions<TInput = any, TOutput = any> = SigloSqsOptions<TInput, TOutput>;
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,IAAI,EAAoB,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAEvE,MAAM,WAAW,cAAc;IAC3B,SAAS,EAAE,MAAM,CAAC;IAClB,aAAa,EAAE,MAAM,CAAC;IACtB,IAAI,EAAE,MAAM,CAAC;IACb,UAAU,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACpC,iBAAiB,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,GAAG,CAAC,CAAC;CAC3C;AAED,MAAM,WAAW,aAAa;IAC1B,OAAO,EAAE,cAAc,EAAE,CAAC;CAC7B;AAED,MAAM,WAAW,eAAe,CAAC,MAAM,GAAG,GAAG,EAAE,OAAO,GAAG,GAAG;IACxD,KAAK,CAAC,EAAE,CAAC,MAAM,EAAE,cAAc,KAAK,OAAO,CAAC,MAAM,CAAC,GAAG,MAAM,CAAC;IAC7D,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,cAAc,KAAK,OAAO,CAAC,OAAO,CAAC,GAAG,OAAO,CAAC;IAC7E,gBAAgB,CAAC,EAAE,gBAAgB,CAAC;IACpC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,SAAS,CAAC,EAAE,MAAM,GAAG,CAAC,CAAC,MAAM,EAAE,cAAc,KAAK,MAAM,GAAG,SAAS,CAAC,CAAC;CACzE;AAiBD,MAAM,WAAW,gBAAgB;IAC7B,iBAAiB,EAAE,KAAK,CAAC;QAAE,cAAc,EAAE,MAAM,CAAA;KAAE,CAAC,CAAC;IACrD,cAAc,EAAE,MAAM,CAAC;CAC1B;AAED;;GAEG;AACH,wBAAgB,qBAAqB,CAAC,MAAM,GAAG,GAAG,EAAE,OAAO,GAAG,GAAG,EAC7D,QAAQ,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,CAAC,EAC/B,OAAO,GAAE,eAAe,CAAC,MAAM,EAAE,OAAO,CAAM,WAEzB,aAAa,KAAG,OAAO,CAAC,gBAAgB,CAAC,CA2DjE;AAGD,MAAM,MAAM,gBAAgB,GAAG,cAAc,CAAC;AAC9C,MAAM,MAAM,eAAe,GAAG,aAAa,CAAC;AAC5C,MAAM,MAAM,iBAAiB,CAAC,MAAM,GAAG,GAAG,EAAE,OAAO,GAAG,GAAG,IAAI,eAAe,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC"}

package/dist/index.js

@@ -0,0 +1,78 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.createSqsBatchHandler = createSqsBatchHandler;
+function unwrapContext(obj) {
+    if (!obj || typeof obj !== 'object') {
+        return obj;
+    }
+    const clean = {};
+    for (const key of Object.keys(obj)) {
+        if (key.startsWith('$')) {
+            continue;
+        }
+        const val = typeof obj[key] === 'function' ? (obj[key].val !== undefined ? obj[key].val : (obj[key].length === 0 ? obj[key]() : obj[key])) : obj[key];
+        clean[key] = val;
+    }
+    return clean;
+}
+/**
+ * Creates an AWS Lambda compatible SQS batch message handler for rivulet workflows.
+ */
+function createSqsBatchHandler(workflow, options = {}) {
+    return async (event) => {
+        const batchItemFailures = [];
+        let processedCount = 0;
+        for (const record of event.Records || []) {
+            const idempKey = `sqs:${record.messageId}`;
+            if (options.idempotencyStore) {
+                const status = await options.idempotencyStore.claim(idempKey, options.idempotencyTtlMs);
+                if (status === 'completed') {
+                    processedCount++;
+                    continue;
+                }
+            }
+            try {
+                let parsedInput = {};
+                if (options.input) {
+                    parsedInput = await options.input(record);
+                }
+                else if (record.body) {
+                    try {
+                        parsedInput = JSON.parse(record.body);
+                    }
+                    catch {
+                        parsedInput = { body: record.body };
+                    }
+                }
+                let namespace;
+                if (typeof options.namespace === 'function') {
+                    namespace = options.namespace(record);
+                }
+                else if (options.namespace) {
+                    namespace = options.namespace;
+                }
+                else if (record.messageAttributes?.namespace?.stringValue) {
+                    namespace = record.messageAttributes.namespace.stringValue;
+                }
+                const execOptions = {
+                    timeout: options.timeout,
+                    namespace,
+                };
+                const rawResult = await workflow.run(parsedInput, execOptions);
+                const cleanResult = unwrapContext(rawResult);
+                if (options.idempotencyStore) {
+                    await options.idempotencyStore.complete(idempKey, cleanResult);
+                }
+                processedCount++;
+            }
+            catch (err) {
+                if (options.idempotencyStore) {
+                    await options.idempotencyStore.fail(idempKey, err);
+                }
+                batchItemFailures.push({ itemIdentifier: record.messageId });
+            }
+        }
+        return { batchItemFailures, processedCount };
+    };
+}
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";;AA8CA,sDA+DC;AAtFD,SAAS,aAAa,CAAC,GAAQ;IAC3B,IAAI,CAAC,GAAG,IAAI,OAAO,GAAG,KAAK,QAAQ,EAAE,CAAC;QAClC,OAAO,GAAG,CAAC;IACf,CAAC;IACD,MAAM,KAAK,GAAwB,EAAE,CAAC;IACtC,KAAK,MAAM,GAAG,IAAI,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC;QACjC,IAAI,GAAG,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;YACtB,SAAS;QACb,CAAC;QACD,MAAM,GAAG,GAAG,OAAO,GAAG,CAAC,GAAG,CAAC,KAAK,UAAU,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,SAAS,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,MAAM,KAAK,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACtJ,KAAK,CAAC,GAAG,CAAC,GAAG,GAAG,CAAC;IACrB,CAAC;IACD,OAAO,KAAK,CAAC;AACjB,CAAC;AAOD;;GAEG;AACH,SAAgB,qBAAqB,CACjC,QAA+B,EAC/B,UAA4C,EAAE;IAE9C,OAAO,KAAK,EAAE,KAAoB,EAA6B,EAAE;QAC7D,MAAM,iBAAiB,GAAsC,EAAE,CAAC;QAChE,IAAI,cAAc,GAAG,CAAC,CAAC;QAEvB,KAAK,MAAM,MAAM,IAAI,KAAK,CAAC,OAAO,IAAI,EAAE,EAAE,CAAC;YACvC,MAAM,QAAQ,GAAG,OAAO,MAAM,CAAC,SAAS,EAAE,CAAC;YAE3C,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;gBAC3B,MAAM,MAAM,GAAG,MAAM,OAAO,CAAC,gBAAgB,CAAC,KAAK,CAAC,QAAQ,EAAE,OAAO,CAAC,gBAAgB,CAAC,CAAC;gBACxF,IAAI,MAAM,KAAK,WAAW,EAAE,CAAC;oBACzB,cAAc,EAAE,CAAC;oBACjB,SAAS;gBACb,CAAC;YACL,CAAC;YAED,IAAI,CAAC;gBACD,IAAI,WAAW,GAAQ,EAAE,CAAC;gBAC1B,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;oBAChB,WAAW,GAAG,MAAM,OAAO,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;gBAC9C,CAAC;qBAAM,IAAI,MAAM,CAAC,IAAI,EAAE,CAAC;oBACrB,IAAI,CAAC;wBACD,WAAW,GAAG,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,CAAC;oBAC1C,CAAC;oBAAC,MAAM,CAAC;wBACL,WAAW,GAAG,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC;oBACxC,CAAC;gBACL,CAAC;gBAED,IAAI,SAA6B,CAAC;gBAClC,IAAI,OAAO,OAAO,CAAC,SAAS,KAAK,UAAU,EAAE,CAAC;oBAC1C,SAAS,GAAG,OAAO,CAAC,SAAS,CAAC,MAAM,CAAC,CAAC;gBAC1C,CAAC;qBAAM,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC;oBAC3B,SAAS,GAAG,OAAO,CAAC,SAAS,CAAC;gBAClC,CAAC;qBAAM,IAAI,MAAM,CAAC,iBAAiB,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;oBAC1D,SAAS,GAAG,MAAM,CAAC,iBAAiB,CAAC,SAAS,CAAC,WAAW,CAAC;gBAC/D,CAAC;gBAED,MAAM,WAAW,GAAqB;oBAClC,OAAO,EAAE,OAAO,CAAC,OAAO;oBACxB,SAAS;iBACZ,CAAC;gBAEF,MAAM,SAAS,GAAQ,MAAM,QAAQ,CAAC,GAAG,CAAC,WAAW,EAAE,WAAW,CAAC,CAAC;gBACpE,MAAM,WAAW,GAAG,aAAa,CAAC,SAAS,CAAC,CAAC;gBAE7C,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBAC3B,MAAM,OAAO,CAAC,gBAAgB,CAAC,QAAQ,CAAC,QAAQ,EAAE,WAAW,CAAC,CAAC;gBACnE,CAAC;gBAED,cAAc,EAAE,CAAC;YACrB,CAAC;YAAC,OAAO,GAAQ,EAAE,CAAC;gBAChB,IAAI,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBAC3B,MAAM,OAAO,CAAC,gBAAgB,CAAC,IAAI,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC;gBACvD,CAAC;gBACD,iBAAiB,CAAC,IAAI,CAAC,EAAE,cAAc,EAAE,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC;YACjE,CAAC;QACL,CAAC;QAED,OAAO,EAAE,iBAAiB,EAAE,cAAc,EAAE,CAAC;IACjD,CAAC,CAAC;AACN,CAAC"}

package/package.json

@@ -1,6 +1,61 @@
 {
   "name": "@siglo/sqs",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+  "version": "1.0.0",
+  "description": "AWS SQS message batch processor and event consumer adapter for Siglo workflows",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "require": "./dist/index.js",
+      "default": "./dist/index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/"
+  },
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "prepublishOnly": "npm run build",
+    "test": "npm run build && mocha -r tsx 'src/**/*.test.ts'"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/jsoftgem/siglo.git",
+    "directory": "packages/sqs"
+  },
+  "author": "Jerico de Guzman",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/jsoftgem/siglo/issues"
+  },
+  "homepage": "https://github.com/jsoftgem/siglo#readme",
+  "dependencies": {
+    "@siglo/core": "^1.0.0"
+  },
+  "peerDependencies": {
+    "@aws-sdk/client-sqs": "^3.0.0"
+  },
+  "peerDependenciesMeta": {
+    "@aws-sdk/client-sqs": {
+      "optional": true
+    }
+  },
+  "devDependencies": {
+    "@types/chai": "^5.2.3",
+    "@types/mocha": "^10.0.10",
+    "@types/node": "^26.6.5",
+    "chai": "^3.5.0",
+    "mocha": "^12.0.3",
+    "tsx": "^4.23.15",
+    "typescript": "~5.7.0"
+  }
+}