@motiadev/adapter-redis-streams 0.8.2-beta.140-709523

package/CHANGELOG.md

@@ -0,0 +1,17 @@
+# Changelog
+
+## [0.1.0] - 2025-10-22
+
+### Added
+- Initial release of Redis streams adapter for Motia
+- Support for distributed stream management across multiple instances
+- Real-time pub/sub for stream updates
+- Support for querying with filtering, sorting, and pagination
+- Separate Redis clients for pub/sub operations
+- Efficient SCAN operations for data retrieval
+- Type-safe generic implementation
+- Configurable key prefixes
+- Automatic reconnection handling
+- Full TypeScript support with type definitions
+- Comprehensive documentation and examples
+

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Motia
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

package/README.md

@@ -0,0 +1,168 @@
+# @motiadev/adapter-redis-streams
+
+Redis streams adapter for Motia framework, enabling distributed stream management with real-time updates across multiple instances.
+
+## Installation
+
+```bash
+npm install @motiadev/adapter-redis-streams
+```
+
+## Usage
+
+Configure the Redis streams adapter in your `motia.config.ts`:
+
+```typescript
+import { config } from '@motiadev/core'
+import { RedisStreamAdapter } from '@motiadev/adapter-redis-streams'
+
+export default config({
+  adapters: {
+    streams: new RedisStreamAdapter({
+      host: process.env.REDIS_HOST || 'localhost',
+      port: parseInt(process.env.REDIS_PORT || '6379'),
+      password: process.env.REDIS_PASSWORD,
+      keyPrefix: 'motia:stream:',
+    }),
+  },
+})
+```
+
+## Configuration Options
+
+### RedisStreamAdapterConfig
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `host` | `string` | `'localhost'` | Redis server host |
+| `port` | `number` | `6379` | Redis server port |
+| `password` | `string` | `undefined` | Redis authentication password |
+| `username` | `string` | `undefined` | Redis authentication username |
+| `database` | `number` | `0` | Redis database number |
+| `keyPrefix` | `string` | `'motia:stream:'` | Prefix for all stream keys |
+| `socket.reconnectStrategy` | `function` | Auto-retry | Custom reconnection strategy |
+| `socket.connectTimeout` | `number` | `10000` | Connection timeout in milliseconds |
+
+## Features
+
+- **Distributed Streams**: Share stream data across multiple Motia instances
+- **Real-time Updates**: Pub/sub for instant data synchronization
+- **Efficient Querying**: Support for filtering, sorting, and pagination
+- **Automatic Reconnection**: Handles connection failures gracefully
+- **Connection Pooling**: Separate clients for pub/sub operations
+- **Type Safety**: Full TypeScript support with generics
+
+## Key Namespacing
+
+The adapter uses the following patterns:
+- **Stream data**: `{keyPrefix}{groupId}:{id}`
+- **Events**: `motia:stream:events:{groupId}` or `motia:stream:events:{groupId}:{id}`
+
+For example:
+```
+motia:stream:users:user-123
+motia:stream:orders:order-456
+motia:stream:events:users
+motia:stream:events:users:user-123
+```
+
+## Example
+
+```typescript
+import { RedisStreamAdapter } from '@motiadev/adapter-redis-streams'
+
+const adapter = new RedisStreamAdapter({
+  host: 'redis.example.com',
+  port: 6379,
+  password: 'your-secure-password',
+  keyPrefix: 'myapp:stream:',
+  socket: {
+    connectTimeout: 5000,
+    reconnectStrategy: (retries) => {
+      if (retries > 20) {
+        return new Error('Too many retries')
+      }
+      return Math.min(retries * 50, 2000)
+    },
+  },
+})
+```
+
+## Real-time Updates
+
+The adapter uses Redis pub/sub to notify all instances about stream changes:
+
+```typescript
+await adapter.subscribe(
+  { groupId: 'users' },
+  async (event) => {
+    console.log('Stream event:', event.type, event.data)
+  }
+)
+
+await adapter.set('users', 'user-123', { name: 'John', email: 'john@example.com' })
+```
+
+## Querying
+
+The adapter supports flexible querying:
+
+```typescript
+const results = await adapter.query('users', {
+  where: { status: 'active' },
+  orderBy: 'createdAt',
+  orderDirection: 'desc',
+  limit: 10,
+  offset: 0,
+})
+```
+
+## Environment Variables
+
+```bash
+REDIS_HOST=localhost
+REDIS_PORT=6379
+REDIS_PASSWORD=your-password
+REDIS_DATABASE=0
+STREAM_KEY_PREFIX=motia:stream:
+```
+
+## Performance Considerations
+
+- **Pub/Sub Overhead**: Real-time updates add slight latency (~10-50ms)
+- **Key Scanning**: Uses SCAN to avoid blocking on large datasets
+- **Connection Pooling**: Maintains separate connections for pub/sub
+- **Batch Operations**: Uses multi-get for group operations
+- **Memory Usage**: Monitor Redis memory with large stream datasets
+
+## Troubleshooting
+
+### Connection Issues
+
+If you experience connection problems:
+1. Verify Redis is running and accessible
+2. Check your host, port, and credentials
+3. Ensure firewall rules allow connections
+4. Review Redis logs for errors
+
+### Subscription Issues
+
+If real-time updates aren't working:
+1. Verify pub/sub client is connected
+2. Check subscription handlers are registered
+3. Monitor Redis pub/sub channels with `PUBSUB CHANNELS`
+4. Review network latency between instances
+
+### Performance Issues
+
+If you experience slow operations:
+1. Monitor Redis server performance
+2. Check network latency
+3. Review query patterns and add appropriate filtering
+4. Consider Redis clustering for large datasets
+5. Monitor pub/sub message rates
+
+## License
+
+MIT
+

package/dist/index.d.ts

@@ -0,0 +1,3 @@
+export { RedisStreamAdapter } from './redis-stream-adapter';
+export type { RedisStreamAdapterConfig } from './types';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,wBAAwB,CAAA;AAC3D,YAAY,EAAE,wBAAwB,EAAE,MAAM,SAAS,CAAA"}

package/dist/index.js

@@ -0,0 +1,5 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.RedisStreamAdapter = void 0;
+var redis_stream_adapter_1 = require("./redis-stream-adapter");
+Object.defineProperty(exports, "RedisStreamAdapter", { enumerable: true, get: function () { return redis_stream_adapter_1.RedisStreamAdapter; } });

package/dist/redis-stream-adapter.d.ts

@@ -0,0 +1,28 @@
+import type { BaseStreamItem, StateStreamEvent, StateStreamEventChannel } from '@motiadev/core';
+import { StreamAdapter, type StreamQueryFilter } from '@motiadev/core';
+import type { RedisStreamAdapterConfig } from './types';
+export declare class RedisStreamAdapter<TData> extends StreamAdapter<TData> {
+    private client;
+    private pubClient;
+    private subClient?;
+    private keyPrefix;
+    private connected;
+    private subscriptions;
+    constructor(config: RedisStreamAdapterConfig);
+    private connect;
+    private ensureConnected;
+    private makeKey;
+    private makeChannelKey;
+    get(groupId: string, id: string): Promise<BaseStreamItem<TData> | null>;
+    set(groupId: string, id: string, data: TData): Promise<BaseStreamItem<TData>>;
+    delete(groupId: string, id: string): Promise<BaseStreamItem<TData> | null>;
+    getGroup(groupId: string): Promise<BaseStreamItem<TData>[]>;
+    send<T>(channel: StateStreamEventChannel, event: StateStreamEvent<T>): Promise<void>;
+    subscribe<T>(channel: StateStreamEventChannel, handler: (event: StateStreamEvent<T>) => void | Promise<void>): Promise<void>;
+    unsubscribe(channel: StateStreamEventChannel): Promise<void>;
+    clear(groupId: string): Promise<void>;
+    query(groupId: string, filter: StreamQueryFilter<TData>): Promise<BaseStreamItem<TData>[]>;
+    private scanKeys;
+    cleanup(): Promise<void>;
+}
+//# sourceMappingURL=redis-stream-adapter.d.ts.map

package/dist/redis-stream-adapter.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"redis-stream-adapter.d.ts","sourceRoot":"","sources":["../src/redis-stream-adapter.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,gBAAgB,EAAE,uBAAuB,EAAE,MAAM,gBAAgB,CAAA;AAC/F,OAAO,EAAE,aAAa,EAAE,KAAK,iBAAiB,EAAE,MAAM,gBAAgB,CAAA;AAEtE,OAAO,KAAK,EAAE,wBAAwB,EAAE,MAAM,SAAS,CAAA;AAEvD,qBAAa,kBAAkB,CAAC,KAAK,CAAE,SAAQ,aAAa,CAAC,KAAK,CAAC;IACjE,OAAO,CAAC,MAAM,CAAiB;IAC/B,OAAO,CAAC,SAAS,CAAiB;IAClC,OAAO,CAAC,SAAS,CAAC,CAAiB;IACnC,OAAO,CAAC,SAAS,CAAQ;IACzB,OAAO,CAAC,SAAS,CAAQ;IACzB,OAAO,CAAC,aAAa,CAAiF;gBAE1F,MAAM,EAAE,wBAAwB;YA8C9B,OAAO;YAWP,eAAe;IAM7B,OAAO,CAAC,OAAO;IAIf,OAAO,CAAC,cAAc;IAMhB,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC;IAOvE,GAAG,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,KAAK,GAAG,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,CAAC;IAW7E,MAAM,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC;IAW1E,QAAQ,CAAC,OAAO,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,EAAE,CAAC;IAW3D,IAAI,CAAC,CAAC,EAAE,OAAO,EAAE,uBAAuB,EAAE,KAAK,EAAE,gBAAgB,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAMpF,SAAS,CAAC,CAAC,EACf,OAAO,EAAE,uBAAuB,EAChC,OAAO,EAAE,CAAC,KAAK,EAAE,gBAAgB,CAAC,CAAC,CAAC,KAAK,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC,GAC5D,OAAO,CAAC,IAAI,CAAC;IAmCV,WAAW,CAAC,OAAO,EAAE,uBAAuB,GAAG,OAAO,CAAC,IAAI,CAAC;IAa5D,KAAK,CAAC,OAAO,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAUrC,KAAK,CAAC,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,iBAAiB,CAAC,KAAK,CAAC,GAAG,OAAO,CAAC,cAAc,CAAC,KAAK,CAAC,EAAE,CAAC;YA+BlF,QAAQ;IAgBhB,OAAO,IAAI,OAAO,CAAC,IAAI,CAAC;CAY/B"}

package/dist/redis-stream-adapter.js

@@ -0,0 +1,209 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.RedisStreamAdapter = void 0;
+const core_1 = require("@motiadev/core");
+const redis_1 = require("redis");
+class RedisStreamAdapter extends core_1.StreamAdapter {
+    constructor(config) {
+        super();
+        this.connected = false;
+        this.subscriptions = new Map();
+        this.keyPrefix = config.keyPrefix || 'motia:stream:';
+        const clientConfig = {
+            socket: {
+                host: config.host || 'localhost',
+                port: config.port || 6379,
+                reconnectStrategy: config.socket?.reconnectStrategy ||
+                    ((retries) => {
+                        if (retries > 10) {
+                            return new Error('Redis connection retry limit exceeded');
+                        }
+                        return Math.min(retries * 100, 3000);
+                    }),
+                connectTimeout: config.socket?.connectTimeout || 10000,
+            },
+            password: config.password,
+            username: config.username,
+            database: config.database || 0,
+        };
+        this.client = (0, redis_1.createClient)(clientConfig);
+        this.pubClient = (0, redis_1.createClient)(clientConfig);
+        this.client.on('error', (err) => {
+            console.error('[Redis Stream] Client error:', err);
+        });
+        this.client.on('connect', () => {
+            this.connected = true;
+        });
+        this.client.on('disconnect', () => {
+            console.warn('[Redis Stream] Disconnected');
+            this.connected = false;
+        });
+        this.pubClient.on('error', (err) => {
+            console.error('[Redis Stream] Pub client error:', err);
+        });
+        this.connect();
+    }
+    async connect() {
+        if (!this.connected) {
+            try {
+                await Promise.all([this.client.connect(), this.pubClient.connect()]);
+            }
+            catch (error) {
+                console.error('[Redis Stream] Failed to connect:', error);
+                throw error;
+            }
+        }
+    }
+    async ensureConnected() {
+        if (!this.client.isOpen || !this.pubClient.isOpen) {
+            await this.connect();
+        }
+    }
+    makeKey(groupId, id) {
+        return id ? `${this.keyPrefix}${groupId}:${id}` : `${this.keyPrefix}${groupId}`;
+    }
+    makeChannelKey(channel) {
+        return channel.id
+            ? `motia:stream:events:${channel.groupId}:${channel.id}`
+            : `motia:stream:events:${channel.groupId}`;
+    }
+    async get(groupId, id) {
+        await this.ensureConnected();
+        const key = this.makeKey(groupId, id);
+        const value = await this.client.get(key);
+        return value ? JSON.parse(value) : null;
+    }
+    async set(groupId, id, data) {
+        await this.ensureConnected();
+        const key = this.makeKey(groupId, id);
+        const item = { ...data, id };
+        await this.client.set(key, JSON.stringify(item));
+        await this.send({ groupId, id }, { type: 'update', data: item });
+        return item;
+    }
+    async delete(groupId, id) {
+        await this.ensureConnected();
+        const item = await this.get(groupId, id);
+        if (item) {
+            const key = this.makeKey(groupId, id);
+            await this.client.del(key);
+            await this.send({ groupId, id }, { type: 'delete', data: item });
+        }
+        return item;
+    }
+    async getGroup(groupId) {
+        await this.ensureConnected();
+        const pattern = `${this.makeKey(groupId)}:*`;
+        const keys = await this.scanKeys(pattern);
+        if (keys.length === 0)
+            return [];
+        const values = await this.client.mGet(keys);
+        return values.filter((v) => v !== null).map((v) => JSON.parse(v));
+    }
+    async send(channel, event) {
+        await this.ensureConnected();
+        const channelKey = this.makeChannelKey(channel);
+        await this.pubClient.publish(channelKey, JSON.stringify(event));
+    }
+    async subscribe(channel, handler) {
+        const channelKey = this.makeChannelKey(channel);
+        if (!this.subClient) {
+            const socketConfig = this.client.options?.socket;
+            const clientConfig = {
+                socket: {
+                    host: socketConfig?.host || 'localhost',
+                    port: socketConfig?.port || 6379,
+                },
+                password: this.client.options?.password,
+                username: this.client.options?.username,
+                database: this.client.options?.database || 0,
+            };
+            this.subClient = (0, redis_1.createClient)(clientConfig);
+            this.subClient.on('error', (err) => {
+                console.error('[Redis Stream] Sub client error:', err);
+            });
+            await this.subClient.connect();
+        }
+        this.subscriptions.set(channelKey, handler);
+        await this.subClient.subscribe(channelKey, async (message) => {
+            try {
+                const event = JSON.parse(message);
+                await handler(event);
+            }
+            catch (error) {
+                console.error('[Redis Stream] Error processing subscription message:', error);
+            }
+        });
+    }
+    async unsubscribe(channel) {
+        if (!this.subClient)
+            return;
+        const channelKey = this.makeChannelKey(channel);
+        this.subscriptions.delete(channelKey);
+        try {
+            await this.subClient.unsubscribe(channelKey);
+        }
+        catch (error) {
+            console.error('[Redis Stream] Error unsubscribing:', error);
+        }
+    }
+    async clear(groupId) {
+        await this.ensureConnected();
+        const pattern = `${this.makeKey(groupId)}:*`;
+        const keys = await this.scanKeys(pattern);
+        if (keys.length > 0) {
+            await this.client.del(keys);
+        }
+    }
+    async query(groupId, filter) {
+        let items = await this.getGroup(groupId);
+        if (filter.where) {
+            items = items.filter((item) => {
+                return Object.entries(filter.where).every(([key, value]) => {
+                    return item[key] === value;
+                });
+            });
+        }
+        if (filter.orderBy) {
+            items.sort((a, b) => {
+                const aVal = a[filter.orderBy];
+                const bVal = b[filter.orderBy];
+                const comparison = aVal < bVal ? -1 : aVal > bVal ? 1 : 0;
+                return filter.orderDirection === 'desc' ? -comparison : comparison;
+            });
+        }
+        if (filter.offset) {
+            items = items.slice(filter.offset);
+        }
+        if (filter.limit) {
+            items = items.slice(0, filter.limit);
+        }
+        return items;
+    }
+    async scanKeys(pattern) {
+        const keys = [];
+        let cursor = 0;
+        do {
+            const result = await this.client.scan(cursor, {
+                MATCH: pattern,
+                COUNT: 100,
+            });
+            cursor = result.cursor;
+            keys.push(...result.keys);
+        } while (cursor !== 0);
+        return keys;
+    }
+    async cleanup() {
+        if (this.subClient?.isOpen) {
+            await this.subClient.quit();
+        }
+        if (this.pubClient.isOpen) {
+            await this.pubClient.quit();
+        }
+        if (this.client.isOpen) {
+            await this.client.quit();
+        }
+        this.subscriptions.clear();
+    }
+}
+exports.RedisStreamAdapter = RedisStreamAdapter;

package/dist/types.d.ts

@@ -0,0 +1,13 @@
+export interface RedisStreamAdapterConfig {
+    host?: string;
+    port?: number;
+    password?: string;
+    username?: string;
+    database?: number;
+    keyPrefix?: string;
+    socket?: {
+        reconnectStrategy?: (retries: number) => number | Error;
+        connectTimeout?: number;
+    };
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,MAAM,WAAW,wBAAwB;IACvC,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,IAAI,CAAC,EAAE,MAAM,CAAA;IACb,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,QAAQ,CAAC,EAAE,MAAM,CAAA;IACjB,SAAS,CAAC,EAAE,MAAM,CAAA;IAClB,MAAM,CAAC,EAAE;QACP,iBAAiB,CAAC,EAAE,CAAC,OAAO,EAAE,MAAM,KAAK,MAAM,GAAG,KAAK,CAAA;QACvD,cAAc,CAAC,EAAE,MAAM,CAAA;KACxB,CAAA;CACF"}

package/dist/types.js

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/package.json

@@ -0,0 +1,23 @@
+{
+  "name": "@motiadev/adapter-redis-streams",
+  "description": "Redis streams adapter for Motia framework, enabling distributed stream management with real-time updates.",
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "version": "0.8.2-beta.140-709523",
+  "dependencies": {
+    "redis": "^4.7.0",
+    "@motiadev/core": "0.8.2-beta.140-709523"
+  },
+  "devDependencies": {
+    "@types/node": "^22.10.2",
+    "typescript": "^5.7.2"
+  },
+  "peerDependencies": {
+    "@motiadev/core": "^0.8.0"
+  },
+  "scripts": {
+    "build": "rm -rf dist && tsc",
+    "lint": "biome check .",
+    "watch": "tsc --watch"
+  }
+}