@sockethub/data-layer 1.0.0-alpha.3 → 1.0.0-alpha.5

package/API.md

@@ -0,0 +1,23 @@
+# API Documentation
+
+This package provides the data layer for Sockethub, including job queue management and secure credential storage.
+
+## Main Classes
+
+### JobQueue
+Redis-backed job queue for managing ActivityStreams message processing. Creates isolated queues per platform instance and session, providing reliable message delivery and processing coordination between Sockethub server and platform workers.
+
+### JobWorker  
+Worker for processing jobs from a Redis queue within platform child processes. Connects to the same queue as its corresponding JobQueue instance and processes jobs using a platform-specific handler function.
+
+### CredentialsStore
+Secure, encrypted storage for user credentials with session-based isolation. Provides automatic encryption/decryption of credential objects stored in Redis, ensuring that sensitive authentication data is never stored in plaintext.
+
+## Complete API Reference
+
+For detailed API documentation including method signatures, parameters, and examples, see the [generated TypeDoc documentation](./docs/index.html).
+
+## Related Documentation
+
+- [Queue Architecture](./QUEUE.md) - Conceptual overview of the queueing system
+- [Main Sockethub Documentation](https://github.com/sockethub/sockethub)

package/QUEUE.md

@@ -0,0 +1,130 @@
+# Sockethub Queue System
+
+The queue system is Sockethub's core infrastructure that enables reliable, scalable
+communication between web clients and protocol platforms.
+
+## Why a Queue System?
+
+Web applications need to communicate with network protocols (IRC, XMPP, RSS) that:
+
+- Have slow or unreliable connections
+- Require persistent state that browsers can't maintain  
+- Need to operate across multiple concurrent user sessions
+
+The queue system solves this by:
+
+- **Decoupling** web clients from protocol complexity
+- **Ensuring reliability** through message persistence
+- **Enabling concurrency** via process isolation
+- **Maintaining security** through encrypted message handling
+
+## Architecture Overview
+
+```
+┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
+│   Web Clients   │    │  Sockethub      │    │   Protocol      │
+│                 │    │  Queue System   │    │   Platforms     │
+│ • Browser Apps  │◄──►│                 │◄──►│                 │
+│ • Mobile Apps   │    │ • Job Queues    │    │ • IRC Client    │
+│ • Desktop Apps  │    │ • Message       │    │ • XMPP Client   │
+│ • APIs          │    │   Routing       │    │ • RSS Parser    │
+│                 │    │ • Session Mgmt  │    │ • Metadata      │
+└─────────────────┘    └─────────────────┘    └─────────────────┘
+```
+
+## Message Flow
+
+### 1. Web Request → Queue
+
+```
+Web App → Socket.IO → Sockethub Server → Redis Queue
+```
+
+- Client sends ActivityStreams message
+- Server validates and queues the request
+- Client gets immediate acknowledgment
+
+### 2. Queue → Platform Processing  
+
+```
+Redis Queue → Platform Worker → External Protocol → Response
+```
+
+- Platform worker picks up job
+- Processes protocol-specific logic in isolation
+- Connects to external services (IRC, XMPP, etc.)
+
+### 3. Response → Client
+
+```
+Platform Result → Queue Event → Sockethub Server → Socket.IO → Web App
+```
+
+- Results flow back through the queue
+- Real-time updates delivered via Socket.IO
+- Clients receive responses and ongoing events
+
+## Key Design Principles
+
+**Process Isolation**: Each protocol runs in its own child process
+
+- Crashed platforms don't affect others
+- Memory leaks and resource issues are contained
+- Different platforms can use different dependencies
+
+**Session-Based Queues**: Each client gets dedicated queue infrastructure  
+
+- Complete privacy between users
+- Natural load distribution
+- Session-specific configurations possible
+
+**Message Durability**: All messages persist until processed
+
+- Server restarts don't lose messages
+- Failed operations can be retried
+- Audit trail for debugging
+
+## Real-World Example
+
+A chat application connecting to both IRC and XMPP:
+
+```
+User message → Queue → [IRC Platform] + [XMPP Platform]
+                          ↓               ↓
+                     IRC Network    XMPP Network
+```
+
+**Benefits:**
+
+- Both protocols process concurrently
+- Slow IRC doesn't block XMPP messages  
+- Platform crashes only affect that protocol
+- Independent delivery confirmations
+
+## Operational Characteristics
+
+**Reliability**: Messages are never lost, even during:
+
+- Server restarts
+- Platform process crashes  
+- Temporary network failures
+- Redis downtime
+
+**Scalability**: Natural horizontal scaling:
+
+- Multiple server instances share Redis
+- Per-session queue isolation
+- No coordination between servers required
+
+**Performance**: Asynchronous design provides:
+
+- Immediate web request responses
+- Concurrent protocol processing
+- Efficient resource utilization
+
+## Related Documentation
+
+- [Data Layer Package README](README.md) - Package overview and usage
+- [Main Sockethub README](../../README.md) - Overall project documentation
+- [BullMQ Documentation](https://bullmq.io/) - Underlying queue library
+- [ActivityStreams Specification](https://www.w3.org/TR/activitystreams-core/) - Message format standard

package/README.md

@@ -0,0 +1,114 @@
+# @sockethub/data-layer
+
+Redis-based data storage and job queue infrastructure for Sockethub, providing reliable
+message processing and secure credential storage.
+
+## About
+
+The data layer package provides the core infrastructure components that enable
+Sockethub's distributed architecture:
+
+- **Job Queue System**: Redis-backed message queuing using BullMQ for reliable ActivityStreams processing
+- **Credentials Store**: Encrypted storage for user credentials and authentication data
+- **Session Management**: Per-client data isolation and security
+- **Redis Integration**: Connection management and health checking
+
+## Core Components
+
+### JobQueue & JobWorker
+
+Handles asynchronous message processing between Sockethub server and platform instances.
+The JobQueue manages job creation and queuing on the server side, while JobWorker
+processes jobs within platform child processes.
+
+### CredentialsStore
+
+Provides secure, encrypted storage for user credentials with automatic
+encryption/decryption and session-based isolation.
+
+## Key Features
+
+### Security
+
+- **End-to-end encryption** of all stored data using AES-256
+- **Session-based isolation** prevents credential cross-contamination
+- **Secure key management** with per-session secrets
+- **No plaintext storage** in Redis or logs
+
+### Reliability
+
+- **Message persistence** ensures no data loss during processing
+- **Automatic retries** for failed operations
+- **Connection pooling** with Redis for efficiency
+- **Graceful error handling** and recovery
+
+### Scalability
+
+- **Horizontal scaling** across multiple server instances
+- **Per-session queues** for natural load distribution
+- **Process isolation** for platform-specific processing
+- **Redis clustering** support for high availability
+
+## Configuration
+
+**Redis Configuration**: Requires a Redis URL in the format `redis://host:port` or `redis://user:pass@host:port`
+
+**Dependencies:**
+
+- Redis server (6.0+)
+- `@sockethub/crypto` for encryption
+- `@sockethub/schemas` for type definitions
+
+## API Documentation
+
+For detailed API documentation including method signatures, parameters, and examples,
+generate the API docs from JSDoc comments:
+
+```bash
+bun run doc
+```
+
+## Architecture Integration
+
+The data layer sits between Sockethub's main server and platform instances:
+
+```
+[Sockethub Server] ←→ [Data Layer] ←→ [Platform Instances]
+       ↑                   ↑                   ↑
+   Socket.IO          Redis Storage      Protocol Logic
+   Connections        Job Queues         (IRC, XMPP, etc.)
+```
+
+**Server Integration**: Creates job queues and credential stores per client session
+
+**Platform Integration**: Workers process jobs and access stored credentials securely
+
+**Redis Storage**: Centralized, persistent storage for all data layer operations
+
+## Error Handling
+
+The data layer provides comprehensive error handling:
+
+- **Connection failures**: Automatic retry with exponential backoff
+- **Encryption errors**: Clear error messages for key management issues  
+- **Job failures**: Failed jobs are logged and can be retried
+- **Resource cleanup**: Proper shutdown procedures prevent memory leaks
+
+## Performance Considerations
+
+**Memory Management:**
+
+- Completed jobs are automatically cleaned up
+- Connection pooling minimizes resource usage
+- Per-session isolation prevents memory leaks
+
+**Network Efficiency:**
+
+- Redis pipelining for batch operations
+- Compressed job payloads for large messages
+- Keep-alive connections reduce overhead
+
+## Related Documentation
+
+- [Main Sockethub README](../../README.md) - Overall project documentation
+- [Queue System Architecture](QUEUE.md) - Overview of the Queue system

package/package.json

@@ -1,21 +1,20 @@
 {
   "name": "@sockethub/data-layer",
   "description": "Storing and RPC of data for Sockethub",
-  "version": "1.0.0-alpha.3",
+  "version": "1.0.0-alpha.5",
+  "type": "module",
   "private": false,
   "author": "Nick Jennings <nick@silverbucket.net>",
   "license": "MIT",
-  "main": "dist/index.js",
-  "types": "dist/index.d.ts",
-  "files": [
-    "dist/",
-    "src/"
-  ],
+  "main": "src/index.ts",
+  "engines": {
+    "bun": ">=1.2"
+  },
   "keywords": [
     "sockethub",
     "messaging",
     "redis",
-    "data lyer",
+    "data layer",
     "rpc",
     "data store"
   ],
@@ -25,44 +24,24 @@
     "directory": "packages/data-layer"
   },
   "homepage": "https://github.com/sockethub/sockethub/tree/master/packages/data-layer",
-  "dependencies": {
-    "@sockethub/crypto": "^1.0.0-alpha.3",
-    "@sockethub/schemas": "^3.0.0-alpha.3",
-    "bull": "^4.0.0",
-    "debug": "^4.3.1",
-    "secure-store-redis": "^1.4.7"
-  },
   "scripts": {
-    "clean": "npx rimraf dist coverage",
-    "clean:deps": "npx rimraf node_modules",
-    "compliance": "yarn run lint && yarn run test && yarn run coverage",
-    "test": "c8 -x \"src/**/*.test.*\" mocha -r ts-node/register src/*.test.ts",
-    "coverage": "c8 check-coverage --statements 85 --branches 90 --functions 100 --lines 85",
-    "lint": "eslint \"**/*.ts\"",
-    "lint:fix": "eslint --fix \"**/*.ts\"",
-    "build": "tsc"
+    "doc": "typedoc --options typedoc.json"
   },
-  "engines": {
-    "node": ">= 14"
+  "dependencies": {
+    "@sockethub/crypto": "1.0.0-alpha.5",
+    "@sockethub/schemas": "3.0.0-alpha.5",
+    "bullmq": "5.66.4",
+    "debug": "4.3.4",
+    "ioredis": "5.3.2",
+    "secure-store-redis": "3.0.7"
   },
   "devDependencies": {
-    "@types/chai": "4.3.3",
-    "@types/debug": "4.1.7",
-    "@types/eslint": "8.4.6",
-    "@types/mocha": "9.1.1",
-    "@types/node": "17.0.13",
-    "@types/proxyquire": "1.3.28",
-    "@types/sinon": "10.0.13",
-    "@typescript-eslint/parser": "5.37.0",
-    "c8": "7.12.0",
-    "chai": "4.3.6",
-    "eslint": "8.23.1",
-    "eslint-cli": "1.1.1",
-    "mocha": "10.0.0",
-    "proxyquire": "2.1.3",
-    "sinon": "14.0.0",
-    "ts-node": "10.9.1",
-    "typescript": "4.8.3"
+    "@types/bun": "latest",
+    "@types/debug": "4.1.12",
+    "@types/sinon": "17.0.2",
+    "sinon": "17.0.1",
+    "typedoc": "^0.28.14",
+    "typedoc-plugin-markdown": "^4.9.0"
   },
-  "gitHead": "1fa39894aafb49dfea7a7425bdaf196325e81e5d"
+  "gitHead": "341ea9eeca6afd1442fe6e01457bc21d112b91a4"
 }

package/src/credentials-store.test.ts

@@ -1,138 +1,155 @@
-import proxyquire from 'proxyquire';
-import { expect } from 'chai';
-import * as sinon from 'sinon';
+import { beforeEach, describe, expect, it } from "bun:test";
+import * as sinon from "sinon";
 
-proxyquire.noPreserveCache();
-proxyquire.noCallThru();
+import { CredentialsStore } from "./credentials-store";
 
-describe('CredentialsStore', () => {
-  let credentialsStore, MockSecureStore, MockStoreGet, MockStoreSave, MockObjectHash;
-  beforeEach(() => {
-    MockStoreGet = sinon.stub().callsArgWith(1, undefined, 'credential foo');
-    MockStoreSave = sinon.stub().callsArgWith(2, undefined);
-    MockObjectHash = sinon.stub();
-    MockSecureStore = sinon.stub().returns({
-      get: MockStoreGet,
-      save: MockStoreSave,
+describe("CredentialsStore", () => {
+    let credentialsStore,
+        MockSecureStore,
+        MockStoreGet,
+        MockStoreSave,
+        MockObjectHash;
+    beforeEach(() => {
+        MockStoreGet = sinon.stub().returns("credential foo");
+        MockStoreSave = sinon.stub();
+        MockObjectHash = sinon.stub();
+        MockSecureStore = sinon.stub().returns({
+            get: MockStoreGet,
+            save: MockStoreSave,
+        });
+        class TestCredentialsStore extends CredentialsStore {
+            initCrypto() {
+                this.objectHash = MockObjectHash;
+            }
+            initSecureStore(secret, redisConfig) {
+                this.store = MockSecureStore({
+                    namespace: "foo",
+                    secret: secret,
+                    redis: redisConfig,
+                });
+            }
+        }
+        credentialsStore = new TestCredentialsStore(
+            "a parent id",
+            "a session id",
+            "a secret must be 32 chars and th",
+            { url: "redis config" },
+        );
     });
-    const StoreMod = proxyquire('./credentials-store', {
-      'secure-store-redis': MockSecureStore,
-      '@sockethub/crypto': {
-        objectHash: MockObjectHash
-      }
-    });
-    const CredentialsStore = StoreMod.default;
-    credentialsStore = new CredentialsStore('a parent id', 'a session id', 'a secret', 'redis config');
-  });
 
-  it('returns a valid CredentialsStore object', () => {
-    sinon.assert.calledOnce(MockSecureStore);
-    sinon.assert.calledWith(MockSecureStore, {
-      namespace: 'sockethub:data-layer:credentials-store:a parent id:a session id',
-      secret: 'a secret',
-      redis: 'redis config'
+    it("returns a valid CredentialsStore object", () => {
+        sinon.assert.calledOnce(MockSecureStore);
+        sinon.assert.calledWith(MockSecureStore, {
+            namespace: "foo",
+            secret: "a secret must be 32 chars and th",
+            redis: { url: "redis config" },
+        });
+        expect(typeof credentialsStore).toEqual("object");
+        expect(credentialsStore.uid).toEqual(
+            `sockethub:data-layer:credentials-store:a parent id:a session id`,
+        );
+        expect(typeof credentialsStore.get).toEqual("function");
+        expect(typeof credentialsStore.save).toEqual("function");
     });
-    expect(typeof credentialsStore).to.equal('object');
-    expect(credentialsStore.uid).to.equal(`sockethub:data-layer:credentials-store:a parent id:a session id`);
-    expect(typeof credentialsStore.get).to.equal('function');
-    expect(typeof credentialsStore.save).to.equal('function');
-  });
 
-  describe('get', () => {
-    it('handles correct params', async () => {
-      let res = await credentialsStore.get('an actor');
-      sinon.assert.calledOnce(MockStoreGet);
-      sinon.assert.calledWith(MockStoreGet, 'an actor');
-      sinon.assert.notCalled(MockObjectHash);
-      sinon.assert.notCalled(MockStoreSave);
-      expect(res).to.equal('credential foo');
-    });
+    describe("get", () => {
+        it("handles correct params", async () => {
+            const res = await credentialsStore.get("an actor");
+            sinon.assert.calledOnce(MockStoreGet);
+            sinon.assert.calledWith(MockStoreGet, "an actor");
+            sinon.assert.notCalled(MockObjectHash);
+            sinon.assert.notCalled(MockStoreSave);
+            expect(res).toEqual("credential foo");
+        });
 
-    it('handles no credentials found', async () => {
-      MockStoreGet.callsArgWith(1, undefined, undefined);
-      let res = await credentialsStore.get('an non-existent actor');
-      sinon.assert.calledOnce(MockStoreGet);
-      sinon.assert.calledWith(MockStoreGet, 'an non-existent actor');
-      sinon.assert.notCalled(MockObjectHash);
-      sinon.assert.notCalled(MockStoreSave);
-      expect(res).to.be.undefined;
-    });
+        it("handles no credentials found", async () => {
+            MockStoreGet.returns(undefined);
+            expect(async () => {
+                await credentialsStore.get("a non-existent actor");
+            }).toThrow("credentials not found for a non-existent actor");
+            sinon.assert.calledOnce(MockStoreGet);
+            sinon.assert.calledWith(MockStoreGet, "a non-existent actor");
+            sinon.assert.notCalled(MockObjectHash);
+            sinon.assert.notCalled(MockStoreSave);
+        });
 
-    it('handles an unexpected error', async () => {
-      MockStoreGet.callsArgWith(1, 'sumting bad happen', undefined);
-      let res;
-      try {
-        res = await credentialsStore.get('a problem actor');
-        throw new Error('should not reach this spot');
-      } catch (err) {
-        expect(err).to.equal('sumting bad happen');
-      }
-      sinon.assert.calledOnce(MockStoreGet);
-      sinon.assert.calledWith(MockStoreGet, 'a problem actor');
-      sinon.assert.notCalled(MockObjectHash);
-      sinon.assert.notCalled(MockStoreSave);
-      expect(res).to.be.undefined;
-    });
+        it("handles an unexpected error", async () => {
+            MockStoreGet.returns(undefined);
+            try {
+                await credentialsStore.get("a problem actor");
+                expect(false).toEqual(true);
+            } catch (err) {
+                expect(err.toString()).toEqual(
+                    "Error: credentials not found for a problem actor",
+                );
+            }
+            sinon.assert.calledOnce(MockStoreGet);
+            sinon.assert.calledWith(MockStoreGet, "a problem actor");
+            sinon.assert.notCalled(MockObjectHash);
+            sinon.assert.notCalled(MockStoreSave);
+        });
 
-    it('validates credentialsHash when provided', async () => {
-      MockObjectHash.returns('a credentialHash string');
-      MockStoreGet.callsArgWith(1, undefined, {
-        object: 'a credential'
-      });
-      let res = await credentialsStore.get('an actor', 'a credentialHash string');
-      sinon.assert.calledOnce(MockStoreGet);
-      sinon.assert.calledWith(MockStoreGet, 'an actor');
-      sinon.assert.calledOnce(MockObjectHash);
-      sinon.assert.calledWith(MockObjectHash, 'a credential');
-      sinon.assert.notCalled(MockStoreSave);
-      expect(res).to.eql({object: 'a credential'});
-    });
+        it("validates credentialsHash when provided", async () => {
+            MockObjectHash.returns("a credentialsHash string");
+            MockStoreGet.returns({
+                object: "a credential",
+            });
+            const res = await credentialsStore.get(
+                "an actor",
+                "a credentialsHash string",
+            );
+            sinon.assert.calledOnce(MockStoreGet);
+            sinon.assert.calledWith(MockStoreGet, "an actor");
+            sinon.assert.calledOnce(MockObjectHash);
+            sinon.assert.calledWith(MockObjectHash, "a credential");
+            sinon.assert.notCalled(MockStoreSave);
+            expect(res).toEqual({ object: "a credential" });
+        });
 
-    it('invalidates credentialsHash when provided', async () => {
-      MockObjectHash.returns('the original credentialHash string');
-      MockStoreGet.callsArgWith(1, undefined, {
-        object: 'a credential'
-      });
-      let res;
-      try {
-        res = await credentialsStore.get('an actor', 'a different credentialHash string');
-        throw new Error('should not reach this spot');
-      } catch (err) {
-        expect(err).to.equal('provided credentials do not match existing platform instance for actor an actor');
-      }
-      sinon.assert.calledOnce(MockStoreGet);
-      sinon.assert.calledWith(MockStoreGet, 'an actor');
-      sinon.assert.calledOnce(MockObjectHash);
-      sinon.assert.calledWith(MockObjectHash, 'a credential');
-      sinon.assert.notCalled(MockStoreSave);
-      expect(res).to.be.undefined;
+        it("invalidates credentialsHash when provided", async () => {
+            MockObjectHash.returns("the original credentialsHash string");
+            MockStoreGet.returns({
+                object: "a credential",
+            });
+            try {
+                expect(
+                    await credentialsStore.get(
+                        "an actor",
+                        "a different credentialsHash string",
+                    ),
+                ).toBeUndefined();
+                expect(false).toEqual(true);
+            } catch (err) {
+                expect(err.toString()).toEqual(
+                    "Error: invalid credentials for an actor",
+                );
+            }
+            sinon.assert.calledOnce(MockStoreGet);
+            sinon.assert.calledWith(MockStoreGet, "an actor");
+            sinon.assert.calledOnce(MockObjectHash);
+            sinon.assert.calledWith(MockObjectHash, "a credential");
+            sinon.assert.notCalled(MockStoreSave);
+        });
     });
-  });
 
-  describe('save', () => {
-    it('handles success', (done) => {
-      const creds = {foo:'bar'};
-      credentialsStore.save('an actor', creds, (err) => {
-        sinon.assert.calledOnce(MockStoreSave);
-        sinon.assert.calledWith(MockStoreSave, 'an actor', creds);
-        sinon.assert.notCalled(MockObjectHash);
-        sinon.assert.notCalled(MockStoreGet);
-        expect(err).to.be.undefined;
-        done();
-      });
-    });
+    describe("save", () => {
+        it("handles success", async () => {
+            const creds = { foo: "bar" };
+            await credentialsStore.save("an actor", creds);
+            sinon.assert.calledOnce(MockStoreSave);
+            sinon.assert.calledWith(MockStoreSave, "an actor", creds);
+            sinon.assert.notCalled(MockObjectHash);
+            sinon.assert.notCalled(MockStoreGet);
+        });
 
-    it('handles failure', (done) => {
-      const creds = {foo:'bar'};
-      MockStoreSave.callsArgWith(2, 'an error');
-      credentialsStore.save('an actor', creds, (err) => {
-        sinon.assert.calledOnce(MockStoreSave);
-        sinon.assert.calledWith(MockStoreSave, 'an actor', creds);
-        sinon.assert.notCalled(MockObjectHash);
-        sinon.assert.notCalled(MockStoreGet);
-        expect(err).to.equal('an error');
-        done();
-      });
+        it("handles failure", async () => {
+            const creds = { foo: "bar" };
+            MockStoreSave.returns(undefined);
+            await credentialsStore.save("an actor", creds);
+            sinon.assert.calledOnce(MockStoreSave);
+            sinon.assert.calledWith(MockStoreSave, "an actor", creds);
+            sinon.assert.notCalled(MockObjectHash);
+            sinon.assert.notCalled(MockStoreGet);
+        });
     });
-  });
 });