worker-que 1.0.0

package/README.md

@@ -0,0 +1,414 @@
+# que-ts
+
+[![Test](https://github.com/Duke-Engineering/que-ts/actions/workflows/test.yml/badge.svg)](https://github.com/Duke-Engineering/que-ts/actions/workflows/test.yml)
+[![Coverage](https://github.com/Duke-Engineering/que-ts/actions/workflows/coverage.yml/badge.svg)](https://github.com/Duke-Engineering/que-ts/actions/workflows/coverage.yml)
+[![Security](https://github.com/Duke-Engineering/que-ts/actions/workflows/security.yml/badge.svg)](https://github.com/Duke-Engineering/que-ts/actions/workflows/security.yml)
+[![npm version](https://badge.fury.io/js/que-ts.svg)](https://badge.fury.io/js/que-ts)
+
+A TypeScript job queue library for PostgreSQL, compatible with Ruby Que and que-go implementations.
+
+## Features
+
+- **Cross-language compatibility**: Works with [Que (Ruby)](https://github.com/chanks/que) and [que-go](https://github.com/bgentry/que-go) job queues
+- **PostgreSQL advisory locks**: Reliable job processing with no duplicate execution
+- **TypeScript support**: Full type safety with comprehensive interfaces
+- **Retry logic**: Exponential backoff for failed jobs
+- **Multiple queues**: Support for named queues and priorities
+- **Transaction support**: Enqueue jobs within existing database transactions
+- **Web Dashboard**: Real-time monitoring and management UI (Express.js integration)
+- **SSL/TLS Support**: Secure connections with client certificates
+
+## Installation
+
+### From npm (when published)
+```bash
+npm install que-ts
+```
+
+### From GitHub (development)
+```bash
+npm install github:Duke-Engineering/que-ts#master
+```
+
+**Note**: When installing from GitHub, the package will automatically build from TypeScript source using the `prepare` script.
+
+### Troubleshooting GitHub Installation
+
+If you encounter "Cannot find module 'que-ts'" errors when installing from GitHub:
+
+1. **Check the installation completed successfully**:
+   ```bash
+   cd node_modules/que-ts
+   ls dist/  # Should show compiled JavaScript files
+   ```
+
+2. **Manual build if needed**:
+   ```bash
+   cd node_modules/que-ts
+   npm run build
+   ```
+
+3. **Verify installation**:
+   ```bash
+   cd node_modules/que-ts
+   node test-install.js
+   ```
+
+4. **Alternative: Use specific tag**:
+   ```bash
+   npm install github:Duke-Engineering/que-ts#v1.0.0
+   ```
+
+## Quick Start
+
+### Database Setup
+
+#### Option 1: Using Docker (Recommended for Development)
+
+```bash
+# Start PostgreSQL with Docker
+npm run docker:up
+
+# Run tests
+npm test
+
+# Stop when done
+npm run docker:down
+```
+
+For detailed Docker usage, see [DOCKER.md](DOCKER.md).
+
+#### Option 2: Manual Database Setup
+
+Create the required database table:
+
+```sql
+CREATE TABLE que_jobs (
+    priority    smallint    NOT NULL DEFAULT 100,
+    run_at      timestamptz NOT NULL DEFAULT now(),
+    job_id      bigserial   NOT NULL,
+    job_class   text        NOT NULL,
+    args        json        NOT NULL DEFAULT '[]'::json,
+    error_count integer     NOT NULL DEFAULT 0,
+    last_error  text,
+    queue       text        NOT NULL DEFAULT '',
+    PRIMARY KEY (queue, priority, run_at, job_id)
+);
+```
+
+### Basic Usage
+
+```typescript
+import { Client, Worker } from 'que-ts';
+
+// Create a client
+const client = new Client({
+  host: 'localhost',
+  port: 5432,
+  database: 'myapp',
+  user: 'postgres',
+  password: 'password'
+});
+
+// Enqueue jobs
+await client.enqueue('SendEmail', ['user@example.com', 'Welcome!']);
+
+await client.enqueue('ProcessPayment', [{ amount: 100, currency: 'USD' }], {
+  priority: 10,
+  runAt: new Date(Date.now() + 60000) // Run in 1 minute
+});
+
+// Create and start a worker
+const worker = new Worker({
+  host: 'localhost',
+  port: 5432,
+  database: 'myapp',
+  user: 'postgres',
+  password: 'password'
+});
+
+// Register job handlers
+worker.register('SendEmail', async (job) => {
+  const [email, message] = job.args;
+  console.log(`Sending email to ${email}: ${message}`);
+  // Email sending logic here
+});
+
+worker.register('ProcessPayment', async (job) => {
+  const paymentData = job.args[0];
+  console.log(`Processing payment of ${paymentData.amount} ${paymentData.currency}`);
+  // Payment processing logic here
+});
+
+// Start processing jobs
+await worker.work();
+
+// Graceful shutdown
+process.on('SIGINT', async () => {
+  await worker.shutdown();
+  await client.close();
+});
+```
+
+## Web Dashboard
+
+que-ts includes a beautiful, real-time web dashboard for monitoring and managing your job queue.
+
+```typescript
+import express from 'express';
+import { Pool } from 'pg';
+import { createDashboard } from 'que-ts/dashboard';
+
+const app = express();
+const pool = new Pool({ /* your config */ });
+
+// Mount dashboard at /admin/queue
+app.use('/admin/queue', createDashboard(pool, {
+  title: 'My App Queue',
+  refreshInterval: 3000,
+  auth: (req, res, next) => {
+    // Add your authentication logic
+    return req.isAuthenticated();
+  }
+}));
+
+app.listen(3000);
+// Visit http://localhost:3000/admin/queue
+```
+
+**Dashboard Features:**
+- 📊 Real-time statistics (total, ready, scheduled, failed jobs)
+- 📈 Visual analytics (charts by queue and job class)
+- 🔍 Advanced filtering (status, queue, job class)
+- 🔄 Job management (retry failed jobs, delete jobs)
+- 🔐 Built-in authentication support
+- 📱 Responsive design
+
+For complete dashboard documentation, see [DASHBOARD.md](DASHBOARD.md).
+
+## SSL Configuration
+
+For detailed SSL configuration including AWS RDS, Google Cloud SQL, Azure, and certificate setup, see [SSL.md](SSL.md).
+
+## API Reference
+
+### Client
+
+#### Constructor
+
+```typescript
+new Client(config?: ClientConfig)
+```
+
+#### Methods
+
+- `enqueue(jobClass: string, args?: any[], options?: EnqueueOptions): Promise<Job>`
+- `enqueueInTx(client: PoolClient, jobClass: string, args?: any[], options?: EnqueueOptions): Promise<Job>`
+- `lockJob(queue?: string): Promise<Job | null>`
+- `close(): Promise<void>`
+
+### Worker
+
+#### Constructor
+
+```typescript
+new Worker(clientConfig?: ClientConfig, options?: WorkerOptions)
+```
+
+#### Methods
+
+- `register(jobClass: string, workFunc: WorkFunction): void`
+- `work(): Promise<void>`
+- `workOne(): Promise<boolean>`
+- `shutdown(): Promise<void>`
+
+### Job
+
+#### Properties
+
+- `id: number`
+- `queue: string`
+- `priority: number`
+- `runAt: Date`
+- `jobClass: string`
+- `args: any[]`
+- `errorCount: number`
+- `lastError?: string`
+
+#### Methods
+
+- `done(): Promise<void>`
+- `delete(): Promise<void>`
+- `error(errorMessage: string): Promise<void>`
+
+## Configuration
+
+### ClientConfig
+
+```typescript
+interface ClientConfig {
+  connectionString?: string;
+  host?: string;
+  port?: number;
+  database?: string;
+  user?: string;
+  password?: string;
+  ssl?: boolean | SSLConfig;  // See SSL Configuration below
+  maxConnections?: number;
+}
+```
+
+### SSL Configuration
+
+que-ts supports SSL/TLS connections with client certificates:
+
+```typescript
+import { Client } from 'que-ts';
+import * as fs from 'fs';
+
+// Basic SSL
+const client = new Client({
+  host: 'your-database.com',
+  port: 5432,
+  database: 'mydb',
+  user: 'myuser',
+  password: 'mypassword',
+  ssl: {
+    rejectUnauthorized: true,
+  }
+});
+
+// SSL with client certificates
+const clientWithCerts = new Client({
+  host: 'your-database.com',
+  port: 5432,
+  database: 'mydb',
+  user: 'myuser',
+  password: 'mypassword',
+  ssl: {
+    rejectUnauthorized: true,
+    cert: fs.readFileSync('./certs/client-cert.pem'),
+    key: fs.readFileSync('./certs/client-key.pem'),
+    ca: fs.readFileSync('./certs/ca-cert.pem'),
+    passphrase: 'optional-key-passphrase', // For encrypted keys
+  }
+});
+```
+
+For detailed SSL configuration including AWS RDS, Google Cloud SQL, Azure, and certificate setup, see [SSL.md](SSL.md).
+
+### WorkerOptions
+
+```typescript
+interface WorkerOptions {
+  queue?: string;        // Queue name to process (default: '')
+  interval?: number;     // Polling interval in ms (default: 5000)
+  maxAttempts?: number;  // Max retry attempts (default: 5)
+}
+```
+
+### EnqueueOptions
+
+```typescript
+interface EnqueueOptions {
+  priority?: number;  // Job priority (lower = higher priority, default: 100)
+  runAt?: Date;       // When to run the job (default: now)
+  queue?: string;     // Queue name (default: '')
+}
+```
+
+## Error Handling and Retries
+
+Failed jobs are automatically retried with exponential backoff:
+
+- 1st retry: after 1 second
+- 2nd retry: after 16 seconds  
+- 3rd retry: after 81 seconds
+- 4th retry: after 256 seconds
+- etc.
+
+Jobs that exceed the maximum number of retries remain in the queue for manual inspection.
+
+## Queues and Priorities
+
+Jobs can be organized into named queues and assigned priorities:
+
+```typescript
+// High priority job in 'critical' queue
+await client.enqueue('ProcessPayment', [paymentData], {
+  queue: 'critical',
+  priority: 1
+});
+
+// Low priority job in 'background' queue
+await client.enqueue('SendNewsletter', [newsletterData], {
+  queue: 'background', 
+  priority: 500
+});
+
+// Worker processing only critical queue
+const criticalWorker = new Worker(config, { queue: 'critical' });
+```
+
+## Cross-Language Compatibility
+
+que-ts is designed to be fully compatible with:
+
+- **[Ruby Que](https://github.com/chanks/que)** - The original Ruby implementation
+- **[que-go](https://github.com/bgentry/que-go)** - Golang port (currently unmaintained)
+
+You can enqueue jobs in one language and process them in another, or run workers in multiple languages simultaneously.
+
+## Related Projects
+
+### Official Implementations
+- **[Que (Ruby)](https://github.com/chanks/que)** - The original and most mature implementation
+- **[que-go](https://github.com/bgentry/que-go)** - Go implementation (unmaintained, but stable)
+- **[que-ts](https://github.com/Duke-Engineering/que-ts)** - This TypeScript/Node.js implementation
+
+## Development
+
+### Using Docker (Recommended)
+
+```bash
+# Install dependencies
+npm install
+
+# Start PostgreSQL with Docker
+npm run docker:up
+
+# Run tests
+npm test
+
+# Run tests in watch mode
+npm run test:watch
+
+# Build
+npm run build
+
+# Lint
+npm run lint
+
+# Stop Docker containers
+npm run docker:down
+```
+
+### Docker Commands
+
+- `npm run docker:up` - Start PostgreSQL and Adminer
+- `npm run docker:down` - Stop containers  
+- `npm run docker:logs` - View PostgreSQL logs
+- `npm run docker:clean` - Remove containers and volumes
+- `npm run test:docker` - Full test cycle with Docker
+
+Access database admin at http://localhost:8080 (user: `que_user`, password: `que_password`)
+
+See [DOCKER.md](DOCKER.md) for detailed Docker documentation.
+
+### Manual Setup
+
+If you prefer not to use Docker, ensure PostgreSQL is running and create the database schema manually using `migrations/schema.sql`.
+
+## License
+
+MIT

package/SSL-QUICK-REFERENCE.md

@@ -0,0 +1,225 @@
+# SSL Quick Reference
+
+## Quick SSL Setup Examples
+
+### 1. Basic SSL (No Certificates)
+
+```typescript
+import { Client } from 'que-ts';
+
+const client = new Client({
+  host: 'db.example.com',
+  port: 5432,
+  database: 'mydb',
+  user: 'myuser',
+  password: 'mypass',
+  ssl: { rejectUnauthorized: true }
+});
+```
+
+### 2. SSL with Client Certificates
+
+```typescript
+import * as fs from 'fs';
+
+const client = new Client({
+  host: 'db.example.com',
+  port: 5432,
+  database: 'mydb',
+  user: 'myuser',
+  password: 'mypass',
+  ssl: {
+    rejectUnauthorized: true,
+    cert: fs.readFileSync('./client-cert.pem'),
+    key: fs.readFileSync('./client-key.pem'),
+    ca: fs.readFileSync('./ca-cert.pem'),
+  }
+});
+```
+
+### 3. Using Environment Variables
+
+```typescript
+// .env file:
+// DB_SSL_CERT=/path/to/client-cert.pem
+// DB_SSL_KEY=/path/to/client-key.pem
+// DB_SSL_CA=/path/to/ca-cert.pem
+
+import * as fs from 'fs';
+import * as dotenv from 'dotenv';
+dotenv.config();
+
+const client = new Client({
+  host: process.env.DB_HOST,
+  port: parseInt(process.env.DB_PORT || '5432'),
+  database: process.env.DB_NAME,
+  user: process.env.DB_USER,
+  password: process.env.DB_PASSWORD,
+  ssl: {
+    rejectUnauthorized: true,
+    cert: fs.readFileSync(process.env.DB_SSL_CERT!),
+    key: fs.readFileSync(process.env.DB_SSL_KEY!),
+    ca: fs.readFileSync(process.env.DB_SSL_CA!),
+  }
+});
+```
+
+## Cloud Providers
+
+### AWS RDS
+
+```typescript
+// Download CA bundle: https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem
+const client = new Client({
+  host: 'instance.region.rds.amazonaws.com',
+  ssl: {
+    rejectUnauthorized: true,
+    ca: fs.readFileSync('./rds-ca-bundle.pem'),
+  }
+});
+```
+
+### Google Cloud SQL
+
+```typescript
+const client = new Client({
+  host: 'instance-ip',
+  ssl: {
+    rejectUnauthorized: true,
+    ca: fs.readFileSync('./server-ca.pem'),
+    cert: fs.readFileSync('./client-cert.pem'),
+    key: fs.readFileSync('./client-key.pem'),
+  }
+});
+```
+
+### Azure PostgreSQL
+
+```typescript
+// Download cert: https://learn.microsoft.com/en-us/azure/postgresql/
+const client = new Client({
+  host: 'server.postgres.database.azure.com',
+  user: 'user@server',
+  ssl: {
+    rejectUnauthorized: true,
+    ca: fs.readFileSync('./BaltimoreCyberTrustRoot.crt.pem'),
+  }
+});
+```
+
+### Heroku Postgres
+
+```typescript
+const client = new Client({
+  connectionString: process.env.DATABASE_URL,
+  ssl: { rejectUnauthorized: false } // Heroku uses self-signed
+});
+```
+
+## Certificate File Permissions
+
+```bash
+chmod 600 client-key.pem      # Private key (read/write owner only)
+chmod 644 client-cert.pem     # Certificate (readable by all)
+chmod 644 ca-cert.pem          # CA certificate (readable by all)
+```
+
+## Generate Self-Signed Certificates (Dev/Testing)
+
+```bash
+# CA certificate
+openssl req -new -x509 -days 365 -nodes -text \
+  -out ca-cert.pem -keyout ca-key.pem
+
+# Client certificate and key
+openssl req -new -nodes -text \
+  -out client.csr -keyout client-key.pem
+
+openssl x509 -req -in client.csr -text -days 365 \
+  -CA ca-cert.pem -CAkey ca-key.pem -CAcreateserial \
+  -out client-cert.pem
+```
+
+## Troubleshooting
+
+| Error | Solution |
+|-------|----------|
+| "self signed certificate" | Add `ca` certificate or set `rejectUnauthorized: false` (dev only) |
+| "unable to verify first certificate" | Include complete certificate chain in `ca` |
+| "certificate has expired" | Renew certificates |
+| "ENOENT: no such file" | Check file paths are correct and files exist |
+
+## Test SSL Connection
+
+```typescript
+async function testConnection() {
+  const client = new Client({
+    host: 'your-host',
+    ssl: {
+      rejectUnauthorized: true,
+      cert: fs.readFileSync('./client-cert.pem'),
+      key: fs.readFileSync('./client-key.pem'),
+      ca: fs.readFileSync('./ca-cert.pem'),
+    }
+  });
+
+  try {
+    const job = await client.enqueue('Test', []);
+    console.log('✓ Connected! Job ID:', job.id);
+    await client.close();
+  } catch (err) {
+    console.error('✗ Failed:', err.message);
+  }
+}
+```
+
+## Security Checklist
+
+- [ ] Use `rejectUnauthorized: true` in production
+- [ ] Never commit certificates to git
+- [ ] Use environment variables for paths/secrets
+- [ ] Set correct file permissions (600 for keys)
+- [ ] Rotate certificates regularly
+- [ ] Use strong key passphrases
+- [ ] Add `*.pem`, `*.key`, `*.crt` to `.gitignore`
+
+## Complete Example
+
+```typescript
+// config.ts
+import { Client, SSLConfig } from 'que-ts';
+import * as fs from 'fs';
+
+export function createSecureClient(): Client {
+  const sslConfig: SSLConfig = {
+    rejectUnauthorized: true,
+    cert: fs.readFileSync(process.env.DB_SSL_CERT || './certs/client-cert.pem'),
+    key: fs.readFileSync(process.env.DB_SSL_KEY || './certs/client-key.pem'),
+    ca: fs.readFileSync(process.env.DB_SSL_CA || './certs/ca-cert.pem'),
+  };
+
+  return new Client({
+    host: process.env.DB_HOST || 'localhost',
+    port: parseInt(process.env.DB_PORT || '5432'),
+    database: process.env.DB_NAME || 'mydb',
+    user: process.env.DB_USER || 'postgres',
+    password: process.env.DB_PASSWORD,
+    ssl: process.env.NODE_ENV === 'production' ? sslConfig : false,
+    maxConnections: 10,
+  });
+}
+
+// usage.ts
+import { createSecureClient } from './config';
+
+const client = createSecureClient();
+
+async function main() {
+  await client.enqueue('SendEmail', ['user@example.com']);
+  await client.close();
+}
+```
+
+---
+
+For full documentation, see [SSL.md](SSL.md)