replicas-engine 0.1.0

package/README.md

@@ -0,0 +1,286 @@
+# Replicas Engine
+
+A lightweight REST API server that runs on each Replicas workspace, providing programmatic access to workspace functionality without requiring SSH access.
+
+## Overview
+
+The Replicas Engine is automatically installed on every workspace and runs as a systemd service. It exposes a simple HTTP API protected by a secret key, allowing the Replicas backend to interact with workspaces for operations like reading logs, checking status, and more.
+
+## Features
+
+- **Lightweight**: Built with Hono for minimal overhead
+- **Secure**: All endpoints (except health check) require authentication via secret header
+- **Automatic**: Installed and configured automatically on workspace creation
+- **Reliable**: Runs as a systemd service with automatic restart
+
+## Installation
+
+The engine is automatically installed during workspace initialization. For manual installation:
+
+```bash
+yarn global add replicas-engine
+```
+
+## Configuration
+
+The engine requires the following environment variables:
+
+- `PORT` - Port to listen on (default: 3737)
+- `REPLICAS_ENGINE_SECRET` - Secret key for authentication (required)
+- `NODE_ENV` - Environment mode (production/development)
+- `WORKSPACE_HOME` - Working directory for Codex (default: `/home/ubuntu`)
+
+### Running as a Service
+
+The engine automatically runs as a systemd service named `replicas-engine`. You can manage it using:
+
+```bash
+# Check status
+systemctl status replicas-engine
+
+# View logs
+journalctl -u replicas-engine -f
+
+# Restart service
+sudo systemctl restart replicas-engine
+
+# Stop service
+sudo systemctl stop replicas-engine
+```
+
+## API Endpoints
+
+### Health Check
+
+**GET /health**
+
+Public endpoint for health checks (no authentication required).
+
+```bash
+curl http://localhost:3737/health
+```
+
+Response:
+```json
+{
+  "status": "ok",
+  "timestamp": "2025-10-26T04:00:00.000Z"
+}
+```
+
+### Ping
+
+**GET /ping**
+
+Authenticated endpoint for testing connectivity.
+
+```bash
+curl http://localhost:3737/ping \
+  -H "X-Replicas-Engine-Secret: your-secret-here"
+```
+
+Response:
+```json
+{
+  "message": "pong",
+  "timestamp": "2025-10-26T04:00:00.000Z"
+}
+```
+
+---
+
+## Codex Integration
+
+The engine provides remote control of Codex coding agents running on the workspace.
+
+### Send Message (with Streaming)
+
+**POST /codex/send**
+
+Send a message to Codex and stream events back via Server-Sent Events (SSE).
+
+```bash
+curl -N http://localhost:3737/codex/send \
+  -H "X-Replicas-Engine-Secret: your-secret-here" \
+  -H "Content-Type: application/json" \
+  -d '{"message": "Add error handling to the login function"}'
+```
+
+**Request Body:**
+```json
+{
+  "message": "Your instruction to Codex"
+}
+```
+
+**Response (SSE Stream):**
+```
+event: thread.started
+data: {"thread_id":"019a1caa-bceb-7731-b607-e9d22c1933a5"}
+
+event: turn.started
+data: {}
+
+event: item.started
+data: {"item":{"id":"...","type":"agent_message","text":"..."}}
+
+event: item.completed
+data: {"item":{"id":"...","type":"agent_message","text":"I'll help add error handling..."}}
+
+event: turn.completed
+data: {"usage":{"input_tokens":100,"output_tokens":50,"cached_input_tokens":0}}
+
+event: done
+data: {}
+```
+
+**Event Types:**
+- `thread.started` - New thread created (includes thread_id)
+- `turn.started` - Turn begins
+- `item.started` - New item (message, command, file change, etc.)
+- `item.updated` - Item updated
+- `item.completed` - Item finished
+- `turn.completed` - Turn ends (includes token usage)
+- `error` - Error occurred
+- `done` - Stream completed
+
+### Get Conversation History
+
+**GET /codex/history**
+
+Retrieve the full conversation history from the JSONL session file.
+
+```bash
+curl http://localhost:3737/codex/history \
+  -H "X-Replicas-Engine-Secret: your-secret-here"
+```
+
+**Response:**
+```json
+{
+  "thread_id": "019a1caa-bceb-7731-b607-e9d22c1933a5",
+  "events": [
+    {
+      "timestamp": "2025-10-25T18:39:02.919Z",
+      "type": "session_meta",
+      "payload": {...}
+    },
+    {
+      "timestamp": "2025-10-25T18:52:11.923Z",
+      "type": "response_item",
+      "payload": {"type": "message", "role": "user", ...}
+    }
+  ]
+}
+```
+
+### Get Thread Status
+
+**GET /codex/status**
+
+Get information about the current Codex thread.
+
+```bash
+curl http://localhost:3737/codex/status \
+  -H "X-Replicas-Engine-Secret: your-secret-here"
+```
+
+**Response:**
+```json
+{
+  "has_active_thread": true,
+  "thread_id": "019a1caa-bceb-7731-b607-e9d22c1933a5",
+  "session_file": "/home/ubuntu/.codex/sessions/2025/10/25/rollout-2025-10-25T18:39:02.891Z-019a1caa.jsonl",
+  "working_directory": "/home/ubuntu"
+}
+```
+
+### Reset Thread
+
+**POST /codex/reset**
+
+Clear the current thread and start a fresh conversation.
+
+```bash
+curl -X POST http://localhost:3737/codex/reset \
+  -H "X-Replicas-Engine-Secret: your-secret-here"
+```
+
+**Response:**
+```json
+{
+  "message": "Thread reset successfully",
+  "success": true
+}
+```
+
+### Codex Usage Notes
+
+- **Single Session**: Currently supports one active thread per workspace
+- **Thread Continuity**: Subsequent calls to `/codex/send` continue the same conversation
+- **Persistence**: Threads are automatically saved to `~/.codex/sessions/`
+- **Working Directory**: Codex runs in `WORKSPACE_HOME` (default: `/home/ubuntu`)
+- **Git Repository**: Git check is skipped to allow running in any directory
+
+---
+
+## Authentication
+
+All endpoints (except `/health`) require the `X-Replicas-Engine-Secret` header:
+
+```
+X-Replicas-Engine-Secret: <workspace-engine-secret>
+```
+
+Requests without this header or with an invalid secret will receive a 401 Unauthorized response.
+
+## Development
+
+```bash
+# Install dependencies
+yarn install
+
+# Run in development mode
+yarn dev
+
+# Build for production
+yarn build
+
+# Run built version
+yarn start
+```
+
+## Security
+
+- The engine only listens on port 3737
+- All requests require a secret key
+- The secret is generated uniquely per workspace
+- The service runs as the workspace user (no root access)
+
+## Troubleshooting
+
+### Engine not responding
+
+1. Check if the service is running:
+   ```bash
+   systemctl status replicas-engine
+   ```
+
+2. Check the logs:
+   ```bash
+   journalctl -u replicas-engine -n 50
+   ```
+
+3. Verify the port is listening:
+   ```bash
+   sudo netstat -tlnp | grep 3737
+   ```
+
+### Authentication errors
+
+- Ensure you're using the correct `engine_secret` from the workspace record
+- Verify the `X-Replicas-Engine-Secret` header is being sent
+
+## License
+
+MIT

package/dist/index.js

@@ -0,0 +1,24 @@
+#!/usr/bin/env node
+import 'dotenv/config';
+import { serve } from '@hono/node-server';
+import { Hono } from 'hono';
+import { authMiddleware } from './middleware/auth.js';
+import ping from './routes/ping.js';
+import codex from './routes/codex.js';
+const app = new Hono();
+// Health check endpoint (no auth required)
+app.get('/health', (c) => {
+    return c.json({ status: 'ok', timestamp: new Date().toISOString() });
+});
+// Apply auth middleware to all routes below
+app.use('*', authMiddleware);
+// Protected routes
+app.route('/ping', ping);
+app.route('/codex', codex);
+const port = Number(process.env.PORT) || 3737;
+serve({
+    fetch: app.fetch,
+    port,
+}, (info) => {
+    console.log(`Replicas Engine running on port ${info.port}`);
+});

package/dist/middleware/auth.js

@@ -0,0 +1,14 @@
+export const authMiddleware = async (c, next) => {
+    const secret = c.req.header('X-Replicas-Engine-Secret');
+    const expectedSecret = process.env.REPLICAS_ENGINE_SECRET;
+    if (!expectedSecret) {
+        return c.json({ error: 'Server configuration error: REPLICAS_ENGINE_SECRET not set' }, 500);
+    }
+    if (!secret) {
+        return c.json({ error: 'Unauthorized: X-Replicas-Engine-Secret header required' }, 401);
+    }
+    if (secret !== expectedSecret) {
+        return c.json({ error: 'Unauthorized: Invalid secret' }, 401);
+    }
+    await next();
+};

package/dist/routes/codex.js

@@ -0,0 +1,112 @@
+import { Hono } from 'hono';
+import { stream } from 'hono/streaming';
+import { CodexManager } from '../services/codex-manager.js';
+const codex = new Hono();
+// Create a singleton instance of CodexManager
+const codexManager = new CodexManager();
+/**
+ * POST /codex/send
+ * Send a message to Codex and stream events back via Server-Sent Events (SSE)
+ */
+codex.post('/send', async (c) => {
+    try {
+        const body = await c.req.json();
+        const { message } = body;
+        if (!message || typeof message !== 'string') {
+            return c.json({ error: 'Message is required and must be a string' }, 400);
+        }
+        // Stream events using SSE
+        return stream(c, async (stream) => {
+            // Set SSE headers
+            stream.onAbort(() => {
+                console.log('Client aborted SSE connection');
+            });
+            try {
+                // Stream Codex events
+                for await (const event of codexManager.sendMessage(message)) {
+                    // Format as Server-Sent Event
+                    const sseData = `event: ${event.type}\ndata: ${JSON.stringify(event)}\n\n`;
+                    await stream.write(sseData);
+                }
+                // Send a final 'done' event to signal completion
+                await stream.write('event: done\ndata: {}\n\n');
+            }
+            catch (error) {
+                console.error('Error during Codex streaming:', error);
+                const errorData = `event: error\ndata: ${JSON.stringify({
+                    message: error instanceof Error ? error.message : 'Unknown error occurred',
+                })}\n\n`;
+                await stream.write(errorData);
+            }
+        });
+    }
+    catch (error) {
+        console.error('Error in /codex/send:', error);
+        return c.json({
+            error: 'Failed to process message',
+            details: error instanceof Error ? error.message : 'Unknown error',
+        }, 500);
+    }
+});
+/**
+ * GET /codex/history
+ * Get the conversation history from the JSONL session file
+ */
+codex.get('/history', async (c) => {
+    try {
+        const history = await codexManager.getHistory();
+        if (!history.thread_id) {
+            return c.json({
+                message: 'No active thread',
+                thread_id: null,
+                events: [],
+            });
+        }
+        return c.json(history);
+    }
+    catch (error) {
+        console.error('Error in /codex/history:', error);
+        return c.json({
+            error: 'Failed to retrieve history',
+            details: error instanceof Error ? error.message : 'Unknown error',
+        }, 500);
+    }
+});
+/**
+ * GET /codex/status
+ * Get current thread status and information
+ */
+codex.get('/status', async (c) => {
+    try {
+        const status = await codexManager.getStatus();
+        return c.json(status);
+    }
+    catch (error) {
+        console.error('Error in /codex/status:', error);
+        return c.json({
+            error: 'Failed to retrieve status',
+            details: error instanceof Error ? error.message : 'Unknown error',
+        }, 500);
+    }
+});
+/**
+ * POST /codex/reset
+ * Reset the current thread and start fresh
+ */
+codex.post('/reset', async (c) => {
+    try {
+        codexManager.reset();
+        return c.json({
+            message: 'Thread reset successfully',
+            success: true,
+        });
+    }
+    catch (error) {
+        console.error('Error in /codex/reset:', error);
+        return c.json({
+            error: 'Failed to reset thread',
+            details: error instanceof Error ? error.message : 'Unknown error',
+        }, 500);
+    }
+});
+export default codex;

package/dist/routes/ping.js

@@ -0,0 +1,9 @@
+import { Hono } from 'hono';
+const ping = new Hono();
+ping.get('/', (c) => {
+    return c.json({
+        message: 'pong',
+        timestamp: new Date().toISOString(),
+    });
+});
+export default ping;

package/dist/services/codex-manager.js

@@ -0,0 +1,103 @@
+import { Codex, Thread } from '@openai/codex-sdk';
+import { findSessionFile, readJSONL } from '../utils/jsonl-reader.js';
+/**
+ * Manages Codex thread lifecycle and session state
+ * Currently supports a single active thread per engine instance
+ */
+export class CodexManager {
+    codex;
+    currentThreadId = null;
+    currentThread = null;
+    workingDirectory;
+    constructor(workingDirectory) {
+        this.codex = new Codex();
+        this.workingDirectory = workingDirectory || process.env.WORKSPACE_HOME || process.env.HOME || '/home/ubuntu';
+    }
+    /**
+     * Send a message to Codex and stream events back
+     * Creates a new thread on first call, resumes existing thread on subsequent calls
+     */
+    async *sendMessage(message) {
+        // Initialize or resume thread
+        if (!this.currentThread) {
+            if (this.currentThreadId) {
+                console.log(`Resuming thread ${this.currentThreadId}`);
+                this.currentThread = this.codex.resumeThread(this.currentThreadId, {
+                    workingDirectory: this.workingDirectory,
+                    skipGitRepoCheck: true,
+                });
+            }
+            else {
+                console.log('Starting new thread');
+                this.currentThread = this.codex.startThread({
+                    workingDirectory: this.workingDirectory,
+                    skipGitRepoCheck: true,
+                });
+            }
+        }
+        // Stream events from Codex
+        const { events } = await this.currentThread.runStreamed(message);
+        for await (const event of events) {
+            // Capture thread ID when thread starts
+            if (event.type === 'thread.started') {
+                this.currentThreadId = event.thread_id;
+                console.log(`Thread started: ${this.currentThreadId}`);
+            }
+            yield event;
+        }
+    }
+    /**
+     * Get conversation history by reading the JSONL session file
+     */
+    async getHistory() {
+        if (!this.currentThreadId) {
+            return {
+                thread_id: null,
+                events: [],
+            };
+        }
+        const sessionFile = await findSessionFile(this.currentThreadId);
+        if (!sessionFile) {
+            console.warn(`Session file not found for thread ${this.currentThreadId}`);
+            return {
+                thread_id: this.currentThreadId,
+                events: [],
+            };
+        }
+        console.log(`Reading session file: ${sessionFile}`);
+        const events = await readJSONL(sessionFile);
+        return {
+            thread_id: this.currentThreadId,
+            events,
+        };
+    }
+    /**
+     * Get current thread status
+     */
+    async getStatus() {
+        let sessionFile = null;
+        if (this.currentThreadId) {
+            sessionFile = await findSessionFile(this.currentThreadId);
+        }
+        return {
+            has_active_thread: this.currentThreadId !== null,
+            thread_id: this.currentThreadId,
+            session_file: sessionFile,
+            working_directory: this.workingDirectory,
+        };
+    }
+    /**
+     * Reset the current thread (start fresh conversation)
+     */
+    reset() {
+        console.log('Resetting Codex thread');
+        this.currentThread = null;
+        this.currentThreadId = null;
+    }
+    /**
+     * Get the current thread ID
+     */
+    getThreadId() {
+        return this.currentThreadId;
+    }
+}

package/dist/utils/jsonl-reader.js

@@ -0,0 +1,127 @@
+import { readFile, readdir, stat } from 'fs/promises';
+import { join } from 'path';
+import { homedir } from 'os';
+/**
+ * Read and parse a JSONL file
+ */
+export async function readJSONL(filePath) {
+    try {
+        const content = await readFile(filePath, 'utf-8');
+        return content
+            .split('\n')
+            .filter((line) => line.trim())
+            .map((line) => {
+            try {
+                return JSON.parse(line);
+            }
+            catch (e) {
+                console.error(`Failed to parse JSONL line: ${line}`, e);
+                return null;
+            }
+        })
+            .filter((event) => event !== null);
+    }
+    catch (error) {
+        console.error(`Failed to read JSONL file ${filePath}:`, error);
+        return [];
+    }
+}
+/**
+ * Find the session file for a given thread ID
+ * Sessions are stored in ~/.codex/sessions/YYYY/MM/DD/rollout-*-{threadId}.jsonl
+ */
+export async function findSessionFile(threadId) {
+    const sessionsDir = join(homedir(), '.codex', 'sessions');
+    try {
+        // Get current date for searching
+        const now = new Date();
+        const year = now.getFullYear();
+        const month = String(now.getMonth() + 1).padStart(2, '0');
+        const day = String(now.getDate()).padStart(2, '0');
+        // Search in today's directory first
+        const todayDir = join(sessionsDir, String(year), month, day);
+        const file = await findFileInDirectory(todayDir, threadId);
+        if (file)
+            return file;
+        // If not found, search recent days (last 7 days)
+        for (let daysAgo = 1; daysAgo <= 7; daysAgo++) {
+            const date = new Date(now);
+            date.setDate(date.getDate() - daysAgo);
+            const searchYear = date.getFullYear();
+            const searchMonth = String(date.getMonth() + 1).padStart(2, '0');
+            const searchDay = String(date.getDate()).padStart(2, '0');
+            const searchDir = join(sessionsDir, String(searchYear), searchMonth, searchDay);
+            const file = await findFileInDirectory(searchDir, threadId);
+            if (file)
+                return file;
+        }
+        return null;
+    }
+    catch (error) {
+        console.error('Error finding session file:', error);
+        return null;
+    }
+}
+/**
+ * Search for a file containing the thread ID in a specific directory
+ */
+async function findFileInDirectory(directory, threadId) {
+    try {
+        const files = await readdir(directory);
+        for (const file of files) {
+            if (file.endsWith('.jsonl') && file.includes(threadId)) {
+                const fullPath = join(directory, file);
+                const stats = await stat(fullPath);
+                if (stats.isFile()) {
+                    return fullPath;
+                }
+            }
+        }
+        return null;
+    }
+    catch (error) {
+        // Directory might not exist, which is fine
+        return null;
+    }
+}
+/**
+ * Get the most recent session file path (for debugging/info)
+ */
+export async function getMostRecentSessionFile() {
+    const sessionsDir = join(homedir(), '.codex', 'sessions');
+    try {
+        const now = new Date();
+        // Search last 7 days for any session file
+        for (let daysAgo = 0; daysAgo <= 7; daysAgo++) {
+            const date = new Date(now);
+            date.setDate(date.getDate() - daysAgo);
+            const year = date.getFullYear();
+            const month = String(date.getMonth() + 1).padStart(2, '0');
+            const day = String(date.getDate()).padStart(2, '0');
+            const searchDir = join(sessionsDir, String(year), month, day);
+            try {
+                const files = await readdir(searchDir);
+                const jsonlFiles = files
+                    .filter((f) => f.endsWith('.jsonl'))
+                    .map((f) => join(searchDir, f));
+                if (jsonlFiles.length > 0) {
+                    // Return the most recent file based on modification time
+                    const stats = await Promise.all(jsonlFiles.map(async (f) => ({
+                        path: f,
+                        mtime: (await stat(f)).mtime,
+                    })));
+                    stats.sort((a, b) => b.mtime.getTime() - a.mtime.getTime());
+                    return stats[0].path;
+                }
+            }
+            catch {
+                continue;
+            }
+        }
+        return null;
+    }
+    catch (error) {
+        console.error('Error finding most recent session file:', error);
+        return null;
+    }
+}

package/package.json

@@ -0,0 +1,38 @@
+{
+  "name": "replicas-engine",
+  "version": "0.1.0",
+  "description": "Lightweight API server for Replicas workspaces",
+  "type": "module",
+  "main": "dist/index.js",
+  "bin": {
+    "replicas-engine": "./dist/index.js"
+  },
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "dev": "tsx watch src/index.ts",
+    "build": "tsc && node scripts/postbuild.js",
+    "start": "node dist/index.js",
+    "prepublishOnly": "yarn build"
+  },
+  "keywords": [
+    "replicas",
+    "workspace",
+    "api",
+    "hono"
+  ],
+  "author": "Replicas",
+  "license": "MIT",
+  "dependencies": {
+    "@hono/node-server": "^1.19.5",
+    "@openai/codex-sdk": "^0.50.0",
+    "dotenv": "^17.2.3",
+    "hono": "^4.10.3"
+  },
+  "devDependencies": {
+    "@types/node": "^20.11.17",
+    "tsx": "^4.7.1",
+    "typescript": "^5.8.3"
+  }
+}