api-ape 3.0.2 → 4.1.0

package/server/lib/files.md

@@ -0,0 +1,111 @@
+# Server Lib Module Files
+
+This is the core implementation of api-ape's server functionality. It orchestrates everything needed to transform a standard HTTP server into a real-time WebSocket API server with automatic controller routing, client management, and message handling.
+
+## Guidelines
+
+- **Runtime abstraction** — All code must work across Node.js, Bun, and Deno; use `wsProvider.js` for detection
+- **No external dependencies** — Use the built-in WebSocket polyfill (`ws/`) instead of external packages
+- **Controller loading** — New controller loading logic should go in `loader.js`; maintain the file-path-to-endpoint mapping convention
+- **Client tracking** — Use `broadcast.js` for client registration; don't maintain separate client lists
+- **Binary transfers** — Coordinate with `fileTransfer.js` for all binary data; use the tag system consistently
+- **HTTP fallback** — Changes to WebSocket handling should have corresponding long-polling support
+
+## Directory Structure
+
+```
+lib/
+├── main.js              # Server initialization entry point
+├── loader.js            # Controller file auto-loader
+├── broadcast.js         # Client tracking & broadcast utilities
+├── bun.js               # Bun.serve() high-level integration
+├── fileTransfer.js      # Binary file transfer manager
+├── fileTransfer.test.js # File transfer test suite
+├── httpUtils.js         # Shared HTTP utilities
+├── longPolling.js       # HTTP streaming fallback coordinator
+├── wiring.js            # WebSocket connection lifecycle handler
+├── wsProvider.js        # Runtime detection & WebSocket provider selection
+├── fileTransfer/        # File transfer sub-modules
+├── longPolling/         # Long-polling sub-modules
+├── runtimes/            # Runtime-specific server initialization
+└── ws/                  # RFC 6455 WebSocket polyfill
+```
+
+## Files
+
+### `main.js`
+
+Main entry point for initializing api-ape on an HTTP server. Detects the runtime environment, creates core handlers, and delegates to the appropriate runtime-specific initializer.
+
+### `loader.js`
+
+Recursively loads controller files from a directory, mapping file paths to endpoint names:
+- `api/users.js` → `controllers['users']`
+- `api/users/profile.js` → `controllers['users/profile']`
+- `api/users/index.js` → `controllers['users']`
+
+### `broadcast.js`
+
+Manages connected clients and provides messaging utilities:
+- `broadcast(type, data)` — Send to all clients
+- `broadcastOthers(type, data, excludeClientId)` — Send to all except one
+- `publish(channel, data)` — Send to all subscribers of a channel
+- `subscribe(clientId, channel)` — Subscribe a client to a channel
+- `unsubscribe(clientId, channel)` — Unsubscribe a client from a channel
+- `clients` — Read-only Map of connected clients with `send()` method
+
+### `bun.js`
+
+High-level Bun integration that returns `fetch` and `websocket` handlers ready for `Bun.serve()`. Handles WebSocket upgrades, client bundle serving, and all api-ape routes.
+
+### `fileTransfer.js`
+
+Manages binary file uploads and downloads:
+- Registers pending uploads with timeout handling
+- Validates upload authorization via client session
+- Coordinates streaming file transfers between clients
+
+### `httpUtils.js`
+
+Shared HTTP utilities used across the server:
+- `matchRoute(path, pattern)` — URL pattern matching with parameter extraction
+- `sendJson(res, status, data)` — JSON response helper
+- `getCookie(headers, name)` — Cookie extraction
+- `isSecure(req)` / `isLocalhost(host)` — Security checks
+
+### `longPolling.js`
+
+Coordinates HTTP long-polling as a WebSocket fallback. Creates and manages the GET (streaming) and POST (messaging) handlers that share client state.
+
+### `wiring.js`
+
+Sets up WebSocket connection lifecycle:
+- Generates unique client IDs
+- Parses User-Agent for client info
+- Registers clients in broadcast system
+- Invokes `onConnect` callback with lifecycle hooks
+- Routes messages to controllers via `socket/receive.js`
+
+### `wsProvider.js`
+
+Detects runtime environment and returns the appropriate WebSocket implementation:
+1. Deno → Native `Deno.upgradeWebSocket()`
+2. Bun → Native Bun WebSocket
+3. Node.js 24+ → Native `node:ws` module
+4. Fallback → Built-in RFC 6455 polyfill
+
+### `fileTransfer/`
+
+File transfer sub-modules for streaming transfers. See [`fileTransfer/files.md`](./fileTransfer/files.md).
+
+### `longPolling/`
+
+HTTP long-polling handlers for GET (streaming) and POST (messaging). See [`longPolling/files.md`](./longPolling/files.md).
+
+### `runtimes/`
+
+Runtime-specific server initialization for Node.js and Bun. See [`runtimes/files.md`](./runtimes/files.md).
+
+### `ws/`
+
+RFC 6455 WebSocket polyfill and runtime adapters. See [`ws/files.md`](./ws/files.md).

package/server/lib/httpUtils.js

@@ -0,0 +1,283 @@
+/**
+ * @fileoverview Shared HTTP Utilities for api-ape Server
+ *
+ * This module provides common HTTP utility functions used across the api-ape
+ * server implementation. These utilities handle:
+ *
+ * - URL route matching with parameter extraction
+ * - JSON response formatting
+ * - Cookie parsing
+ * - Security checks (localhost detection, HTTPS validation)
+ * - Static file serving (client bundle and source maps)
+ *
+ * @module server/lib/httpUtils
+ * @see {@link module:server/lib/main} - Main server module using these utilities
+ * @see {@link module:server/lib/runtimes/node} - Node.js runtime using these utilities
+ *
+ * @example
+ * const { matchRoute, sendJson, getCookie, isSecure } = require('./httpUtils')
+ *
+ * // Match a route with parameters
+ * const params = matchRoute('/api/ape/data/abc123', '/api/ape/data/:hash')
+ * console.log(params) // { hash: 'abc123' }
+ *
+ * // Send JSON response
+ * sendJson(res, 200, { success: true })
+ *
+ * // Check security
+ * if (!isSecure(req) && !isLocalhost(req.headers.host)) {
+ *     sendJson(res, 403, { error: 'HTTPS required' })
+ * }
+ */
+
+const path = require("path");
+const fs = require("fs");
+
+/**
+ * Parses URL path parameters by matching against a pattern.
+ *
+ * Supports route patterns with colon-prefixed parameters like Express.js.
+ * Returns null if the path doesn't match the pattern.
+ *
+ * @function matchRoute
+ * @param {string} pathname - The actual URL pathname to match
+ * @param {string} pattern - The route pattern with parameters (e.g., '/api/:id')
+ * @returns {Object|null} Object with extracted parameters, or null if no match
+ *
+ * @example
+ * // Basic parameter extraction
+ * matchRoute('/api/users/123', '/api/users/:id')
+ * // Returns: { id: '123' }
+ *
+ * @example
+ * // Multiple parameters
+ * matchRoute('/api/users/123/posts/456', '/api/users/:userId/posts/:postId')
+ * // Returns: { userId: '123', postId: '456' }
+ *
+ * @example
+ * // No match (different segment count)
+ * matchRoute('/api/users', '/api/users/:id')
+ * // Returns: null
+ *
+ * @example
+ * // No match (different static segment)
+ * matchRoute('/api/posts/123', '/api/users/:id')
+ * // Returns: null
+ */
+function matchRoute(pathname, pattern) {
+  const patternParts = pattern.split("/");
+  const pathParts = pathname.split("/");
+
+  // Must have same number of segments
+  if (patternParts.length !== pathParts.length) return null;
+
+  const params = {};
+
+  for (let i = 0; i < patternParts.length; i++) {
+    if (patternParts[i].startsWith(":")) {
+      // Parameter segment - extract value
+      params[patternParts[i].slice(1)] = pathParts[i];
+    } else if (patternParts[i] !== pathParts[i]) {
+      /* istanbul ignore next - static segment mismatch path */
+      // Static segment doesn't match
+      return null;
+    }
+  }
+
+  return params;
+}
+
+/**
+ * Sends a JSON response with the specified status code.
+ *
+ * Sets the Content-Type header to application/json and stringifies the data.
+ *
+ * @function sendJson
+ * @param {http.ServerResponse} res - The HTTP response object
+ * @param {number} statusCode - HTTP status code (e.g., 200, 400, 500)
+ * @param {*} data - Data to JSON-stringify and send
+ *
+ * @example
+ * // Success response
+ * sendJson(res, 200, { users: [...] })
+ *
+ * @example
+ * // Error response
+ * sendJson(res, 404, { error: 'User not found' })
+ *
+ * @example
+ * // Validation error
+ * sendJson(res, 400, { error: 'Invalid input', fields: ['email'] })
+ */
+function sendJson(res, statusCode, data) {
+  res.writeHead(statusCode, { "Content-Type": "application/json" });
+  res.end(JSON.stringify(data));
+}
+
+/**
+ * Extracts a cookie value from request headers.
+ *
+ * Supports both Node.js style headers (object) and Fetch API style headers
+ * (Headers object with .get() method).
+ *
+ * @function getCookie
+ * @param {Object|Headers} headers - Request headers object
+ * @param {string} name - Cookie name to extract
+ * @returns {string|null} Cookie value, or null if not found
+ *
+ * @example
+ * // Node.js style headers
+ * const sessionId = getCookie(req.headers, 'sessionId')
+ *
+ * @example
+ * // Fetch API style headers
+ * const sessionId = getCookie(request.headers, 'sessionId')
+ *
+ * @example
+ * // Cookie not present
+ * const missing = getCookie(req.headers, 'nonexistent')
+ * // Returns: null
+ */
+function getCookie(headers, name) {
+  // Support both Node.js style and Fetch API Headers
+  const cookies =
+    typeof headers.get === "function" ? headers.get("cookie") : headers.cookie;
+
+  if (!cookies) return null;
+
+  // Match cookie name followed by = and capture value until ; or end
+  const match = cookies.match(new RegExp(`(?:^|;\\s*)${name}=([^;]*)`));
+  return match ? match[1] : null;
+}
+
+/**
+ * Checks if the host is localhost.
+ *
+ * Used to allow certain operations (like non-HTTPS file transfers)
+ * during local development.
+ *
+ * Recognized localhost values:
+ * - 'localhost'
+ * - '127.0.0.1'
+ * - '[::1]' (IPv6 localhost)
+ *
+ * @function isLocalhost
+ * @param {string} host - Host header value (may include port)
+ * @returns {boolean} True if the host is localhost
+ *
+ * @example
+ * isLocalhost('localhost:3000')     // true
+ * isLocalhost('127.0.0.1:8080')     // true
+ * isLocalhost('[::1]:3000')         // true
+ * isLocalhost('example.com')        // false
+ * isLocalhost('192.168.1.1:3000')   // false
+ */
+function isLocalhost(host) {
+  const hostname = host?.split(":")[0] || "";
+  return ["localhost", "127.0.0.1", "[::1]"].includes(hostname);
+}
+
+/**
+ * Checks if the request is using HTTPS (secure connection).
+ *
+ * Checks multiple indicators:
+ * 1. X-Forwarded-Proto header (for reverse proxies like nginx)
+ * 2. Socket encryption (direct HTTPS)
+ *
+ * Supports both Node.js request objects and Fetch API Request objects.
+ *
+ * @function isSecure
+ * @param {http.IncomingMessage|Request} req - The HTTP request object
+ * @returns {boolean} True if the connection is secure
+ *
+ * @example
+ * // Check if HTTPS is required
+ * if (!isSecure(req) && !isLocalhost(req.headers.host)) {
+ *     return sendJson(res, 403, { error: 'HTTPS required' })
+ * }
+ *
+ * @example
+ * // Behind a reverse proxy
+ * // Request with header: X-Forwarded-Proto: https
+ * isSecure(req) // true
+ */
+function isSecure(req) {
+  // Fetch API style (Headers object)
+  if (typeof req.headers?.get === "function") {
+    return req.headers.get("x-forwarded-proto") === "https";
+  }
+
+  // Node.js style - check socket encryption or X-Forwarded-Proto
+  return (
+    req.socket?.encrypted || req.headers?.["x-forwarded-proto"] === "https"
+  );
+}
+
+/**
+ * Serves the api-ape client JavaScript bundle.
+ *
+ * Reads the pre-built client bundle from the dist directory and sends it
+ * with the appropriate Content-Type header.
+ *
+ * @function serveClientBundle
+ * @param {string} clientPath - The request path (used for error logging)
+ * @param {http.ServerResponse} res - The HTTP response object
+ *
+ * @example
+ * // In request handler
+ * if (pathname === '/api/ape.js') {
+ *     return serveClientBundle('/api/ape.js', res)
+ * }
+ */
+/* istanbul ignore next 11 - client bundle serving only used by browser clients */
+function serveClientBundle(clientPath, res) {
+  const filePath = path.join(__dirname, "../../dist/ape.js");
+
+  fs.readFile(filePath, (err, data) => {
+    if (err) {
+      sendJson(res, 500, { error: "Failed to read client bundle" });
+      return;
+    }
+    res.writeHead(200, { "Content-Type": "application/javascript" });
+    res.end(data);
+  });
+}
+
+/**
+ * Serves the api-ape client source map.
+ *
+ * Reads the source map file from the dist directory for debugging support.
+ * Returns 404 if the source map doesn't exist.
+ *
+ * @function serveSourceMap
+ * @param {http.ServerResponse} res - The HTTP response object
+ *
+ * @example
+ * // In request handler
+ * if (pathname === '/api/ape.js.map') {
+ *     return serveSourceMap(res)
+ * }
+ */
+/* istanbul ignore next 11 - source map serving only used by browser clients */
+function serveSourceMap(res) {
+  const filePath = path.join(__dirname, "../../dist/ape.js.map");
+
+  fs.readFile(filePath, (err, data) => {
+    if (err) {
+      sendJson(res, 404, { error: "Source map not found" });
+      return;
+    }
+    res.writeHead(200, { "Content-Type": "application/json" });
+    res.end(data);
+  });
+}
+
+module.exports = {
+  matchRoute,
+  sendJson,
+  getCookie,
+  isLocalhost,
+  isSecure,
+  serveClientBundle,
+  serveSourceMap,
+};

package/server/lib/loader.js

@@ -1,10 +1,211 @@
-const deeprequire = require('../utils/deepRequire')
-const path = require('path')
+/**
+ * @fileoverview Controller Loader for api-ape Server
+ *
+ * This module provides the functionality to automatically load API controller
+ * files from a directory structure. Controllers are JavaScript files that
+ * export functions to handle specific API endpoints.
+ *
+ * ## How It Works
+ *
+ * The loader recursively scans a directory and loads all JavaScript files,
+ * mapping their file paths to endpoint names:
+ *
+ * ```
+ * Directory Structure:           Endpoint Mapping:
+ * ─────────────────────          ─────────────────
+ * api/
+ * ├── users.js            →     controllers['users']
+ * ├── users/
+ * │   ├── profile.js      →     controllers['users/profile']
+ * │   └── settings.js     →     controllers['users/settings']
+ * ├── chat.js             →     controllers['chat']
+ * └── admin/
+ *     └── dashboard.js    →     controllers['admin/dashboard']
+ * ```
+ *
+ * ## Controller File Format
+ *
+ * Each controller file should export a function (sync or async) that handles
+ * requests to that endpoint:
+ *
+ * ```javascript
+ * // api/users.js
+ * module.exports = async function(data) {
+ *   // `this` contains embed values from onConnect
+ *   const { userId, permissions } = this
+ *
+ *   // `data` is the payload sent by the client
+ *   const { action, query } = data
+ *
+ *   // Return value is sent back to client
+ *   return await db.users.find(query)
+ * }
+ * ```
+ *
+ * ## Index Files
+ *
+ * Files named `index.js` are mapped to their parent directory:
+ * - `api/users/index.js` → `controllers['users']`
+ *
+ * ## Duplicate Detection
+ *
+ * The loader detects duplicate endpoints and throws an error:
+ * - `api/users.js` and `api/users/index.js` would both map to 'users'
+ * - This is caught and reported with both file paths
+ *
+ * @module server/lib/loader
+ * @see {@link module:server/utils/deepRequire} for the underlying loader implementation
+ * @see {@link module:server/lib/main} for how controllers are used
+ *
+ * @example <caption>Basic Usage</caption>
+ * const loader = require('./loader')
+ *
+ * // Load all controllers from ./api directory
+ * const controllers = loader('api')
+ *
+ * // controllers = {
+ * //   'users': [Function],
+ * //   'users/profile': [Function],
+ * //   'chat': [Function],
+ * //   'admin/dashboard': [Function]
+ * // }
+ *
+ * @example <caption>Calling a Controller</caption>
+ * const controllers = loader('api')
+ *
+ * // This is how api-ape invokes controllers internally
+ * const handler = controllers['users']
+ * const context = { userId: 123, permissions: ['read'] }
+ * const result = await handler.call(context, { action: 'list' })
+ *
+ * @example <caption>Controller Implementation</caption>
+ * // api/messages.js
+ * module.exports = async function(data) {
+ *   // Available context via `this`:
+ *   // - this.clientId     - Unique client identifier
+ *   // - this.sessionId    - Session ID from cookies
+ *   // - this.req          - Original HTTP request (WebSocket upgrade)
+ *   // - this.send         - Function to send messages to this client
+ *   // - this.broadcast    - Function to broadcast to all clients
+ *   // - this.broadcastOthers - Broadcast excluding this client
+ *   // - this.clients      - Map of all connected clients
+ *   // - ...embed values   - Custom values from onConnect
+ *
+ *   const { roomId, text } = data
+ *
+ *   // Save to database
+ *   const message = await db.messages.create({
+ *     roomId,
+ *     text,
+ *     userId: this.userId,  // From embed
+ *     createdAt: new Date()
+ *   })
+ *
+ *   // Broadcast to other users in the room
+ *   this.broadcastOthers('new-message', {
+ *     roomId,
+ *     message
+ *   })
+ *
+ *   return message
+ * }
+ *
+ * @example <caption>Nested Routes</caption>
+ * // api/admin/users/ban.js → endpoint: 'admin/users/ban'
+ * module.exports = async function({ userId, reason }) {
+ *   // Check admin permissions
+ *   if (!this.permissions?.includes('admin')) {
+ *     throw new Error('Permission denied')
+ *   }
+ *
+ *   await db.users.ban(userId, reason)
+ *   return { success: true }
+ * }
+ */
 
-// Use the current working directory (where node was started)
-// This ensures 'where' folder is relative to the calling application
-const currentDir = process.cwd()
+const deeprequire = require("../utils/deepRequire");
+const path = require("path");
 
+/**
+ * Current working directory where Node.js was started
+ *
+ * This ensures that the 'where' folder path is resolved relative to
+ * the application root, not relative to this module's location.
+ *
+ * @constant {string}
+ * @private
+ */
+const currentDir = process.cwd();
+
+/**
+ * Load all controller files from a directory
+ *
+ * Recursively scans the specified directory for JavaScript files and
+ * loads them as controller functions. Each file's path (relative to the
+ * directory) becomes its endpoint name.
+ *
+ * ## Path Resolution
+ *
+ * The `dirname` parameter is resolved relative to `process.cwd()` (where
+ * Node.js was started), not relative to this module. This allows the
+ * calling application to specify paths relative to its own root.
+ *
+ * ## File Selection
+ *
+ * By default, all `.js` files are loaded. Use the optional `selector`
+ * parameter to customize which files are included.
+ *
+ * ## Error Handling
+ *
+ * - Throws if `dirname` doesn't exist
+ * - Throws if duplicate endpoints are detected (e.g., `users.js` and `users/index.js`)
+ * - Throws if a controller file has syntax errors
+ *
+ * @param {string} dirname - Directory name relative to current working directory
+ *                           (e.g., 'api', 'controllers', 'src/api')
+ * @param {string[]} [selector=['js']] - File extensions to include (without dots)
+ * @returns {Object.<string, Function>} Object mapping endpoint paths to controller functions
+ * @throws {Error} If directory doesn't exist or contains duplicate endpoints
+ *
+ * @example
+ * // Load from ./api directory
+ * const controllers = loader('api')
+ *
+ * @example
+ * // Load from nested directory
+ * const controllers = loader('src/api/v1')
+ *
+ * @example
+ * // Custom file extension selector
+ * const controllers = loader('api', ['js', 'mjs'])
+ *
+ * @example
+ * // Resulting controller object structure
+ * const controllers = loader('api')
+ * // {
+ * //   'users': function(data) { ... },
+ * //   'users/profile': function(data) { ... },
+ * //   'users/settings': function(data) { ... },
+ * //   'chat': function(data) { ... },
+ * //   'chat/rooms': function(data) { ... },
+ * //   'admin/stats': function(data) { ... }
+ * // }
+ *
+ * @example
+ * // Manual controller invocation (internal use)
+ * const controllers = loader('api')
+ *
+ * async function handleRequest(endpoint, data, context) {
+ *   const handler = controllers[endpoint]
+ *
+ *   if (!handler) {
+ *     throw new Error(`Endpoint not found: ${endpoint}`)
+ *   }
+ *
+ *   // Call with context bound to `this`
+ *   return await handler.call(context, data)
+ * }
+ */
 module.exports = function (dirname, selector) {
-  return deeprequire(path.join(currentDir, dirname), selector)
-}
+  return deeprequire(path.join(currentDir, dirname), selector);
+};

package/server/lib/longPolling/README.md

@@ -0,0 +1,63 @@
+# Long Polling Module
+
+## Overview
+
+The longPolling module provides HTTP-based fallback communication when WebSocket connections are unavailable. Many corporate networks, firewalls, and proxy servers block WebSocket connections, so api-ape automatically falls back to HTTP long-polling to maintain real-time bidirectional communication.
+
+**Key capabilities:**
+
+- **Streaming responses** — Server holds GET requests open and streams JSON events as they occur
+- **Message posting** — Client sends messages via POST requests with immediate responses
+- **Session management** — Tracks clients via `apeClientId` cookie across requests
+- **Heartbeat keepalive** — Prevents proxy timeouts with periodic heartbeat messages
+- **Connection recycling** — Automatically closes and reopens connections every ~25 seconds
+- **Broadcast integration** — Long-polling clients receive broadcasts just like WebSocket clients
+
+The transport is transparent to application code—controllers work identically regardless of whether clients connect via WebSocket or HTTP long-polling.
+
+> **Contributing?** See [`files.md`](./files.md) for directory structure and file descriptions.
+
+## How It Works
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                       Client                                │
+├─────────────────────────────────────────────────────────────┤
+│                                                             │
+│  GET /api/ape/poll ──────────────────► Streaming Response   │
+│  (reconnects every ~25s)                ◄── JSON events     │
+│                                         ◄── Heartbeats      │
+│                                                             │
+│  POST /api/ape/poll ─────────────────► Request/Response     │
+│  { type: '/users/list', data: {} }                          │
+│                                         ◄── { data: [...] } │
+│                                                             │
+└─────────────────────────────────────────────────────────────┘
+```
+
+## Response Headers
+
+The GET handler sets these headers for proper streaming:
+
+```
+Content-Type: application/json
+Cache-Control: no-cache
+Connection: keep-alive
+X-Accel-Buffering: no    # Disables nginx/proxy buffering
+```
+
+## When Is This Used?
+
+Long-polling activates automatically when:
+
+1. WebSocket connection fails to establish
+2. WebSocket upgrade is rejected by a proxy or firewall
+3. Network doesn't support the WebSocket protocol
+4. Client explicitly requests HTTP transport
+
+## See Also
+
+- [`../longPolling.js`](../longPolling.js) — Main long-polling coordinator
+- [`../wiring.js`](../wiring.js) — WebSocket wiring (primary transport)
+- [`../broadcast.js`](../broadcast.js) — Client tracking for long-polling clients
+- [`../../../client/transports/README.md`](../../../client/transports/README.md) — Client-side streaming transport