bulkhead-runtime 0.1.0 → 2026.4.5-beta.1

package/README.md

@@ -1,27 +1,14 @@
-<p align="center">
-  <br />
-  <img src="https://img.shields.io/badge/%E2%96%88%E2%96%88%E2%96%88_BULKHEAD-RUNTIME_%E2%96%88%E2%96%88%E2%96%88-000?style=for-the-badge&labelColor=0d1117&color=00ff41" alt="Bulkhead Runtime" />
-  <br /><br />
-  <strong>Watertight isolation for multi-tenant AI agents.</strong>
-  <br /><br />
-  <a href="https://www.npmjs.com/package/bulkhead-runtime"><img src="https://img.shields.io/npm/v/bulkhead-runtime?style=flat-square&color=00ff41&labelColor=0d1117" alt="npm" /></a>
-  <img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square&labelColor=0d1117" alt="MIT" />
-  <img src="https://img.shields.io/badge/node-%3E%3D22.12-brightgreen?style=flat-square&labelColor=0d1117&logo=node.js&logoColor=white" alt="node" />
-  <img src="https://img.shields.io/badge/runtime_deps-3-00ff41?style=flat-square&labelColor=0d1117" alt="deps" />
-  <img src="https://img.shields.io/badge/tests-88_passing-00ff41?style=flat-square&labelColor=0d1117" alt="tests" />
-  <img src="https://img.shields.io/badge/sandbox-5_isolation_layers-ff6b6b?style=flat-square&labelColor=0d1117" alt="isolation" />
-  <img src="https://img.shields.io/badge/crypto-AES--256--GCM-blueviolet?style=flat-square&labelColor=0d1117" alt="crypto" />
-</p>
-
-<br />
-
-<p align="center">
-  Run 1,000 AI agents on a single Linux box.<br />
-  Each in its own OS namespace. Each with private memory, encrypted credentials, and an isolated filesystem.<br />
-  <b>No Docker. No cloud. One <code>npm install</code>.</b>
-</p>
-
-<br />
+![Bulkhead Runtime](docs/assets/hero-banner.png)
+
+[![npm](https://img.shields.io/npm/v/bulkhead-runtime?style=flat-square&color=00ff41&labelColor=0d1117)](https://www.npmjs.com/package/bulkhead-runtime) ![MIT](https://img.shields.io/badge/license-MIT-blue?style=flat-square&labelColor=0d1117) ![node](https://img.shields.io/badge/node-%3E%3D22.12-brightgreen?style=flat-square&labelColor=0d1117&logo=node.js&logoColor=white) ![deps](https://img.shields.io/badge/runtime_deps-3-00ff41?style=flat-square&labelColor=0d1117) ![tests](https://img.shields.io/badge/tests-279_passing-00ff41?style=flat-square&labelColor=0d1117) ![isolation](https://img.shields.io/badge/sandbox-5_isolation_layers-ff6b6b?style=flat-square&labelColor=0d1117) ![crypto](https://img.shields.io/badge/crypto-AES--256--GCM-blueviolet?style=flat-square&labelColor=0d1117)
+
+**Run 1,000 AI agents on a single Linux box.**
+Each in its own OS namespace. Each with private memory, encrypted credentials, and an isolated filesystem.
+**No Docker. No cloud. One `npm install`.**
+
+*Built on production-hardened internals from [OpenClaw](https://github.com/nicepkg/openclaw) — failover, SSRF protection, embedding pipelines, session indexing, and more.*
+
+![Features](docs/assets/features.png)
 
 ---
 
@@ -36,10 +23,9 @@ import { createPlatform } from "bulkhead-runtime";
 
 const platform = createPlatform({
   stateDir: "/var/bulkhead-runtime",
-  credentialPassphrase: process.env.CREDENTIAL_KEY,
+  credentialPassphrase: process.env.BULKHEAD_CREDENTIAL_KEY,
 });
 
-// Create isolated workspaces — one per user, team, or tenant
 const workspace = await platform.createWorkspace("user-42", {
   provider: "anthropic",
   model: "claude-sonnet-4-20250514",
@@ -49,27 +35,27 @@ const result = await workspace.run({
   message: "Refactor the auth module to use JWT",
   sessionId: "project-alpha",
 });
-// Runs in a Linux namespace sandbox.
-// Private memory, encrypted credentials, isolated filesystem.
-// No other workspace can see this agent's data. Ever.
 ```
 
 > **Requires:** Linux + Node.js 22.12+
 >
 > **macOS / Windows dev:**
+>
 > ```bash
 > git clone https://github.com/tonga54/bulkhead-runtime.git && cd bulkhead-runtime
 > docker compose run dev bash
-> pnpm test  # 88 tests, all green
+> pnpm test  # 279 tests, all green
 > ```
 
 ---
 
 ## Use Cases
 
-<table>
-<tr>
-<td width="50%">
+
+|     |
+| --- |
+|     |
+
 
 **One agent per customer in your SaaS**
 
@@ -86,9 +72,6 @@ app.post("/api/agent", async (req, res) => {
 });
 ```
 
-</td>
-<td width="50%">
-
 **Per-team agents inside your company**
 
 Engineering, ops, and data each get their own agent. Each team's agent connects to their own tools — different GitHub orgs, different databases, different cloud accounts. No credential leaks between teams.
@@ -107,11 +90,6 @@ await ops.credentials.store("pagerduty", { token: "pd_..." });
 await data.credentials.store("gcp", { key: "..." });
 ```
 
-</td>
-</tr>
-<tr>
-<td width="50%">
-
 **Client-isolated agents in consulting / agencies**
 
 Each client project gets its own workspace. The agent knows that client's stack, their conventions, their infra. When you offboard a client, `deleteWorkspace()` wipes everything — memory, credentials, sessions.
@@ -126,13 +104,9 @@ await acme.run({
   sessionId: "daily-ops",
 });
 
-// Client offboarded — clean removal
 await platform.deleteWorkspace("client-acme");
 ```
 
-</td>
-<td width="50%">
-
 **Ephemeral agents for CI / PR review / task runners**
 
 Spin up a workspace per job, per PR, or per deploy. The agent runs, does its thing, and the workspace is destroyed. No state leaks between runs.
@@ -151,53 +125,13 @@ const result = await ws.run({
 await platform.deleteWorkspace(jobId);
 ```
 
-</td>
-</tr>
-</table>
-
 **The common thread:** you have multiple tenants (users, teams, clients, jobs) and each one needs an AI agent with its own secrets, tools, and memory — on the same server, without any cross-contamination.
 
 ---
 
 ## How It Works
 
-```mermaid
-graph TB
-  subgraph HOST["HOST PROCESS"]
-    subgraph WA["Workspace A"]
-      WA_mem["memory.db"]
-      WA_cred["creds.enc"]
-      WA_sess["sessions/"]
-      WA_skill["skills[]"]
-    end
-    subgraph WB["Workspace B"]
-      WB_mem["memory.db"]
-      WB_cred["creds.enc"]
-      WB_sess["sessions/"]
-      WB_skill["skills[]"]
-    end
-    subgraph WN["Workspace N"]
-      WN_mem["memory.db"]
-      WN_cred["creds.enc"]
-      WN_sess["sessions/"]
-      WN_skill["skills[]"]
-    end
-
-    subgraph SA["Sandbox A — user ns · mount ns · pid ns · net ns · cgroup v2"]
-      AgentA["Agent A"]
-    end
-    subgraph SB["Sandbox B — user ns · mount ns · pid ns · net ns · cgroup v2"]
-      AgentB["Agent B"]
-    end
-    subgraph SN["Sandbox N — user ns · mount ns · pid ns · net ns · cgroup v2"]
-      AgentN["Agent N"]
-    end
-  end
-
-  WA -->|"JSON-RPC IPC"| SA
-  WB -->|"JSON-RPC IPC"| SB
-  WN -->|"JSON-RPC IPC"| SN
-```
+![Architecture](docs/assets/architecture.png)
 
 > Agent A cannot see Agent B's files, memory, credentials, or processes. Not by policy. **By kernel enforcement.**
 
@@ -205,17 +139,23 @@ graph TB
 
 ## Why Bulkhead Over Alternatives
 
-| | Docker per user | E2B / Cloud | **Bulkhead Runtime** |
-|:---|:---:|:---:|:---:|
-| **Isolation mechanism** | Container per user | Cloud VM per session | **Linux namespaces** |
-| **Credential security** | DIY | Not built-in | **AES-256-GCM, never exposed to agent** |
-| **Persistent memory** | DIY | DIY | **SQLite + vector embeddings per tenant** |
-| **Skills with secret injection** | DIY | DIY | **Credentials injected server-side** |
-| **Per-workspace skill config** | DIY | DIY | **Enable/disable per tenant** |
-| **Infrastructure** | Docker daemon | Cloud API + billing | **Single npm package** |
-| **Cold start** | ~2s | ~5-10s | **~50ms** |
-| **Embeddable in your app** | No | No | **Yes — it's a library** |
-| **License** | — | Proprietary | **MIT** |
+
+|                                  | Docker per user    | E2B / Cloud          | **Bulkhead Runtime**                                      |
+| -------------------------------- | ------------------ | -------------------- | --------------------------------------------------------- |
+| **Isolation mechanism**          | Container per user | Cloud VM per session | **Linux namespaces**                                      |
+| **Credential security**          | DIY                | Not built-in         | **AES-256-GCM, never exposed to agent**                   |
+| **Persistent memory**            | DIY                | DIY                  | **SQLite + vector embeddings per tenant**                 |
+| **Embedding cache + batch**      | DIY                | DIY                  | **SQLite cache, batch with retry**                        |
+| **SSRF protection**              | DIY                | DIY                  | **DNS-resolve-and-pin, private-IP blocking, fail-closed** |
+| **Model fallback**               | DIY                | DIY                  | **Automatic multi-model fallback chains**                 |
+| **API key rotation**             | DIY                | DIY                  | **Multi-key with rate-limit rotation**                    |
+| **Parallel subagents**           | DIY                | DIY                  | **Built-in with depth limiting**                          |
+| **Skills with secret injection** | DIY                | DIY                  | **Credentials injected server-side**                      |
+| **Infrastructure**               | Docker daemon      | Cloud API + billing  | **Single npm package**                                    |
+| **Cold start**                   | ~2s                | ~5-10s               | **~50ms**                                                 |
+| **Embeddable in your app**       | No                 | No                   | **Yes — it's a library**                                  |
+| **License**                      | —                  | Proprietary          | **MIT**                                                   |
+
 
 ---
 
@@ -249,10 +189,9 @@ import { createPlatform } from "bulkhead-runtime";
 
 const platform = createPlatform({
   stateDir: "/var/bulkhead-runtime",
-  credentialPassphrase: process.env.CREDENTIAL_KEY,
+  credentialPassphrase: process.env.BULKHEAD_CREDENTIAL_KEY,
 });
 
-// Each workspace is a universe unto itself
 const alice = await platform.createWorkspace("alice", {
   provider: "anthropic",
   model: "claude-sonnet-4-20250514",
@@ -268,33 +207,7 @@ const bob = await platform.createWorkspace("bob", {
 
 When `workspace.run()` executes, Bulkhead spawns a **child process** with 5 layers of kernel isolation. The agent **never runs in your application's process**.
 
-```mermaid
-sequenceDiagram
-    participant App as Your App
-    participant Host as Host Process
-    participant Sandbox as Sandbox (isolated)
-
-    App->>Host: workspace.run({ message })
-    Host->>Host: Load session history
-    Host->>Host: Resolve enabled skills
-    Host->>Host: Detect sandbox capabilities
-    Host->>Sandbox: Spawn child process (unshare)
-    activate Sandbox
-    Note over Sandbox: Enter user namespace<br/>pivot_root → new filesystem<br/>PID namespace (own procs)<br/>Network ns (loopback only)<br/>cgroup v2 (mem/cpu/pid cap)
-    Sandbox->>Host: IPC: memory.search
-    Host-->>Sandbox: search results
-    Sandbox->>Host: IPC: memory.store
-    Host-->>Sandbox: store confirmation
-    Sandbox->>Host: IPC: skill.execute
-    Host->>Host: Decrypt credentials (AES-256-GCM)
-    Host-->>Sandbox: skill output (no credentials)
-    Sandbox-->>Host: Agent response
-    deactivate Sandbox
-    Host->>Host: Kill child process
-    Host->>Host: Fire lifecycle hooks
-    Host->>Host: Update session store
-    Host-->>App: { response, session }
-```
+![Execution Flow](docs/assets/execution-flow.png)
 
 The agent gets coding tools (bash, file read/write/edit) because the mount namespace restricts its entire filesystem view. **It literally cannot see anything outside its sandbox.**
 
@@ -305,33 +218,11 @@ The agent gets coding tools (bash, file read/write/edit) because the mount names
 Credentials are **AES-256-GCM encrypted** at rest. PBKDF2 key derivation with 100k iterations (SHA-512). The agent **never** sees raw secrets — not through tools, not through IPC, not through environment variables.
 
 ```typescript
-// Store encrypted credentials for Alice
 await alice.credentials.store("github", { token: "ghp_alice_secret" });
 await alice.credentials.store("openai", { apiKey: "sk-..." });
-
-// When the agent uses a skill that needs credentials:
-// 1. Host decrypts credentials server-side
-// 2. Injects them as env vars into the skill process
-// 3. Agent receives the skill's output — never the credential itself
 ```
 
-```mermaid
-sequenceDiagram
-    participant Agent as Agent (sandboxed)
-    participant Host as Host Process
-    participant Skill as Skill Script
-
-    Agent->>Host: skill_execute({ skillId: "github-issues" })
-    Host->>Host: Resolve skill directory
-    Host->>Host: Decrypt credentials (AES-256-GCM)
-    Host->>Skill: Spawn script with credentials as env vars
-    activate Skill
-    Skill->>Skill: process.env.token → use API
-    Skill-->>Host: stdout → JSON result
-    deactivate Skill
-    Host-->>Agent: Result string (no credentials)
-    Note over Agent: ✗ Never sees the token<br/>✗ Cannot read credential file<br/>✗ Cannot intercept env vars
-```
+![Credential Flow](docs/assets/credential-flow.png)
 
 System environment variables (`PATH`, `HOME`, `NODE_ENV`) are protected from credential key collision. Skill IDs are validated against prototype pollution.
 
@@ -342,24 +233,13 @@ System environment variables (`PATH`, `HOME`, `NODE_ENV`) are protected from cre
 Skills are registered globally and **enabled per workspace**. Each workspace gets exactly the capabilities it needs — nothing more.
 
 ```typescript
-// Register skills globally
-// skills/
-//   github-issues/
-//     SKILL.md         ← LLM reads this to understand the skill
-//     execute.js       ← Runs with credentials injected as env vars
-//   db-migration/
-//     SKILL.md
-//     execute.sh
-
-// Enable different skills per workspace
 const frontend = await platform.createWorkspace("team-frontend");
 const backend  = await platform.createWorkspace("team-backend");
 
 frontend.skills.enable("github-issues");
 backend.skills.enable("github-issues");
-backend.skills.enable("db-migration");   // only backend gets this
+backend.skills.enable("db-migration");
 
-// Store different credentials per workspace
 await frontend.credentials.store("github", { token: "ghp_frontend_token" });
 await backend.credentials.store("github", { token: "ghp_backend_token" });
 await backend.credentials.store("database", { url: "postgres://prod:5432/app" });
@@ -368,7 +248,7 @@ await backend.credentials.store("database", { url: "postgres://prod:5432/app" })
 ```javascript
 // skills/github-issues/execute.js
 const params = JSON.parse(await readStdin());
-const token = process.env.token;  // Injected from encrypted store — never over IPC
+const token = process.env.token;
 const res = await fetch(`https://api.github.com/repos/${params.repo}/issues`, {
   headers: { Authorization: `Bearer ${token}` },
 });
@@ -411,17 +291,19 @@ await runtime.run({
 // Agent searches memory → finds preferences → scaffolds TypeScript project
 ```
 
-**Search engine under the hood:**
+**Hybrid search engine under the hood:**
+
 
-| Stage | Algorithm |
-|:---|:---|
-| **Vector search** | Cosine similarity against stored embeddings |
-| **Keyword search** | SQLite FTS5 with BM25 ranking |
-| **Fusion** | Weighted merge of vector + keyword scores |
-| **Temporal decay** | Exponential time-based score attenuation |
-| **Diversity** | MMR (Maximal Marginal Relevance) re-ranking |
+| Stage               | Algorithm                                            |
+| ------------------- | ---------------------------------------------------- |
+| **Vector search**   | Cosine similarity against stored embeddings          |
+| **Keyword search**  | SQLite FTS5 with BM25 ranking                        |
+| **Fusion**          | Weighted merge of vector + keyword scores            |
+| **Temporal decay**  | Exponential time-based score attenuation             |
+| **Diversity**       | MMR (Maximal Marginal Relevance) re-ranking          |
 | **Query expansion** | 7-language keyword extraction (EN/ES/PT/ZH/JA/KO/AR) |
 
+
 Works without any embedding API key — falls back to FTS5 keyword search.
 
 ---
@@ -447,23 +329,34 @@ Sessions are per-workspace, stored as JSONL transcripts with async locking.
 
 ## Subagent Orchestration
 
-A tool can spawn a child agent. The parent blocks until the child finishes.
+Agents can spawn sub-agents for parallel or complex work. Sub-agents run concurrently with controlled parallelism and depth limiting.
 
 ```typescript
 const result = await runtime.run({
   message: "Review this PR for security and performance",
-  tools: [{
-    name: "specialist",
-    description: "Delegate a subtask to a specialist agent",
-    parameters: { task: { type: "string" }, role: { type: "string" } },
-    async execute(_id, params) {
-      const r = await runtime.run({
-        message: params.task,
-        systemPrompt: `You are a ${params.role} expert.`,
-      });
-      return { resultForAssistant: r.response };
-    },
-  }],
+  enableSubagents: true,
+});
+```
+
+Or programmatically:
+
+```typescript
+import { spawnSubagentsParallel } from "bulkhead-runtime";
+
+const results = await spawnSubagentsParallel({
+  tasks: [
+    { id: "security", task: "Audit for SQL injection and XSS", label: "Security" },
+    { id: "perf", task: "Profile hot paths and suggest optimizations", label: "Performance" },
+    { id: "docs", task: "Check all public APIs have JSDoc", label: "Documentation" },
+  ],
+  maxConcurrent: 3,
+  run: async (task) => {
+    const r = await runtime.run({
+      message: task.task,
+      systemPrompt: `You are a ${task.label} expert.`,
+    });
+    return r.response;
+  },
 });
 ```
 
@@ -485,47 +378,212 @@ workspace.hooks.register("after_agent_end", async ({ sessionId, result }) => {
 
 ---
 
+## Model Fallback & API Key Rotation
+
+When a model fails, Bulkhead automatically falls back to the next candidate. The error classification engine — ported from [OpenClaw](https://github.com/nicepkg/openclaw)'s battle-tested failover system — covers rate limits, billing, auth, overload, timeout, model not found, context overflow, and provider-specific patterns (Bedrock, Groq, Azure, Ollama, Mistral, etc.). Providers in cooldown are skipped automatically. When a key is rate-limited, it rotates to the next one.
+
+```typescript
+const result = await runtime.run({
+  message: "Analyze this codebase",
+  provider: "anthropic",
+  model: "claude-sonnet-4-20250514",
+
+  fallbacks: ["openai/gpt-4o", "google/gemini-2.5-flash"],
+
+  apiKeys: [process.env.ANTHROPIC_KEY_1!, process.env.ANTHROPIC_KEY_2!],
+});
+```
+
+Or set keys via environment variables:
+
+```bash
+ANTHROPIC_API_KEY=sk-ant-...
+ANTHROPIC_API_KEYS=sk-ant-key1,sk-ant-key2,sk-ant-key3
+ANTHROPIC_API_KEY_1=sk-ant-...
+ANTHROPIC_API_KEY_2=sk-ant-...
+```
+
+---
+
+## Context Window Guards
+
+Prevents silent failures from models with insufficient context windows.
+
+```typescript
+import { resolveContextWindowInfo, evaluateContextWindowGuard } from "bulkhead-runtime";
+
+const info = resolveContextWindowInfo({
+  modelContextWindow: model.contextWindow,
+  configContextTokens: 16_000,
+});
+
+const guard = evaluateContextWindowGuard({ info });
+// guard.shouldWarn  → true if below 32,000 tokens
+// guard.shouldBlock → true if below 16,000 tokens
+```
+
+The runtime applies these guards automatically before every agent execution.
+
+---
+
+## Retry with Compaction
+
+Transient errors (rate limits, timeouts, context overflow) trigger automatic retry with exponential backoff.
+
+```typescript
+const result = await runtime.run({
+  message: "Refactor the auth module",
+  maxRetries: 3,
+});
+// On context overflow → SDK compaction reduces history
+// On 429/5xx → exponential backoff + jitter
+```
+
+---
+
+## Embedding Pipeline
+
+### Embedding Cache
+
+Embeddings are cached in SQLite to avoid re-embedding unchanged content. Enabled by default.
+
+```typescript
+const memory = createSimpleMemoryManager({
+  dbDir: "/var/data/memory",
+  embeddingProvider: createEmbeddingProvider({ provider: "openai", apiKey: "..." }),
+  enableEmbeddingCache: true,
+  maxCacheEntries: 50_000,
+});
+```
+
+### Batch Embedding with Retry
+
+```typescript
+import { embedBatchWithRetry } from "bulkhead-runtime";
+
+const result = await embedBatchWithRetry(
+  ["text 1", "text 2", "text 3"],
+  {
+    provider: embeddingProvider,
+    cache: memory.embeddingCache ?? undefined,
+    batchSize: 100,
+    concurrency: 2,
+    retryAttempts: 3,
+  },
+);
+// result: { embeddings: [...], cached: 150, computed: 50, errors: 0 }
+```
+
+### SSRF Protection
+
+All embedding provider HTTP calls are protected against Server-Side Request Forgery by default. The SSRF engine — ported from [OpenClaw](https://github.com/nicepkg/openclaw) — resolves DNS and pins IPs before connecting, blocks private/link-local ranges, and enforces a hostname allowlist. Fail-closed by default.
+
+```typescript
+import { validateUrl, buildBaseUrlPolicy } from "bulkhead-runtime";
+
+const provider = createEmbeddingProvider({
+  provider: "openai",
+  apiKey: "...",
+  enableSsrf: true,
+});
+
+await validateUrl("https://api.openai.com/v1/embeddings"); // OK
+await validateUrl("http://169.254.1.1/steal");              // throws SSRF error
+```
+
+---
+
+## File-based Memory Indexing
+
+Automatically watches `MEMORY.md` and `memory/` directory for changes and re-indexes them into the memory system.
+
+```typescript
+import { createFileIndexer } from "bulkhead-runtime";
+
+const indexer = createFileIndexer({
+  workspaceDir: "/path/to/workspace",
+  memory,
+  watchPaths: ["docs/"],
+  debounceMs: 2000,
+});
+
+indexer.start();
+indexer.stop();
+```
+
+---
+
+## Session Transcript Indexing
+
+Indexes session transcripts into memory for cross-session search. Supports post-compaction re-indexing. Ported from [OpenClaw](https://github.com/nicepkg/openclaw)'s memory-core extension.
+
+```typescript
+import { createSessionIndexer } from "bulkhead-runtime";
+
+const indexer = createSessionIndexer({
+  sessionsDir: path.join(stateDir, "sessions"),
+  memory,
+  deltaBytes: 4096,
+  deltaMessages: 10,
+});
+
+await indexer.indexAllSessions();
+indexer.onTranscriptUpdate(sessionFile);
+```
+
+---
+
+## Structured Logging
+
+Structured JSON logging with file output, rotation, and configurable levels.
+
+```typescript
+import { configureLogger, createSubsystemLogger } from "bulkhead-runtime";
+
+configureLogger({
+  level: "debug",
+  file: "/var/log/bulkhead-runtime.log",
+  maxFileBytes: 10 * 1024 * 1024,
+  json: true,
+});
+
+const log = createSubsystemLogger("my-module");
+log.info("agent started", { userId: "alice", model: "claude-sonnet-4-20250514" });
+```
+
+Or via environment:
+
+```bash
+BULKHEAD_LOG_LEVEL=debug
+BULKHEAD_LOG_FILE=/var/log/bulkhead-runtime.log
+```
+
+---
+
 ## Security Architecture
 
 ### 5 Layers of Sandbox Isolation
 
 All layers are **fail-closed** — if any layer can't be applied, the sandbox refuses to start.
 
-```mermaid
-block-beta
-  columns 1
-  block:L1["Layer 1 — User Namespace · unprivileged creation via unshare(2)"]
-    columns 1
-    block:L2["Layer 2 — Mount Namespace · pivot_root → new root · old root unmounted"]
-      columns 1
-      block:L3["Layer 3 — PID Namespace · agent only sees its own processes"]
-        columns 1
-        block:L4["Layer 4 — Network Namespace · loopback only · no external access"]
-          columns 1
-          L5["Layer 5 — cgroups v2 · memory.max · pids.max · cpu.weight"]
-        end
-      end
-    end
-  end
-  AGENT["Agent process — can only see its own isolated world"]
-
-  L1 --> AGENT
-```
+![Security Layers](docs/assets/security-layers.png)
 
 ### Defense in Depth
 
-| Defense | Mechanism |
-|:---|:---|
-| **Env allowlist** | Only `PATH`, `HOME`, `NODE_ENV` + the single API key the agent needs. Everything else dropped. |
-| **Credential proxy** | Secrets decrypted server-side, injected into skill execution. Never sent over IPC. |
-| **Path traversal blocklist** | `/proc`, `/sys`, `/home/`, `/etc/shadow`, `/run/docker.sock`, and more are blocked from bind mounts. |
-| **Symlink rejection** | `additionalBinds` sources must not be symlinks (prevents TOCTOU attacks). |
-| **IPC rate limiting** | 200 calls/sec per method. Prevents resource exhaustion from rogue agents. |
-| **IPC buffer limit** | 50 MB max. Peer stops on overflow to prevent memory exhaustion. |
-| **Prototype pollution guard** | `__proto__`, `constructor`, `prototype` rejected as skill/credential IDs. |
-| **Stdout interception** | IPC uses a dedicated fd. All other stdout is redirected to stderr. |
-| **Sensitive path validation** | `workspaceDir`, `projectDir`, `nodeExecutable`, `additionalBinds` all validated. |
-| **Atomic writes** | Config, credentials, sessions, skill state — all use tmp+rename pattern. |
+
+| Defense                       | Mechanism                                                                                            |
+| ----------------------------- | ---------------------------------------------------------------------------------------------------- |
+| **Env allowlist**             | Only `PATH`, `HOME`, `NODE_ENV` + the single API key the agent needs. Everything else dropped.       |
+| **Credential proxy**          | Secrets decrypted server-side, injected into skill execution. Never sent over IPC.                   |
+| **Path traversal blocklist**  | `/proc`, `/sys`, `/home/`, `/etc/shadow`, `/run/docker.sock`, and more are blocked from bind mounts. |
+| **Symlink rejection**         | `additionalBinds` sources must not be symlinks (prevents TOCTOU attacks).                            |
+| **IPC rate limiting**         | 200 calls/sec per method. Prevents resource exhaustion from rogue agents.                            |
+| **IPC buffer limit**          | 50 MB max. Peer stops on overflow to prevent memory exhaustion.                                      |
+| **Prototype pollution guard** | `__proto__`, `constructor`, `prototype` rejected as skill/credential IDs.                            |
+| **Stdout interception**       | IPC uses a dedicated fd. All other stdout is redirected to stderr.                                   |
+| **Sensitive path validation** | `workspaceDir`, `projectDir`, `nodeExecutable`, `additionalBinds` all validated.                     |
+| **Atomic writes**             | Config, credentials, sessions, skill state — all use tmp+rename pattern.                             |
+
 
 ---
 
@@ -535,27 +593,31 @@ block-beta
 
 Any provider supported by [pi-ai](https://github.com/nicepkg/pi-ai):
 
-| Provider | Example Model |
-|:---|:---|
+
+| Provider      | Example Model              |
+| ------------- | -------------------------- |
 | **Anthropic** | `claude-sonnet-4-20250514` |
-| **Google** | `gemini-2.5-flash` |
-| **OpenAI** | `gpt-4o` |
-| **Groq** | `llama-3.3-70b-versatile` |
-| **Cerebras** | `llama-3.3-70b` |
-| **Mistral** | `mistral-large-latest` |
-| **xAI** | `grok-3` |
+| **Google**    | `gemini-2.5-flash`         |
+| **OpenAI**    | `gpt-4o`                   |
+| **Groq**      | `llama-3.3-70b-versatile`  |
+| **Cerebras**  | `llama-3.3-70b`            |
+| **Mistral**   | `mistral-large-latest`     |
+| **xAI**       | `grok-3`                   |
+
 
 ### Embedding Providers
 
 Optional — keyword search works without any API key.
 
-| Provider | Default Model | Local |
-|:---|:---|:---:|
-| **OpenAI** | `text-embedding-3-small` | |
-| **Gemini** | `gemini-embedding-001` | |
-| **Voyage** | `voyage-3-lite` | |
-| **Mistral** | `mistral-embed` | |
-| **Ollama** | `nomic-embed-text` | **Yes** |
+
+| Provider    | Default Model            | Local   |
+| ----------- | ------------------------ | ------- |
+| **OpenAI**  | `text-embedding-3-small` |         |
+| **Gemini**  | `gemini-embedding-001`   |         |
+| **Voyage**  | `voyage-3-lite`          |         |
+| **Mistral** | `mistral-embed`          |         |
+| **Ollama**  | `nomic-embed-text`       | **Yes** |
+
 
 ---
 
@@ -571,23 +633,40 @@ src/
 │   ├── cgroup.ts          cgroups v2 resource limits (fail-closed)
 │   ├── rootfs.ts          Minimal rootfs with bind mounts
 │   ├── ipc.ts             Bidirectional JSON-RPC 2.0 over stdio
-│   ├── seccomp.ts         BPF syscall filter profiles (prepared)
+│   ├── seccomp.ts         BPF syscall filter profiles
+│   ├── seccomp-apply.ts   seccomp-BPF application via C helper
 │   ├── proxy-tools.ts     memory/skill tools proxied to host via IPC
 │   └── worker.ts          Agent entry point inside sandbox
 ├── credentials/       AES-256-GCM encrypted store + credential proxy
 ├── skills/            Global registry + per-workspace enablement
 ├── runtime/           createRuntime() — single-user mode
+│   ├── failover-error.ts   Error classification (ported from OpenClaw)
+│   ├── model-fallback.ts   Fallback chains + cooldown tracking
+│   ├── context-guard.ts    Context window guards (16K/32K thresholds)
+│   ├── retry.ts            Exponential backoff with jitter + retryAfterMs
+│   ├── api-key-rotation.ts Per-provider key rotation
+│   ├── subagent.ts         Parallel execution + lifecycle + registry
+│   ├── session-pruning.ts  Trim old tool results in context
+│   ├── tool-result-truncation.ts  Truncate oversized tool output
+│   ├── memory-flush.ts     Pre-compaction memory save
+│   └── stream-adapters.ts  Per-provider stream configuration
 ├── memory/            Hybrid search engine
 │   ├── hybrid.ts          Vector + FTS5 fusion scoring
 │   ├── mmr.ts             Maximal Marginal Relevance re-ranking
 │   ├── temporal-decay.ts  Exponential time-based scoring
-│   └── query-expansion.ts 7-language keyword expansion
+│   ├── query-expansion.ts 7-language keyword expansion
+│   ├── embedding-cache.ts SQLite embedding cache
+│   ├── embedding-batch.ts Batch embedding with retry
+│   ├── file-indexer.ts    File-based memory indexing with fs.watch
+│   ├── session-indexer.ts Session transcript indexing
+│   └── ssrf.ts            SSRF protection for HTTP calls
 ├── hooks/             6 lifecycle hook points
+├── logging/           Structured JSON logging with file output
 ├── sessions/          File-based store with async locking
 └── config/            Configuration loading
 ```
 
-**59 source files** · **3 runtime deps** · Sandbox, crypto, and IPC use **zero external deps** — all Node.js built-ins.
+**78 source files** · **3 runtime deps** · Sandbox, crypto, IPC, logging, and SSRF use **zero external deps** — all Node.js built-ins.
 
 ---
 
@@ -614,6 +693,30 @@ Every file is workspace-scoped. No shared state between tenants.
 
 ---
 
+## Built on OpenClaw
+
+Bulkhead Runtime stands on the shoulders of [OpenClaw](https://github.com/nicepkg/openclaw). Several production-critical subsystems were ported directly from OpenClaw's codebase:
+
+
+| Subsystem                                 | Origin                                                                    |
+| ----------------------------------------- | ------------------------------------------------------------------------- |
+| **Model fallback & error classification** | `src/agents/model-fallback.ts`, `failover-error.ts`, `failover-policy.ts` |
+| **API key rotation**                      | `src/agents/api-key-rotation.ts`, `live-auth-keys.ts`                     |
+| **Context window guards**                 | `src/agents/context-window-guard.ts`                                      |
+| **Retry with backoff**                    | `src/infra/retry.ts`                                                      |
+| **SSRF protection**                       | `src/infra/net/ssrf.ts`, `fetch-guard.ts`                                 |
+| **Embedding cache & batch**               | `extensions/memory-core/src/memory/manager-embedding-ops.ts`              |
+| **File indexer**                          | `extensions/memory-core/src/memory/manager-sync-ops.ts`                   |
+| **Session transcript indexer**            | `extensions/memory-core/src/memory/manager-sync-ops.ts`                   |
+| **Subagent orchestration**                | `src/agents/subagent-*.ts`                                                |
+| **Structured logging**                    | `src/logging/logger.ts`, `subsystem.ts`, `levels.ts`                      |
+| **Transcript events**                     | `src/sessions/transcript-events.ts`                                       |
+
+
+OpenClaw solved these problems in production. We extracted, adapted, and integrated them into Bulkhead's multi-tenant architecture. Full credit where it's due.
+
+---
+
 ## Requirements
 
 - **Linux** — bare metal, VM, or container with `--privileged`
@@ -622,4 +725,4 @@ Every file is workspace-scoped. No shared state between tenants.
 
 ## License
 
-[MIT](LICENSE)
+[MIT](LICENSE)