@assemblyline-agents/docs 2.0.0

package/dist/corpus.json

@@ -0,0 +1 @@
+{"schemaVersion":1,"frameworkVersion":"2.0.0","revision":"d7e261d341a5da9c","pages":[{"id":"adapters","sourcePath":"adapters.md","title":"Adapters","description":"Choose interchangeable runtime, state, storage, deployment, channel, and sandbox providers.","url":"https://assemblyline.artificialillumination.co/docs/adapters","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/adapters.md","headings":[{"depth":1,"title":"Adapters (Substrate Reference)","anchor":"adapters-substrate-reference"},{"depth":2,"title":"Provider Docs Used","anchor":"provider-docs-used"},{"depth":2,"title":"Gateway","anchor":"gateway"},{"depth":2,"title":"Community Plugin Providers","anchor":"community-plugin-providers"},{"depth":2,"title":"Scheduler","anchor":"scheduler"},{"depth":2,"title":"Capabilities","anchor":"capabilities"},{"depth":3,"title":"Subagents And Model Routing","anchor":"subagents-and-model-routing"},{"depth":3,"title":"LiveKit Voice","anchor":"livekit-voice"},{"depth":2,"title":"Channels","anchor":"channels"},{"depth":3,"title":"Channel-owned ingress auth and attachment resolution","anchor":"channel-owned-ingress-auth-and-attachment-resolution"},{"depth":3,"title":"Attachment materialization","anchor":"attachment-materialization"},{"depth":2,"title":"Connections","anchor":"connections"},{"depth":2,"title":"Sandboxes","anchor":"sandboxes"},{"depth":2,"title":"Blob Storage","anchor":"blob-storage"},{"depth":2,"title":"Database","anchor":"database"},{"depth":2,"title":"Observability","anchor":"observability"}],"content":"# Adapters (Substrate Reference)\n\nThis page is the reference for consuming the substrate adapters that ship\nwith Assembly Line. To discover and install plugins, including the full catalog\nand `assembly-line add`, start at [Plugins](plugins.md).\n\nAssembly Line adapters are small. `gateway.ts` chooses where the\nruntime runs and which durable services it uses; channel files choose how\nexternal events become Assembly Line turns; sandbox files choose where isolated code\nand shell work runs. These choices are independent.\n\nAdapters are substitutable substrate: each role (channel, sandbox, blob, state,\nscheduler, gateway/deploy) has a generic contract with interchangeable\nproviders, so the same agent runs unchanged on different infrastructure.\nCapabilities are a separate tier. A capability is a single-vendor feature that\ngives an agent something new to do rather than somewhere new to run. It exposes\nthat vendor's own surface and has no generic contract to swap behind. Capability\npackages live under `capabilities/` (see [Capabilities](#capabilities)), still\nride the same open provider seam, and describe themselves through provider\nmetadata so tools like the no-code builder can scaffold them automatically.\n\nAdapter support levels:\n\n- Supported means Assembly Line has a real runtime instantiation path, docs, and\n  regression coverage in this repo.\n- Preview means the public helper, compiler metadata, and local regression\n  coverage exist, but the adapter still needs live provider smoke coverage or\n  more provider hardening before it should be announced as fully supported.\n- Planned means the docs may name the direction, but the adapter is not a\n  production claim yet.\n\nThe current matrix is:\n\n| Role | Supported | Preview | Planned |\n| --- | --- | --- | --- |\n| Channels | Slack, Discord, Telegram, Microsoft Teams, Photon/Spectrum | - | - |\n| Sandboxes | local dev/test, Docker, Daytona, E2B | Modal | - |\n| Blob storage | local dev/test, R2, generic S3-compatible storage | - | - |\n| Durable state | local dev/test files, Postgres for production, with run/event/checkpoint, FIFO `ConversationTurnStore`, and atomic `AgentStateStore` facets; Neon, Railway, Supabase, local, and custom presets | - | other database families |\n| Scheduler | local in-process loop, gateway-triggered cloud scheduler, Postgres-backed multi-worker loop | - | - |\n| Gateway/deploy | local, Railway, generic VPS | Docker, Fly | provider-managed VPS provisioning and other gateway families |\n\nOther database families, gateway families, and sandbox providers are not part\nof this initial adapter set.\n\nA state adapter may claim durable generalized-hook support only when it passes\nthe agent-state conformance contract: bounded snapshot reads, atomic set/update,\ndelete, revision compare-and-set behavior, and conversation isolation. Omitted\nstate facets fall back to memory for embedding hosts and are reported as\ndegraded; production hooks that must survive restarts need File or Postgres\nstate.\n\nPrimary model engines are not an adapter role in this matrix. Pi remains the\ndefault. Separately, the Node host includes a preview `openai-codex/*`\nprovider-prefix route backed by the official Codex app-server. That route is\nimplemented in `@assemblyline-agents/codex` but is not provider-package registration and\ndoes not add a public `harness:` field to `agent.ts`.\n\nCapabilities are tracked separately from the adapter roles above because they\nare not substitutable substrate:\n\n| Capability | Supported | Preview | Planned |\n| --- | --- | --- | --- |\n| Voice / telephony | - | LiveKit voice dispatch and SIP calls | - |\n\n## Provider Docs Used\n\nThe adapter shapes follow current provider docs for:\n\n- [Discord interactions](https://docs.discord.com/developers/interactions/receiving-and-responding)\n- [Discord Gateway](https://discord.com/developers/docs/events/gateway)\n- [Slack agents](https://docs.slack.dev/ai/agents)\n- [Slack Events API](https://docs.slack.dev/apis/events-api/)\n- [Slack request verification](https://docs.slack.dev/authentication/verifying-requests-from-slack/)\n- [Telegram Bot API](https://core.telegram.org/bots/api)\n- [GitHub App authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app)\n- [Microsoft Bot Connector authentication](https://learn.microsoft.com/en-us/azure/bot-service/rest-api/bot-framework-rest-connector-authentication?view=azure-bot-service-4.0)\n- [LiveKit Agents telephony](https://docs.livekit.io/agents/start/telephony/)\n- [LiveKit agent dispatch](https://docs.livekit.io/agents/worker/agent-dispatch/)\n- [LiveKit SIP API](https://docs.livekit.io/sip/api/)\n- [LiveKit access tokens](https://docs.livekit.io/home/server/generating-tokens/)\n- [Docker containers](https://docs.docker.com/engine/containers/run/)\n- [Docker resource constraints](https://docs.docker.com/engine/containers/resource_constraints/)\n- [Daytona sandboxes](https://www.daytona.io/docs/en/sandboxes)\n- [E2B sandboxes](https://e2b.dev/docs/sandbox)\n- [Modal sandboxes](https://modal.com/docs/guide/sandboxes)\n- [Modal sandbox files](https://modal.com/docs/guide/sandbox-files)\n- [Modal sandbox snapshots](https://modal.com/docs/guide/sandbox-snapshots)\n- [R2 S3 compatibility](https://developers.cloudflare.com/r2/api/s3/api/)\n- [Amazon S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html)\n- [Neon Postgres connections](https://neon.com/docs/connect/connect-from-any-app)\n- [Supabase Postgres connections](https://supabase.com/docs/guides/database/connecting-to-postgres)\n- [Railway CLI](https://docs.railway.com/cli)\n- [Fly deploy](https://fly.io/docs/flyctl/deploy/)\n- [OpenSSH client](https://man.openbsd.org/ssh)\n\n## Gateway\n\nGateway adapters answer one question: where does the compiled Assembly Line runtime\nservice run?\n\nThey do not choose your state, blob, sandbox, channels, or connections. Those\nare separate settings in the same `gateway.ts` file.\n\n```ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\nimport { neonPostgres } from \"@assemblyline-agents/postgres\";\nimport { r2Blob } from \"@assemblyline-agents/s3\";\nimport { dockerSandbox } from \"@assemblyline-agents/docker\";\n\nexport default defineGateway({\n  deploy: adapter(\"railway\"),\n  runtime: adapter(\"node\"),\n  state: neonPostgres(),\n  blob: r2Blob(),\n  sandbox: dockerSandbox()\n});\n```\n\nDeploy target status:\n\n- Supported: `adapter(\"railway\")` or `railwayDeploy()` runs the hosted Node\n  runtime through the `@assemblyline-agents/railway` deploy publisher.\n- Preview: `adapter(\"docker\")` or `dockerDeploy()` builds the compiled\n  artifact as a Docker image through the `@assemblyline-agents/docker` deploy publisher\n  and can optionally run it locally.\n- Preview: `adapter(\"fly\")` or `flyDeploy()` generates `fly.toml` and\n  publishes the artifact through the `@assemblyline-agents/fly` deploy publisher and\n  `flyctl deploy`.\n- Supported: `adapter(\"vps\")` or `vpsDeploy()` deploys to a named AMD64\n  Ubuntu 24.04/26.04 or Debian 12 host over fingerprint-pinned SSH. It manages\n  per-agent containers, networks, storage, database access, secrets, Caddy\n  routing, and transactional blue/green releases without mounting the Docker\n  socket into an agent. Hetzner hosts can also be securely created or adopted\n  through `assembly-line hosts bootstrap`.\n\nUseful CLI flags:\n\n```sh\nassembly-line deploy --target docker --docker-image assembly-line/my-agent --serve\nassembly-line deploy --target fly --fly-app my-agent --fly-region iad\nassembly-line deploy --target railway --railway-project prj_x --railway-service svc_y\nassembly-line deploy --target vps --vps-host production-eu --sync-secrets\n```\n\n## Community Plugin Providers\n\nAny npm package can supply a state, blob, sandbox, or deploy provider that\nagents select with `adapter(kind, options, { package })`, see\n[Authoring Plugins](authoring-adapters.md) for the contract, preflight, and\nartifact-packaging behavior.\n\n## Scheduler\n\nSchedules compile to registration metadata, but the scheduler adapter chooses\nwhere the clock lives.\n\n```ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\n\nexport default defineGateway({\n  scheduler: adapter(\"gateway\")\n});\n```\n\nSupported scheduler adapters:\n\n- `adapter(\"local\")`: starts an in-process polling loop with the Node host. Use\n  this for local development, tests, and simple single-process hosts.\n- `adapter(\"gateway\")`: does not start a local loop. Use a platform cron,\n  cloud scheduler, Durable Object alarm, queue worker, or gateway route to call\n  `GET` or `POST /assembly-line/automations/tick`, or call `runtime.runDueAutomations()`\n  from host code.\n- `adapter(\"postgres\")`: starts the polling loop and coordinates duplicate\n  workers through the Postgres state adapter's idempotency and dynamic automation\n  leases. Pair it with `state: adapter(\"postgres\")`.\n\nFor gateway-triggered production schedulers, set `ASSEMBLY_LINE_SCHEDULER_SECRET`\nand send it as `Authorization: Bearer <secret>` or\n`x-assembly-line-scheduler-secret`.\n\n## Capabilities\n\nCapabilities live under `capabilities/` and are single-vendor features, not\nsubstitutable adapters. They expose focused clients, tools, definitions, and\nconnection metadata instead of pretending to implement an interchangeable\nruntime role. A no-code builder or catalog can use that metadata to render a\nplugin card and credential form without making the capability an agent engine.\n\n### Subagents And Model Routing\n\nEvery primary agent and child agent runs through Pi. Subagents are isolation\nand delegation boundaries, not engine adapters; their definitions may narrow\nthe model, workspace, tools, and connections, while omitted models inherit the\nprimary model.\n\n`@assemblyline-agents/codex` also exports `codexAgentHarness()` for the Node host's preview\nprimary-model route. It exposes Assembly Line tools to app-server as dynamic tools,\nkeeps execution and approvals in the Assembly Line runtime, and persists the Codex\nthread id as opaque continuation state. Authentication stays inside the Codex\nCLI session. App-server reports whether that session is using ChatGPT or an\nAPI key; Assembly Line records the billing rail and provider token/credit snapshots\nwithout reading either credential. For Railway deployments, selecting an\n`openai-codex/*` primary model packages the official CLI, provisions persistent\n`/data/codex` storage, and supports direct remote device authentication through\n`assembly-line auth codex`; no OAuth token is copied into Railway variables.\n\nThe Pi/OpenRouter route records native token and charged-credit receipts, with\ngeneration-ID reconciliation when settlement is delayed. Codex reports\nper-turn tokens but not per-turn dollars; API-key cash is therefore reconciled\nas an OpenAI organization control total and is attributed to an agent only\nwhen an operator provides a dedicated project or API-key mapping.\n\n### LiveKit Voice\n\nAssembly Line does not run realtime media inside the text-model runtime. Instead,\n`@assemblyline-agents/livekit` signs LiveKit server tokens,\ncalls the LiveKit Agent Dispatch and SIP Twirp APIs, and lets an Assembly Line turn\nstart or route a LiveKit voice session.\n\nUse a tool when the parent agent should decide to dial:\n\n```ts\n// tools/start_call.ts\nimport { defineLiveKitOutboundCallTool } from \"@assemblyline-agents/livekit\";\n\nexport default defineLiveKitOutboundCallTool({\n  agentName: \"support-voice\",\n  outboundTrunkId: \"ST_outbound\"\n});\n```\n\nDeclare the connection when the agent or a Pi-backed subagent should receive\nLiveKit credentials and preflight requirements:\n\n```ts\n// connections/livekit.ts\nimport { defineLiveKitConnection } from \"@assemblyline-agents/livekit\";\n\nexport default defineLiveKitConnection();\n```\n\nThe LiveKit agent named by `agentName` must be running in your LiveKit Agents\nworker. For outbound phone calls, Assembly Line dispatches that worker into a room\nand calls `CreateSIPParticipant` to dial the callee through your outbound SIP\ntrunk. Incoming phone calls should be routed in LiveKit with SIP dispatch\nrules to the LiveKit agent worker; Assembly Line can still be used by that worker as\nan app/runtime layer, but the phone media path remains LiveKit.\n\nThis keeps the boundary explicit: LiveKit owns rooms, media, dispatch, and SIP;\nAssembly Line owns Pi reasoning, typed tool calls, and durable run state.\n\nRequired environment:\n\n| Purpose | Env |\n| --- | --- |\n| LiveKit server API | `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET` |\n| Default outbound dialing | optional `LIVEKIT_OUTBOUND_TRUNK_ID` |\n| Default voice worker dispatch | optional `LIVEKIT_VOICE_AGENT_NAME` |\n\n## Channels\n\nChannel packages export one-line helpers for normal use. They verify provider\nauth, normalize the provider event into `ChannelTurn`, preserve provider IDs in\nmetadata/delivery, and send the final reply through provider APIs.\n\nAfter a deploy, point each channel's provider at the deployed ingress URL with:\n\n```sh\nassembly-line channels wire <agentRoot> --url https://your-service.example.com\n```\n\nIt computes each channel's ingress URL from the compiled route table and, for\nTelegram, calls `setWebhook` directly (needs `TELEGRAM_BOT_TOKEN`, and uses\n`TELEGRAM_WEBHOOK_SECRET` when set). For providers that configure their endpoint\nin a console (Slack Request URL, Discord Interactions Endpoint, Teams messaging\nendpoint), it prints the exact URL to paste. Output is a structured\n`ChannelWireResult[]` (`action: \"set\" | \"manual\"`).\n\n```ts\n// channels/slack.ts\nimport { defineSlackChannel } from \"@assemblyline-agents/slack\";\n\nexport default defineSlackChannel();\n```\n\nTo share one employee identity across Slack, web, and other surfaces, configure\nthe helper's optional `resolvePrincipal(turn, ctx)` callback. Return a canonical\nuser principal with internal id, issuer, and string/string-array attributes\nsuch as `roles`, `teams`, or `tenantId`. Throw to reject an unmapped sender.\n\n```ts\n// channels/discord.ts\nimport { defineDiscordChannel } from \"@assemblyline-agents/discord\";\n\nexport default defineDiscordChannel({\n  gateway: true,\n  requireMention: true\n});\n```\n\n```ts\n// channels/telegram.ts\nimport { defineTelegramChannel } from \"@assemblyline-agents/telegram\";\n\nexport default defineTelegramChannel();\n```\n\n```ts\n// channels/teams.ts\nimport { defineTeamsChannel } from \"@assemblyline-agents/teams\";\n\nexport default defineTeamsChannel();\n```\n\nRequired channel environment:\n\n| Channel | Required env |\n| --- | --- |\n| Slack | `SLACK_SIGNING_SECRET`, `SLACK_BOT_TOKEN`, optional `SLACK_BOT_USER_ID`, optional `SLACK_ASSISTANT_ENABLED` |\n| Discord | `DISCORD_PUBLIC_KEY`, `DISCORD_APPLICATION_ID`, `DISCORD_BOT_TOKEN`, optional `DISCORD_GATEWAY_ENABLED`, optional `DISCORD_GATEWAY_INTENTS`, optional `DISCORD_BOT_USER_ID` |\n| Telegram | `TELEGRAM_BOT_TOKEN`; `TELEGRAM_WEBHOOK_SECRET` is required outside `devMode` |\n| Teams | `MICROSOFT_APP_ID`, `MICROSOFT_APP_PASSWORD`, optional `ASSEMBLY_LINE_TEAMS_ALLOWED_TENANTS`, optional `ASSEMBLY_LINE_TEAMS_ALLOWED_SERVICE_URLS` |\n\n### Channel-owned ingress auth and attachment resolution\n\nChannels declare their own production ingress-auth requirements and resolve\ntheir own provider attachments; the runtime stays a generic dispatcher.\n\n- `ChannelDefinition.ingress.requiredSecretEnv` is a list of any-of groups of\n  env var names: production boot succeeds when every var in at least one group\n  is set (for example Photon declares\n  `[[\"PHOTON_WEBHOOK_SIGNING_SECRET\"], [\"PHOTON_INGRESS_TOKEN\"]]`). The\n  compiler stamps the declaration into `CompiledChannel.metadata.ingress`; in\n  production the runtime refuses to boot when no group is satisfied, and in\n  `devMode` it logs a warning instead. The built-in helpers\n  (`defineSlackChannel`, `defineTelegramChannel`, `defineDiscordChannel`,\n  `defineTeamsChannel`, `definePhotonChannel`) declare\n  this automatically; custom channels can set `ingress` on their channel config\n  and re-export the same shape as `ingressAuth` on the module.\n- `ChannelModule.resolveAttachment(attachment, ctx)` turns a turn attachment\n  into a download request (`{ url, headers, filename? }`). The runtime performs\n  the download, applies size/type limits and timeouts, and stores the blob; the\n  channel owns auth lookups (Slack `files.info`, Telegram `getFile`, the Teams\n  Bot Framework token flow, Photon bridge bearer headers) and host allowlists.\n  Return `undefined` to fall back to a generic unauthenticated URL download.\n- Trust boundary: the runtime only calls `resolveAttachment` on the module of\n  the channel that produced the turn, and only when the turn's declared\n  provider matches that channel, a hostile attachment claiming another\n  provider can never route to that provider's credentials. Inside a resolver,\n  use `attachmentTrustedForProvider(attachment, \"<provider>\")` and\n  `attachmentRemoteUrl(attachment)` from `@assemblyline-agents/runtime` to honor the\n  per-attachment `remote.trusted` markers before attaching credentials.\n\n### Attachment materialization\n\nInbound files follow the same normalized `ChannelTurn.attachments` contract on\nevery channel. Adapters should preserve provider file identity plus one of:\n\n- inline content (`content`, `text`, `body`, or base64 `data`)\n- a safe remote reference (`url`, `downloadUrl`, `contentUrl`, or\n  `remote: { provider, auth?, url }`)\n- provider-specific lookup metadata such as Telegram `file_id`\n\nThe runtime materializes remote attachments after the webhook ACK, stores the\noriginal bytes through the configured blob adapter, records them as read-only\n`/files/original/...` resources, and lists them in `/files/manifest.json`.\nPrivate download details and auth hints are stripped from model-visible\nattachment context after storage. Text and Markdown attachments read back as\nUTF-8; binary files remain byte-accurate for resource projection. ZIP uploads\nare also expanded when possible: the original archive remains available under\n`/files/original/...`, and safe entries are exposed as read-only resources under\n`/files/extracted/<archive-name>/...` for the core file tools. Unsupported, encrypted, oversized, or path-escaping ZIP\nentries are skipped or reported without dropping the original uploaded archive.\n\nStored PNG, JPEG, GIF, WebP, MP4, MPEG, MOV, and WebM attachments are\nadditionally supplied to the primary harness as typed model media. Adapters\nshould therefore preserve the correct MIME type instead of labeling every\nupload as `application/octet-stream`. The runtime never exposes private\nattachment URLs to the model; it reads the configured blob and gives the\nharness bounded base64 bytes. A harness sends complete video only when the\nselected model advertises native video support. Otherwise Assembly Line reports the\nlimitation and does not sample frames unless the user explicitly requests that\napproximation.\n\nSlack uses the Events API route `/slack/events`. It verifies the raw request\nbody with Slack's signing secret, handles `url_verification` inline, returns a\n2xx ACK for accepted events before model work, and uses Slack `event_id` as the\ndelivery idempotency key. It starts turns only for intentional agent entry\npoints: app mentions, user DM messages, and assistant-thread user messages when\nassistant mode is enabled. Delivery always uses the preserved Slack channel and\nthread target from the normalized turn.\n\nSlack context augmentation runs after ACK and before default context bundle\nconstruction. It supplies the current conversation history plus up to three\nrecent Assembly Line Slack conversations for the same user from the last 24 hours as\nshort summaries and structured metadata. It does not fetch workspace-wide Slack\nhistory during ingress.\n\nAgent communication channel status:\n\n- Slack and Photon/Spectrum are reference agent channels. Slack maps app\n  mentions, DMs, and assistant-thread user messages into durable Assembly Line turns;\n  Photon/Spectrum maps iMessage bridge events into the same lifecycle and adds\n  native actions such as reactions, polls, app cards, and backgrounds.\n- Discord is an agent-oriented communication channel when Gateway ingress is\n  enabled. Interactions still cover slash commands, components, modals, and\n  autocomplete; Gateway ingress covers DMs, mentions, and thread/channel\n  messages. Discord starts typing indicators for Gateway turns and delivers\n  through either interaction responses or bot-token channel messages.\n- Telegram is an agent-oriented communication channel over Bot API webhooks. It\n  supports messages, edited messages, callback queries, channel posts, business\n  messages, forum topics, typing actions, inline keyboards, callback answers,\n  media sends, and `getFile` attachment materialization.\n- Microsoft Teams is an agent-oriented communication channel over Bot Framework\n  activities. It supports message activities, Adaptive Card invoke submissions,\n  mention stripping, tenant/service URL constraints, typing activities,\n  Adaptive Card replies, suggested actions, and protected attachment\n  materialization.\n## Connections\n\nConnection helpers remain available when agents need provider capabilities\nbeyond receiving messages. Channels own inbound events and reply delivery;\nconnections own typed provider capabilities, credential requirements, and\nremote tool discovery.\n\nGitHub is connection-only: use it for repository, issue, pull-request, and\nworkflow tools rather than as an inbound communication channel.\n\n```ts\n// connections/github.ts\nimport { defineGitHubConnection } from \"@assemblyline-agents/github\";\n\nexport default defineGitHubConnection();\n```\n\n```ts\n// connections/teams.ts\nimport { defineTeamsConnection } from \"@assemblyline-agents/teams\";\n\nexport default defineTeamsConnection();\n```\n\nGitHub App connections require `GITHUB_APP_ID` and `GITHUB_PRIVATE_KEY`. Teams\nuses Bot Framework credentials.\n\nLive smoke evidence for Discord, Telegram, and Teams is written locally under\n`docs/internal/evidence/channels/` (a gitignored working directory). The root smoke scripts skip without\ncredentials and point to those provider checklists when credentials are present.\n\n## Sandboxes\n\nThe sandbox adapter is acquired lazily when a tool or capability asks for a\nsandbox. Normal channel receipt, model turns without sandbox-backed tools, skill\nactivation, memory reads/writes, and final delivery do not need to pay sandbox\nstartup cost.\n\n```ts\n// sandbox/default.ts\nimport { dockerSandbox } from \"@assemblyline-agents/docker\";\n\nexport default dockerSandbox({\n  image: \"node:22-slim\",\n  network: \"none\"\n});\n```\n\nSandbox adapter status:\n\n- Supported: `adapter(\"local\")` for trusted dev/test only.\n- Supported: `adapter(\"docker\")` or `dockerSandbox()` for local container\n  isolation, one container per acquired session, cleanup on dispose, and\n  a physical `/workspace` container cwd.\n- Supported: `adapter(\"daytona\")` for hosted Daytona sandboxes.\n- Supported: `adapter(\"e2b\")` or `e2bSandbox()` for hosted E2B sandboxes.\n- Preview: `adapter(\"modal\")` or `modalSandbox()` for hosted Modal\n  sandboxes. Its JavaScript SDK, filesystem, lifecycle, readiness probe, image,\n  tag, and snapshot bindings are compile-checked against the installed Modal\n  SDK.\n\nAll hosted built-in sandbox adapters expose the same physical namespace:\nshells start in `/workspace`, absolute `/workspace/...` shell paths and provider\nfile APIs address the same files, and create/connect/wake fail if that invariant\ndoes not hold. Docker uses its native container workdir, Daytona builds the\nconfigured image with `WORKDIR /workspace`, Modal extends its image and passes\nthe native Sandbox `workdir`, and E2B idempotently provisions `/workspace`\nthrough its root command facility before returning to the template's ordinary\ncommand user. `..` traversal is rejected. Recursive listings return canonical\nabsolute paths and include contents for runtime persistence.\n\nThe filesystem contract is versioned in provider metadata, labels/tags,\nprovider-safe names, runtime manifests, and sync jobs. A runtime never\nreconnects or restores a snapshot from an obsolete contract. The Local adapter\nis explicitly a trusted dev/test logical emulation over a host temporary\ndirectory; Docker is the local conformance path when physical `/workspace`\nsemantics matter.\n\nDirty sandbox sessions are retained when async sandbox sync is pending:\nDocker stops the container, Daytona pauses/stops, and E2B pauses with configurable\nmemory retention. Modal detaches retained sessions and terminates clean ones.\nClean sessions are removed,\ndeleted, killed, or terminated through the provider lifecycle API. The sync worker\nreconnects first and then calls `wake()`/start when a retained sandbox is warm\nor paused.\n\nHosted adapters do not silently fall back to local execution. Daytona local\nfallback exists only for explicit development/test opt-in with\n`ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_FALLBACK=true`.\n\nThe local sandbox is for trusted dev/test execution. Production Node runtime\nconstruction rejects `adapter(\"local\")` for sandboxes unless\n`ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true` is set to acknowledge that the\nhost process, filesystem, and network are not isolated. Docker defaults\n`ASSEMBLY_LINE_DOCKER_NETWORK` to `none`; Daytona supports\n`ASSEMBLY_LINE_DAYTONA_NETWORK_BLOCK_ALL` and allow/domain lists for explicit egress\npolicy.\n\nProvider env:\n\n| Sandbox | Required env | Lifecycle and policy env |\n| --- | --- | --- |\n| Docker | Docker CLI/daemon available | Optional `DOCKER_HOST`, `ASSEMBLY_LINE_DOCKER_NETWORK` (defaults to `none`), `ASSEMBLY_LINE_DOCKER_CPUS`, `ASSEMBLY_LINE_DOCKER_MEMORY`, `ASSEMBLY_LINE_DOCKER_PULL_POLICY`, `ASSEMBLY_LINE_DOCKER_COMMAND_TIMEOUT_MS` |\n| Daytona | `DAYTONA_API_KEY` | Optional `DAYTONA_API_URL`, `DAYTONA_TARGET`, `ASSEMBLY_LINE_DAYTONA_CREATE_TIMEOUT_SECONDS`, `ASSEMBLY_LINE_DAYTONA_LIFECYCLE_TIMEOUT_SECONDS`, `ASSEMBLY_LINE_DAYTONA_AUTO_STOP_MINUTES`, `ASSEMBLY_LINE_DAYTONA_AUTO_ARCHIVE_MINUTES`, `ASSEMBLY_LINE_DAYTONA_AUTO_DELETE_MINUTES`, `ASSEMBLY_LINE_DAYTONA_EPHEMERAL`, `ASSEMBLY_LINE_DAYTONA_NETWORK_BLOCK_ALL`, `ASSEMBLY_LINE_DAYTONA_NETWORK_ALLOW_LIST`, `ASSEMBLY_LINE_DAYTONA_DOMAIN_ALLOW_LIST` |\n| E2B | `E2B_API_KEY` | Optional `E2B_TEMPLATE`, `ASSEMBLY_LINE_E2B_TIMEOUT_MS`, `ASSEMBLY_LINE_E2B_RETAIN_TIMEOUT_MS`, `ASSEMBLY_LINE_E2B_REQUEST_TIMEOUT_MS`, `ASSEMBLY_LINE_E2B_PAUSE_KEEP_MEMORY`, `ASSEMBLY_LINE_E2B_ALLOW_INTERNET_ACCESS` |\n| Modal | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Optional `MODAL_APP_NAME`, `ASSEMBLY_LINE_MODAL_TIMEOUT_MS`, `ASSEMBLY_LINE_MODAL_WAIT_READY` |\n\nProvider snapshots are opt-in through the Assembly Line sandbox snapshot policy.\nDocker reports snapshots as unsupported. Daytona and E2B expose snapshot\ncreation only when the installed provider SDK exposes it; Modal uses its\nfilesystem snapshot image API. Normal\nproduction persistence remains Assembly Line state/blob sync.\n\nLive sandbox smoke is opt-in:\n\n```sh\npnpm smoke:sandbox -- --provider=docker\npnpm smoke:sandbox -- --provider=daytona\npnpm smoke:sandbox -- --provider=e2b\npnpm smoke:sandbox -- --provider=modal\n```\n\nDocker smoke runs when Docker is available. Hosted smoke skips cleanly when\ndisposable provider credentials are missing. Smoke evidence is written locally under\n`docs/internal/smoke/` (gitignored) and records provider, lifecycle result, safe file hashes,\nand snapshot status without secrets.\n\nDeploy parity checks are also available without hosted-provider usage:\n\n```sh\npnpm smoke:deploy:docker\npnpm smoke:deploy:fly\npnpm smoke:deploy:fly:live\n```\n\nThe Docker check performs a real local image build, container recreation,\nremote command, and persistent-volume lifecycle. The Fly check gives the\ngenerated `fly.toml` to the installed `flyctl` local parser and verifies the\ndeploy, volume, Machine, and SSH flags used by the publisher. It uses no valid\nFly credential and cannot create provider resources. The explicit `:live`\nvariant creates one ephemeral Fly app and volume, tests a real deploy, HTTP\nhealth, secret sync, SSH, redeploy persistence, and publisher-owned teardown,\nthen verifies the app is absent. Hosted Fly and Modal smoke require credentials\nand may incur provider usage.\n\n## Blob Storage\n\nProduction blob storage is S3-compatible. R2 remains first-class through the R2\nwrapper, but the runtime contract is the same for R2, AWS S3, and MinIO-style\nendpoints.\n\n```ts\nimport { defineGateway } from \"@assemblyline-agents/core\";\nimport { s3Blob, r2Blob, minioBlob } from \"@assemblyline-agents/s3\";\n\nexport default defineGateway({\n  blob: r2Blob()\n  // or blob: s3Blob()\n  // or blob: minioBlob()\n});\n```\n\nGeneric S3 env:\n\n- `S3_BUCKET`\n- `S3_REGION`\n- `S3_ACCESS_KEY_ID`\n- `S3_SECRET_ACCESS_KEY`\n- optional `S3_ENDPOINT`\n- optional `S3_FORCE_PATH_STYLE`\n- optional `S3_PREFIX`\n- optional `S3_PUBLIC_BASE_URL`\n\nR2 env:\n\n- `R2_ACCOUNT_ID`\n- `R2_BUCKET`\n- `R2_ACCESS_KEY_ID`\n- `R2_SECRET_ACCESS_KEY`\n- optional `R2_PREFIX`\n- optional `R2_PUBLIC_BASE_URL`\n\n`@assemblyline-agents/r2` still exports `r2Adapter()`, `R2BlobAdapter`, and\n`InMemoryR2Bucket` for existing imports.\n\n`S3_PUBLIC_BASE_URL` and `R2_PUBLIC_BASE_URL` do not make every blob public.\nBlob writes are private by default; adapters return public HTTP URLs only when\nthe write explicitly uses `{ visibility: \"public\" }`.\n\n## Database\n\nAssembly Line stays opinionated here: Postgres is the only production durable state\nplane in this phase. That keeps runs, messages, tool traces, schedules, dynamic\nconnections, durable skills and their full-body revision history, pending skill\nchanges, leased background-review jobs, idempotency, and migrations on one auditable\ndatabase contract.\n\nSQLite is not included because it would create a second production state shape\njust as deploy targets and channels are expanding. File-backed state remains for\nlocal dev/demo/test, not hosted production: it is single-process by design (its\nidempotency reservations live in process memory, so two hosts sharing one state\nfile cannot coordinate leases or claims). Its writes are crash-safe: temp file,\nfsync, then atomic rename. An unreadable state file is backed up to\n`<path>.corrupt-<timestamp>` and replaced with a fresh state instead of\ncrashing the host, but multi-replica guarantees always require Postgres.\n\nConnection grant and authorization-session stores follow the same production\nrule. Postgres implements those stores directly. File-backed connection\ncredential stores are for local development or deliberately small deployments;\nproduction Node hosts using them must set `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` or\n`ASSEMBLY_LINE_SECRET` to a stable secret of at least 32 characters. The known local\ndevelopment fallback is rejected outside `devMode: true`.\n\n```ts\nimport { defineGateway } from \"@assemblyline-agents/core\";\nimport {\n  neonPostgres,\n  railwayPostgres,\n  supabasePostgres,\n  localPostgres,\n  postgresAdapter\n} from \"@assemblyline-agents/postgres\";\n\nexport default defineGateway({\n  state: neonPostgres()\n  // or state: railwayPostgres()\n  // or state: supabasePostgres()\n  // or state: localPostgres()\n  // or state: postgresAdapter({ provider: \"custom\" })\n});\n```\n\nDefault env:\n\n- `DATABASE_URL`\n- optional `ASSEMBLY_LINE_POSTGRES_CONNECTION_ENV`\n- optional `ASSEMBLY_LINE_POSTGRES_SSL_REJECT_UNAUTHORIZED` (defaults to `true`; set `false` for self-signed/proxied certs)\n\nNeon, Railway, Supabase, local Postgres, and custom Postgres all run the same\nAssembly Line migrations and schema. On a Railway deploy, `railwayPostgres()` uses the\nRailway CLI to reuse the configured database service or provision a real\nPostgres service, then sets the application service's `DATABASE_URL` to a\nprivate Railway reference variable before publishing. Its defaults are\n`{ databaseService: \"Postgres\", provision: true }`; set `provision: false` to\nselect a named pre-existing service instead. Automatic creation uses Railway's\ndefault `Postgres` service name. Railway's official Postgres image uses a\ngenerated certificate, so this preset keeps TLS enabled but defaults\n`sslRejectUnauthorized` to `false`. For Supabase, copy a direct connection\nstring for a long-lived IPv6-capable host, or a session-pooler connection string\nwhen the host requires IPv4. Store either one as `DATABASE_URL`.\n\n## Observability\n\nTelemetry is not a gateway adapter. It lives in `agent/instrumentation.ts`.\n`@assemblyline-agents/otlp` provides an OTLP/HTTP GenAI telemetry sink you wire in the\n`setup` callback; see\n[Customizing Agents → Observability](customization.md#observability) for the\nsetup, capture-detail (`captureContent`) options, and the Langfuse recipe.\n"},{"id":"agent-stack/agent-ts","sourcePath":"agent-stack/agent-ts.md","title":"agent.ts","description":"Compose an Assembly Line agent with static policy and synchronous runtime capability selection.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/agent-ts","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/agent-ts.md","headings":[{"depth":1,"title":"agent.ts","anchor":"agentts"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Static Fields","anchor":"static-fields"},{"depth":2,"title":"Built-in Composition Functions","anchor":"built-in-composition-functions"},{"depth":2,"title":"Conditional Capabilities","anchor":"conditional-capabilities"},{"depth":2,"title":"Durable State And Re-evaluation","anchor":"durable-state-and-re-evaluation"},{"depth":2,"title":"Event Reactions","anchor":"event-reactions"},{"depth":2,"title":"Structured Outputs","anchor":"structured-outputs"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# agent.ts\n\n`agent.ts` is the agent-composition entrypoint. Its `defineAgent({...})` object\nholds static identity, policy, and limits. Its synchronous `setup()` function\nselects the compiled capabilities available to the next model request.\n\n`instructions.md` remains required and always trusted. `useInstructions()`\nonly appends conditional guidance; it never replaces that permanent identity.\n\n## Minimal Example\n\n```ts\nimport { defineAgent, useModel, useTool } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  id: \"minimal-agent\",\n  name: \"Minimal Agent\",\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useTool(\"echo\");\n  }\n});\n```\n\n`setup()` must be synchronous and side-effect free. The runtime preloads run\ndata and conversation state, validates the result against the compiled catalog\nand static policy, and records the complete capability snapshot before using\nit. The composition API provides no asynchronous setup phase. Keep network,\nfilesystem, and other side effects in tools or adapters.\n\n## Static Fields\n\n| Field | Meaning |\n| --- | --- |\n| `id`, `name`, `description` | Stable identity and display metadata. |\n| `maxIterations` | Hard positive agent-loop iteration limit. |\n| `maxReasoning` | Ceiling for `useReasoning()`. |\n| `selfImprovement` | Permission policy for durable skill authoring. |\n| `dynamicAutomations` | Permission policy for runtime-created automations. |\n| `dynamicConnections` | Permission and host policy for adopted connections. |\n| `context` | Trusted context policy, when a custom `context.ts` policy is required. |\n| `metadata` | Static JSON metadata. |\n| `setup()` | Synchronous runtime capability declaration. |\n\nModel, reasoning selection, visible tools, output schema, skills, connections,\nsubagents, sandbox profile, and conditional instructions do\nnot belong in static fields.\n\n## Built-in Composition Functions\n\n| Function | Effect |\n| --- | --- |\n| `useRun()` | Reads immutable run, canonical principal/initiator, message, channel, conversation, metadata, and attachment metadata. |\n| `usePersistentState(key, initial)` | Reads conversation-scoped JSON control state and returns an async setter. |\n| `useModel(model)` | Selects exactly one model. |\n| `useReasoning(level)` | Selects effort within `maxReasoning`. |\n| `useInstructions(text)` | Appends trusted instructions in call order. |\n| `useTool(name)` | Enables a compiled tool. |\n| `useSkill(name)` | Activates a compiled skill. |\n| `useConnection(name)` | Selects a compiled connection; grants and approvals still apply. |\n| `useSubagent(name)` | Exposes a compiled subagent. |\n| `useSandbox(name)` | Selects a compiled sandbox profile; acquisition stays lazy. |\n| `useOutputSchema(schema)` | Selects the runtime-enforced final-output contract. |\n\nSet-like composition calls deduplicate by compiled name. Repeated `useModel()`,\n`useReasoning()`, `useSandbox()`, or `useOutputSchema()` calls must agree.\nConditional calls are valid:\nstate identity comes from explicit keys, not call position.\n\nCapability names, models, reasoning levels, and state keys passed to composition\nfunctions must be string literals so the compiler can audit them. Custom\ncomposition helpers are ordinary synchronous functions:\n\n```ts\nfunction useVerifiedCustomer() {\n  const [verified] = usePersistentState(\"customer.verified\", false);\n  useTool(\"verify_customer\");\n  if (verified) useTool(\"issue_refund\");\n  return verified;\n}\n```\n\nCalling a built-in composition function outside `setup()` (or a function called\nby it) throws an actionable error. Cross-cutting event reactions belong in the\nseparate [`hooks/`](hooks.md) directory.\n\n## Conditional Capabilities\n\n```ts\nimport {\n  defineAgent,\n  useInstructions,\n  useModel,\n  useRun,\n  useTool\n} from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  setup() {\n    const run = useRun();\n    const roles = run.principal?.type === \"user\"\n      ? run.principal.attributes?.roles\n      : undefined;\n    const finance = Array.isArray(roles) && roles.includes(\"finance\");\n    if (!finance) {\n      useModel(\"openai/gpt-5.4-mini\");\n      useTool(\"search_docs\");\n      useInstructions(\"Do not access billing information.\");\n      return;\n    }\n    useModel(\"openai/gpt-5.4\");\n    useTool(\"search_docs\");\n    useTool(\"lookup_account\");\n  }\n});\n```\n\n`run.principal` is the current authenticated actor; `run.initiator` is the\nactor that originally opened the conversation. `run.userId` is the canonical\ncurrent user id retained for compatibility. Channel or host authentication\nmust resolve roles and teams before synchronous `setup()` runs; composition never\nfetch identity or policy themselves.\n\nEvery possible named capability must already exist in the compiled catalog.\nComposition can narrow static policy but cannot bypass host restrictions, connection\nauthorization, tool approvals, sandbox policy, or subagent declarations.\n\n## Durable State And Re-evaluation\n\n`usePersistentState()` stores small JSON control values such as workflow\nstages. Do not use it for secrets, files, transcripts, or long-form memory.\nEach key may contain at most 200 characters. Each value may serialize to at\nmost 16,384 characters. A conversation snapshot may contain at most 256 keys\nand 262,144 serialized characters.\n\nThe function returns the current value and an async setter. Call the setter from a\ntool or event handler, never during `setup()`. Tools can also update the same\nstate through `ctx.agentState`:\n\n```ts\nasync execute(input, ctx) {\n  await saveDiagnosis(input);\n  await ctx.agentState.set(\"triage.stage\", \"report\");\n  return { saved: true };\n}\n```\n\nThe write is atomic, increments the conversation revision, emits an\n`agent.state_changed` event with the key, revision, and either a value hash or\na deletion marker, and marks\nthe active snapshot dirty. Pass `expectedRevision` to `ctx.agentState` writes\nwhen concurrent changes must fail instead of overwriting each other. The\nruntime lets the current tool finish, then re-evaluates `setup()` before the\nnext model request. Model, prompt, tools, connections, subagents, sandbox, and\noutput schema change only at that boundary. One run may record at most 50\ncapability snapshots.\n\n## Event Reactions\n\nAuthor post-persist event reactions as `defineHook({ events: ... })` files under\n[`hooks/`](hooks.md). The deprecated `useEvent()` composition call remains\naccepted for compatibility but emits a compiler migration warning.\n\n## Structured Outputs\n\nCall `useOutputSchema(schema)` in `setup()`. The runtime adds the schema to the\ntrusted prompt, validates the final JSON, performs bounded corrective retries,\nand exposes the parsed value as `RunAgentResult.output`. `maxIterations` remains\na static hard limit.\n\n## Related Docs\n\n- [Configuration Reference](../config-reference.md#defineagent-agentts)\n- [Context](context-ts.md)\n- [Subagents](subagents.md)\n- [Runtime And Deployment](../runtime-and-deployment.md)\n"},{"id":"agent-stack/automations","sourcePath":"agent-stack/automations.md","title":"automations/","description":"Run agents from time-based schedules or normalized external events.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/automations","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/automations.md","headings":[{"depth":1,"title":"automations/","anchor":"automations"},{"depth":2,"title":"Scheduled Automation","anchor":"scheduled-automation"},{"depth":2,"title":"Event Automation","anchor":"event-automation"},{"depth":2,"title":"Inline Lifecycle","anchor":"inline-lifecycle"},{"depth":2,"title":"Dynamic Automations","anchor":"dynamic-automations"},{"depth":2,"title":"Delivery And Reliability","anchor":"delivery-and-reliability"},{"depth":2,"title":"Legacy Compatibility","anchor":"legacy-compatibility"}],"content":"# automations/\n\nAutomations start durable agent work without a conversational prompt. Every\nautomation declares a trigger and may invoke the default agent, a skill, an\nagent target, or a playbook. Triggers are either time-based schedules or\nnormalized external events.\n\n## Scheduled Automation\n\n```ts\n// automations/morning_brief.ts\nimport { defineAutomation } from \"@assemblyline-agents/core\";\n\nexport default defineAutomation({\n  description: \"Run a daily brief.\",\n  trigger: {\n    type: \"schedule\",\n    cron: \"0 8 * * *\",\n    timezone: \"America/Chicago\"\n  },\n  idempotencyKey: \"starter-agent:morning-brief\",\n  message: \"Prepare the morning brief.\",\n  target: { type: \"skill\", name: \"morning-brief\" }\n});\n```\n\nSchedule automations require `idempotencyKey`. The runtime appends the due\ntimestamp to this prefix, reserves the resulting key before dispatch, and\nrecords one durable run for each cron occurrence.\n\n## Event Automation\n\n```ts\n// automations/process_client_email.ts\nimport { defineAutomation } from \"@assemblyline-agents/core\";\n\nexport default defineAutomation({\n  description: \"Process important client email.\",\n  trigger: {\n    type: \"event\",\n    source: \"gmail\",\n    event: \"email.received\",\n    connection: \"gmail\",\n    filter: {\n      label: \"important\"\n    }\n  },\n  message: \"Review the email and extract the required actions.\",\n  target: { type: \"skill\", name: \"process-client-email\" }\n});\n```\n\nEvent filters use recursive JSON-subset matching. Every key in `filter` must\nexist with the same value in the normalized event payload; extra payload keys\nare allowed. Event automations default their idempotency prefix to\n`automation:<filename>`, then append the provider's stable `eventId`.\n\nTrusted hosts can submit normalized events directly:\n\n```http\nPOST /assembly-line/automations/events\nAuthorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>\nContent-Type: application/json\n\n{\n  \"source\": \"gmail\",\n  \"event\": \"email.received\",\n  \"eventId\": \"provider-message-id\",\n  \"occurredAt\": \"2026-07-24T13:30:00Z\",\n  \"payload\": {\n    \"label\": \"important\",\n    \"subject\": \"Contract follow-up\"\n  }\n}\n```\n\nProvider channel modules can return `{ kind: \"event\", event }` from\n`normalizeHttp()` after verifying the provider signature. Long-lived channel\nlisteners can call `emit.automation(event)`. Both paths use the same filtering,\ncapacity, idempotency, and durable run path as direct host dispatch through\n`runtime.dispatchAutomationEvent(event)`.\n\n## Inline Lifecycle\n\nAn automation may prepare memory and resources before the model turn, select a\ntarget, and finalize application state afterward without exposing orchestration\ntools to the model. Keep that lifecycle beside its trigger so the automation is\nauditable as one file.\n\n```ts\n// automations/llm_wiki_dream.ts\nimport { defineAutomation } from \"@assemblyline-agents/core\";\n\nexport default defineAutomation({\n  trigger: {\n    type: \"schedule\",\n    cron: \"15 8 * * *\",\n    timezone: \"UTC\"\n  },\n  idempotencyKey: \"system-routine:wiki-dream\",\n  target: { type: \"skill\", name: \"personal-wiki-update\" },\n\n  async prepare(ctx) {\n    const bundle = await ctx.resources.collect({\n      sources: [\"memory\", \"history\", \"connections\"],\n      limit: 100\n    });\n\n    return {\n      target: { type: \"skill\", name: \"personal-wiki-update\" },\n      promptContext: {\n        triggerKind: ctx.trigger.kind,\n        sourceBundle: bundle.markdown\n      }\n    };\n  },\n\n  async finalize(ctx, result) {\n    await ctx.emit(\"wiki.automation_finished\", {\n      ok: result.ok,\n      status: result.status ?? \"unknown\"\n    });\n  }\n});\n```\n\nThe lifecycle context exposes run identity, trigger metadata, memory, resources,\nblob storage, environment access, routine-run bookkeeping, durable events,\nidempotency keys, and replayable `ctx.step()` execution. For event automations,\n`ctx.trigger.event` contains the normalized event envelope.\n\n## Dynamic Automations\n\n`dynamicAutomations` in `agent.ts` controls runtime-created automations:\n\n```ts\nimport { defineAgent, useModel } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  dynamicAutomations: {\n    dynamic: true,\n    approval: false\n  },\n  setup() {\n    useModel(\"openai/gpt-5.4\");\n  }\n});\n```\n\nTools use `ctx.automationManager`:\n\n```ts\nawait ctx.automationManager?.createAutomation({\n  message: \"Prepare a weekly review.\",\n  cron: \"0 16 * * 5\",\n  timezone: \"America/Chicago\"\n});\n```\n\nDynamic automation creation currently supports time-based schedules. Event\nautomation definitions remain reviewed source because provider subscription,\nauthentication, and filtering policy are trusted application concerns.\nAutomations created through `ctx.automationManager` capture the run's canonical\n`principal` and `initiator`; scheduled execution restores both before hooks,\nmemory, tools, or user-subject connections run. Lifecycle handlers receive the\nsame values as `ctx.principal` and `ctx.initiator`.\nModel-visible updates cannot change an automation's owner principal or scope.\n\nHosts trigger due time-based work through `runtime.runDueAutomations()` or\n`GET/POST /assembly-line/automations/tick`. The deprecated `runDueSchedules()`\nmethod alias still works (it forwards to `runDueAutomations()`). The\nlegacy `/openeve/*` route aliases return `404`.\n\n## Delivery And Reliability\n\n- Provider event IDs and schedule occurrence IDs are reserved durably.\n- Capacity is checked before consuming an event's idempotency key.\n- Provider delivery is at least once, so external side effects must still use\n  `ctx.idempotencyKey()` or destination-level deduplication.\n- `delivery` on a normalized event can route the final result through an\n  originating provider. Omitting it runs the automation silently.\n- Channels remain conversational ingress. Automations are operational ingress\n  and do not require a conversation or reply.\n\n## Legacy Compatibility\n\n`schedules/`, `triggers/`, `defineSchedule()`, `defineTriggerHandler()`,\n`dynamicSchedules`, and `ctx.scheduleManager` remain accepted for compatibility\nand emit compiler deprecation warnings where applicable. New agents should use\n`automations/`, inline `prepare`/`finalize`, `defineAutomation()`,\n`dynamicAutomations`, and `ctx.automationManager`. The former\n`automation-handlers/`, `defineAutomationHandler()`, and `lifecycle.handler`\nsurfaces are also deprecated and compile with migration warnings.\n"},{"id":"agent-stack/channels","sourcePath":"agent-stack/channels.md","title":"channels/","description":"Receive external events and deliver replies through provider channels.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/channels","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/channels.md","headings":[{"depth":1,"title":"channels/","anchor":"channels"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"Provider Helpers","anchor":"provider-helpers"},{"depth":2,"title":"Normalization And Ingress","anchor":"normalization-and-ingress"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# channels/\n\nChannels normalize external events into Assembly Line turns and deliver replies back\nto the provider. Add a channel file when the agent should receive HTTP, Slack,\nDiscord, Teams, Telegram, Photon, or custom events.\n\n## Minimal Example\n\n```ts\n// channels/http.ts\nimport { defineChannel } from \"@assemblyline-agents/core\";\n\nexport default defineChannel({\n  description: \"Receive a local HTTP message.\",\n  transport: \"http\",\n  route: \"/message\",\n  methods: [\"POST\"]\n});\n```\n\nThe raw HTTP shape is intended for local development or authenticated host\nintegrations. In production, a generic HTTP channel without `normalizeHttp()`\nis rejected: the runtime returns\n`403 Channel <name> requires normalizeHttp() or trusted host authorization in production.`\nunless the request is host-trusted.\n\n## Full Options\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `description` | `string` | None | Human and manifest metadata. |\n| `transport` | `\"http\"` \\| `\"local\"` \\| `\"webhook\"` \\| `\"queue\"` (required) | None | How events reach the channel. |\n| `route` | `string` | None | HTTP route the runtime serves for this channel. |\n| `methods` | `string[]` | None | Accepted HTTP methods. |\n| `connection` | `string` | None | Connection this channel uses for provider credentials. |\n| `ingress` | `{ requiredSecretEnv: string[][] }` | None | Ingress-auth secrets required in production. |\n| `metadata` | JSON object | None | Structured app-specific metadata. |\n\n`ingress.requiredSecretEnv` is a list of any-of groups of env var names:\nproduction ingress auth is satisfied when every var in at least one group is\nset. `[[\"TELEGRAM_WEBHOOK_SECRET\"]]` requires that one var;\n`[[\"PHOTON_WEBHOOK_SIGNING_SECRET\"], [\"PHOTON_INGRESS_TOKEN\"]]` accepts either.\n\n## Provider Helpers\n\nProvider helpers keep common webhook wiring to one file:\n\n```ts\n// channels/slack.ts\nimport { defineSlackChannel } from \"@assemblyline-agents/slack\";\n\nexport default defineSlackChannel();\n```\n\nAssembly Line ships helpers for Slack, Discord, Telegram, Microsoft Teams, and\nPhoton/Spectrum. Provider helpers stamp route, required env,\ningress, normalization, and delivery behavior.\n\n`assembly-line add` installs a channel plugin and scaffolds the channel file for\nSlack, Discord, Telegram, and Teams, then prints the required env vars;\nexisting channel files are left untouched:\n\n```sh\nassembly-line add slack agent\n```\n\nPhoton channel files are written by hand with `definePhotonChannel()`.\n\nGitHub repository access remains available through\n`defineGitHubConnection()` or `defineGitHubMcpConnection()` in\n`connections/`; GitHub is not an inbound channel.\n\n## Normalization And Ingress\n\nA channel module can export `normalizeHttp` to verify and normalize the\nprovider request before a turn starts:\n\n```ts\nnormalizeHttp?: (request: ChannelHttpRequest, ctx: ChannelContext) =>\n  Promise<ChannelHttpResult> | ChannelHttpResult;\n```\n\nA normalized turn is a `ChannelTurn`:\n\n```ts\ninterface ChannelTurn {\n  eventId: string;\n  channel: string;\n  conversationId: string;\n  userId?: string;\n  principal?: AgentPrincipal;\n  initiator?: AgentPrincipal;\n  message: string;\n  attachments?: JsonObject[];\n  delivery?: JsonObject;\n  metadata?: JsonObject;\n  recentHistory?: string[];\n}\n```\n\nAfter provider authentication, a channel may map the provider sender to the\napplication's canonical user with `resolvePrincipal(turn, ctx)`. The runtime\ninvokes it before queueing or starting work:\n\n```ts\n// channels/slack.ts\nimport { defineSlackChannel } from \"@assemblyline-agents/slack\";\n\nexport default defineSlackChannel({\n  async resolvePrincipal(turn) {\n    const employee = await lookupEmployeeBySlackId(turn.userId!);\n    if (!employee) throw new Error(\"Unknown Slack user.\");\n    return {\n      type: \"user\",\n      id: employee.id,\n      issuer: \"company-directory\",\n      attributes: { roles: employee.roles, teams: employee.teams }\n    };\n  }\n});\n```\n\nVerified provider helpers supply a provider-scoped principal by default. Use a\nresolver when memory, connections, automations, or capability hooks must share\none internal identity across channels. The runtime preserves the first\nprincipal as `initiator` when later messages in the same conversation come\nfrom another user.\n\nChannel modules can also export `startIngress(ctx, emit)` for long-lived\nprovider listeners such as Discord Gateway. The Node host starts these listeners\nbeside the scheduler and stops them on server shutdown.\n\n`conversationId` is the concurrency boundary as well as the transcript key.\nNormalize it to the smallest provider object that users experience as one\nsession: a Slack thread root, Discord DM/channel/thread (or one interaction\ncommand when no thread exists), Telegram chat/forum topic, Teams conversation,\nor Photon space. Namespace raw provider IDs so unrelated channels cannot\ncollide. The runtime then permits one running or parked turn for that\nnormalized conversation, queues later turns FIFO, and leases other\nconversations in parallel.\n\n## Conventions\n\nChannel files own provider event semantics:\n\n- Verify signatures, tokens, and route-auth secrets.\n- Normalize provider payloads into `ChannelTurn`.\n- Resolve provider identities to canonical principals before agent work.\n- Use stable provider delivery IDs for idempotency.\n- Use a stable, provider-namespaced conversation boundary.\n- Return fast ACKs for retrying webhook providers when needed.\n- Deliver replies through provider APIs.\n- Keep provider routing here, not in tools.\n\n## Related Docs\n\n- [Adapters: Channels](../adapters.md#channels)\n- [Photon iMessage Channel](../photon.md)\n- [Connections](connections.md)\n- [Configuration Reference: channels](../config-reference.md#channelsts)\n"},{"id":"agent-stack/connections","sourcePath":"agent-stack/connections.md","title":"connections/","description":"Declare external capabilities and credential contracts.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/connections","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/connections.md","headings":[{"depth":1,"title":"connections/","anchor":"connections"},{"depth":2,"title":"Live Tool Connections","anchor":"live-tool-connections"},{"depth":2,"title":"Install A Provider Plugin","anchor":"install-a-provider-plugin"},{"depth":2,"title":"Plugin Transports","anchor":"plugin-transports"},{"depth":2,"title":"One-Time Binding Packets","anchor":"one-time-binding-packets"},{"depth":2,"title":"Missing Credentials At Runtime","anchor":"missing-credentials-at-runtime"},{"depth":2,"title":"Authorizing Before The First Run","anchor":"authorizing-before-the-first-run"},{"depth":2,"title":"MCP Transports","anchor":"mcp-transports"},{"depth":2,"title":"Dynamic Connections","anchor":"dynamic-connections"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# connections/\n\nConnections declare external capabilities and credential requirements. Raw\nsecrets and refresh tokens stay outside the agent folder, model context, and\nsandbox filesystem.\n\n```ts\n// connections/github.ts\nimport { adapter, defineConnection } from \"@assemblyline-agents/core\";\n\nexport default defineConnection({\n  description: \"GitHub capability contract.\",\n  provider: \"github\",\n  binding: adapter(\"env\"),\n  scopes: [\"repo:read\"],\n  capabilities: [\"issues:read\", \"pull_requests:read\"],\n  subject: \"user\",\n  required: false\n});\n```\n\n## Live Tool Connections\n\nUse a protocol helper when a connection should expose tools through the\ndeferred discovery path:\n\n- `defineMcpClientConnection()`\n- `defineOpenAPIConnection()`\n- `defineHttpApiConnection()`\n- `defineSandboxCliConnection()`\n\nConnection tools appear through `tool_search`, `tool_describe`, and `tool_call`.\nConcrete remote schemas are not injected into every prompt.\nIf deferred connection modules cannot load, `tool_search` records a failed,\nrecoverable tool call and returns the loader error to the model; it does not\ncount the failure as an available tool match. Partial searches may still return\nhealthy matches while reporting other connection loader errors separately.\n\nThe abstraction is the external capability and account, not its wire protocol.\nMCP, OpenAPI, HTTP, and sandbox CLI are transports beneath the same connection\nselection, access, approval, tracing, and subagent-scoping model. A sandbox CLI\nconnection runs its reviewed command inside the active agent sandbox so it can\nsee `/files` and `/workspace`; it does not launch the CLI on the gateway host.\n\nEvery live connection must declare tool-level access. Unclassified tools are\nnot discoverable, and write matches take precedence over read matches:\n\n```ts\nexport default defineMcpClientConnection({\n  url: \"https://mcp.example.com\",\n  description: \"Example service.\",\n  access: {\n    read: { tools: [\"list_items\", \"get_item\"] },\n    write: {\n      tools: [\"create_item\", \"update_item\", \"delete_item\"],\n      approval: \"always\"\n    }\n  }\n});\n```\n\n## Install A Provider Plugin\n\nConnection plugins generate this policy rather than hiding it:\n\n```sh\nassembly-line add notion agent\nassembly-line add slack agent --role connection\n```\n\nThe generated provider helper uses the package's reviewed read/write tool-name\npatterns and starts with `read: true, write: false`. To enable mutations, edit\nthe file explicitly:\n\n```ts\nimport { defineNotionConnection } from \"@assemblyline-agents/notion\";\n\nexport default defineNotionConnection({\n  access: {\n    read: true,\n    write: { approval: \"always\" }\n  }\n});\n```\n\nAssembly Line does not create the Notion integration or OAuth application. The\ndeployment owner supplies the token, a custom `auth` definition, or an endpoint\noverride. This same boundary applies to all official connection plugins.\n\n## Plugin Transports\n\nEvery official connection plugin follows one of four transports. The\n[plugin catalog](../plugins.md#connection-plugins) lists each plugin's\nendpoint and credential env vars; each package README documents\nprovider-specific setup.\n\n| Transport | What runs where | Exemplars |\n| --- | --- | --- |\n| Hosted MCP (Streamable HTTP) | The provider's (or a developer-operated) MCP server; credential from a `<PROVIDER>_MCP_TOKEN`-style env var, custom header, OAuth flow, or host-redeemed one-time binding packet | Notion (above), Linear, Browser Use, Arcads, Margins, Mirror, Provenance |\n| Direct OpenAPI | The provider's HTTP API called from the runtime using a bundled or referenced OpenAPI spec | SoundCloud |\n| Stdio MCP | A packaged bridge or separately installed binary launched by the Assembly Line runtime host, trusted configuration, never model-chosen | FFmpeg, Remotion, Orgo, Peekaboo |\n| Sandbox CLI | A provider CLI executed inside the active run sandbox with reviewed, individually quoted arguments | Higgsfield |\n\nOne exemplar for each remaining transport:\n\n```ts\n// connections/soundcloud.ts, direct OpenAPI with OAuth 2.1 PKCE.\n// Register a SoundCloud app; set SOUNDCLOUD_CLIENT_ID and SOUNDCLOUD_CLIENT_SECRET.\nimport { defineSoundCloudConnection } from \"@assemblyline-agents/soundcloud\";\n\nexport default defineSoundCloudConnection({\n  access: { read: true, write: false }\n});\n```\n\n```ts\n// connections/ffmpeg.ts, stdio bridge; install ffmpeg/ffprobe on the runtime host.\n// All media paths stay inside workspaceRoot; the model never supplies flags or shell.\nimport { defineFfmpegConnection } from \"@assemblyline-agents/ffmpeg\";\n\nexport default defineFfmpegConnection({\n  workspaceRoot: \"/workspace/media\",\n  access: {\n    read: true,\n    write: { approval: \"always\" }\n  }\n});\n```\n\n```ts\n// connections/higgsfield.ts, sandbox CLI transport.\n// Install the official CLI in the sandbox image; run `higgsfield auth login`\n// interactively inside each persistent, user-scoped sandbox.\nimport { defineHiggsfieldConnection } from \"@assemblyline-agents/higgsfield\";\n\nexport default defineHiggsfieldConnection({\n  access: {\n    read: true,\n    write: { approval: \"always\" }\n  }\n});\n```\n\nStdio and sandbox-CLI definitions are trusted application configuration:\nprocess command, arguments, working directory, and environment come from the\nchecked-in connection source, never from the model. Local binaries and project\ndependencies must be provisioned on each runtime host where a stdio\nconnection executes.\n\n## One-Time Binding Packets\n\nConnections such as Margins and Mirror can expose a host-side pairing\nredeemer. While such a connection is unauthorized, connection discovery still\nadvertises `<connection>__pair`. When a user pastes provider-generated binding\ninstructions, pass the complete text through that tool's `secret` field. The\nruntime redacts the field from tool-call evidence, sends the one-time claim\nonly to the connection's configured provider origin, and persists the returned\naccess and refresh credentials in the host grant store. Do not run a packet's\nshell-like line inside the sandbox or copy the claim into an agent file.\n\nFor Margins, the binding packet snapshots one page, one folder and its\ndescendants, or the whole workspace plus `suggest` or `edit` permission. Call\n`margins_status` with the packet's expected fields immediately after pairing.\nAssembly Line's connection policy remains an outer ceiling: comments,\nsuggestions, page creation, and direct edits are write tools and stay hidden\nuntil `write` is enabled with an approval policy. Margins then applies its own\nscope, live-share, permission, stale-head, and revocation checks.\n\nSubagents receive only their declared connection set. The static `connections`\narray is a ceiling. The child must also select each active connection with\n`useConnection()`:\n\n```ts\nimport { defineAgent, useConnection, useModel } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  description: \"Analyze sources, cut social video, and render approved outputs.\",\n  connections: [\"arcads\", \"higgsfield\", \"ffmpeg\", \"remotion\"],\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useConnection(\"arcads\");\n    useConnection(\"higgsfield\");\n    useConnection(\"ffmpeg\");\n    useConnection(\"remotion\");\n  }\n});\n```\n\n## Missing Credentials At Runtime\n\nWhen a tool call reaches a live connection that has no usable credential, the\nruntime raises `ConnectionAuthorizationRequiredError` and parks the run\ninstead of failing it:\n\n- The run's status becomes `waiting_for_connection` and an\n  `authorization.required` event records the connection name, reason, and the\n  authorization challenge (including a session ID for OAuth flows). When a\n  required connection is missing before the model turn, the runtime parks the\n  run and emits `connection.required`.\n- For OAuth and interactive authorization, completing the provider flow at\n  `GET`/`POST /assembly-line/connections/callback`\n  (`runtime.completeConnectionAuthorizationCallback` for embedders) stores\n  the grant and resumes the parked tool call. The resume is idempotency-\n  claimed, so a replayed or double-fired callback never executes the gated\n  tool twice.\n- Env-token connections are configuration, not authorization: a missing\n  `<PROVIDER>_MCP_TOKEN`-style variable surfaces as an explicit\n  `Missing <VAR>` error naming the variable to set.\n- In `tool_search` results, a connection needing authorization is reported\n  with `needsAuthorization: true` rather than silently omitted. Discovery and\n  sandbox credential probing are read-only: they do not create durable\n  authorization sessions.\n\nCallback-URL construction always emits `/assembly-line/connections/callback`;\nthe legacy `/openeve/connections/callback` path is no longer served (`404`).\n\n## Authorizing Before The First Run\n\nOperator surfaces can start (or probe) authorization without parking a run:\n`GET /assembly-line/connections/authorize?connection=<name>` on the node host\n(`runtime.beginConnectionAuthorization()` for embedders). The route is\nauthenticated under the agent-control class and returns JSON. The dashboard\nopens the returned consent URL, and the provider redirects to the public\ncallback:\n\n- `{ \"status\": \"authorize\", \"url\": \"…\" }`: send the user's browser here. A\n  pending session's challenge is reused, so repeated calls never mint\n  duplicate sessions and the same call doubles as a poll while consent is in\n  flight. Expired sessions are pruned before a new one is created.\n- `{ \"status\": \"connected\" }`: a usable grant already exists.\n- `{ \"status\": \"unavailable\", \"reason\": \"…\" }` (409): the connection has no\n  auth definition, an env-token variable is missing (`Missing <VAR>`), or a\n  user-subject connection was called without an identity.\n\nConnections with `subject: \"user\"` key their grant by canonical principal. For\nlegacy runs, pass the same `userId`/`channel` query params. When a channel maps\nusers to an internal principal, also pass its `issuer` so the minted grant is\nthe one that run resolves. Embedders may pass the complete principal directly\nto `runtime.beginConnectionAuthorization()`. Grants live in the runtime's own\nstore, so users authorize once per environment.\n\nConnection auth and header resolvers also receive `ctx.session.principal` and\n`ctx.session.initiator`. An agent-to-agent connection may use those claims to\nmint a signed downstream assertion after explicitly trusting the caller; raw\nOAuth tokens and connection credentials must not be forwarded.\n\n## MCP Transports\n\n`defineMcpClientConnection()` is the backwards-compatible Streamable HTTP\nform; `transport: \"http\"` is optional. `defineMcpStdioConnection()` declares a\nstatic process with `transport: \"stdio\"`, `command`, optional `args`, `cwd`, and\nexplicit environment overrides. `defineMcpRelayConnection()` declares a\nstatic, HTTPS device relay with `transport: \"relay\"`, `url`, `credentialEnv`,\nand an optional bounded timeout. Assembly Line uses the MCP TypeScript SDK for HTTP\nand stdio, routes HTTP/relay traffic through the host request policy, and\nlaunches stdio commands directly without a shell. Each registry keeps one lazy\nclient/process per connection and closes it during runtime shutdown.\n\nProcess- and device-backed MCP definitions are trusted application\nconfiguration. Dynamic connections remain ordinary URL-backed HTTP MCP only\nand cannot supply commands, arguments, working directories, process\nenvironments, relay credentials, or a paired device target.\n\n## Dynamic Connections\n\nDynamic connections are off by default. When enabled in [`agent.ts`](agent-ts.md),\na tool calling `ctx.connectionManager` can persist a new URL-backed MCP, OpenAPI, or HTTP\nconnection definition into the durable connection registry.\n\nSaving is approval-gated by default and restricted by `allowedHosts`. Credentials\nroute through host APIs or authorization flows, never model-visible tool input,\nthe agent folder, sandbox, or prompt.\n\n## Related Docs\n\n- [Plugins: Connection Plugin Catalog](../plugins.md#connection-plugins)\n- [Adapters: Connections](../adapters.md#connections)\n- [Authoring Plugins: Connection Plugins](../authoring-adapters.md#connection-plugins-assemblylineplugin)\n- [Configuration Reference: connections](../config-reference.md#connectionsts)\n- [Customization: Dynamic Connections](../customization.md#dynamic-connections)\n"},{"id":"agent-stack/context-ts","sourcePath":"agent-stack/context-ts.md","title":"context.ts","description":"Customize the default Assembly Line context bundle.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/context-ts","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/context-ts.md","headings":[{"depth":1,"title":"context.ts","anchor":"contextts"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"Custom Policies","anchor":"custom-policies"},{"depth":2,"title":"Wiring Styles","anchor":"wiring-styles"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# context.ts\n\n`context.ts` declares the agent's context bundle policy. Add it when the\ndefault history bounds need tuning or a product needs a named custom policy;\nwhen it is absent, Assembly Line uses `defaultContext()`.\n\nThe default context bundle includes trusted instructions, the active event,\nbounded recent history, memory shape, a file manifest, attachments, compact\nskills and capability catalogs, channel metadata, and trust boundaries. Deferred\ntool schemas and skill bodies are not injected up front.\n\n## Minimal Example\n\nTune the default bundle by default-exporting `defaultContext(options)`:\n\n```ts\nimport { defaultContext } from \"@assemblyline-agents/core\";\n\nexport default defaultContext({\n  recentHistory: { maxMessages: 8 }\n});\n```\n\n## Full Options\n\nA context policy is a `ContextPolicy` object; `defaultContext(options)` and\n`defineContext(policy)` both return one.\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `kind` | `string` | `\"default\"` from `defaultContext()` | Policy identity recorded in the manifest. Any other value names a custom policy. |\n| `name` | `string` | `\"defaultContext\"` from `defaultContext()` | Attribution name; also the export the runtime looks up in `context.ts`. |\n| `options` | JSON object | `{}` | Policy options. Deep-merged across the `extends` chain. |\n| `extends` | `ContextPolicy` | None | Base policy. The runtime flattens the chain into one policy with merged options. |\n\nThe default bundle interprets exactly one options key:\n`recentHistory.maxMessages`, a bound on the recent-history messages included in\nthe bundle. Every other key (for example a `files:` block) is carried into the\nflattened policy and recorded in the manifest, but the default bundle does not\ninterpret it, such keys only have an effect when a custom host or policy\nconsumer reads them.\n\n## Custom Policies\n\nUse `defineContext()` when a product needs a named context policy:\n\n```ts\nimport { defaultContext, defineContext } from \"@assemblyline-agents/core\";\n\nexport default defineContext({\n  kind: \"custom\",\n  name: \"caseContext\",\n  extends: defaultContext({ recentHistory: { maxMessages: 5 } }),\n  options: {\n    includeCaseSummary: true\n  }\n});\n```\n\nA custom `kind` changes attribution, not built-in behavior: the runtime\nflattens the `extends` chain, deep-merges `options`, and records the policy\n(with source attribution) in the manifest. Built-in bundle assembly still reads\nonly `recentHistory.maxMessages`; the host interprets any custom options.\n\n## Wiring Styles\n\nTwo equivalent wirings are supported:\n\n- **Default export in `context.ts`.** The compiler records `context.ts` as the\n  policy source, and the runtime loads its default export at run time.\n- **Named export referenced from `agent.ts`.** Export a named policy from\n  `context.ts` and pass it to `context:` in `agent.ts`. The compiler stamps the\n  identifier name into the manifest, and the runtime resolves that export first,\n  then the default export.\n\n```ts\n// context.ts\nimport { defaultContext, defineContext } from \"@assemblyline-agents/core\";\n\nexport const customContext = defineContext({\n  kind: \"custom\",\n  name: \"customContext\",\n  extends: defaultContext({ recentHistory: { maxMessages: 5 } }),\n  options: { includeTestMarker: true }\n});\n```\n\n```ts\n// agent.ts\nimport { defineAgent, useModel } from \"@assemblyline-agents/core\";\nimport { customContext } from \"./context\";\n\nexport default defineAgent({\n  context: customContext,\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n  }\n});\n```\n\n## Conventions\n\nContext policy is trusted app-runtime behavior. Keep prompt trust clear:\n\n- `instructions.md` and loaded skills are trusted instructions.\n- Files, memory, history, attachments, webpages, search results, and tool output\n  are untrusted context.\n- `/files` and `/history` are read-only projections.\n- `/workspace` is where generated artifacts and modified input copies should go.\n\n## Related Docs\n\n- [Customization: Context](../customization.md#context)\n- [Configuration Reference: defineContext](../config-reference.md#definecontext--defaultcontext-contextts)\n- [agent.ts](agent-ts.md)\n- [Sandbox](sandbox.md)\n"},{"id":"agent-stack/evals","sourcePath":"agent-stack/evals.md","title":"evals/","description":"Define engagement-owned golden cases and run them through the production Assembly Line runtime path.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/evals","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/evals.md","headings":[{"depth":1,"title":"evals/","anchor":"evals"},{"depth":2,"title":"Case contract","anchor":"case-contract"},{"depth":3,"title":"Multi-turn conversations","anchor":"multi-turn-conversations"},{"depth":3,"title":"Tool mocks","anchor":"tool-mocks"},{"depth":3,"title":"Fixtures","anchor":"fixtures"},{"depth":3,"title":"Tool trajectories","anchor":"tool-trajectories"},{"depth":3,"title":"Custom evaluators","anchor":"custom-evaluators"},{"depth":2,"title":"Running evals","anchor":"running-evals"},{"depth":3,"title":"Remote gateways","anchor":"remote-gateways"},{"depth":2,"title":"Experiment artifacts","anchor":"experiment-artifacts"},{"depth":2,"title":"Baselines and regression gates","anchor":"baselines-and-regression-gates"},{"depth":2,"title":"CI","anchor":"ci"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# evals/\n\n`evals/` is the convention for an agent's golden dataset. Assembly Line ships the\nrunner and case contract; cases and scoring opinions remain developer-owned.\nEval files are excluded from `manifest.files`, `sources.json`, and\n`agentRevision`, so changing test data does not create a deployment revision.\n\nUse one JSON file per case:\n\n```json\n{\n  \"name\": \"Escalates an unsafe request\",\n  \"description\": \"The agent should ask for approval before the write.\",\n  \"tags\": [\"high-risk\"],\n  \"input\": {\n    \"message\": \"Apply the account change\",\n    \"approve\": false\n  },\n  \"expect\": {\n    \"status\": \"waiting_for_approval\",\n    \"toolsCalled\": [\"update_account\"],\n    \"toolsNotCalled\": [\"delete_account\"],\n    \"maxCostUsd\": 0.05\n  }\n}\n```\n\n`name` defaults to the filename stem (including subfolders: cases are\ndiscovered recursively, so `evals/billing/refund.json` defaults to\n`billing/refund`; the `evals/evaluators/` subtree is never treated as cases).\nA useful starting suite balances `normal`, `edge`, `ambiguous`, and\n`high-risk` cases.\n\n## Case contract\n\n`input` accepts:\n\n- `message` for a single-turn case, or `turns` for a scripted conversation\n  (exactly one of the two);\n- `channel` and `recentHistory` (multi-turn cases apply `recentHistory` to the\n  first turn only; later turns hydrate history from the conversation itself);\n- `tool` and `toolInput` for a forced authored-tool run;\n- `approve` to override the command-wide `--approve` setting.\n\nA case may also declare top-level `mocks` (canned tool results),\n`fixtures` (starting sandbox or memory state; see below), and `repetitions`\n(integer 1..100): how many times this case runs per suite, winning over the\n`--repetitions` flag. Each repetition produces its own report entry carrying\n`repetitionsPlanned`, so clients can render a per-case pass distribution\nrather than a single binary.\n\n`expect` accepts:\n\n- `status`: any durable run status, including waiting states;\n- `output.contains`: `output.matches`, or `output.equals`;\n- `output.json.valid`: `output.json.jsonSchema`, and `output.json.fields` using\n  dot paths or JSON Pointer paths;\n- `toolsCalled` and `toolsNotCalled`;\n- `trajectory`: for ordered, exact, present, or forbidden tool-call checks;\n- `evaluators`: for developer-owned scoring modules;\n- `maxCostUsd` across the case's recorded model usage;\n- `judge`: `{ \"criteria\": \"...\", \"model\": \"provider/model\",\n  \"threshold\": 0.7 }`.\n\nWhen `agent.ts` declares `outputSchema`, `output.json` assertions read the\nalready-validated `RunAgentResult.output`. Otherwise the assertion layer parses\nthe response as JSON.\n\n### Multi-turn conversations\n\n`input.turns` scripts a conversation. Every turn is its own durable run, and\nall turns share one conversation: the runtime hydrates recent history and\nreuses the conversation-scoped sandbox session exactly as a production channel\nturn would, so behavior across turns (memory of earlier answers, files written\nin turn one and read in turn two) is evaluated for real.\n\n```json\n{\n  \"name\": \"Clarifies before acting\",\n  \"input\": {\n    \"turns\": [\n      {\n        \"message\": \"Change the plan on my account\",\n        \"expect\": { \"toolsCalled\": [\"ask_question\"] }\n      },\n      {\n        \"message\": \"The pro plan, for user@example.com\",\n        \"approve\": true,\n        \"expect\": { \"status\": \"completed\", \"toolsCalled\": [\"update_account\"] }\n      }\n    ]\n  },\n  \"expect\": {\n    \"status\": \"completed\",\n    \"toolsNotCalled\": [\"delete_account\"],\n    \"judge\": { \"criteria\": \"The agent asked before acting and confirmed the change.\" }\n  }\n}\n```\n\nEach turn accepts `message`, `approve`, `tool`, and `toolInput`, plus an\noptional per-turn `expect` limited to the deterministic assertions (`status`,\n`output`, `toolsCalled`, `toolsNotCalled`, `trajectory`, `maxCostUsd`);\nper-turn failures are reported prefixed `turn N:`. `judge` and `evaluators`\nstay case-level.\n\nThe case-level `expect` evaluates the conversation as a whole: `status`,\n`output`, and `response` come from the final turn; `toolsCalled`,\n`toolsNotCalled`, and `trajectory` span every turn in order; `maxCostUsd`\ncovers the summed cost. The judge receives the full transcript, and results\nrecord a `turns` array (per-turn status, response, tools, cost, and run id).\nThe `--timeout` budget applies per turn. A turn that ends waiting (for\napproval or input) parks that run; the next turn starts a new run in the same\nconversation, in-conversation approval resumption is not simulated, so use\nper-turn `approve` to model the granted-approval path.\n\n### Tool mocks\n\n`mocks.tools` replaces named tool executions with canned results, so cases\nthat would otherwise hit external services run deterministically and\nside-effect free. The authored tool module is never imported; approval gates,\ntool-call recording, and run events still apply, so trajectory and approval\nassertions keep working against mocked tools.\n\n```json\n{\n  \"mocks\": {\n    \"tools\": {\n      \"search_accounts\": { \"result\": { \"accounts\": [{ \"id\": \"a1\" }] } },\n      \"flaky_api\": { \"results\": [{ \"attempt\": 1 }, { \"attempt\": 2 }] },\n      \"billing_api\": { \"error\": \"upstream unavailable\" }\n    }\n  }\n}\n```\n\nEach entry declares exactly one of `result` (every call), `results` (consumed\nper call; the last repeats), or `error` (the tool call throws). Sequences\nreset for every attempt and repetition; because per-run sequence cursors are\nin-memory, a crash-resumed run or a subagent child run restarts its sequence.\nBuilt-in sandbox tools can be mocked by name too. Connection tools cannot, so\nthey execute for real wherever the suite runs.\n\nSingle-turn mocks also work against remote gateways that enable\n`ASSEMBLY_LINE_ENABLE_EVAL_RUNS` (see Remote gateways below). Multi-turn cases keep\ntheir mocks local so sequences span turns.\n\n### Fixtures\n\n`fixtures` seeds starting state before the first turn, for cases that assume\nthe agent has already accumulated files or memory:\n\n```json\n{\n  \"fixtures\": {\n    \"sandbox\": { \"notes.txt\": \"existing workspace file\" },\n    \"memory\": { \"/memory/USER.md\": \"Prefers concise answers.\" }\n  }\n}\n```\n\n`sandbox` paths are seeded into the case's sandbox session (relative paths\nland under `/workspace`). `memory` paths must start with `/memory/` and are\nwritten to the memory store in the same scope the case's runs read from.\nCases with sandbox fixtures (like multi-turn cases) run conversation-scoped\nso every turn sees the seeded session.\n\n### Tool trajectories\n\n`toolsCalled` answers only whether a tool appeared. Use `trajectory` when order,\narguments, completion status, or returned values matter:\n\n```json\n{\n  \"expect\": {\n    \"trajectory\": {\n      \"mode\": \"ordered\",\n      \"steps\": [\n        {\n          \"tool\": \"search_accounts\",\n          \"arguments\": { \"email\": \"person@example.com\" },\n          \"argumentsMatch\": \"partial\",\n          \"status\": \"completed\"\n        },\n        {\n          \"tool\": \"update_account\",\n          \"arguments\": { \"plan\": \"pro\" },\n          \"result\": { \"updated\": true }\n        }\n      ]\n    }\n  }\n}\n```\n\nModes are:\n\n- `contains`: every expected step must match a different call, in any order;\n- `ordered`: expected steps must appear in order, with unrelated calls allowed;\n- `exact`: call count, order, and every expected step must match;\n- `forbid`: no listed step may match.\n\nArgument and result matching is recursive and partial by default. Set\n`argumentsMatch` or `resultMatch` to `exact` when extra object fields should\nfail the case.\n\n### Custom evaluators\n\nPut custom scoring logic in `evals/evaluators/<name>.ts` (JavaScript and MTS are\nalso accepted). The framework owns loading, validation, and reporting; the\nagent developer owns the scoring opinion. Wrap the module in `defineEvaluator`\nfrom `@assemblyline-agents/cli/eval`, or export a plain object with\n`satisfies EvalEvaluator`:\n\n```ts\nimport type { EvalEvaluator } from \"@assemblyline-agents/cli/eval\";\n\nexport default {\n  name: \"citation-quality\",\n  async evaluate({ actual, config }) {\n    const minimum = typeof config.minimum === \"number\" ? config.minimum : 1;\n    const count = (actual.response.match(/https:\\/\\//gu) ?? []).length;\n    return {\n      key: \"citations\",\n      score: Math.min(1, count / minimum),\n      pass: count >= minimum,\n      comment: `${count} citations found`\n    };\n  }\n} satisfies EvalEvaluator;\n```\n\nReference it from a case:\n\n```json\n{\n  \"expect\": {\n    \"evaluators\": [\n      {\n        \"name\": \"citation-quality\",\n        \"config\": { \"minimum\": 2 },\n        \"metric\": \"citations\",\n        \"minScore\": 0.8\n      }\n    ]\n  }\n}\n```\n\nThe evaluator context has: `evalCase` (the case), `actual` (the actual\nresponse, parsed output, and per-turn records for conversations), `timeline`\n(the final turn's run timeline), `timelines` (one timeline per turn),\n`config` (the case-owned JSON config), `repetition`, `attempt`, and\n`complete`, a model completer bound to the suite's judge configuration:\n\n```ts\nconst graded = await complete({\n  prompt: \"Rate the citations in this answer...\",\n  systemPrompt: \"Return PASS or FAIL.\", // optional\n  model: \"anthropic/claude-sonnet-5\"     // optional; defaults to the judge model\n});\n// graded.text, graded.costUsd (accounted into the case's judge cost)\n```\n\nThis makes model-based evaluators (rubric panels, pairwise comparisons,\nensemble judging) first-class without wiring a provider client; the scoring\nopinion still lives entirely in the evaluator. An evaluator returns one\nmetric or an array of uniquely named metrics. A metric can expose a numeric\n`score`, JSON `value`, boolean `pass`, and a short `comment`. `requirePass`\ndefaults to true. Evaluators are eval-only code and are not packaged into the\ndeployed agent.\n\n## Running evals\n\n```sh\nassembly-line eval ./agent\nassembly-line eval ./agent --filter escalation\nassembly-line eval ./agent --tag high-risk --concurrency 4\nassembly-line eval ./agent --repetitions 3\nassembly-line eval ./agent --json > eval-report.json\n```\n\nFlags:\n\n- `--judge-model` overrides `ASSEMBLY_LINE_EVAL_JUDGE_MODEL`; a case-level model wins\n  over both. Without any override, the agent model is used.\n- `--timeout` is the per-run timeout in seconds (default 120), applied to each\n  turn of a conversation case. A timeout requests cooperative cancellation and\n  reports an errored case.\n- `--repetitions` runs every valid case multiple times (default 1); a\n  case-level `repetitions` field wins for that case. Malformed case files are\n  reported once rather than repeated.\n- `--retry-attempts` is the number of retries after the first transient\n  provider failure (default 2); `--retry-backoff-ms` sets the initial\n  exponential delay (default 250 ms). Set retry attempts to 0 to disable it.\n- `--fail-fast` stops scheduling new cases after the first failure or error.\n- `--approve` enables gated tools unless `input.approve` overrides it. An\n  approving turn records `tool.approval_requested` plus\n  `tool.approval_auto_resolved` durably and executes the gated tool in the\n  same tick, approval assertions keep working, without parking the run.\n- `--json` emits the stable CI report and suppresses the human table.\n- `--url <gatewayUrl>` runs the cases against a deployed gateway instead of the\n  in-process runtime (see below). `--token` supplies the admin token, falling\n  back to `ASSEMBLY_LINE_ADMIN_TOKEN`.\n\nCases run sequentially by default. Every repetition and retry gets fresh file\nstate, blob, and sandbox roots under `.assembly-line/eval-results/` in the build\nartifact. The runtime uses record-only delivery: it creates and settles the\nnormal durable delivery obligation but never invokes a channel sender.\n\n### Remote gateways\n\n`--url` points the suite at a deployed runtime: runs are created through the\nnode host's `POST /runs` (the target must set `ASSEMBLY_LINE_ENABLE_API_RUNS=true`),\ngraded from `GET /runs/:id/timeline`, and cancelled on timeout through\n`POST /runs/:id/cancel`. Assertions, trajectories, judges, and cost metrics\ngrade the deployed runtime's real timeline.\n\nThe runner probes the target's `/healthz` `capabilities` before sending\nanything eval-specific. Servers that set `ASSEMBLY_LINE_ENABLE_EVAL_RUNS=true` (or\n`allowEvalRuns`) advertise `eval-runs` and accept a per-run `eval` block on\n`POST /runs`: single-turn tool mocks, approval auto-resolve, and record-only\ndelivery all work remotely. The block is persisted in the run's durable input.\nEvery stubbed run is auditable in run detail and marked by its\n`tool.approval_auto_resolved` events. Against servers without the capability,\ncases that need mocks or auto-approval are refused upfront with the reason\nrather than silently degraded: an old server ignoring a stub it never\nreceived cannot execute a real tool by accident.\n\nStill refused remotely regardless of capability: multi-turn conversations,\nseeded `recentHistory`, and fixtures. They depend on local adapters or\n`run()` inputs the HTTP surface does not accept. Case metadata tagging is not\ntransmitted remotely.\n\nSecurity notes. `ASSEMBLY_LINE_ENABLE_EVAL_RUNS` is separate from\n`ASSEMBLY_LINE_ENABLE_API_RUNS` because stubbed runs are a testing surface: enable\nit on dedicated test environments (`assembly-line deploy agent --env test`), not on\nproduction. The normal run-create authentication (admin token or host auth\npolicy) still applies. Connection tools are never stubbable and auto-approval\nmakes gated connection tools execute. Do not point remote evals at\nagents holding live production connections.\n\nFor local (in-process) runs, each judged attempt also appends an\n`eval.judged` event to the run's durable event log with the score, threshold,\npass status, and reasoning. The verdict travels with the run, not just the\nreport.\nRemote runs record verdicts only in the report.\n\nRetries are deliberately narrow. They cover common provider conditions such\nas HTTP 408/425/429/5xx responses, connection resets, DNS retry signals,\ntimeouts reported by a provider, rate limits, and temporarily unavailable or\noverloaded services. Assertion failures, malformed judge responses, local\nevaluator failures, and eval case timeouts are not retried. A transient judge\ncall is retried without rerunning the agent.\n\nThe report distinguishes assertion failures (`failed`) from execution errors\n(`errored`, such as missing provider credentials, model failures, malformed\ncase files, judge parse errors, or timeouts). Either produces a non-zero exit\ncode. Costs are split between agent runs and judge calls, with per-tag results\nincluded in both human and JSON output.\n\n## Experiment artifacts\n\nEvery invocation creates a new experiment directory and never overwrites an\nold one:\n\n```text\n.assembly-line/eval-results/<experiment-id>/\n├── experiment.json\n├── results.json\n├── evaluator-cache/\n└── cases/\n```\n\n`experiment.json` is written before cases run. It records the framework and\nagent revisions, selected case count, planned execution count, safe execution\nconfiguration, and SHA-256 suite/config fingerprints. The suite fingerprint\ncovers selected case bytes and bundled source for referenced custom evaluators,\nincluding local imports. Environment values and provider credentials are never\ncaptured. `results.json` is written once after the run and contains the\nmanifest, per-execution outcomes and metrics, costs, and artifact paths. A crash\ntherefore leaves an inspectable manifest without pretending the experiment\ncompleted.\n\n## Baselines and regression gates\n\nCompare a run with a prior `results.json`, `experiment.json`, or experiment\ndirectory:\n\n```sh\nassembly-line eval ./agent \\\n  --baseline .assembly-line/eval-results/<experiment-id> \\\n  --max-regressions 0 \\\n  --max-cost-increase-percent 20\n```\n\nFor each case present in both experiments, Assembly Line compares the fraction of\nrepetitions that passed. A lower fraction is a regression; a higher fraction is\nan improvement. Added and removed case IDs are listed separately. The default\ngate allows zero regressions. The cost-growth gate is opt-in; a positive cost\nagainst a zero-cost baseline fails it as an unbounded percentage increase.\nFailed gates set the CLI exit status to non-zero even when every current case\npasses. This keeps framework policy neutral: teams choose the baseline,\nrepetition count, and acceptable cost budget.\n\n## CI\n\nStore provider credentials in the CI secret store, never in case files. A\nminimal gate is:\n\n```sh\nassembly-line eval ./agent --json \\\n  --baseline .assembly-line/eval-results/<approved-experiment-id>\n```\n\nFor deterministic PR checks, omit `expect.judge`; run judge cases in a separate\njob when model variability or provider availability should not block every\ncode change. Repetitions are also opt-in because they multiply runtime and\nprovider cost; use them for nondeterministic or release-critical suites rather\nthan every fast local check.\n\n## Related Docs\n\n- [agent.ts: Structured Outputs](agent-ts.md#structured-outputs)\n- [Tools](tools.md)\n- [Runtime And Deployment](../runtime-and-deployment.md)\n- [Configuration Reference](../config-reference.md)\n"},{"id":"agent-stack/gateway-ts","sourcePath":"agent-stack/gateway-ts.md","title":"gateway.ts","description":"Declare the portable runtime stack for an Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/gateway-ts","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/gateway-ts.md","headings":[{"depth":1,"title":"gateway.ts","anchor":"gatewayts"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"Provider Independence","anchor":"provider-independence"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# gateway.ts\n\n`gateway.ts` is the portable stack declaration. Add it when you choose where\nthe compiled runtime runs and which adapters provide state, blobs, sandboxing,\nand schedules.\n\n## Minimal Example\n\n```ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\n\nexport default defineGateway({\n  deploy: adapter(\"railway\"),\n  runtime: adapter(\"node\"),\n  state: adapter(\"postgres\"),\n  blob: adapter(\"r2\"),\n  sandbox: adapter(\"docker\"),\n  scheduler: adapter(\"postgres\")\n});\n```\n\n## Full Options\n\nEvery slot is optional. Compiler defaults are `adapter(\"local\")`, except\n`runtime`, which defaults to `adapter(\"node\")`.\n\n| Slot | Chooses | Default |\n| --- | --- | --- |\n| `deploy` | Where the compiled runtime service runs. | `adapter(\"local\")` |\n| `runtime` | The HTTP/runtime host. | `adapter(\"node\")` |\n| `state` | Durable run, event, checkpoint, delivery, memory, and schedule state. | `adapter(\"local\")` |\n| `blob` | Attachments, extracted text, generated artifacts, and sandbox sync bundles. | `adapter(\"local\")` |\n| `sandbox` | Default isolated filesystem and command backend. | `adapter(\"local\")` |\n| `scheduler` | Where scheduled ticks are coordinated. | `adapter(\"local\")` |\n\n`assembly-line add <kind>` edits this file for you when the added plugin fills a\ngateway role (`deploy`, `runtime`, `state`, `blob`, `sandbox`, `scheduler`),\nwiring the slot to the installed adapter.\n\n## Provider Independence\n\nDeploy choice does not imply a state, blob, sandbox, or scheduler vendor. A\nRailway, Docker, Fly, or VPS deployment can use Postgres state, R2 blobs, a hosted\nsandbox, and gateway-triggered schedules. Assembly Line keeps these boundaries\nexplicit so the agent can move between hosts. Deployment planning refuses the\nunconfined local sandbox on non-local targets unless it is explicitly\nacknowledged, and warns when local state or blobs would be container-local.\n\nArtifacts may also declare runtime requirements independently of a provider.\nFor example, `openai-codex/*` requires a sensitive persistent `/data` directory\nand remote command execution. Any deploy plugin advertising the corresponding\ncapabilities can satisfy the same artifact contract.\n\n## Conventions\n\nThe compiler reads `gateway.ts` from the TypeScript AST, so the file must stay\nstatically analyzable: no computed values, conditionals, or dynamic imports.\nUse statically readable `defineGateway({ ... })`, `adapter(\"kind\")`, or\nprovider helper calls such as `railwayDeploy()`, `neonPostgres()`,\n`railwayPostgres()`, `supabasePostgres()`, `r2Blob()`, and `dockerSandbox()`.\n\n## Related Docs\n\n- [Adapters](../adapters.md)\n- [Runtime And Deployment](../runtime-and-deployment.md)\n- [Configuration Reference: defineGateway](../config-reference.md#definegateway-gatewayts-and-adapter)\n"},{"id":"agent-stack/hooks","sourcePath":"agent-stack/hooks.md","title":"hooks/","description":"React to durable runtime events without changing the run being observed.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/hooks","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/hooks.md","headings":[{"depth":1,"title":"hooks/","anchor":"hooks"},{"depth":2,"title":"Runtime Contract","anchor":"runtime-contract"},{"depth":2,"title":"Compatibility","anchor":"compatibility"}],"content":"# hooks/\n\nHooks are filesystem-authored, agent-level reactions to durable runtime events.\nUse them for audit records, application metrics, notifications, analytics, and\nother cross-cutting side effects that should run regardless of which channel or\nautomation started the run.\n\n```ts\n// hooks/audit.ts\nimport { defineHook } from \"@assemblyline-agents/core\";\n\nexport default defineHook({\n  description: \"Record completed runs in the application audit log.\",\n  events: {\n    async \"run.completed\"(event, ctx) {\n      await auditLog.record({\n        runId: ctx.runId,\n        agent: ctx.agent.name,\n        hook: ctx.hook?.name,\n        status: event.data.status ?? \"completed\"\n      });\n    }\n  }\n});\n```\n\nThe filename is the compiled hook name. Each key in `events` is a durable run\nevent type; `\"*\"` observes every event. Exact handlers run before wildcard\nhandlers. Callbacks receive the persisted event plus run, agent, and hook\nidentity.\n\n## Runtime Contract\n\nThe runtime persists and publishes the event before invoking hooks. A thrown\nhook records `agent.event_handler_failed` and is logged, but it does not change\nthe originating run's result. Hooks may perform real side effects, so protect\nat-least-once external writes with an application idempotency key.\n\nHooks are reactors, not interceptors:\n\n- They cannot replace the run message, target, model, tools, or result.\n- They do not contribute model context.\n- They should not own work required for a run to be considered successful.\n\nPut required automation preparation or finalization directly in the owning\n[`automations/`](automations.md) file. Put capability composition in\n[`agent.ts`](agent-ts.md), provider ingress in [`channels/`](channels.md), and\ntelemetry exporter configuration in [`instrumentation.ts`](instrumentation.md).\n\n## Compatibility\n\n`useEvent(type, handler)` in `agent.ts` remains accepted for older agents but\nemits a compiler deprecation warning. Move those callbacks to `hooks/*.ts` and\nwrap their `events` map with `defineHook()`.\n"},{"id":"agent-stack/instructions","sourcePath":"agent-stack/instructions.md","title":"instructions.md","description":"Write the always-on trusted instructions for an Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/instructions","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/instructions.md","headings":[{"depth":1,"title":"instructions.md","anchor":"instructionsmd"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":1,"title":"Identity","anchor":"identity"},{"depth":1,"title":"Operating Rules","anchor":"operating-rules"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Runtime Context","anchor":"runtime-context"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# instructions.md\n\n`instructions.md` is the agent's always-on trusted prompt: identity, stable\noperating rules, safety boundaries, and product behavior that applies to every\nturn. Every agent needs this file; it is one of the two required files.\n\n## Minimal Example\n\n```md\n# Identity\n\nYou are a concise Assembly Line agent. Use tools when they are available.\n\n# Operating Rules\n\n- Ask a clarifying question when a required input is missing.\n- Use durable tools for durable side effects.\n- Write generated artifacts under /workspace.\n```\n\nThere are no options: the whole file is Markdown prompt text. The compiler\nrecords its hash in the manifest, and the runtime loads the body verbatim as\ntrusted instructions. Skill bodies loaded from `skills/` join it at the same\ntrust level.\n\n## Conventions\n\nWhat belongs here:\n\n- Agent role, tone, and durable behavioral rules.\n- Domain-specific rules the model should always follow.\n- Boundaries between trusted instructions and untrusted context.\n- Stable escalation, approval, or handoff guidance.\n\nWhat does not belong here:\n\n- Dynamic user memory, secrets, access tokens, or provider credentials.\n- Large procedures that are only relevant sometimes; put those in\n  [`skills/`](skills.md).\n- Provider webhook parsing; put that in [`channels/`](channels.md).\n- Deployment or storage choices; put those in [`gateway.ts`](gateway-ts.md).\n\nKeep the file short enough to read in one sitting, a page or two. Instructions\nare injected into every turn, so every extra paragraph costs tokens on every\nrun; move sometimes-useful procedures into skills.\n\n## Runtime Context\n\nAfter `instructions.md`, the runtime adds a stable filesystem contract that\nnames logical paths:\n\n| Path | Mutability | Purpose |\n| --- | --- | --- |\n| `/memory` | writable by policy | Durable memory documents. |\n| `/skills` | writable when self-improvement allows it | Durable skill files. |\n| `/history` | read-only | Bounded conversation history projection. |\n| `/files` | read-only | Input files and attachment projections. |\n| `/workspace` | writable | Generated artifacts, scripts, and modified copies. |\n\nCore file tools use absolute logical paths such as `/workspace/report.txt`.\nHosted `bash` starts in the physical `/workspace` directory, so absolute and\nrelative workspace paths agree. The Local adapter is a trusted host-directory\nemulation; use Docker when a local run must reproduce absolute `/workspace`\nshell semantics.\n\nFiles, history, memory, webpages, search results, attachments, and tool output\nare context, not instructions. Keep that distinction explicit in the agent's\nprompt when the domain has sensitive decisions.\n\n## Related Docs\n\n- [Skills](skills.md)\n- [Context](context-ts.md)\n- [Sandbox](sandbox.md)\n- [Configuration Reference](../config-reference.md#per-file-contracts-the-compiler-validates)\n"},{"id":"agent-stack/instrumentation","sourcePath":"agent-stack/instrumentation.md","title":"instrumentation.ts","description":"Configure telemetry and content capture for an Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/instrumentation","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/instrumentation.md","headings":[{"depth":1,"title":"instrumentation.ts","anchor":"instrumentationts"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"Agent Runs","anchor":"agent-runs"},{"depth":2,"title":"OTLP Export","anchor":"otlp-export"},{"depth":2,"title":"Capture Levels","anchor":"capture-levels"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# instrumentation.ts\n\n`instrumentation.ts` is the single home for telemetry, auto-discovered and run\nonce at startup before the first agent turn. Add it when you want an external\ntelemetry sink or explicit content-capture controls.\n\n## Minimal Example\n\n```ts\nimport { defineInstrumentation } from \"@assemblyline-agents/core\";\nimport { createOtlpSinkFromEnv } from \"@assemblyline-agents/otlp\";\n\nexport default defineInstrumentation({\n  serviceName: \"starter-agent\",\n  captureContent: \"usage\",\n  setup: ({ env }) => createOtlpSinkFromEnv(env)\n});\n```\n\n## Full Options\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `serviceName` | `string` | agent name | Service name stamped on spans. |\n| `setup` | `({ agentName, manifest, env }) => sink` | None | Runs once at startup; return a telemetry sink to export spans. |\n| `captureContent` | level string \\| policy object | `\"usage\"` | Content-capture detail; a bare level is shorthand for `{ level }`. |\n| `recordInputs` / `recordOutputs` | `boolean` | `false` | Record run inputs/outputs on spans. |\n\n`captureContent` also accepts a policy object:\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `level` | `off` \\| `usage` \\| `content` \\| `full` | None | Capture level (required in object form). |\n| `maxChars` | `number` | ~8000 (relaxed at `full`) | Per-field character cap before truncation. |\n| `redact` | `boolean` | `true` (even at `full`) | Key-based redaction of sensitive values. |\n| `redactKeys` | `string[]` | built-in set | Extra sensitive key names to redact. |\n| `sampleRate` | `number` 0..1 | `1` | Per-trace sampling of content; usage/metadata always export. |\n| `includeToolIO` | `boolean` | `true` at `content`/`full` | Capture tool arguments and results. |\n\n## Agent Runs\n\nAgent Runs-style observability is available without `instrumentation.ts` through\nrun, event, tool, checkpoint, delivery, usage, subagent, and timeline query\ncontracts. The Node host exposes authenticated inspection endpoints under\n`/runs`.\n\n## OTLP Export\n\n`@assemblyline-agents/otlp` provides:\n\n- `createOtlpSink(options)`\n- `createOtlpSinkFromEnv(env)`\n\n`createOtlpSinkFromEnv` reads `OTEL_EXPORTER_OTLP_ENDPOINT`,\n`OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_SERVICE_NAME`,\n`OTEL_EXPORTER_OTLP_TIMEOUT`, `ASSEMBLY_LINE_OTLP_BATCH_MAX`, and\n`ASSEMBLY_LINE_OTLP_FLUSH_MS`. Any OTLP backend works: Langfuse, Phoenix, Grafana,\nand others.\n\n## Capture Levels\n\n| Level | Records |\n| --- | --- |\n| `off` | No span content. |\n| `usage` | Token usage, cost, model, provider, finish reason, and tool-call spans. No message bodies. |\n| `content` | Adds bounded prompt, completion, and tool IO previews with redaction. |\n| `full` | More complete prompt, completion, and tool IO capture; use only in trusted environments. |\n\n## Conventions\n\n- Prefer `usage` in production unless operators have an explicit reason to\n  capture content and the product privacy materials account for it.\n- Host- or gateway-supplied instrumentation overrides win over\n  `instrumentation.ts`.\n\n## Related Docs\n\n- [Customization: Observability](../customization.md#observability)\n- [Configuration Reference: defineInstrumentation](../config-reference.md#defineinstrumentation-instrumentationts)\n- [Runtime And Deployment: Node Runtime HTTP API](../runtime-and-deployment.md#node-runtime-http-api)\n"},{"id":"agent-stack/overview","sourcePath":"agent-stack/overview.md","title":"Agent Build Stack","description":"The folder-first map for building an Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/overview","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/overview.md","headings":[{"depth":1,"title":"Agent Build Stack","anchor":"agent-build-stack"},{"depth":2,"title":"File Map","anchor":"file-map"},{"depth":2,"title":"Design Rules","anchor":"design-rules"},{"depth":2,"title":"What Happens At Build Time","anchor":"what-happens-at-build-time"}],"content":"# Agent Build Stack\n\nAn Assembly Line agent is a directory. Each file or folder has one job, and the\ncompiler turns that directory into a manifest, runtime artifact, route table,\nschedule table, and deploy plan.\n\nOnly `instructions.md` and `agent.ts` are required. Add the rest as the agent\ngrows. Start here: [`instructions.md`](instructions.md) and\n[`agent.ts`](agent-ts.md). Run `assembly-line init` to scaffold both, then edit\nthem.\n\n```txt\nagent/\n  instructions.md\n  agent.ts\n  context.ts\n  gateway.ts\n  tools/\n  skills/\n  channels/\n  automations/\n  hooks/\n  connections/\n  sandbox/\n  subagents/\n  evals/\n  instrumentation.ts\n```\n\n## File Map\n\n| Path | Purpose | Add it when |\n| --- | --- | --- |\n| [`instructions.md`](instructions.md) | Always-on trusted instructions. | You define the agent's role, tone, rules, and durable behavior. |\n| [`agent.ts`](agent-ts.md) | Static identity/policy plus synchronous composition functions. | You select models, instructions, tools, skills, connections, subagents, sandboxes, or schemas. |\n| [`context.ts`](context-ts.md) | Context bundle policy. | The default history, memory, files, or trust-boundary behavior needs tuning. |\n| [`gateway.ts`](gateway-ts.md) | Portable runtime stack choices. | You choose deploy, state, blob, sandbox, scheduler, or runtime adapters. |\n| [`tools/`](tools.md) | Typed actions the model can call. | The agent needs to do work through reviewed app-runtime code. |\n| [`skills/`](skills.md) | On-demand procedures and reference material. | Guidance is useful only sometimes, or the agent should improve by editing skills. |\n| [`channels/`](channels.md) | External entrypoints and reply delivery. | The same agent should receive HTTP, Slack, Discord, Teams, Telegram, Photon, or custom events. |\n| [`automations/`](automations.md) | Time- or event-triggered runs with optional inline preparation and finalization. | Work should run on cron or in response to an external event. |\n| [`hooks/`](hooks.md) | Cross-cutting reactions to durable runtime events. | You need audit records, metrics, notifications, analytics, or application synchronization without changing run control flow. |\n| [`connections/`](connections.md) | External service capability and credential contracts. | Tools need GitHub, MCP, OpenAPI, HTTP APIs, OAuth, or workspace/user credentials. |\n| [`sandbox/`](sandbox.md) | Isolated filesystem and command execution backend. | The agent needs shell/file work outside the trusted app process. |\n| [`subagents/`](subagents.md) | Pi-backed child agents. | A task should run with a narrower identity, model, tool set, workspace, connection set, or durable run boundary. |\n| [`evals/`](evals.md) | Engagement-owned golden dataset. | You need regression checks for agent behavior, tool choices, structured output, cost, or qualitative criteria. |\n| [`instrumentation.ts`](instrumentation.md) | Telemetry setup and capture policy. | You want OTLP export or explicit content-capture controls. |\n\n## Design Rules\n\n- Let `agent.ts` decide what the agent can do now; keep how capabilities work in their small modules.\n- Put provider event parsing in `channels/`, not tools.\n- Put deployment and storage choices in `gateway.ts`, not `agent.ts`.\n- Put realtime service access in `connections/` and typed tools, not subagent engine configuration.\n- Put reusable procedures in `skills/`, not long tool descriptions.\n- Put secrets in environment variables or credential stores, never in model-visible files.\n- Treat files, history, memory, search results, webpages, and tool output as untrusted context.\n- Write generated artifacts and modified input copies under `/workspace`.\n- In `bash`, use paths relative to the workspace cwd for local-sandbox portability.\n\n## What Happens At Build Time\n\nAssembly Line statically reads the agent directory. The compiler validates known\ntop-level files, extracts `define*` declarations from TypeScript, records source\nhashes, and emits `.assembly-line/`. `evals/` is excluded from the revision hash, so\nchanging test cases never creates a new deployment revision.\n\n```txt\nagent/ source\n  -> compiler validation\n  -> manifest.json\n  -> route-table.json\n  -> schedules.json\n  -> preflight.json\n  -> server/boot.js\n```\n\nFor the full artifact shape, see [Runtime And Deployment](../runtime-and-deployment.md#build-artifact).\nFor every config field, see [Configuration Reference](../config-reference.md).\n"},{"id":"agent-stack/sandbox","sourcePath":"agent-stack/sandbox.md","title":"sandbox/","description":"Choose the isolated filesystem and command backend for an agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/sandbox","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/sandbox.md","headings":[{"depth":1,"title":"sandbox/","anchor":"sandbox"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":3,"title":"Multiple Sandbox Files","anchor":"multiple-sandbox-files"},{"depth":2,"title":"Filesystem Contract","anchor":"filesystem-contract"},{"depth":2,"title":"Adapter Choices","anchor":"adapter-choices"},{"depth":2,"title":"Snapshots","anchor":"snapshots"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# sandbox/\n\nSandbox files choose the agent computer for isolated file and shell work. Add\none when the agent needs shell or file work outside the trusted app process.\n\n## Minimal Example\n\n```ts\n// sandbox/default.ts\nimport { defineSandbox } from \"@assemblyline-agents/core\";\n\nexport default defineSandbox({\n  adapter: \"docker\",\n  image: \"node:22\",\n  workingDirectory: \"/workspace\",\n  env: [\"OPENAI_API_KEY\"]\n});\n```\n\n## Full Options\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `adapter` | `string` (required) | None | Sandbox backend kind: `local`, `docker`, `daytona`, `e2b`, `modal`, or a plugin kind. |\n| `image` | `string` | adapter default | Container/base image for containerized adapters. |\n| `workingDirectory` | `\"/workspace\"` | `\"/workspace\"` | Compatibility field for the canonical Assembly Line workspace. Other values are rejected. |\n| `env` | `string[]` | None | Names of env vars forwarded into the sandbox. |\n| `snapshot` | `{ mode, retainLast?, reason? }` | `mode: \"never\"` | Infrastructure checkpoint policy (see Snapshots). |\n| `metadata` | JSON object | None | Structured app-specific metadata. |\n\n### Multiple Sandbox Files\n\nEvery `.ts`/`.js` file in `sandbox/` compiles into the manifest's sandbox list\n(sorted by filename; the file stem becomes the entry name, and the entry's\n`adapter` falls back to the helper kind or the file stem when not declared). At\nruntime the host selects the entry whose `adapter` matches the gateway's\n`sandbox` adapter kind, and otherwise falls back to the first entry. So the\nfilename itself does not select a sandbox. Declare one file per adapter kind\nyou configure, and let `gateway.ts` choose between them.\n\n## Filesystem Contract\n\nAssembly Line's hosted sandbox contract is a real, physical `/workspace` directory:\n\n- the default shell cwd is `/workspace`;\n- absolute shell paths such as `/workspace/report.txt` address that directory;\n- provider file APIs and shell commands address the same files; and\n- create, connect, and wake validate the contract before returning a session.\n\nThe contract is versioned in provider metadata and runtime session manifests.\nAssembly Line does not reconnect a sandbox created under an older or missing\ncontract, and provider names include the contract version to avoid collisions.\n`workingDirectory` may be omitted or set to `/workspace`; a different path is\nrejected because a file-API alias cannot rewrite absolute paths inside shell\ncommands. `/runtime` remains retired and rejected.\n\n| Path | Mutability | Purpose |\n| --- | --- | --- |\n| `/memory` | writable by policy | Durable memory documents. |\n| `/skills` | writable when self-improvement allows it | Durable skill files. |\n| `/history` | read-only | Bounded conversation history projection. |\n| `/files` | read-only | Input files and attachment projections. |\n| `/workspace` | writable | Generated artifacts, scripts, and modified copies. |\n\nCore file tools accept these absolute paths, and hosted `bash` commands may use\nthe same absolute paths directly.\n\nSandboxes are acquired lazily when a sandbox-backed tool or capability asks for\none. Channel lifecycle events and final delivery do not require sandbox\nhydration.\n\n## Adapter Choices\n\n- `local`: trusted development and tests only. It maps the logical paths onto a\n  host temporary directory, so it cannot provide a physical host `/workspace`.\n  Use Docker locally when exact shell namespace parity matters.\n- `docker`: local isolation baseline.\n- `daytona`: `e2b`: hosted sandbox choices.\n- `modal`: supported hosted Modal sandbox.\n\nUse Docker or hosted sandboxes when untrusted or model-generated code needs an\nisolation boundary.\n\n## Snapshots\n\nSnapshots are infrastructure checkpoints, not normal turn persistence. Runtime\nstate, memory, tool traces, delivery, and `/workspace` artifacts persist through\nAssembly Line state and blob sync.\n\n```ts\nexport default defineSandbox({\n  adapter: \"daytona\",\n  image: \"node:22\",\n  snapshot: {\n    mode: \"manual\",\n    retainLast: 3,\n    reason: \"operator-requested checkpoint\"\n  }\n});\n```\n\nSupported modes are `never`, `manual`, `on_failure`, and `always`. The default\nis `never`.\n\n## Conventions\n\n- Keep the local sandbox for trusted development and tests.\n- Run Docker for local conformance testing of absolute `/workspace` shell paths.\n- Write generated artifacts and modified input copies under `/workspace`.\n- Forward only the env vars the sandboxed work actually needs.\n\n## Related Docs\n\n- [Adapters: Sandboxes](../adapters.md#sandboxes)\n- [Configuration Reference: sandbox](../config-reference.md#sandboxts)\n- [Runtime And Deployment: Sandbox Sync](../runtime-and-deployment.md#sandbox-sync)\n- [gateway.ts](gateway-ts.md)\n"},{"id":"agent-stack/skills","sourcePath":"agent-stack/skills.md","title":"skills/","description":"Add on-demand procedures and self-improvement surfaces.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/skills","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/skills.md","headings":[{"depth":1,"title":"skills/","anchor":"skills"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":1,"title":"Note Taking","anchor":"note-taking"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"How Skills Load","anchor":"how-skills-load"},{"depth":2,"title":"Self-Improvement","anchor":"self-improvement"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# skills/\n\nSkills live at `skills/<name>/SKILL.md`. They are procedures or reference\nmaterial loaded on demand, not always-on prompt text. Add a skill when guidance\nis useful only sometimes, or when the agent should improve by editing skills.\n\nThere is no `defineSkill`. The `SKILL.md` file is the whole contract:\n\n```sh\nmkdir -p skills/note-taking\n$EDITOR skills/note-taking/SKILL.md\n```\n\n## Minimal Example\n\n```md\n---\ndescription: Capture durable notes for the user\nallowed-tools: [record_note]\n---\n\n# Note Taking\n\nUse this skill when the user shares a fact, preference, or decision that should\noutlive the conversation.\n```\n\n## Full Options\n\nFrontmatter is the skill's entire option surface:\n\n| Key | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `description` | `string` | `Skill <name>` | Model-facing summary in the compact skill index; drives skill selection. |\n| `allowed-tools` | `string[]` | None | Tools the procedure is expected to use. |\n| `tags` | `string[]` | None | Discovery tags in the capability catalog. |\n| `aliases` | `string[]` | None | Alternate names for capability lookup. |\n\nThe body below the frontmatter is the skill's trusted instructions, loaded at\nthe same trust level as `instructions.md` when the skill is selected.\n\n## How Skills Load\n\n`useSkill()` selects the compact skill entries available in a capability\nsnapshot. Select `load_skill` with `useTool()` when the model should retrieve a\nselected skill's full body on demand. `load_skill` cannot load an unselected\nskill and never widens the snapshot's tool set.\n\nIf a sandbox already exists, the selected skill may also be projected into\n`/skills/<name>/SKILL.md` for filesystem inspection.\n\n## Self-Improvement\n\nSelf-improvement reviews completed runs and saves reusable learning to durable\nmemory or skills. It is enabled by default; configure it in\n[`agent.ts`](agent-ts.md) to tune review triggers or require owner approval:\n\n```ts\nimport { defineAgent, useModel, useSkill } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  id: \"research-agent\",\n  selfImprovement: {\n    enabled: true,\n    writeApproval: false,\n    reviewEveryTurns: 10,\n    reviewMinToolCalls: 5,\n    reviewModel: \"inherit\"\n  },\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useSkill(\"research\");\n  }\n});\n```\n\nThe reviewer sees bounded evidence and receives only memory and skill tools.\nWith `writeApproval: false` it writes directly. Approval-gated changes are\nstored separately from the active skill. Each skill mutation appends a full-body\nrevision; archive replaces permanent deletion, and rollback copies an older body\ninto a new revision. Compiled `skills/` seed the writable durable store, while\n`externalDirs` are read-only. Set `enabled: false` for a static skill surface.\n`writable` remains a deprecated alias.\n\n## Conventions\n\n- Keep skills focused on repeatable procedures.\n- Put durable product data in memory or state, not in skill bodies.\n- Prefer a short `description` that helps the model choose the skill.\n- Keep `allowed-tools` aligned with the tools the procedure actually needs.\n\n## Related Docs\n\n- [Configuration Reference: skills](../config-reference.md#skillsnameskillmd)\n- [Customization: Self-Improvement](../customization.md#self-improvement)\n- [Tools](tools.md)\n- [instructions.md](instructions.md)\n"},{"id":"agent-stack/subagents","sourcePath":"agent-stack/subagents.md","title":"subagents/","description":"Delegate focused work to hook-composed child agents.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/subagents","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/subagents.md","headings":[{"depth":1,"title":"subagents/","anchor":"subagents"},{"depth":2,"title":"Exposing And Invoking A Subagent","anchor":"exposing-and-invoking-a-subagent"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# subagents/\n\nSubagents live under `subagents/<name>/`, each with its own `instructions.md`\nand `agent.ts`. A child uses the same `defineAgent()` plus synchronous hook\nmodel as its parent, but gets an isolated durable run and conversation-scoped\ncontrol state.\n\nChild runs automatically inherit the parent's canonical `principal`,\n`initiator`, project, and metadata. A child's `useRun()` and tool contexts can\ntherefore apply the same role policy, and user-subject connections resolve for\nthe same person. Credentials are never copied into the child run.\n\n```ts\n// subagents/researcher/agent.ts\nimport { defineAgent, useModel, useReasoning, useTool } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  description: \"Use this subagent for scoped research tasks.\",\n  maxReasoning: \"low\",\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useReasoning(\"low\");\n    useTool(\"search_docs\");\n  }\n});\n```\n\nThe description is parent-facing when-to-use guidance. Every child must select\nexactly one model in `setup()`; models do not implicitly inherit from the\nparent.\n\n`description` is required for a subagent. `workspace` and `connections` add\nchild-specific limits. Other static fields, including `maxReasoning` and\n`maxIterations`, use the same contract as the parent.\n\n| Field | Effect |\n| --- | --- |\n| `description` | Required guidance shown to the parent. |\n| `workspace` | Static sandbox/workspace adapter ceiling. |\n| `connections` | Names the child is permitted to select with `useConnection()`. |\n| `maxReasoning`, `maxIterations` | Shared agent fields that set hard runtime ceilings. |\n\nThe child selects tools, skills, connections, output schema, sandbox profile,\nreasoning, and model through hooks. Its named references must exist in the\nparent artifact's compiled capability catalog.\n\n```ts\nimport { adapter, defineAgent, useConnection, useModel, useTool } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  description: \"Use this for durable coding sessions.\",\n  workspace: adapter(\"local\"),\n  connections: [\"github\"],\n  setup() {\n    useModel(\"openai/gpt-5-codex\");\n    useConnection(\"github\");\n    useTool(\"read\");\n  }\n});\n```\n\n## Exposing And Invoking A Subagent\n\nThe parent must call `useSubagent(\"researcher\")` before that child is visible.\nThe parent model then calls the snapshot-scoped `delegate_<name>` tool; authored tools may call\n`ctx.spawnSubagent(...)` directly:\n\n```ts\nawait ctx.spawnSubagent({\n  name: \"researcher\",\n  task: \"Summarize the three most recent issues.\",\n  expectedOutput: \"A markdown summary with issue links.\",\n  constraints: [\"Read-only: do not comment on issues.\"]\n});\n```\n\nRealtime services remain connections and typed tools, not child-agent engines.\nThe runtime enforces connection authorization and tool approvals independently\nfor the child.\n\n## Related Docs\n\n- [agent.ts](agent-ts.md)\n- [Connections](connections.md)\n- [Tools](tools.md)\n- [Runtime And Deployment](../runtime-and-deployment.md)\n"},{"id":"agent-stack/tools","sourcePath":"agent-stack/tools.md","title":"tools/","description":"Add typed model-callable actions to an Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/agent-stack/tools","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/agent-stack/tools.md","headings":[{"depth":1,"title":"tools/","anchor":"tools"},{"depth":2,"title":"Minimal Example","anchor":"minimal-example"},{"depth":2,"title":"Full Options","anchor":"full-options"},{"depth":2,"title":"Execution Model","anchor":"execution-model"},{"depth":2,"title":"Approval Gates","anchor":"approval-gates"},{"depth":2,"title":"Durable Steps","anchor":"durable-steps"},{"depth":2,"title":"Safe Model Output","anchor":"safe-model-output"},{"depth":2,"title":"Execution Context","anchor":"execution-context"},{"depth":2,"title":"Conventions","anchor":"conventions"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# tools/\n\nEach file in `tools/` becomes one model-facing tool. Add a tool when the agent\nneeds to do work through reviewed app-runtime code.\n\nThe filename stem is the tool name: `tools/record_note.ts` compiles to the\n`record_note` tool. Use `snake_case` filenames so tool names match what the\nmodel sees.\n\n## Minimal Example\n\n```ts\n// tools/echo.ts\nimport { defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Echo a message back to the caller.\",\n  inputSchema: {\n    type: \"object\",\n    properties: {\n      message: { type: \"string\" }\n    },\n    required: [\"message\"]\n  },\n  async execute(input: { message: string }) {\n    return { message: input.message };\n  }\n});\n```\n\n## Full Options\n\n| Field | Type / values | Default | Effect |\n| --- | --- | --- | --- |\n| `description` | `string` (required) | None | Model-facing purpose statement; drives tool selection. |\n| `inputSchema` | JSON Schema (required) | None | Argument contract enforced before `execute`. |\n| `outputSchema` | JSON Schema | None | Declared result contract. |\n| `execute` | `(input, ctx) => result` | None | Tool body; runs in the trusted app runtime. |\n| `toModelOutput` | `(output) => projection` | None | Bounded projection shown to the model; full result is persisted. |\n| `needsApproval` | `boolean` \\| `ApprovalPolicy` | `false` | Approval gate. `approvalRequired(reason, sideEffect?)` builds an always-approve policy; `sideEffect` defaults to `\"external\"`. |\n| `defaultEnabled` | `boolean` | inferred | Forces the tool visible up front instead of behind deferred discovery. |\n| `sideEffect` | `\"none\"` \\| `\"idempotent\"` \\| `\"external\"` | None | Side-effect class recorded for approval and audit surfaces. |\n| `capability` | `{ visibility?, execution?, namespace?, tags?, aliases? }` | None | Discovery metadata. `visibility`: `auto` \\| `always` \\| `deferred` \\| `skill` \\| `hidden`. `execution`: `auto` \\| `direct` \\| `sandbox` \\| `both`. |\n\nReturn values must be JSON-serializable. A thrown error marks the tool call\n`failed`, surfaces `{ error }` to the model, and fails the run. A thrown error\nmeans the agent's own code broke. Return an error-shaped result\ninstead of throwing when the condition is something the model should recover\nfrom.\n\n## Execution Model\n\nTool code runs in the trusted app runtime by default. Use `ctx.getSandbox()`\nonly when the tool needs isolated filesystem or shell work.\n\n`useTool(\"name\")` in `agent.ts` makes a compiled tool visible in the current\ncapability snapshot. Tool metadata still controls discovery, execution, and\napproval behavior; selecting a tool never bypasses a host restriction or its\n`needsApproval` policy.\n\n## Approval Gates\n\nUse approval gates for durable side effects or sensitive operations:\n\n```ts\nimport { approvalRequired, defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Record a durable note.\",\n  inputSchema: {\n    type: \"object\",\n    properties: { note: { type: \"string\" } },\n    required: [\"note\"]\n  },\n  needsApproval: approvalRequired(\"Recording a note is a durable side effect.\", \"idempotent\"),\n  async execute(input: { note: string }, ctx) {\n    await ctx.emit(\"note.recorded\", {\n      note: input.note,\n      idempotencyKey: ctx.idempotencyKey(\"record-note\")\n    });\n    return { recorded: true };\n  }\n});\n```\n\nAuthored tools that perform non-idempotent external writes should use\n`ctx.idempotencyKey(...)` or a destination-level dedupe key derived from the\ntool-call id.\n\n## Durable Steps\n\nUse `ctx.step(...)` to cache completed substeps inside the current run. If the\nsame run is retried or resumed and the same step key is reached again, Assembly Line\nreturns the persisted JSON result and records `durable_step.replayed` instead of\nrunning the body again.\n\n```ts\nimport { defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Hydrate a customer profile once per run.\",\n  inputSchema: {\n    type: \"object\",\n    properties: { customerId: { type: \"string\" } },\n    required: [\"customerId\"]\n  },\n  async execute(input: { customerId: string }, ctx) {\n    const profile = await ctx.step(\n      `hydrate-customer:${input.customerId}`,\n      async () => {\n        const response = await fetch(`https://api.example.com/customers/${input.customerId}`, {\n          headers: { \"Idempotency-Key\": ctx.idempotencyKey(`customer:${input.customerId}`) }\n        });\n        return response.json();\n      },\n      { metadata: { customerId: input.customerId } }\n    );\n\n    return { profile };\n  }\n});\n```\n\nStep keys are scoped to the current run, and step results must be\nJSON-serializable. A crash inside the step body can still run the body again, so\nexternal writes should still use `ctx.idempotencyKey(...)` or a destination\ndedupe key. `ctx.step(...)` is completed-step replay, not universal deterministic\nworkflow replay.\n\n## Safe Model Output\n\nUse `toModelOutput` when the runtime should persist rich results but show the\nmodel only a bounded projection:\n\n```ts\nexport default defineTool({\n  description: \"Look up a record and return a safe summary.\",\n  inputSchema: {\n    type: \"object\",\n    properties: { lookup: { type: \"string\" } },\n    required: [\"lookup\"]\n  },\n  async execute(input: { lookup: string }) {\n    return {\n      summary: `Found ${input.lookup}`,\n      internalScore: 0.98,\n      internalTrace: [\"vector\", \"rerank\", \"policy\"]\n    };\n  },\n  toModelOutput(output: { summary: string }) {\n    return { summary: output.summary };\n  }\n});\n```\n\n## Execution Context\n\n`execute(input, ctx)` receives a `ToolExecutionContext`:\n\n| Member | Effect |\n| --- | --- |\n| `ctx.runId`, `ctx.agentRevision` | Identity of the current run and compiled revision. |\n| `ctx.principal`, `ctx.initiator` | Canonical current and conversation-initiating actors for authorization checks. |\n| `ctx.approvedToolCall` | `true` when this call has passed the runtime approval gate. |\n| `ctx.askQuestion(question, options?)` | Suspends the run to ask the user and returns `Promise<never>`. Code after it never runs in this call. |\n| `ctx.emit(eventType, data?)` | Records a durable run event. |\n| `ctx.idempotencyKey(scope)` | Stable per-run dedupe key for external writes. |\n| `ctx.step(key, fn, options?)` | Completed-step replay cache (see Durable Steps). |\n| `ctx.getSandbox()` | Acquires the run sandbox for isolated file/shell work. |\n| `ctx.agentState` | Reads or atomically updates conversation-scoped hook control state. Writes trigger capability re-evaluation before the next model request. |\n| `ctx.blob`, `ctx.memory`, `ctx.resources` | Blob, durable memory, and resource APIs when available. |\n| `ctx.spawnSubagent({ name, task, expectedOutput?, constraints? })` | Runs a compiled subagent; see [Subagents](subagents.md). |\n| `ctx.connections`, `ctx.channel` | Resolved connections and the originating channel view. |\n| `ctx.selfImprovement`, `ctx.automationManager`, `ctx.connectionManager` | Skill-writing, dynamic-automation, and dynamic-connection APIs; present only when the matching `agent.ts` policy enables them. |\n\n## Conventions\n\n- One tool per file; keep each tool focused on one capability.\n- Put provider event parsing in [`channels/`](channels.md), not tools.\n- Put reusable procedures in [`skills/`](skills.md), not long tool descriptions.\n- Gate non-idempotent external writes behind `needsApproval` and use\n  idempotency keys.\n- Recheck role and tenant authorization inside sensitive tools. Hook-based\n  visibility improves routing but is not an authorization boundary.\n\n## Related Docs\n\n- [Configuration Reference: tools/*.ts](../config-reference.md#toolsts)\n- [Customization: Tool Discovery](../customization.md#tool-discovery-and-capability-metadata)\n- [Sandbox](sandbox.md)\n- [Subagents](subagents.md)\n"},{"id":"architecture","sourcePath":"architecture.md","title":"Architecture","description":"Follow Assembly Line from an authored agent folder to a durable runtime service.","url":"https://assemblyline.artificialillumination.co/docs/architecture","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/architecture.md","headings":[{"depth":1,"title":"Assembly Line Architecture","anchor":"assembly-line-architecture"},{"depth":2,"title":"System Shape","anchor":"system-shape"},{"depth":2,"title":"Trust Boundaries","anchor":"trust-boundaries"},{"depth":2,"title":"Package Boundaries","anchor":"package-boundaries"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# Assembly Line Architecture\n\nAssembly Line turns a declarative agent folder into a durable runtime service. This\npage shows the shape of that system: the pipeline from folder to running\nservice, the trust boundaries the runtime enforces, and the package boundaries\nthat keep the framework core small.\n\n## System Shape\n\n```txt\n        agent folder                    .assembly-line artifact\n  +----------------------+         +------------------------+\n  | instructions.md      |         | manifest.json          |\n  | agent.ts  gateway.ts |  build  | agent-revision.json    |\n  | tools/  skills/      | ------> | route-table.json       |\n  | channels/ automations/|compiler| automations.json       |\n  | hooks/ connections/   |         | preflight.json         |\n  | sandbox/              |         |                        |\n  | subagents/ evals/    |         | server/boot.js  ...    |\n  +----------------------+         +------------------------+\n                                               |\n                                               | boot\n                                               v\n  channels ---> +------------------------------------------+\n  (Slack, HTTP, |              runtime host                |\n   schedules,   |  durable runs - approvals - deliveries   |\n   direct API)  | capability snapshots - recovery - workers |\n                +------------------------------------------+\n                   |          |          |           |\n                   v          v          v           v\n                 state       blob     sandbox     engine\n              (Postgres,  (S3/R2,   (Docker,    (Pi model\n               local)      local)    Daytona,     loop)\n                                     E2B, ...)\n```\n\n- The **compiler** reads the folder statically (TypeScript AST, never executing\n  config code), validates it, and emits a deterministic manifest plus a\n  self-contained runtime artifact. The same source always produces the same\n  `agentRevision`.\n- The **runtime host** loads the artifact and executes runs durably: every run\n  is persisted before execution, every pause (approval, human input,\n  suspension) is a durable state, and final delivery is an idempotent\n  obligation backed by a queue.\n- **Adapters** make each infrastructure choice substitutable per role: deploy,\n  runtime, state, blob, sandbox, scheduler, channel, and connection. The same\n  agent folder moves between providers without rewrites.\n\n[Runtime And Deployment](runtime-and-deployment.md) documents the artifact\ntree, HTTP API, and lifecycle; the [Framework Guide](framework.md) explains\neach concept in depth.\n\n## Trust Boundaries\n\nAssembly Line separates three levels of trust and keeps them separate at runtime:\n\n| Zone | Contains | Treated as |\n| --- | --- | --- |\n| Trusted app code | `agent.ts`, `gateway.ts`, tool implementations, inline automation lifecycle code, hooks, context policy, channel normalizers, instrumentation | Reviewed source. Runs in the host process with access to adapters and secrets-by-reference. |\n| Untrusted context | Memory, history, files, attachments, webpages, search results, tool output, remote tool descriptions | Data, never instructions. Projected read-only where possible; never grants capabilities. |\n| Sandbox | Shell commands, generated code, CLI connections, subagent workspaces | Isolated execution. Sees an allowlisted projection of logical paths, not the host filesystem. |\n\nKey consequences:\n\n- **Instructions vs. context.** Only `instructions.md` and durable skills are\n  trusted instructions. Everything the agent reads at runtime, including MCP\n  tool descriptions and dynamic-connection metadata, is untrusted context.\n- **Credentials never enter the model.** Connection secrets live in encrypted\n  grant stores or the state adapter; they are resolved by trusted code and are\n  never placed in model context, tool input, the agent folder, or the sandbox\n  (except as explicitly configured short-lived materializations for sandbox\n  CLI connections).\n- **The sandbox is a projection, not a mount.** `/memory`, `/history`,\n  `/files`, and `/workspace` are logical paths hydrated on demand; `/history`\n  and `/files` are read-only, and writeback flows through a durable sync queue\n  rather than direct host writes.\n- **Agents never author trusted code.** Self-improvement is scoped to skills\n  (instructions). Dynamic automations can only reference reviewed compiled\n  automation lifecycle code, and dynamic connections are URL-only, host-allowlisted,\n  and approval-gated. Net-new typed tools are always a reviewed source change.\n- **Ingress is authenticated per class.** Control-plane routes require host\n  auth or an admin token; provider webhook routes verify provider signatures;\n  the scheduler tick requires its shared secret in production.\n\n## Package Boundaries\n\nThe framework core stays small; everything provider-specific is an optional\nplugin package.\n\n- `@assemblyline-agents/core`: definitions, contracts, manifest types, and `define*` helpers.\n- `@assemblyline-agents/compiler`: folder discovery, validation, manifests, revisions, artifacts, routes, schedules, and deploy plans.\n- `@assemblyline-agents/cli`: command-line developer path, including coding-agent skill installation and version-matched docs commands.\n- `@assemblyline-agents/docs`: generated developer-docs corpus, focused search/read APIs, diagnostic links, and read-only MCP transport.\n- `@assemblyline-agents/sdk`: public CLI/meta package that re-exports the core framework APIs.\n- `@assemblyline-agents/runtime`: durable lifecycle, context bundles, local adapters, engine-neutral harness loop, tool execution, approvals, human input, replay, and delivery obligations.\n- `@assemblyline-agents/pi`: the default multi-provider model loop behind the internal `AgentHarness` contract.\n- `@assemblyline-agents/node`: hosted-container HTTP runtime host, including the preview `openai-codex/*` provider-prefix route.\n- `@assemblyline-agents/codex`: official Codex app-server primary-model bridge.\n- `@assemblyline-agents/railway`: Railway deploy adapter helper and publisher.\n- `@assemblyline-agents/postgres`: Postgres state adapter migrations, query-client implementation, and provider presets.\n- `@assemblyline-agents/s3`: S3-compatible blob adapter plus AWS, MinIO, and R2 helpers.\n- `@assemblyline-agents/r2`: R2 compatibility wrapper.\n- `@assemblyline-agents/otlp`: OTLP/HTTP telemetry sink for `instrumentation.ts`.\n- `@assemblyline-agents/daytona`: Daytona sandbox adapter boundary.\n- `@assemblyline-agents/docker`: Docker deploy and sandbox adapters.\n- `@assemblyline-agents/e2b`: E2B sandbox adapter boundary.\n- `@assemblyline-agents/modal`: Modal sandbox adapter boundary.\n- `@assemblyline-agents/fly`: Fly deploy adapter helper and publisher.\n- `@assemblyline-agents/vps`: supported provider-neutral existing-VPS deploy\n  helper and SSH/Docker publisher.\n- `@assemblyline-agents/slack`: `@assemblyline-agents/photon`, `@assemblyline-agents/discord`, `@assemblyline-agents/telegram`, `@assemblyline-agents/teams`: agent communication channel helpers.\n- `@assemblyline-agents/github`: GitHub App and MCP connection helpers for authenticated repository tooling.\n- `@assemblyline-agents/livekit`: voice/telephony capability package for LiveKit connections, dispatch, and SIP tools.\n\nAssembly Line core does not import product app code. Plugin packages own\nprovider-specific helpers, live adapter behavior, and their own\n`assemblyLineProvider` registration metadata; the CLI and Node host resolve\nbuilt-in kinds from first-party defaults and any other kind through the\nadapter definition's `packageName`, failing with a specific error when a\nplugin package is missing or misshapen.\n\n## Related Docs\n\n- [Framework Guide](framework.md): concepts and contracts.\n- [Runtime And Deployment](runtime-and-deployment.md): CLI, artifact, HTTP API, lifecycle, and deploy targets.\n- [Configuration Reference](config-reference.md): every `define*` shape and environment variable.\n- [Plugins](plugins.md): the extension model and plugin catalog.\n- [Authoring Plugin Providers](authoring-adapters.md): implementation contracts for plugin authors.\n"},{"id":"authoring-adapters","sourcePath":"authoring-adapters.md","title":"Authoring Plugins","description":"Implement provider, connection, channel, and skill contributions and publish them as Assembly Line plugin packages.","url":"https://assemblyline.artificialillumination.co/docs/authoring-adapters","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/authoring-adapters.md","headings":[{"depth":1,"title":"Authoring Plugins","anchor":"authoring-plugins"},{"depth":2,"title":"The Provider Contract (assemblyLineProvider)","anchor":"the-provider-contract-assemblylineprovider"},{"depth":2,"title":"Connection Plugins (assemblyLinePlugin)","anchor":"connection-plugins-assemblylineplugin"},{"depth":3,"title":"ConnectionPluginMetadata","anchor":"connectionpluginmetadata"},{"depth":3,"title":"Tool Classification Patterns","anchor":"tool-classification-patterns"},{"depth":3,"title":"Access Policy","anchor":"access-policy"},{"depth":3,"title":"The define*PluginConnection Factories","anchor":"the-definepluginconnection-factories"},{"depth":2,"title":"Sandbox CLI Tools","anchor":"sandbox-cli-tools"},{"depth":2,"title":"How assembly-line add Reads Your Package","anchor":"how-assembly-line-add-reads-your-package"},{"depth":2,"title":"Bundled Skills","anchor":"bundled-skills"},{"depth":2,"title":"Channel Modules","anchor":"channel-modules"},{"depth":2,"title":"Sandbox Adapters","anchor":"sandbox-adapters"},{"depth":2,"title":"Blob Adapters","anchor":"blob-adapters"},{"depth":2,"title":"Deploy Publishers","anchor":"deploy-publishers"},{"depth":2,"title":"Scheduler Adapters","anchor":"scheduler-adapters"},{"depth":2,"title":"State Adapters","anchor":"state-adapters"},{"depth":2,"title":"Agent Engines Are Not Plugin Providers","anchor":"agent-engines-are-not-plugin-providers"},{"depth":2,"title":"Capability Tier Or Connection Plugin?","anchor":"capability-tier-or-connection-plugin"},{"depth":2,"title":"Publishing To npm","anchor":"publishing-to-npm"},{"depth":2,"title":"Testing Your Plugin","anchor":"testing-your-plugin"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# Authoring Plugins\n\nThis guide is for implementing Assembly Line plugin contributions, providers\n(channels, sandboxes, blob stores, deploy targets, state stores), connection\nplugins, sandbox-CLI tools, and bundled skills, and packaging them so agents\ncan use them with zero Assembly Line core or host edits.\n[Plugins](plugins.md) defines the user-facing plugin model and catalog;\n[Adapters](adapters.md) is the companion reference for *consuming* the\nadapters that ship in this repo.\n\nA plugin package exports up to two well-known symbols:\n\n- `assemblyLineProvider`: provider registrations constructed by the runtime for\n  an adapter role (state, blob, sandbox, deploy, channel, connection).\n- `assemblyLinePlugin`: connection-plugin metadata consumed by the CLI and\n  compiler, paired with a `define<X>Connection` helper agents call in\n  `connections/<kind>.ts`.\n\nContents:\n\n- [The Provider Contract (assemblyLineProvider)](#the-provider-contract-assemblylineprovider)\n- [Connection Plugins (assemblyLinePlugin)](#connection-plugins-assemblylineplugin)\n- [Sandbox CLI Tools](#sandbox-cli-tools)\n- [How assembly-line add Reads Your Package](#how-assembly-line-add-reads-your-package)\n- [Bundled Skills](#bundled-skills)\n- [Channel Modules](#channel-modules)\n- [Sandbox Adapters](#sandbox-adapters)\n- [Blob Adapters](#blob-adapters)\n- [Deploy Publishers](#deploy-publishers)\n- [Scheduler Adapters](#scheduler-adapters)\n- [State Adapters](#state-adapters)\n- [Agent Engines Are Not Plugin Providers](#agent-engines-are-not-plugin-providers)\n- [Capability Tier Or Connection Plugin?](#capability-tier-or-connection-plugin)\n- [Publishing To npm](#publishing-to-npm)\n- [Testing Your Plugin](#testing-your-plugin)\n\n## The Provider Contract (assemblyLineProvider)\n\nProvider registration is open: any npm package can supply a state, blob,\nsandbox, deploy, channel, or connection provider. A plugin with provider\ncontributions exports one well-known symbol:\n\n```ts\nimport type { ProviderModule } from \"@assemblyline-agents/core\";\n\nexport const assemblyLineProvider: ProviderModule = {\n  providers: [\n    {\n      metadata: {\n        kind: \"neon-state\",            // the adapter kind used in gateway.ts\n        role: \"state\",                 // deploy | runtime | state | blob | sandbox | scheduler | channel | connection\n        packageName: \"@acme/assembly-line-neon\",\n        stability: \"preview\",          // supported | preview | planned\n        requiredEnv: [\"NEON_DATABASE_URL\"],\n        optionalEnv: [\"NEON_POOL_SIZE\"],\n        capabilities: [\"durable-state\"]\n      },\n      create(ctx) {\n        // ctx: { role, kind, options, env, fetch, manifest?, artifactRoot?, devMode }\n        return new NeonStateAdapter(ctx.env.NEON_DATABASE_URL!, ctx.options);\n      }\n    }\n  ]\n};\n```\n\nThe plugin package owns its provider metadata; the built-in registry in\n`@assemblyline-agents/core` is only an offline mirror for the adapters that ship in this\nrepo (a test asserts they never drift). `create(ctx)` receives everything\nconstruction needs. This includes adapter options merged with host-supplied\nrole extras, environment, fetch, the compiled manifest, and the artifact root.\nThe function returns the instance for the role: a `StateAdapter` or\n`StateStores` facet,\n`BlobAdapter`, `SandboxAdapter`, `DeployPublisher`, channel/connection\nimplementation, or another matching adapter contract. Telemetry sinks are\nwired in `instrumentation.ts`, not through adapters; see\n[Customizing Agents → Observability](customization.md#observability).)\n\nAgents opt in through the third `adapter()` argument:\n\n```ts\nexport default defineGateway({\n  runtime: adapter(\"node\"),\n  state: adapter(\"neon-state\", { pool: 4 }, { package: \"@acme/assembly-line-neon\" })\n});\n```\n\nAt runtime, local kinds stay in the host and bypass package resolution. Every\nother kind uses an explicit `packageName` when present, then falls back to the\nbuilt-in metadata registry. The Node host imports the package (also resolving\nfrom the deployed artifact's `node_modules`), reads `assemblyLineProvider`, and\ncalls the registration matching the role and kind. Failures are specific and\nearly: no known package for the kind, a package without the\n`assemblyLineProvider` export, or a package without a matching role/kind\nregistration each produce a distinct boot error naming the package.\n`assembly-line deploy` uses the same mechanism for deploy targets beyond\nlocal/railway/docker/fly/vps.\n\nPreflight picks up third-party requirements at build time: when the compiler\nsees a `packageName` it cannot find in the built-in registry, it imports the\npackage from the agent root, stamps the resolved `requiredEnv`, `setup`,\n`capabilities`, and `stability` into `manifest.providerMetadata`, and emits\nthe matching env requirements into `manifest.preflight`, so\n`assembly-line deploy --dry-run` reports missing provider env exactly like it does\nfor built-ins. If the package cannot be resolved, the compiler keeps the\ngeneric unknown-kind requirement and adds a `provider-package-unresolved`\nvalidation warning instead of failing the build; the Node host re-validates at\nboot.\n\nConnection providers whose executable must run on a particular host declare\n`hostRequirements: { deployTargets?, platforms?, message }` in their metadata.\nAssembly Line carries those constraints into the manifest, deployment plan, and\nruntime platform check. Use this for genuine execution requirements, not\nprovider preferences. The built-in Peekaboo plugin (`local` + `darwin`) is\nthe reference.\n\n## Connection Plugins (assemblyLinePlugin)\n\nA connection plugin gives agents a reviewed external capability: an MCP\nserver, OpenAPI service, or provider CLI behind the standard connection access,\napproval, discovery, and subject-scoping model. The package exports two\nthings: a `define<X>Connection` helper agents call in\n`connections/<kind>.ts`, and the `assemblyLinePlugin` module built with\n`definePlugin`:\n\n```ts\nimport type { PluginModule } from \"@assemblyline-agents/core\";\n\nexport interface PluginModule {\n  connections?: ConnectionPluginMetadata[];\n}\n```\n\nThe official Notion plugin is the minimal complete reference, five lines,\nbecause its metadata lives in the built-in catalog:\n\n```ts\n// packages/notion/src/index.ts\nimport { connectionPluginMetadata, defineMcpPluginConnection, definePlugin, type McpPluginConnectionOptions } from \"@assemblyline-agents/core\";\nconst PLUGIN = connectionPluginMetadata(\"notion\")!;\nexport type NotionConnectionOptions = McpPluginConnectionOptions;\nexport function defineNotionConnection(options: NotionConnectionOptions) { return defineMcpPluginConnection(PLUGIN, options); }\nexport const assemblyLinePlugin = definePlugin({ connections: [PLUGIN] });\n```\n\nA community plugin supplies its own metadata object instead of calling\n`connectionPluginMetadata`:\n\n```ts\n// src/index.ts\nimport {\n  defineMcpPluginConnection,\n  definePlugin,\n  type ConnectionPluginMetadata,\n  type McpPluginConnectionOptions\n} from \"@assemblyline-agents/core\";\n\nconst PLUGIN: ConnectionPluginMetadata = {\n  kind: \"acme\",\n  role: \"connection\",\n  packageName: \"@yourscope/assembly-line-acme\",\n  helper: \"defineAcmeConnection\",\n  protocol: \"mcp\",\n  transport: \"http\",\n  provider: \"acme\",\n  description: \"Acme issues and projects through Acme MCP.\",\n  defaultUrl: \"https://mcp.acme.dev/mcp\",\n  urlEnv: \"ACME_MCP_URL\",\n  tokenEnv: \"ACME_MCP_TOKEN\",\n  tokenRequired: true,\n  requiredEnv: [\"ACME_MCP_TOKEN\"],\n  optionalEnv: [\"ACME_MCP_URL\"],\n  readToolPatterns: [\"regex:^(get|list|search)_\"],\n  writeToolPatterns: [\"regex:^(create|update|delete)_\"],\n  skill: \"acme\",\n  setup: [{\n    kind: \"connection\",\n    name: \"acme-developer-configuration\",\n    required: true,\n    message: \"Create an Acme API token and set ACME_MCP_TOKEN.\"\n  }]\n};\n\nexport type AcmeConnectionOptions = McpPluginConnectionOptions;\nexport function defineAcmeConnection(options: AcmeConnectionOptions) {\n  return defineMcpPluginConnection(PLUGIN, options);\n}\nexport const assemblyLinePlugin = definePlugin({ connections: [PLUGIN] });\n```\n\n### ConnectionPluginMetadata\n\n`ConnectionPluginMetadata` extends the generic `AdapterProviderMetadata`\n(`kind`, `role`, `stability`, `packageName`, `requiredEnv`, `optionalEnv`,\n`capabilities`, `setup`) and is shared by the CLI, compiler, and your helper:\n\n| Field | Type | Required | Effect |\n| --- | --- | --- | --- |\n| `kind` | `string` | yes | The connection kind: the `assembly-line add` argument and the `connections/<kind>.ts` filename. |\n| `role` | `\"connection\"` | yes | Always `\"connection\"` for plugin metadata. |\n| `helper` | `string` | yes | Exported helper name; `assembly-line add` writes `import { <helper> } from \"<packageName>\"` into the scaffolded connection file, and factory error messages name it. |\n| `protocol` | `\"mcp\" \\| \"openapi\" \\| \"cli\"` | yes | Selects the factory family; each factory rejects mismatched metadata. |\n| `transport` | `\"http\" \\| \"stdio\" \\| \"relay\" \\| \"sandbox\"` | no | Defaults to Streamable HTTP (`\"http\"`) when omitted. `\"sandbox\"` pairs only with `protocol: \"cli\"`. |\n| `provider` | `string` | yes | Provider identity stamped into `metadata.provider` on every generated definition. |\n| `description` | `string` | yes | Default connection description when the agent author passes none. |\n| `command` | `string` | for `cli` | The sandbox CLI executable name; `defineSandboxCliPluginConnection` fails without it (unless the author overrides). |\n| `defaultUrl` | `string` | no | Endpoint fallback when neither `options.url` nor `urlEnv` supplies one. |\n| `urlEnv` | `string` | no | Env var consulted for the endpoint URL (before `defaultUrl`). Required env when there is no `defaultUrl`. |\n| `defaultSpec` / `specEnv` | `string` | OpenAPI only | OpenAPI specification source fallback and env override. |\n| `defaultBaseUrl` / `baseUrlEnv` | `string` | OpenAPI only | API base URL fallback and env override. |\n| `tokenEnv` | `string` | no | Env var holding the credential. Default auth sends it as a Bearer token. |\n| `tokenRequired` | `boolean` | no | `false` makes `tokenEnv` optional and skips default auth when the var is unset. |\n| `tokenHeader` | `string` | no | Send the `tokenEnv` value verbatim in this header instead of Bearer authorization (for example `x-browser-use-api-key`). |\n| `credentialEnv` | `string` | relay only | Env var containing one opaque, device-scoped relay binding; `defineRelayMcpPluginConnection` fails without it. |\n| `scopes` | `string[]` | no | OAuth scopes advertised for authorization flows. |\n| `subject` | `\"user\" \\| ...` | no | Default connection subject (per-user vs environment credential scoping) when the author passes none. |\n| `tools` | `{ allow: [...] } \\| { block: [...] }` | no | Default tool allow/block filter cloned into generated definitions; entries use the exact, `*` glob, or `regex:` syntax described below, and the author's `tools` option overrides it. |\n| `hostRequirements` | `{ deployTargets?, platforms?, message }` | no | Runtime-host restrictions for process-backed or platform-specific plugins; enforced at validation, deploy planning, and boot. |\n| `readToolPatterns` | `string[]` | yes | Tool-name patterns classified as read authority. |\n| `writeToolPatterns` | `string[]` | yes | Tool-name patterns classified as write authority. An empty array makes the plugin read-only. |\n| `skill` | `string` | no | Single bundled skill directory name copied by `assembly-line add`. |\n| `skills` | `string[]` | no | Multiple bundled skill directory names (use one of `skill`/`skills`). |\n| `requiredEnv` / `optionalEnv` | `string[]` | yes/no | Env vars surfaced by `assembly-line add` and stamped into preflight. Keep them consistent with the URL/token fields above. |\n| `setup` | `{ kind, name, required, message }[]` | no | Human setup steps `assembly-line add` prints and preflight reports. |\n| `stability` | `\"supported\" \\| \"preview\" \\| \"planned\"` | no | Support level surfaced in metadata and manifests. |\n| `packageName` | `string` | yes for community | The npm package `assembly-line add` installs and the scaffold imports from. |\n\n### Tool Classification Patterns\n\n`readToolPatterns` and `writeToolPatterns` classify every discovered tool\nname:\n\n- A bare string matches the exact tool name.\n- A string containing `*` is a glob (`arcads_get_*`).\n- A `regex:` prefix compiles the remainder as a case-insensitive regular\n  expression, e.g. `regex:^(get|list|search)_`.\n\nWrite matches take precedence over read matches. Unclassified tools are not\ndiscoverable. Classification is part of the plugin's security surface, so\nreview it against the provider's real tool list rather than\ntrusting naming conventions.\n\n### Access Policy\n\nAgent authors pass one small, explicit policy to your helper:\n\n```ts\ninterface ConnectionPluginAccessSelection {\n  read: true;\n  write: false | {\n    approval: \"always\" | \"once\" | \"never\" | ConnectionApprovalDefinition;\n  };\n}\n```\n\nEvery factory runs `validatePluginAccess` before building the definition:\n`access` must be present, and enabling `write` on a plugin whose\n`writeToolPatterns` is empty throws\n`<helper>: <provider> is a read-only plugin and cannot enable writes.`\nThe factory then expands the selection into a full\n`ConnectionAccessDefinition` using your metadata's read/write patterns, so\nthe agent folder shows the policy while the plugin owns the classification.\n\n### The define*PluginConnection Factories\n\nChoose the factory matching your metadata; each one rejects mismatched\nprotocol/transport with an error naming the correct factory:\n\n| Metadata | Factory | Produces |\n| --- | --- | --- |\n| `protocol: \"mcp\"`, `transport: \"http\"` (default) | `defineMcpPluginConnection(PLUGIN, options)` | Streamable HTTP MCP connection |\n| `protocol: \"mcp\"`, `transport: \"stdio\"` | `defineStdioMcpPluginConnection(PLUGIN, options)` | Static stdio MCP process (options must pass `command`) |\n| `protocol: \"mcp\"`, `transport: \"relay\"` | `defineRelayMcpPluginConnection(PLUGIN, options)` | E2EE device-relay MCP connection (metadata must declare `credentialEnv`) |\n| `protocol: \"openapi\"` | `defineOpenAPIPluginConnection(PLUGIN, options)` | Direct OpenAPI connection from `defaultSpec`/`specEnv` |\n| `protocol: \"cli\"`, `transport: \"sandbox\"` | `defineSandboxCliPluginConnection(PLUGIN, options)` | Reviewed CLI invocations in the run sandbox (options must pass `tools`) |\n\nShared behavior the factories give you for free:\n\n- **URL resolution**: `options.url`, else the `urlEnv` environment variable,\n  else `defaultUrl`; a missing URL throws an error naming `urlEnv`. OpenAPI\n  resolves `spec` and `baseUrl` the same way from their fields.\n- **Auth**: unless the author passes an `auth` definition (or `auth: false`),\n  the factory builds a Bearer-token auth from `tokenEnv`, skipped when\n  `tokenRequired: false`, or, when `tokenHeader` is set, sends the env value\n  verbatim in that header instead.\n- **Defaults**: `description`, `subject`, and the `tools` filter fall back to\n  metadata; `required` defaults to `true`; `metadata.provider` and\n  `metadata.plugin` are stamped so tracing and audits attribute tool calls to\n  the plugin.\n\n## Sandbox CLI Tools\n\nA sandbox-CLI plugin (`protocol: \"cli\"`, `transport: \"sandbox\"`) exposes a\nprovider's official CLI as a small set of reviewed tools that execute inside\nthe active run sandbox. The process sees `/files` and `/workspace` and never\nruns on the gateway host. `@assemblyline-agents/higgsfield` is the reference\nimplementation.\n\n`defineSandboxCliPluginConnection(PLUGIN, options)` requires\n`options.tools: SandboxCliToolDefinition[]`:\n\n```ts\ninterface SandboxCliToolDefinition {\n  name: string;                 // qualified as <kind>__<name>, e.g. higgsfield__read\n  description: string;\n  inputSchema: JsonSchema;      // the model-facing input contract\n  outputSchema?: JsonSchema;\n  build(input: unknown): MaybePromise<SandboxCliInvocation>;\n}\n\ninterface SandboxCliInvocation {\n  args: string[];               // CLI arguments, excluding the executable\n  cwd?: string;                 // e.g. \"/workspace\"\n  timeoutMs?: number;\n  hydratePaths?: string[];      // logical paths to materialize before launch\n}\n```\n\nContract and safety rules:\n\n- **The command is trusted configuration.** The executable comes from\n  `PLUGIN.command` (or an author override in the connection file), never from\n  the model. `build(input)` receives the model's JSON input and returns the\n  argument vector. The runtime passes every element of `args` as an\n  individually quoted argument and never accepts model-authored shell source.\n- **Validate inside `build`.** Treat it as your allowlist: check the\n  subcommand against a reviewed read or write set, reject credential-printing\n  and auth subcommands, and bound argument count and length. Throw a specific\n  error for anything outside the reviewed surface (see the Higgsfield\n  package's `validateReadCommand`/`validateWriteCommand`).\n- **`hydratePaths`** lists logical paths (`/files/...`, `/workspace/...`)\n  that must exist in the sandbox filesystem before the process starts.\n  Return every attachment path referenced by the arguments so the CLI can\n  read uploaded media.\n- **`timeoutMs`** bounds each invocation; pick separate read and write\n  defaults when writes are long-running jobs.\n- **`maxOutputChars`** (a connection-level option) caps captured\n  stdout/stderr before it reaches the model.\n- Access classification applies to the *tool names*: a common shape is one\n  `read` tool and one `write` tool with `readToolPatterns: [\"read\"]` and\n  `writeToolPatterns: [\"write\"]`, so the write tool stays hidden until the\n  connection enables writes with an approval policy.\n- Credentials that live in the CLI's own login state (like\n  `higgsfield auth login`) belong to a persistent, user-scoped sandbox.\n  Document the interactive login in `setup` and never bake credentials into a\n  shared image.\n\n## How assembly-line add Reads Your Package\n\n`assembly-line add @yourscope/assembly-line-acme <agentRoot>` installs the package, then\nimports it and inspects the two well-known exports (on the module or its\ndefault export):\n\n1. `assemblyLinePlugin.connections`: each entry becomes an installable\n   connection contribution (with `packageName` defaulted to the installed\n   package).\n2. `assemblyLineProvider.providers`: each registration's `metadata` becomes an\n   installable provider contribution.\n\nSelection rules:\n\n- `--role <role>` picks the matching contribution, or fails with\n  `Package <name> has no <role> provider. Available: <role>:<kind>, ...`.\n- Without `--role`, a single-role package is unambiguous. A multi-role\n  package fails with\n  `Package <name> provides multiple roles: <roles>. Pass --role <role>.`\n- A package exporting neither symbol fails with\n  `Plugin package <name> does not export assemblyLinePlugin or\n  assemblyLineProvider, so its contributions cannot be determined.`\n- With `--no-install` and the package absent, the import fails with\n  `Failed to load plugin package <name>: ... Install it first (or rerun\n  without --no-install).`\n\nAfter selection, the CLI wires the agent folder exactly as it does for\nofficial plugins. It writes a connection file, channel file, or `gateway.ts` slot and\nprints your metadata's `requiredEnv`, `optionalEnv`, and `setup` messages.\nSee [Plugins: Install A Plugin](plugins.md#install-a-plugin-with-assembly-line-add)\nfor the per-role scaffold behavior.\n\n## Bundled Skills\n\nA connection plugin should ship at least one skill teaching the agent how to\noperate the integration:\n\n- Put each skill at `skills/<name>/SKILL.md` in the package root (plus any\n  reference files), and include `\"skills\"` in the package.json `files` array\n  so npm publishes it.\n- List the directory names in metadata as `skill: \"<name>\"` or\n  `skills: [\"<a>\", \"<b>\"]`, only listed skills are copied.\n- `assembly-line add` copies each listed directory into the agent's\n  `skills/<name>/`. An existing directory is never overwritten, and a listed\n  skill without a `SKILL.md` is reported as an error line.\n\nSkills must never contain secrets, tokens, or account-specific values.\nCredentials stay in host environment variables and authorization flows.\n\n## Channel Modules\n\nA channel package exports a `ChannelModule` (defined in\n`@assemblyline-agents/runtime`'s `channel.ts`). Every member is optional.\nImplement only what your provider needs:\n\n| Member | Purpose |\n| --- | --- |\n| `normalizeHttp(request, ctx)` | Verify and normalize an inbound HTTP request into a turn. Returns `{ kind: \"run\" }`, `{ kind: \"accepted\" }` (ACK before model work), `{ kind: \"response\" }` (reply without a run, e.g. a `401`), or `{ kind: \"ignored\" }`. |\n| `startIngress(ctx, emit)` | Long-lived provider listener (e.g. Discord Gateway). `emit.accepted({ turn, idempotencyKey, ... })` feeds the same durable run path as HTTP webhooks and reports capacity rejections with `retryAfterMs`. |\n| `startTurn(turn, ctx)` | Turn lifecycle hook, typing indicators and similar. Accepted HTTP ingress starts it after capacity/idempotency admission, concurrently with runtime initialization and durable run creation; other runs start it during setup. The runtime stops it before final delivery. |\n| `augmentContext(turn, ctx)` | Add `recentHistory`/`channelContext` after ACK and before context bundle construction. |\n| `send(delivery, ctx)` | Deliver the final response through the provider API. |\n| `ingressAuth` | `{ requiredSecretEnv?: string[][] }`, production ingress-auth declaration (see below). |\n| `resolveAttachment(attachment, ctx)` | Turn a turn attachment into an authenticated download request (see below). |\n\nEvery member receives the same `ChannelContext`: the compiled channel,\n`agentScope`, resolved `connections`, `env`, `fetch`, `state`, and `blob`.\n\nGuidance that keeps channels durable and safe:\n\n- **Idempotency.** Providers redeliver webhooks. Use the provider's stable\n  delivery id (Slack `event_id`, Spectrum webhook id + message id, Telegram\n  `update_id`) as the turn's `idempotencyKey` so retries never start duplicate\n  runs, and return `{ kind: \"accepted\" }` before model work for providers\n  that enforce fast ACK deadlines.\n- **Verify every request.** `normalizeHttp` owns signature/token\n  verification. Compare secrets with `constantTimeSecretEqual` from\n  `@assemblyline-agents/runtime` and return `{ kind: \"response\", status: 401, ... }` on\n  mismatch. Provider channel routes are public in production, verification\n  is your only gate.\n- **Declare ingress secrets.** Set\n  `ingress: { requiredSecretEnv: [[\"MY_WEBHOOK_SECRET\"], [\"MY_BEARER_TOKEN\"]] }`\n  on the channel config (any-of groups: boot succeeds when every var in at\n  least one group is set) and export the same shape as `ingressAuth` on the\n  module. The compiler stamps it into `CompiledChannel.metadata.ingress`;\n  production boot fails until a group is satisfied, dev mode warns.\n- **Resolve your own attachments.** `resolveAttachment` returns\n  `{ url, headers, filename? }` with your provider's credentials and host\n  allowlist; the runtime performs the download, applies size/type limits and\n  timeouts, and stores the blob. Return `undefined` to fall back to a generic\n  unauthenticated URL download. The trust boundary is enforced by the\n  runtime: only the module of the channel that produced the turn is ever\n  consulted, and only when the attachment's declared provider matches, so a\n  hostile attachment can never route to another channel's credentials. Use\n  `attachmentTrustedForProvider(attachment, \"<provider>\")` and\n  `attachmentRemoteUrl(attachment)` from `@assemblyline-agents/runtime` before attaching\n  credentials.\n- **Delivery results.** A `send` failure is retried in-process and then\n  deferred onto the durable delivery queue when retryable. Throw for\n  transient provider errors; the runtime treats failures as retryable unless\n  marked otherwise. When resending from the queue, the original turn may be\n  gone. Fall back to the persisted `payload.delivery` target.\n- **Outbound HTTP.** Use the shared runtime HTTP client\n  (`fetchWithPolicy`/`fetchJson` from `@assemblyline-agents/runtime`) for provider API\n  calls: per-attempt timeouts, `Retry-After` handling, backoff with jitter,\n  and size-capped bodies come for free, and the `fetchImpl` parameter\n  preserves the `ctx.fetch` injection seam tests rely on.\n\nThe Slack, Telegram, Teams, Discord, and Photon packages are the reference\nimplementations for all of the above.\n\n`assembly-line add` has channel scaffolds only for Slack, Discord, Telegram, and\nTeams. For any other channel kind, including a community channel package, it prints `No channel scaffold is known for \"<kind>\"`; document that users\ncreate `channels/<kind>.ts` exporting your `ChannelDefinition` manually.\n\n## Sandbox Adapters\n\nImplement `SandboxAdapter` from `@assemblyline-agents/runtime`. Only `acquire` is\nrequired; the optional members opt into warm reconnects and the dirty-session\nretention that sandbox sync depends on:\n\n```ts\ninterface SandboxAdapter {\n  provider?: string;\n  acquire(run: RunRecord): Promise<SandboxSession>;\n  create?(run, input?): Promise<SandboxSession>;\n  lookup?(input): Promise<SandboxLookupResult | undefined>;   // { state: \"live\" | \"warm\" }\n  connect?(input): Promise<SandboxSession | undefined>;\n  wake?(input): Promise<SandboxSession | undefined>;\n  pauseOrRetain?(session, input?: { dirty?: boolean; reason?: string }): Promise<void>;\n  disposeClean?(session): Promise<void>;\n}\n```\n\nRules the built-in adapters follow and yours should too:\n\n- Hosted adapters must establish a physical `/workspace`, make it the default\n  shell cwd, and ensure shell operations and provider file APIs address the\n  same files. Validate the invariant after create, connect, and wake with\n  `sandboxFilesystemContractProbeCommand()` and\n  `assertSandboxFilesystemContractProbe()`.\n- Stamp `sandboxFilesystemContractMetadata()` into provider labels/tags and\n  runtime manifests, filter provider lookup by that metadata, require\n  `hasCurrentSandboxFilesystemContract()` before reconnect, and include\n  `sandboxFilesystemContractKey(sessionKey)` in provider resource names. This\n  prevents old namespaces from being silently reused after a contract change.\n- Paths use the canonical Assembly Line namespace. Use the shared helpers from\n  `@assemblyline-agents/core`,   `normalizeSandboxPath`, `normalizeSandboxRoot`, `resolveSandboxPath`\n  (rejects `..` traversal and retired `/runtime`), `canonicalizeSandboxListingPath`,\n  `assertCanonicalSandboxWorkingDirectory`, and `shellQuote`,   instead of reimplementing path handling.\n- Listings return virtual absolute paths and include contents so runtime sync\n  can persist memory and artifacts without provider-specific follow-up reads.\n- `listFiles(path, { excludeTopLevel })` must prune named direct child\n  directories before reading file contents.\n- Implement `deletePath` when possible so the core delete tool does not have to\n  shell out.\n- Dirty sessions are retained/paused (not destroyed) while sandbox sync is\n  pending; clean sessions are disposed through the provider lifecycle API.\n- Never fall back to local execution silently.\n\n`LocalSandboxAdapter` (`@assemblyline-agents/runtime`) is the explicitly dev-only logical\nemulation reference; `@assemblyline-agents/docker` is the reference for the physical\nnamespace and a real isolation boundary.\n\n## Blob Adapters\n\n`BlobAdapter` is two methods:\n\n```ts\ninterface BlobAdapter {\n  put(key, value, contentTypeOrOptions?): Promise<BlobRecord>;\n  get(key): Promise<Uint8Array | undefined>;\n}\n```\n\n`put` accepts a content type string or `{ contentType?, visibility? }` and\nreturns a `BlobRecord` (`id`, `key`, `uri`, `sha256`, `size`, ...). Blobs are\nprivate by default; only return public HTTP URLs when the write explicitly\nused `{ visibility: \"public\" }`. `@assemblyline-agents/s3` is the reference\nimplementation.\n\n## Deploy Publishers\n\nDeploy targets implement `DeployPublisher` from `@assemblyline-agents/core`:\n\n```ts\ninterface DeployPublisher {\n  target: string;\n  preflight?(artifact: DeployArtifact, plan: DeployPlan): Promise<DeployReceipt | void>;\n  prepare?(artifact: DeployArtifact, plan: DeployPlan): Promise<DeployReceipt | void>;\n  publish(artifact: DeployArtifact, plan: DeployPlan): Promise<DeployReceipt>;\n  rollback?(artifact: DeployArtifact, plan: DeployPlan): Promise<DeployReceipt>;\n  destroy?(artifact: DeployArtifact, plan: DeployPlan, options?: { purgeData?: boolean }): Promise<DeployReceipt>;\n  syncSecrets?(secrets: Record<string, string>, plan: DeployPlan): Promise<DeploySecretsReceipt>;\n  runMigrations?(\n    artifact: DeployArtifact,\n    plan: DeployPlan,\n    request: { entries: string[]; command?: string }\n  ): Promise<string>;\n  runRemoteCommand?(\n    artifact: DeployArtifact,\n    plan: DeployPlan,\n    command: string[],\n    options?: { interactive?: boolean; silent?: boolean; release?: \"active\" | \"prepared\" }\n  ): Promise<DeployCommandResult>;\n}\n// DeployArtifact.manifest may include deploymentRequirements:\n// { persistentDirectories: [{ id, path, sensitive }], remoteExecution }\n// DeployPlan: { target, environment, agentRevision, agentId?, defaultEnvironment?, requirements? }\n```\n\nA publisher must isolate resources per `plan.environment`. How is the target's\nchoice: native environments (railway), name scoping via\n`environmentScopedName(base, plan)` from `@assemblyline-agents/core` (docker, fly), or\nidentity hashing (VPS). The helper encodes the compatibility rule. The default\nenvironment keeps unscoped legacy names. Every other environment gets an\n`-<environment>` suffix. Use this helper instead of creating another scheme.\n`destroy` follows the same feature-detection pattern as `rollback`: omit it if\nthe target cannot remove resources safely, re-derive names from the plan (no\nreceipt is available), and keep durable data unless `options.purgeData`.\n\n`publish` receives the built `.assembly-line` artifact root plus the plan, shells\nout to provider tooling (the built-ins take a `RunDeployCommand` so tests can\ninject a fake), and returns a receipt. The CLI adds migration status and writes\nthe final `deployment.json` after `publish` succeeds. `preflight`, `prepare`,\nand `runMigrations` are optional: remote-host publishers can validate the\ntarget, stage an immutable image, and migrate beside a private database before\nactivation. Register the publisher\nwith role `\"deploy\"` in your `assemblyLineProvider` and\n`assembly-line deploy --target <kind>` picks it up through the resolver, no CLI\nedits. `@assemblyline-agents/railway`, `@assemblyline-agents/docker`, `@assemblyline-agents/fly`, and\n`@assemblyline-agents/vps` are the references.\n\n`syncSecrets` is optional and should be implemented only for hosted targets\nthat have a remote secret store. It receives key/value pairs from\n`assembly-line deploy --sync-secrets` and must log or return key names only, never\nsecret values. Provider commands that accept private input should use the\nframework runner contract instead of interpolating values into argv:\n\n```ts\ninterface DeployCommandOptions {\n  cwd?: string;\n  env?: Record<string, string | undefined>;\n  silent?: boolean;\n  stdin?: string;\n}\n```\n\nValidate variable names, reject line breaks and null bytes, skip empty values,\nand abort publishing when synchronization fails. Do not treat a missing local\nkey as a request to delete an existing remote secret.\n\n`runRemoteCommand` is optional for backward compatibility. Implement it when a\npublisher advertises `remote-exec`; execute the exact argument vector without\ninterpolating it unsafely into a shell. A publisher that advertises\n`persistent-storage` must provision every declared persistent directory and\nmust treat entries marked `sensitive` as credential-bearing. Do not place their\ncontents in secrets, logs, artifacts, or receipts. The built-in Codex auth flow\nrequires both capabilities plus this method; ordinary artifacts can still use\nolder community publishers.\n\n## Scheduler Adapters\n\nThe `scheduler` role selects where the schedule clock lives. The runtime\ncontract is `SchedulerAdapter` from `@assemblyline-agents/runtime`:\n\n```ts\ninterface SchedulerAdapter {\n  kind: string;\n  start(runtime: RuntimeSchedulerHost): MaybePromise<RuntimeSchedulerController | undefined>;\n}\n// RuntimeSchedulerHost:      { startSchedulerPollingLoop(), registerManifestSchedules() }\n// RuntimeSchedulerController: { stop(), tick() }\n```\n\nThe built-ins cover the three shapes: `local` and `postgres` start the\nruntime's polling loop (Postgres adds multi-worker lease coordination through\nthe state adapter); `gateway` registers the manifest schedules and returns no\ncontroller, leaving ticks to an external trigger calling\n`/assembly-line/automations/tick` or `runtime.runDueAutomations()`.\n\nAn unknown scheduler kind in `gateway.ts` falls back to gateway-style\nbehavior, schedules are registered, but no in-process loop starts. A custom\nin-process scheduler is therefore an embedder seam rather than a\npackage-resolved provider: hosts constructing the runtime directly pass their\nown `SchedulerAdapter` as `RuntimeOptions.scheduler`. See\n[Adapters: Scheduler](adapters.md#scheduler) for the consumption view.\n\n## State Adapters\n\nDurable state is capability-faceted. Start from `RunStore`, the only\nrequired facet, and add facets as your backend supports them:\n\n- **`RunStore` (required):** runs, run events, tool calls, checkpoints,\n  deliveries, and idempotency keys (`createRun`, `updateRun`, `getRun`,\n  `listRuns`, `appendEvent`, `listEvents`, `createToolCall`,\n  `updateToolCall`, `listToolCalls`, `createCheckpoint`, `listCheckpoints`,\n  `createDelivery`, `updateDelivery`, `listDeliveries`,\n  `reserveIdempotencyKey`).\n- **Optional `RunStore` methods** unlock durability features:\n  `leaseDueDeliveries`, `completeDeliverySend`, `failDeliverySend`, and\n  `recoverExpiredDeliveries` enable the durable delivery queue (make the\n  lease multi-replica safe, Postgres uses `for update skip locked`);\n  `listRunsByStatus` makes orphan sweeps efficient; `touchRun` gives the run\n  heartbeat a guarded write that bumps `updatedAt` only while the run is\n  still `created`/`running`, implement it with a single conditional\n  statement (never read-modify-write) so a heartbeat can never race a status\n  transition; `requestRunControl` and `settleRunControl` must atomically apply\n  cooperative suspend/cancel with cancel precedence across replicas; `close`\n  participates in graceful shutdown.\n- **Optional facets:** `ConversationStore`, `ConversationTurnStore`,\n  `ScheduleStateStore`, `FileIndexStore`, `UsageStore`,\n  `SandboxSessionStore`, `MemoryStateStore`, `RuntimeSettingsStore`, and\n  `AgentStateStore`.\n  Hand them to the runtime as a `StateStores` object\n  (`{ runs, conversations?, conversationTurns?, schedules?, files?, usage?,\n  sandboxSessions?, memory?, settings?, agentState? }`). Missing facets fall\n  back to in-memory implementations with one `state.degraded` boot warning.\n  `ConversationTurnStore` must lease one FIFO turn per conversation for the\n  durable ingress mailbox. `AgentStateStore` must provide bounded snapshot\n  reads, atomic set/update/delete operations, aggregate revision\n  compare-and-set, and conversation isolation. `StateAdapter` remains the\n  backward-compatible monolith; `ResolvedStateAdapter` is the full internal\n  intersection after the runtime fills missing facets.\n\n`UsageStore` is observational. Implement `recordUsage` and `listUsage`, with\n`queryUsage` and `summarizeUsage` for efficient reporting. Writes should be\nidempotent on the provider/request key and should allow a later\nprovider-reconciled receipt to replace an unavailable observation. Usage-store\nfailures must be surfaced through logs and degraded-state reporting, but must\nnot reject model requests or suppress provider responses.\n\nReferences, in order of approachability: the per-facet `InMemory*Store`\nclasses and `FileStateAdapter` in `@assemblyline-agents/runtime`, then\n`PostgresStateAdapter` in `@assemblyline-agents/postgres` (migrations, leases,\nmulti-replica coordination). Capability guards (`isRunStore`,\n`isStateAdapter`, ...) are exported for feature detection.\n\n## Agent Engines Are Not Plugin Providers\n\nThe primary model engine is not an adapter role. Pi (`@assemblyline-agents/pi`) is the\ndefault engine; `agent.ts` rejects a `harness:` slot at validate time. The Node\nhost additionally owns a preview `openai-codex/*` provider-prefix route through\n`RuntimeOptions.modelHarnesses`. That internal host route is not a\nprovider-registered engine role. The internal `AgentHarness` contract in\n`@assemblyline-agents/core` remains the seam the runtime speaks through. It\nkeeps the runtime free of engine types, continuations as opaque JSON, and\ndurability in the runtime. `RuntimeOptions.agentHarness` exposes this seam to\nembedders and tests. The durability suite drives scripted engines through it.\nIt is not a provider-registered extension point.\n\nSubagents use Pi too. Each subagent selects its model, tools, and active\nconnections through the same synchronous composition functions as its parent. Its static\ndefinition can set a workspace adapter and a ceiling for connection names.\nThere is no `subagent` provider role or public harness adapter contract.\nService-specific execution surfaces such as LiveKit should expose typed tools\nand connection definitions instead.\n\n## Capability Tier Or Connection Plugin?\n\nBuild a **connection plugin** when the integration is an external capability\nan agent calls as tools, such as an MCP server, an API, or a reviewed CLI behind\nthe standard access/approval model. Build a **capability package** (under\n`capabilities/`, like LiveKit) only when the integration is a single-vendor\nfeature with its own execution surface, typed tool definitions, clients, and\nconnection metadata that expose the vendor's own concepts rather than an\ninterchangeable adapter role or a discoverable tool catalog. When in doubt,\nprefer a connection plugin: it gets `assembly-line add` scaffolding, read/write\nclassification, approvals, and skills for free. See\n[Adapters: Capabilities](adapters.md#capabilities).\n\n## Publishing To npm\n\n- **Naming.** Use `@yourscope/assembly-line-<thing>` (for example\n  `@acme/assembly-line-neon`). Official packages are `@assemblyline-agents/<kind>`.\n- **ESM with an exports map.** The CLI, compiler, and Node host all load your\n  package with dynamic `import()`. Ship ESM (`\"type\": \"module\"`) with an\n  `exports` entry resolving to your built output, and export\n  `assemblyLinePlugin`/`assemblyLineProvider` from that entry (a default export\n  containing them also works).\n- **Depend on `@assemblyline-agents/core` as a peer dependency** so your metadata and\n  definition types come from the host's single core instance.\n- **Ship your skills.** Include `\"skills\"` (and your `dist`) in the\n  package.json `files` array.\n- **Artifact packaging.** The compiled `.assembly-line/package.json`\n  declares every community plugin package used by the agent as a dependency.\n  It pins the version installed in the agent root when present, else the\n  agent's declared range, else `*`, so `npm install --omit=dev` in the\n  deployed artifact pulls your package without manual edits.\n- **Document alongside the package:** required/optional env, provider setup\n  steps (OAuth app registration, CLI installs), verification behavior,\n  idempotency keys, and the exact read/write tool surface.\n\n## Testing Your Plugin\n\nCopy the patterns from this repo's acceptance tests (Node `node:test` +\n`node:assert/strict` against built `dist/` output):\n\n- `tests/fixtures/provider-fixture/` is a minimal community plugin provider:\n  a `package.json` with a bare `assemblyLineProvider` export plus a\n  deliberate `no-provider` entry for error paths. Model your package (and its\n  tests) on it.\n- `tests/provider-registry.test.mjs` shows the registration contract tests\n  worth having: `adapter(kind, opts, { package })` records `packageName`;\n  your `assemblyLineProvider` metadata matches what you document; `resolveProvider`\n  constructs your adapter with merged options and env; the compiler stamps\n  your `requiredEnv` into preflight; and the exact error messages for\n  missing/misshapen packages.\n- `tests/connection-plugins.test.mjs` shows connection-plugin contract tests:\n  every catalog entry's helper exists, `assemblyLinePlugin.connections` matches,\n  generated definitions carry the expected transport/URL/subject, and\n  read/write classification behaves for representative tool names.\n- `tests/adapters.test.mjs` shows channel-module tests: a custom channel with\n  `ingress.requiredSecretEnv` and `resolveAttachment` compiled and exercised\n  end-to-end with zero runtime edits, plus the cross-channel credential\n  trust-boundary test (a forged attachment must never reach your resolver\n  with credentials attached).\n- `tests/state-stores.test.mjs` shows facet tests: a runs-only `StateStores`\n  boots with one `state.degraded` warning; a monolithic adapter is detected\n  with none; a missing `runs` facet throws.\n- Inject fakes through the seams the contracts already provide: `ctx.fetch`\n  for HTTP, `RunDeployCommand` for deploy CLIs, provider client injection for\n  sandboxes.\n\nA standalone community package can start with a shape test plus a compile\ntest against a fixture agent:\n\n```mjs\nimport assert from \"node:assert/strict\";\n\nconst imported = await import(\"@yourscope/assembly-line-acme\");\n\n// The plugin contract the CLI and runtime rely on.\nassert.equal(typeof imported.defineAcmeConnection, \"function\");\nassert.equal(imported.assemblyLinePlugin.connections[0].kind, \"acme\");\n\n// The helper builds a read-only definition from the explicit policy.\nconst definition = imported.defineAcmeConnection({\n  access: { read: true, write: false }\n});\nassert.equal(definition.access.write, false);\nassert.ok(definition.access.read.tools.length > 0);\n```\n\nFor provider contributions, compile a fixture agent whose config selects your\npackage with `adapter(\"<kind>\", {}, { package: \"@yourscope/assembly-line-acme\" })`\nand assert the manifest carries your `requiredEnv` in preflight.\n\nDocument required/optional env, verification behavior, idempotency keys, and\npreflight requirements alongside the package, [Contributing](contributing.md#adding-a-plugin-provider) has the checklist.\n\n## Related Docs\n\n- [Plugins](plugins.md): the user-facing plugin model, catalog, and `assembly-line add`.\n- [Adapters](adapters.md): consuming the adapters that ship in this repo.\n- [connections/](agent-stack/connections.md): the connection file format agents author.\n- [Contributing](contributing.md): repo conventions and the plugin-provider checklist.\n"},{"id":"building-agents","sourcePath":"building-agents.md","title":"Building Agents","description":"Build an agent from scaffold to gated tools, a channel, an automation, and passing evals.","url":"https://assemblyline.artificialillumination.co/docs/building-agents","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/building-agents.md","headings":[{"depth":1,"title":"Building Agents","anchor":"building-agents"},{"depth":2,"title":"1. Scaffold The Agent","anchor":"1-scaffold-the-agent"},{"depth":2,"title":"2. Write The Instructions","anchor":"2-write-the-instructions"},{"depth":2,"title":"3. Add A Tool With An Approval Gate","anchor":"3-add-a-tool-with-an-approval-gate"},{"depth":2,"title":"4. Add A Channel And Test It Over HTTP","anchor":"4-add-a-channel-and-test-it-over-http"},{"depth":2,"title":"5. Add An Automation","anchor":"5-add-an-automation"},{"depth":2,"title":"6. React To Runtime Events","anchor":"6-react-to-runtime-events"},{"depth":2,"title":"7. Write Two Evals And Run Them","anchor":"7-write-two-evals-and-run-them"},{"depth":2,"title":"8. Ship It","anchor":"8-ship-it"},{"depth":2,"title":"Design Rules","anchor":"design-rules"}],"content":"# Building Agents\n\nAn Assembly Line agent is a folder. This tutorial builds a complete agent one\nfile at a time, from `assembly-line init` to passing evals. Each\nstep gives the exact command, the exact file content, and the expected output,\nthen links to the [Agent Build Stack](agent-stack/overview.md) page that owns\nthat file's full option surface.\n\nCommands use the installed `assembly-line` form. In a source checkout, run them as\n`pnpm assembly-line <command>` (see [Getting Started](getting-started.md)).\n\n1. [Scaffold the agent](#1-scaffold-the-agent)\n2. [Write the instructions](#2-write-the-instructions)\n3. [Add a tool with an approval gate](#3-add-a-tool-with-an-approval-gate)\n4. [Add a channel and test it over HTTP](#4-add-a-channel-and-test-it-over-http)\n5. [Add an automation](#5-add-an-automation)\n6. [Write two evals and run them](#6-write-two-evals-and-run-them)\n7. [Ship it](#7-ship-it)\n\n## 1. Scaffold The Agent\n\n```sh\nassembly-line init agent\n```\n\n```txt\nCreated Assembly Line agent at /path/to/agent\nNext: export OPENAI_API_KEY, then: pnpm assembly-line run /path/to/agent --message \"hello\"\n```\n\nThe scaffold is the smallest useful agent:\n\n```txt\nagent/\n  instructions.md   # trusted, always-on guidance\n  agent.ts          # identity/policy plus runtime capability hooks\n  gateway.ts        # deploy/runtime/state/blob/sandbox/scheduler adapters\n  .env.example      # provider env template\n  tools/\n    echo.ts         # one typed model-callable action\n```\n\nOnly `instructions.md` and `agent.ts` are required; everything else is\noptional. Give the agent a name and description in `agent.ts` with\n`defineAgent`:\n\n```ts\n// agent.ts\nimport { defineAgent, useModel, useTool } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  name: \"notes-agent\",\n  description: \"Records durable notes behind an approval gate.\",\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useTool(\"echo\");\n  }\n});\n```\n\nThe full folder map, including `context.ts`, `skills/`, `connections/`,\n`sandbox/`, `subagents/`, and `instrumentation.ts`, is in the\n[Agent Build Stack overview](agent-stack/overview.md). Every `defineAgent`\nstatic policy and built-in hook, including `useOutputSchema()`, persistent\nworkflow state, `maxIterations`, `selfImprovement`, `dynamicAutomations`, and\n`dynamicConnections`, is on\n[agent.ts](agent-stack/agent-ts.md).\n\n## 2. Write The Instructions\n\n`instructions.md` is trusted, always-on guidance. It is the one prose file the\nmodel always sees. Replace the scaffold line with an identity and three rules:\n\n```md\nYou are a concise note-taking agent.\n\n- Use tools when they are available instead of describing what you would do.\n- Ask before recording anything durable.\n- Keep replies to a few sentences.\n```\n\nValidate after every step:\n\n```sh\nassembly-line validate agent\n```\n\n```txt\nValid Assembly Line agent: /path/to/agent\n```\n\nWhat belongs in instructions versus skills, tool descriptions, or memory is\ncovered in [instructions.md](agent-stack/instructions.md).\n\n## 3. Add A Tool With An Approval Gate\n\nEach file in `tools/` becomes one model-facing tool. The filename is the tool\nname. The scaffolded `tools/echo.ts` has no side effects. Add a second tool\nwith a side effect and require approval:\n\n```ts\n// tools/record_note.ts\nimport { approvalRequired, defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Record a durable note after explicit approval.\",\n  inputSchema: {\n    type: \"object\",\n    properties: {\n      note: { type: \"string\" }\n    },\n    required: [\"note\"]\n  },\n  needsApproval: approvalRequired(\"Recording a note is a durable side effect.\"),\n  async execute(input: { note: string }, ctx) {\n    await ctx.emit(\"note.recorded\", {\n      note: input.note,\n      idempotencyKey: ctx.idempotencyKey(\"record-note\")\n    });\n    return { recorded: true, note: input.note };\n  }\n});\n```\n\nEnable the compiled tool by adding `useTool(\"record_note\")` to `agent.ts`\n`setup()`. Tool files define how actions work; hooks decide which actions are\navailable in the current capability snapshot.\n\nRun it without approval to watch the gate pause the run:\n\n```sh\nassembly-line run agent --tool record_note --input '{\"note\":\"Ship Friday\"}'\n```\n\n```json\n{\n  \"run\": {\n    \"id\": \"3f9d2b1e-…\",\n    \"status\": \"waiting_for_approval\",\n    ...\n  },\n  \"waitingForApproval\": true,\n  ...\n}\n```\n\nApprove it and it completes:\n\n```sh\nassembly-line run agent --tool record_note --input '{\"note\":\"Ship Friday\"}' --approve\n```\n\n```json\n{\n  \"run\": {\n    \"id\": \"8a41c6d0-…\",\n    \"status\": \"completed\",\n    ...\n  },\n  \"response\": \"{\\\"recorded\\\":true,\\\"note\\\":\\\"Ship Friday\\\"}\",\n  ...\n}\n```\n\nApproval policies and side-effect classes, durable steps, sandbox execution,\nand `toModelOutput` projections that keep rich results out of model context\nare on [tools/](agent-stack/tools.md).\n\n## 4. Add A Channel And Test It Over HTTP\n\nChannels normalize external events into agent turns and deliver replies back\nto the provider. Add a raw HTTP channel for local development:\n\n```ts\n// channels/http.ts\nimport { defineChannel } from \"@assemblyline-agents/core\";\n\nexport default defineChannel({\n  description: \"Receive a local/dev HTTP message.\",\n  transport: \"http\",\n  route: \"/message\",\n  methods: [\"POST\"]\n});\n```\n\nA channel turn is a full model turn, so set the provider key for the model in\n`agent.ts`, then serve the agent:\n\n```sh\nexport OPENAI_API_KEY=sk-...\nassembly-line serve agent --port 3000\n```\n\n```txt\nAssembly Line runtime serving 4b0c9a17…\nhttp://127.0.0.1:3000\n```\n\nIn a second terminal, post to the route:\n\n```sh\ncurl -X POST http://127.0.0.1:3000/message \\\n  -H \"content-type: application/json\" \\\n  -d '{\"message\":\"hello\"}'\n```\n\n```json\n{\n  \"runId\": \"3f9d2b1e-…\",\n  \"status\": \"completed\",\n  \"response\": \"Hello! How can I help you today?\",\n  \"waitingForApproval\": false,\n  \"waitingForInput\": false,\n  \"waitingForConnection\": false,\n  \"eventCount\": 8,\n  \"toolCallCount\": 0\n}\n```\n\nThis raw shape is open in dev mode only. In production, generic HTTP channels\nrequire host auth. Provider-facing routes should export `normalizeHttp()` to\nverify and normalize the provider event before starting a turn. Provider\nhelpers for Slack, Discord, Telegram, Microsoft Teams, and Photon/Spectrum keep\nwebhook wiring in one file. Install them with\n`assembly-line add <channel> agent`; see [channels/](agent-stack/channels.md).\n\n## 5. Add An Automation\n\nFiles in `automations/` compile into schedule- or event-triggered durable work:\n\n```ts\n// automations/morning_brief.ts\nimport { defineAutomation } from \"@assemblyline-agents/core\";\n\nexport default defineAutomation({\n  description: \"Run a small daily brief.\",\n  trigger: {\n    type: \"schedule\",\n    cron: \"0 8 * * *\",\n    timezone: \"America/Chicago\"\n  },\n  idempotencyKey: \"notes-agent:morning-brief\",\n  message: \"Summarize yesterday's notes.\"\n});\n```\n\nRebuild and inspect the compiled schedule table:\n\n```sh\nassembly-line build agent\n```\n\n```txt\nBuilt Assembly Line agent revision 9e5d2c80…\nArtifact: /path/to/agent/.assembly-line\n```\n\n`.assembly-line/automations.json` now lists the automation. Event triggers,\ninline preparation/finalization, and runtime-created dynamic automations are covered in\n[automations/](agent-stack/automations.md).\n\n## 6. React To Runtime Events\n\nPut cross-cutting, after-persist reactions in `hooks/`:\n\n```ts\n// hooks/audit.ts\nimport { defineHook } from \"@assemblyline-agents/core\";\n\nexport default defineHook({\n  events: {\n    async \"run.completed\"(event, ctx) {\n      await auditLog.record(ctx.runId, event.data);\n    }\n  }\n});\n```\n\nHook failures are recorded without changing the originating run's result. See\n[`hooks/`](agent-stack/hooks.md) for event ordering and idempotency guidance.\n\n## 7. Write Two Evals And Run Them\n\n`evals/*.json` is the agent's golden dataset, run through the same compiled\nruntime path used in production. Start with one normal case and one high-risk\ncase. Both force a tool run, so they are deterministic and need no provider\nkey.\n\n`evals/echo_roundtrip.json`:\n\n```json\n{\n  \"name\": \"Echo round trip\",\n  \"input\": {\n    \"message\": \"ping\",\n    \"tool\": \"echo\",\n    \"toolInput\": { \"message\": \"ping\" }\n  },\n  \"expect\": {\n    \"status\": \"completed\",\n    \"toolsCalled\": [\"echo\"]\n  }\n}\n```\n\n`evals/approval_gate.json`:\n\n```json\n{\n  \"name\": \"Note requires approval\",\n  \"tags\": [\"high-risk\"],\n  \"input\": {\n    \"message\": \"Record that we ship Friday.\",\n    \"tool\": \"record_note\",\n    \"toolInput\": { \"note\": \"Ship Friday\" },\n    \"approve\": false\n  },\n  \"expect\": {\n    \"status\": \"waiting_for_approval\",\n    \"toolsCalled\": [\"record_note\"]\n  }\n}\n```\n\nRun the suite:\n\n```sh\nassembly-line eval agent\n```\n\n```txt\n✓ Echo round trip  84ms  $0.000000\n✓ Note requires approval  41ms  $0.000000\n\n2/2 executions passed (100.0%); 0 failed; 0 errored\n2 unique cases; 1 repetition requested\nCost: runs $0.000000 + judges $0.000000 = $0.000000\nTags:\n  high-risk        1/1 passed, 0 failed, 0 errored\n```\n\nEach case gets isolated state, blob, and sandbox roots, and channel senders\nare never invoked. The full case contract, multi-turn conversations, output\nand JSON Schema assertions, tool trajectories, tool mocks, state fixtures,\ncustom evaluators, LLM-as-judge, repetitions, baselines, and CI gates, is on\n[evals/](agent-stack/evals.md).\n\n## 8. Ship It\n\n`gateway.ts` declares where the agent runs. The scaffold pins every slot\nlocal:\n\n```ts\n// gateway.ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\n\nexport default defineGateway({\n  deploy: adapter(\"local\"),\n  runtime: adapter(\"node\"),\n  state: adapter(\"local\"),\n  blob: adapter(\"local\"),\n  sandbox: adapter(\"local\"),\n  scheduler: adapter(\"local\")\n});\n```\n\nSwap slots without touching agent code, for example,\n`assembly-line add postgres agent` sets `state: adapter(\"postgres\")`. Preview the\ndeploy plan without publishing:\n\n```sh\nassembly-line deploy agent --dry-run\n```\n\nThis prints the deploy plan JSON: target, environment, required env, and any\nmissing requirements. Adapter roles and gateway conventions are on\n[gateway.ts](agent-stack/gateway-ts.md); deploy targets, host auth, secrets,\nand the production checklist are in\n[Runtime And Deployment](runtime-and-deployment.md).\n\n## Design Rules\n\nThe canonical authoring rules cover file ownership, secrets, and\nuntrusted-context boundaries. They live in the\n[Agent Build Stack overview](agent-stack/overview.md#design-rules).\n"},{"id":"coding-agents","sourcePath":"coding-agents.md","title":"Coding Agents","description":"Give Codex and Claude Code version-matched Assembly Line guidance without loading the whole manual into every turn.","url":"https://assemblyline.artificialillumination.co/docs/coding-agents","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/coding-agents.md","headings":[{"depth":1,"title":"Coding Agents","anchor":"coding-agents"},{"depth":2,"title":"Install The Authoring Integration","anchor":"install-the-authoring-integration"},{"depth":2,"title":"Use Version-Matched Documentation","anchor":"use-version-matched-documentation"},{"depth":2,"title":"Optional MCP Access","anchor":"optional-mcp-access"},{"depth":2,"title":"Machine-Readable Hosted Surfaces","anchor":"machine-readable-hosted-surfaces"},{"depth":2,"title":"Validation Loop","anchor":"validation-loop"}],"content":"# Coding Agents\n\nAssembly Line ships its authoring guidance with the CLI. Codex and Claude Code\ncan therefore use documentation that matches the installed framework version,\neven when the hosted site has moved ahead.\n\nThe integration has four layers:\n\n1. `AGENTS.md` or `CLAUDE.md` contains a short routing rule.\n2. A project-scoped `assembly-line-authoring` skill loads only for Assembly Line work.\n3. `assembly-line docs search` finds the relevant page in the installed documentation corpus.\n4. `assembly-line docs read` loads that page or one section. `assembly-line validate --json` returns structured issues, suggested fixes, and documentation links.\n\nThis is deliberate progressive disclosure. A coding agent does not need the\nfull manual in its standing context.\n\n## Install The Authoring Integration\n\nRun this at the repository root that contains the Assembly Line agent:\n\n```sh\nassembly-line authoring install all .\n```\n\nUse `codex` or `claude` instead of `all` to install one integration. The command\ncreates these project-scoped files:\n\n```txt\nAGENTS.md\nCLAUDE.md\n.agents/skills/assembly-line-authoring/\n.claude/skills/assembly-line-authoring/\n```\n\nCodex reads the shared skill from `.agents/skills`. Claude Code reads its copy\nfrom `.claude/skills`, while `CLAUDE.md` imports the shared `AGENTS.md` routing\nrules. The installer adds small managed blocks to existing instruction files\nand leaves unrelated content unchanged.\n\nAn existing skill or managed routing block is not overwritten by `install`.\nAfter upgrading the Assembly Line CLI, refresh the managed files explicitly:\n\n```sh\nassembly-line authoring update all .\nassembly-line authoring status all . --json\n```\n\nCommit these files when every contributor and CI coding agent should receive\nthe same authoring behavior.\n\n## Use Version-Matched Documentation\n\nThe default commands read the corpus bundled with the installed CLI:\n\n```sh\nassembly-line docs version\nassembly-line docs list\nassembly-line docs search \"approval gated tool\"\nassembly-line docs read agent-stack/tools\nassembly-line docs read agent-stack/tools#conventions\n```\n\nAdd `--json` when another program will consume the result. Search returns a\nsmall ranked catalog with snippets. Read then retrieves one page or one section.\n\nUse hosted documentation only when the task is explicitly about current\nbehavior or an upgrade:\n\n```sh\nassembly-line docs search \"current deploy providers\" --latest --json\n```\n\nThe `--latest` flag fetches the hosted corpus. It is never the default because\nnew documentation may describe APIs that the installed packages do not yet\nprovide.\n\n## Optional MCP Access\n\nThe same local corpus is available as a read-only stdio MCP server:\n\n```sh\nassembly-line docs mcp\n```\n\nIt exposes three tools:\n\n- `assembly_line_docs_search`\n- `assembly_line_docs_read`\n- `assembly_line_docs_list`\n\nConfigure either coding agent to start that command when you want native MCP\ntool discovery. For example, a project-scoped Codex `.codex/config.toml` entry\ncan use:\n\n```toml\n[mcp_servers.assembly_line_docs]\ncommand = \"assembly-line\"\nargs = [\"docs\", \"mcp\"]\n```\n\nA project-scoped Claude Code `.mcp.json` entry can use:\n\n```json\n{\n  \"mcpServers\": {\n    \"assembly-line-docs\": {\n      \"command\": \"assembly-line\",\n      \"args\": [\"docs\", \"mcp\"]\n    }\n  }\n}\n```\n\nIf the binary is not globally available, replace `assembly-line` with the\nproject's package-manager command. The hosted site also exposes a read-only\nStreamable HTTP endpoint at `https://assemblyline.artificialillumination.co/mcp`.\nThat endpoint follows the current hosted docs; prefer the local server while\nediting a project pinned to an older framework version.\n\n## Machine-Readable Hosted Surfaces\n\nThe hosted documentation publishes:\n\n- `/llms.txt` for a compact page catalog.\n- `/api/agent-docs` for the versioned JSON corpus used by `--latest`.\n- `/mcp` for on-demand search, read, and list tools.\n\nThese surfaces are generated from the same Markdown under `docs/developers/`.\nThere is no separate agent-only manual to drift out of date.\n\n## Validation Loop\n\nAsk coding agents to finish every material agent change with:\n\n```sh\nassembly-line validate ./agent --json\nassembly-line build ./agent\nassembly-line eval ./agent --json\n```\n\nThe JSON validation report identifies the installed docs revision and attaches\na focused documentation page to each issue. Build and eval remain separate so\nprojects can choose the appropriate cost and confidence level for each change.\n"},{"id":"config-reference","sourcePath":"config-reference.md","title":"Configuration Reference","description":"Reference every agent-folder configuration shape and Assembly Line environment variable.","url":"https://assemblyline.artificialillumination.co/docs/config-reference","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/config-reference.md","headings":[{"depth":1,"title":"Configuration Reference","anchor":"configuration-reference"},{"depth":2,"title":"`defineAgent` (agent.ts)","anchor":"defineagent-agentts"},{"depth":2,"title":"Subagent `defineAgent` (`subagents/<name>/agent.ts`)","anchor":"subagent-defineagent-subagentsnameagentts"},{"depth":2,"title":"`defineGateway` (gateway.ts) and `adapter()`","anchor":"definegateway-gatewayts-and-adapter"},{"depth":3,"title":"Runtime state facets","anchor":"runtime-state-facets"},{"depth":2,"title":"`defineInstrumentation` (instrumentation.ts)","anchor":"defineinstrumentation-instrumentationts"},{"depth":2,"title":"`defineContext` / `defaultContext` (context.ts)","anchor":"definecontext-defaultcontext-contextts"},{"depth":3,"title":"`transcript` options","anchor":"transcript-options"},{"depth":2,"title":"Per-File Contracts The Compiler Validates","anchor":"per-file-contracts-the-compiler-validates"},{"depth":3,"title":"tools/*.ts","anchor":"toolsts"},{"depth":3,"title":"`skills/<name>/SKILL.md`","anchor":"skillsnameskillmd"},{"depth":3,"title":"channels/*.ts","anchor":"channelsts"},{"depth":3,"title":"automations/*.ts","anchor":"automationsts"},{"depth":3,"title":"hooks/*.ts","anchor":"hooksts"},{"depth":3,"title":"connections/*.ts","anchor":"connectionsts"},{"depth":3,"title":"sandbox/*.ts","anchor":"sandboxts"},{"depth":3,"title":"instrumentation.ts","anchor":"instrumentationts"},{"depth":3,"title":"Community plugin provider stamping","anchor":"community-plugin-provider-stamping"},{"depth":2,"title":"Environment Variable Reference","anchor":"environment-variable-reference"},{"depth":3,"title":"Alphabetical index","anchor":"alphabetical-index"},{"depth":3,"title":"Server and auth (`@assemblyline-agents/node`)","anchor":"server-and-auth-assemblyline-agentsnode"},{"depth":3,"title":"Usage accounting","anchor":"usage-accounting"},{"depth":3,"title":"Concurrency and rate limiting","anchor":"concurrency-and-rate-limiting"},{"depth":3,"title":"Graceful shutdown","anchor":"graceful-shutdown"},{"depth":3,"title":"OTLP telemetry (`@assemblyline-agents/otlp`)","anchor":"otlp-telemetry-assemblyline-agentsotlp"},{"depth":3,"title":"Durability workers and recovery","anchor":"durability-workers-and-recovery"},{"depth":3,"title":"Delivery queue","anchor":"delivery-queue"},{"depth":3,"title":"Sandbox sync and hydration","anchor":"sandbox-sync-and-hydration"},{"depth":3,"title":"Sandbox snapshots","anchor":"sandbox-snapshots"},{"depth":3,"title":"Model loop, memory, and logging","anchor":"model-loop-memory-and-logging"},{"depth":3,"title":"Attachments and resource projection","anchor":"attachments-and-resource-projection"},{"depth":3,"title":"Self-improvement, dynamic automations, dynamic connections","anchor":"self-improvement-dynamic-automations-dynamic-connections"},{"depth":3,"title":"Scheduler","anchor":"scheduler"},{"depth":3,"title":"Secrets and local store paths (`@assemblyline-agents/node`)","anchor":"secrets-and-local-store-paths-assemblyline-agentsnode"},{"depth":3,"title":"Build, deploy, and migrations (CLI and compiler)","anchor":"build-deploy-and-migrations-cli-and-compiler"},{"depth":3,"title":"Provider: Postgres (`@assemblyline-agents/postgres`)","anchor":"provider-postgres-assemblyline-agentspostgres"},{"depth":3,"title":"Provider: Docker (`@assemblyline-agents/docker`)","anchor":"provider-docker-assemblyline-agentsdocker"},{"depth":3,"title":"Provider: VPS (`@assemblyline-agents/vps`)","anchor":"provider-vps-assemblyline-agentsvps"},{"depth":3,"title":"Provider: E2B (`@assemblyline-agents/e2b`)","anchor":"provider-e2b-assemblyline-agentse2b"},{"depth":3,"title":"Provider: Modal (`@assemblyline-agents/modal`)","anchor":"provider-modal-assemblyline-agentsmodal"},{"depth":3,"title":"Provider: Daytona (`@assemblyline-agents/daytona`)","anchor":"provider-daytona-assemblyline-agentsdaytona"},{"depth":3,"title":"Computer Use Relay (`@assemblyline-agents/computer-use`)","anchor":"computer-use-relay-assemblyline-agentscomputer-use"},{"depth":3,"title":"Provider: Microsoft Teams (`@assemblyline-agents/teams`)","anchor":"provider-microsoft-teams-assemblyline-agentsteams"}],"content":"# Configuration Reference\n\nThis page is the single reference for agent-folder configuration shapes and\nevery `ASSEMBLY_LINE_*` environment variable. The guides explain when and why to use\nthese settings:\n\n- [Building Agents](building-agents.md) for the folder shape and authoring flow.\n- [Customizing Agents](customization.md) for context, engine routing, hardening, and channels.\n- [Runtime And Deployment](runtime-and-deployment.md) for the runtime lifecycle and production knobs.\n- [Adapters](adapters.md) for provider helpers and provider-specific env (Slack, Postgres, S3, sandboxes, ...).\n\nThe `define*` helpers from `@assemblyline-agents/core` return typed\ndeclarations. Some connection helpers also run local validation. The compiler\nreads manifest-facing declarations from the TypeScript AST and never executes\nconfig code. Values that affect the manifest must therefore be statically\nreadable.\n\nContents:\n\n- [`defineAgent` (agent.ts)](#defineagent-agentts)\n- [Subagent `defineAgent` (subagents/&lt;name&gt;/agent.ts)](#subagent-defineagent-subagentsnameagentts)\n- [`defineGateway` (gateway.ts) and `adapter()`](#definegateway-gatewayts-and-adapter)\n- [`defineInstrumentation` (instrumentation.ts)](#defineinstrumentation-instrumentationts)\n- [`defineContext` / `defaultContext` (context.ts)](#definecontext--defaultcontext-contextts)\n- [Per-file contracts the compiler validates](#per-file-contracts-the-compiler-validates)\n- [Environment variable reference](#environment-variable-reference): starting\n  with the [alphabetical index](#alphabetical-index) of every variable.\n\n## `defineAgent` (agent.ts)\n\n`agent.ts` default-exports `defineAgent({ ... })`. A synchronous `setup()` is\nrequired and must select exactly one model on the baseline path.\n\n| Field | Type | Required | Meaning |\n| --- | --- | --- | --- |\n| `id` | `string` | no | Stable logical agent identity. Keeps learned skills, usage attribution, and durable state attached to the agent across revisions and redeploys. |\n| `name` | `string` | no | Display name. |\n| `description` | `string` | no | Short description recorded in the manifest. |\n| `context` | `ContextPolicy` | no | Context policy from `defaultContext(...)` or `defineContext(...)`. The compiler stamps `defaultContext()` when omitted. |\n| `setup` | `() => void` | yes | Synchronous composition function. Returning a promise is a compile/runtime error. |\n| `maxReasoning` | reasoning level | no | Static ceiling for `useReasoning()`. |\n| `maxIterations` | positive integer | no | Agent-loop iteration budget for one active execution. Overrides `ASSEMBLY_LINE_MAX_MODEL_ITERATIONS`; a cap hit fails the run instead of delivering a partial response. |\n| `selfImprovement` | object | no | Skill authoring config; see below. |\n| `dynamicAutomations` | object | no | Runtime-created automation config; see below. |\n| `dynamicConnections` | object | no | Runtime-created connection config; see below. |\n| `metadata` | JSON object | no | Free-form metadata recorded in the manifest. |\n\nPi is the default primary engine; `agent.ts` has no `harness:` slot (declaring\none fails validation with `harness-not-configurable`). The Node host's\n`openai-codex/*` route is selected by model prefix rather than a public harness\nfield. Subagents use Pi as well; they do not expose a harness slot.\n\nRuntime choices are made with built-in composition functions inside `setup()`:\n\n| Function | Contract |\n| --- | --- |\n| `useRun()` | Immutable run, conversation, metadata, and attachment metadata. |\n| `usePersistentState(key, initial)` | Conversation-scoped bounded JSON state plus async setter. Key must be literal. |\n| `useModel(model)` | Exactly one compiled possible model. Literal model required. |\n| `useReasoning(level)` | Effort at or below `maxReasoning`. |\n| `useInstructions(text)` | Append trusted text in call order. |\n| `useTool`, `useSkill`, `useConnection`, `useSubagent`, `useSandbox` | Select a named compiled capability. Names must be literals. |\n| `useOutputSchema(schema)` | Runtime-enforced final JSON contract. |\n\nSet-like composition calls deduplicate by name. Conflicting model or sandbox selections\nfail closed. Conditional calls are supported; state identity uses explicit\nkeys rather than call position.\n\n`usePersistentState()` accepts JSON values only. Keys may contain at most 200\ncharacters, and one value may serialize to at most 16,384 characters. The\nruntime caps each conversation snapshot at 256 keys and 262,144 serialized\ncharacters. The returned setter cannot run during `setup()`. Tool code can use\n`ctx.agentState.get()`, `set()`, `update()`, and `delete()` instead. Mutations\nreturn the new aggregate revision and accept `expectedRevision` for\ncompare-and-set behavior.\n\nSub-config blocks (all fields optional):\n\n| Block | Field | Default | Meaning |\n| --- | --- | --- | --- |\n| `selfImprovement` | `enabled` | `true` | Enable durable skill writes and background review. |\n| `selfImprovement` | `writable` | - | Deprecated compatibility alias for `enabled`. |\n| `selfImprovement` | `writeApproval` | `false` | Skill writes require approval. |\n| `selfImprovement` | `reviewEveryTurns` | `10` | Review every N completed foreground turns. |\n| `selfImprovement` | `reviewMinToolCalls` | `5` | Immediately review runs with at least this many tool calls. |\n| `selfImprovement` | `reviewModel` | `inherit` | Reviewer model; `inherit` reuses the source run model. |\n| `selfImprovement` | `externalDirs` | - | Extra directories treated as skill sources. |\n| `dynamicAutomations` | `dynamic` | `true` | Agent may create/manage time-based automations through `ctx.automationManager`. |\n| `dynamicAutomations` | `approval` | `false` | Dynamic automation changes require approval. |\n| `dynamicConnections` | `dynamic` | `false` | Agent may persist runtime-provided MCP/OpenAPI/HTTP connections. |\n| `dynamicConnections` | `approval` | `true` | Saving a dynamic connection requires approval. |\n| `dynamicConnections` | `allowedHosts` | `[]` | Host allowlist; required non-empty when `dynamic` is enabled. |\n\n## Subagent `defineAgent` (`subagents/<name>/agent.ts`)\n\nSubagents use `defineAgent()` and the same hooks. They support the static agent\nfields above, but `description` is required so the parent knows when to\ndelegate. These fields add child-specific limits:\n\n| Field | Type | Meaning |\n| --- | --- | --- |\n| `workspace` | `AdapterDefinition?` | Sandbox/workspace adapter for the subagent (compiled with role `sandbox`). |\n| `connections` | `string[]?` | Static ceiling of connection names the subagent may select with `useConnection()`. |\n| `maxReasoning` | reasoning level | Static ceiling for `useReasoning()`. |\n| `maxIterations` | positive integer | Agent-loop iteration limit for the child run. |\n\nEvery child selects its own model and active capabilities in `setup()`; there\nis no implicit model/tool inheritance. All subagents run through Pi. `harness`\nand the retired `engineConnection` field fail validation.\n\n## `defineGateway` (gateway.ts) and `adapter()`\n\nEvery gateway slot is an optional `AdapterDefinition`. Compiler defaults are\nall `adapter(\"local\")` except `runtime`, which defaults to `adapter(\"node\")`.\n\n| Slot | Chooses | Default |\n| --- | --- | --- |\n| `deploy` | Where the compiled runtime service runs. | `adapter(\"local\")` |\n| `runtime` | Runtime host. | `adapter(\"node\")` |\n| `state` | Durable state adapter. | `adapter(\"local\")` |\n| `blob` | Blob storage adapter. | `adapter(\"local\")` |\n| `sandbox` | Default sandbox adapter. | `adapter(\"local\")` |\n| `scheduler` | Where the schedule clock lives (`local`, `gateway`, `postgres`). | `adapter(\"local\")` |\n\nTelemetry is not a gateway slot. Configure it in `instrumentation.ts` (see\n[`defineInstrumentation`](#defineinstrumentation-instrumentationts)).\n\n`adapter()` builds an `AdapterDefinition`:\n\n```ts\nadapter(kind: string, options?: JsonObject, extras?: { package?: string })\n```\n\n- `kind`: adapter kind, e.g. `\"postgres\"`, `\"railway\"`, `\"pi\"`.\n- `options`: JSON options merged with host-supplied extras at construction.\n- `extras.package`: npm plugin package name that exports `assemblyLineProvider`; it\n  is stored as `packageName` on the definition. This is how community plugin\n  providers plug in with zero core edits:\n\n```ts\nstate: adapter(\"neon-state\", { pool: 4 }, { package: \"@acme/assembly-line-neon\" })\n```\n\n`AdapterRole` values: `deploy`, `runtime`, `state`, `blob`, `sandbox`,\n`scheduler`, `channel`, and `connection`. A subagent `workspace` adapter\ncompiles with role `sandbox`.\nSee [Plugins](plugins.md) for the extension model,\n[Authoring Plugin Providers](authoring-adapters.md) for the provider contract,\nand [Adapters](adapters.md) for the built-in matrix.\n\n### Runtime state facets\n\n`RuntimeOptions.state` may be a full `StateAdapter` or a `StateStores` object.\n`runs` is required. Optional facets are `conversations`, `conversationTurns`,\n`schedules`, `files`, `usage`, `sandboxSessions`, `memory`, `agentState`, and\n`settings`. `conversationTurns` provides the durable per-conversation FIFO\ningress mailbox. `agentState` stores bounded conversation-scoped hook control\nstate with atomic revisions. The `settings`\n(`RuntimeSettingsStore`) facet holds stable-agent operator settings and their\ncontrol-plane audit events. Omitted facets use in-memory fallbacks and produce\none `state.degraded` warning; in particular, an ingress kill switch backed by\nthe fallback does not survive process restart. File and Postgres state adapters\nimplement the durable settings facet.\n\n## `defineInstrumentation` (instrumentation.ts)\n\n`agent/instrumentation.ts` is the single home for telemetry. It is auto-discovered\nand run once at startup; its presence enables telemetry (no separate toggle). Wire\na sink in the `setup` callback and set capture preferences as sibling fields.\n\n| Field | Type | Meaning |\n| --- | --- | --- |\n| `serviceName` | `string?` | `service.name` on spans; defaults to the agent name. |\n| `functionId` | `string?` | Overrides the function id on spans. |\n| `setup` | `(ctx) => sink \\| void` | Runs before the first turn with `{ agentName, manifest, env }`; return a `TelemetrySink` to export spans. |\n| `recordInputs` / `recordOutputs` | `boolean?` | Legacy capture flags. Default `false`. |\n| `captureContent` | `ContentCaptureLevel \\| ContentCapturePolicy?` | Content detail (see below). Default `usage`. |\n\n`@assemblyline-agents/otlp` supplies `createOtlpSink(options)` / `createOtlpSinkFromEnv(env)`\nfor the `setup` callback; see\n[Customizing Agents → Observability](customization.md#observability) for the full\nexample and the Langfuse recipe.\n\n**Capture detail (`captureContent`).** Controls how much of each model call is\nrecorded. Set it separately for each environment:\n\n| Level | Records |\n| --- | --- |\n| `off` | No span content. |\n| `usage` *(default)* | Token usage, cost, model, provider, finish reason, tool-call spans. No message bodies. |\n| `content` | Adds prompt + completion text and tool input/output, **truncated and key-redacted**. |\n| `full` | Same coverage as `content` with truncation relaxed; redaction stays on unless explicitly disabled. |\n\nA `ContentCapturePolicy` object also accepts `maxChars`, `redact`, `redactKeys`,\n`sampleRate`, and `includeToolIO`. Prompt/completion bodies can contain secrets\nand PII and increase telemetry storage/cardinality, treat `full` as\ntrusted-operator-only, and prefer `usage` in production.\n\n## `defineContext` / `defaultContext` (context.ts)\n\n`context.ts` is optional; without it the compiler stamps `defaultContext()`.\n\n- `defaultContext(options?: JsonObject)` returns\n  `{ kind: \"default\", name: \"defaultContext\", options }`. Options tune the\n  default bundle, for example `recentHistory: { maxMessages: 12 }` or\n  `files: { includeManifest: true }`.\n- `defineContext(policy)` declares a custom `ContextPolicy`:\n\n| Field | Type | Meaning |\n| --- | --- | --- |\n| `kind` | `string` | Policy kind, e.g. `\"custom\"`. |\n| `name` | `string?` | Policy name recorded in the manifest. |\n| `options` | JSON object? | Policy options. |\n| `extends` | `ContextPolicy?` | Base policy, usually `defaultContext(...)`. |\n| `sourcePath` | `string?` | Set by the compiler for attribution. |\n\n### `transcript` options\n\n`options.transcript` tunes conversation transcript resume and trimming\n(see [Framework, conversation transcript resume](framework.md)):\n\n| Option | Default | Meaning |\n| --- | --- | --- |\n| `resume` | `true` | Resume conversation follow-ups from the stored harness transcript; `false` always rebuilds from flattened recent history. |\n| `reserveTokens` | `16384` | Tokens reserved below the model context window before compaction triggers. |\n| `keepRecentTokens` | `20000` | Approximate recent-transcript tokens kept verbatim through trimming and compaction. |\n| `toolResultCapChars` | `2000` | Character cap applied to tool-result bodies outside the recent tail at resume time. |\n\n## Per-File Contracts The Compiler Validates\n\nAllowed top-level entries in an agent folder: `instructions.md`, `agent.ts`,\n`context.ts`, `gateway.ts`, `skills/`, `tools/`, `channels/`, `automations/`,\n`hooks/`, `connections/`, `evals/`, `sandbox/`, `subagents/`, and\n`instrumentation.ts`. A `resolvers.ts` file is a direct compile error with\nguidance to move capability composition into `agent.ts`. Anything else produces an\n`unknown-top-level` warning.\n`evals/**/*.json` holds golden eval cases for `assembly-line eval` (discovered\nrecursively; `evals/evaluators/` holds custom evaluator modules); see\n[Evals](agent-stack/evals.md).\n\n### tools/*.ts\n\nThe filename is the tool name (`[A-Za-z0-9_-]`, unique). The default export\nmust declare `description` and `inputSchema` (errors: `invalid-tool-name`,\n`duplicate-tool-name`, `missing-tool-export`, `missing-tool-description`,\n`missing-tool-schema`). Optional fields: `outputSchema`, `needsApproval`,\n`toModelOutput`, `defaultEnabled`, `sideEffect`\n(`\"none\" | \"idempotent\" | \"external\"`, the tool's declared side-effect class;\n`approvalRequired()` stamps `external` and `approvalNever()` stamps `none` on\nthe approval policy), and a `capability` block with `visibility`\n(`auto`/`always`/`deferred`/`skill`/`hidden`) and `execution`\n(`auto`/`direct`/`sandbox`/`both`) plus `namespace`, `tags[]`, `aliases[]`.\n`auto` (the implicit default) uses capability metadata for catalog discovery;\nthe current visible set is selected atomically by `useTool()`.\n\nA file named after a built-in harness tool overrides that built-in (the\ndescription/schema checks are skipped so overrides can spread\n`builtInToolDefaults` from `@assemblyline-agents/runtime`), and it keeps the built-in\nslot's always-on visibility. A `disableTool()` default export (from\n`@assemblyline-agents/core`) removes the built-in of that name; a filename matching no\nbuilt-in raises `invalid-disable-tool`. See\n[Customizing Agents](customization.md#override-wrap-or-disable-built-in-tools).\n\n### `skills/<name>/SKILL.md`\n\nMarkdown with YAML frontmatter. `allowed-tools` entries must name an authored\ntool or a core tool (`read`, `write`, `edit`, `delete`, `list`, `grep`,\n`bash`, `ask_question`, `load_skill`, `tool_search`,\n`tool_describe`, `tool_call`), else `unknown-skill-tool`. Frontmatter\n`description`, `tags`, and `aliases` feed the skill catalog.\n\n### channels/*.ts\n\nFields: `transport` (`http`/`local`/`webhook`/`queue`), `route`, `methods[]`,\n`requiredEnv[]`, `description`, `connection`, `metadata`, and `ingress`.\nCustom HTTP channels must declare a `route` (`missing-channel-route`), and\nchannels with production `requiredEnv` must export `send()`\n(`missing-channel-send`). Provider helpers (`defineSlackChannel`,\n`defineDiscordChannel`, `defineTelegramChannel`, `defineTeamsChannel`,\n`definePhotonChannel`) stamp kind, route,\n`requiredEnv`, and `ingress` automatically.\n\n`ingress` declares production ingress-auth secrets as any-of groups:\n\n```ts\ningress: { requiredSecretEnv: [[\"MY_WEBHOOK_SECRET\"], [\"MY_BEARER_TOKEN\"]] }\n```\n\nProduction boot succeeds when every var in at least one group is set; dev mode\nlogs a warning instead. The compiler stamps the declaration into\n`CompiledChannel.metadata.ingress`.\n\n### automations/*.ts\n\n`defineAutomation()` requires a `trigger`. Schedule triggers declare\n`{ type: \"schedule\", cron, timezone? }` and require a top-level\n`idempotencyKey`. Event triggers declare\n`{ type: \"event\", source, event, connection?, filter? }`; `filter` is a\nrecursive JSON subset matched against the normalized provider payload.\nOptional fields are `description`, `message`, `target`, `prepare`, `finalize`,\n`enabled`, and `metadata`. `prepare(ctx)` may return a message, target, prompt\ncontext, or metadata override. `finalize(ctx, result)` runs after the agent turn.\n\n### hooks/*.ts\n\nThe filename is the hook name. `defineHook({ description?, events })` subscribes\nto persisted runtime events. Event keys are exact event types or `\"*\"`; handlers\nreceive `(event, ctx)`. Hook failures emit `agent.event_handler_failed` and do\nnot fail the originating run.\n\nThe legacy `schedules/`, `triggers/`, `automation-handlers/`,\n`lifecycle.handler`, and `useEvent()` contracts remain accepted with\ndeprecation warnings.\n\n### connections/*.ts\n\nProtocol is inferred from the helper (`defineMcpClientConnection`,\n`defineMcpStdioConnection`, or `defineMcpRelayConnection` -> `mcp`,\n`defineOpenAPIConnection` -> `openapi`, `defineHttpApiConnection` -> `http`,\n`defineSandboxCliConnection` -> `cli`, otherwise `declaration`). Fields: `subject`\n(`user`/`workspace`/`installation`/`environment`, default `user`), `provider`,\n`scopes[]`, `capabilities[]`, `binding` (adapter), `url`/`baseUrl`/`spec`,\n`description`, `required` (default `true`). Provider helpers\n(`defineGitHubConnection`, `defineTeamsConnection`, `defineLiveKitConnection`)\nstamp provider, binding, and subject.\n\nLive MCP, OpenAPI, HTTP, and sandbox CLI definitions require `access`. The generic form is\n`{ read: { tools: string[] }, write: false | { tools: string[], approval } }`.\nTool entries accept exact names, `*` globs, or `regex:` patterns. A remote tool\nthat matches neither class is hidden; a write match wins over a read match.\nOfficial provider helpers expose the simpler author choice\n`{ read: true, write: false | { approval } }` and apply their packaged tool\nclassifiers. CLI-generated provider files default to `write: false`.\n\nMCP definitions discriminate on transport. The existing HTTP shape keeps\n`url` and may omit `transport` (equivalent to `\"http\"`). The stdio shape uses\n`transport: \"stdio\"`, `command`, optional `args`, `cwd`, and string-valued\n`env` overrides. The relay shape uses `transport: \"relay\"`, an HTTPS `url`, a\nfixed `credentialEnv`, and an optional bounded `timeoutMs`. Relay and stdio\ndefinitions are static-file-only; dynamic connection definitions are URL-only\nand reject process-backed or device-relay transports.\n\nSandbox CLI definitions use `transport: \"sandbox\"`, a trusted `command`, and\nreviewed static tool builders. The runtime hydrates declared logical paths and\nexecutes each built argument inside the active run sandbox, not the gateway\nhost. Dynamic connections cannot select this transport.\n\nConnection plugins may use MCP, OpenAPI, HTTP, or sandbox CLI. OpenAPI helpers can package a\nstable remote specification and base URL, while keeping the generated access\npolicy visible in `connections/*.ts`. OAuth definitions accept `tokenType` when\na provider requires an authorization scheme other than `Bearer`; SoundCloud,\nfor example, uses `tokenType: \"OAuth\"`.\n\nProcess-backed media helpers can package their own stdio MCP bridge while\nleaving external executables as trusted host dependencies. The FFmpeg and\nRemotion helpers fix the Node bridge path, child executable/prefix arguments,\nworking directory, and workspace root in source. Their remote tool schemas do\nnot accept arbitrary shell commands or CLI flags, and output paths cannot\nescape the configured root.\n\nPlatform-specific connection plugins may publish `hostRequirements` metadata\nwith `deployTargets`, Node `platforms`, and an actionable `message`. These\nconstraints are compiler-owned metadata rather than model input. Validation\nwarns when the default gateway target is incompatible, deployment planning\nmarks the constraint as a required missing item, and the runtime checks the\nplatform before loading the connection definition. Peekaboo uses\n`deployTargets: [\"local\"]` and `platforms: [\"darwin\"]`.\n\n### sandbox/*.ts\n\n`defineSandbox({ adapter, image?, workingDirectory?, env?, snapshot? })` or a\nprovider helper (`dockerSandbox()`, `e2bSandbox()`, `modalSandbox()`, ...).\n`snapshot` is `{ mode: \"never\" | \"manual\" | \"on_failure\" | \"always\",\nretainLast?, reason? }` and defaults to `never`.\n`workingDirectory` defaults to `/workspace` and may only be `/workspace`.\nHosted adapters establish it as a physical shell directory and validate it\nafter create/connect/wake; arbitrary aliases are rejected because they cannot\nmake absolute paths inside shell commands portable. `/runtime` is retired and\nrejected. The Local adapter is a trusted dev/test emulation over a temporary\nhost directory; use Docker for exact local namespace parity.\n\n### instrumentation.ts\n\n`defineInstrumentation({ serviceName?, recordInputs?,\nrecordOutputs?, captureContent?, functionId?, metadata?, setup? })`. If `setup()` returns a telemetry\nsink, the runtime uses it unless the host supplied telemetry directly.\n\n### Community plugin provider stamping\n\nWhen gateway slots or subagent `harness`/`workspace` definitions carry a\n`packageName` that is not in the built-in registry, the\ncompiler imports that package from the agent root, reads\n`assemblyLineProvider.providers[].metadata`, records it in\n`manifest.providerMetadata`, and emits the provider's `requiredEnv` and\n`setup` entries into `manifest.preflight`. Unresolvable packages produce a\n`provider-package-unresolved` warning (never a build failure) and keep a\ngeneric role-based requirement; the Node host re-validates at boot.\n\n## Environment Variable Reference\n\nEvery `ASSEMBLY_LINE_*` variable read by the packages in this repo. Boolean\nvariables accept `true`/`1` and `false`/`0` unless noted. Provider-specific\nnon-`ASSEMBLY_LINE_` env (API keys, `DATABASE_URL`, `SLACK_*`, ...) is documented in\n[Adapters](adapters.md) and [Runtime And Deployment](runtime-and-deployment.md#preflight).\n\nEvery `ASSEMBLY_LINE_*` variable also accepts its legacy `OPENEVE_*` spelling,\npermanently. This mirror is not a temporary compatibility window and will not\nbe removed.\nThe new name wins, setting both to the same value is fine, and conflicting\nvalues fail at startup.\n\n### Alphabetical index\n\nEvery variable in this reference, with the section that documents it.\n`ASSEMBLY_LINE_ARTIFACT_ROOT`, `ASSEMBLY_LINE_AGENT_REVISION`, and\n`ASSEMBLY_LINE_MIGRATION_FILES` are deploy-time outputs, not knobs; see\n[Build, deploy, and migrations](#build-deploy-and-migrations-cli-and-compiler).\n\n| Variable | Section |\n| --- | --- |\n| `ASSEMBLY_LINE_ADMIN_TOKEN` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_FALLBACK` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_ARTIFACT_PACKAGE_MODE` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_ATTACHMENT_FETCH_TIMEOUT_MS` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_ATTACHMENT_MAX_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_ATTACHMENT_MAX_COUNT` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_AUTOMATIONS_APPROVAL` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_AUTOMATIONS_DYNAMIC` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_AUTO_MIGRATE` | [Provider: Postgres (@assemblyline-agents/postgres)](#provider-postgres-assemblyline-agentspostgres) |\n| `ASSEMBLY_LINE_BASH_TIMEOUT_MAX_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_BASH_TOOL_MODE` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_BLOB_ROOT` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_BACKGROUND_REVIEW_WORKER` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_CHECKPOINT_BLOB_THRESHOLD_BYTES` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_CHECKPOINT_EVERY_ITERATION` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_CHECKPOINT_FAILED_TTL_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_CHECKPOINT_MAX_ACTIVE_PER_RUN_NAME` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_CHECKPOINT_SCHEDULED_TTL_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_CHECKPOINT_TERMINAL_TTL_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_COMPUTER_USE_BINDING` | [Computer Use relay (@assemblyline-agents/computer-use)](#computer-use-relay-assemblyline-agentscomputer-use) |\n| `ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL` | [Computer Use relay (@assemblyline-agents/computer-use)](#computer-use-relay-assemblyline-agentscomputer-use) |\n| `ASSEMBLY_LINE_CONNECTIONS_ALLOWED_HOSTS` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_CONNECTIONS_APPROVAL` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_CONNECTIONS_DYNAMIC` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_CONNECTION_AUTH_SESSIONS_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_CONNECTION_CALLBACK_BASE_URL` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_CONNECTION_DEFINITIONS_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_CONNECTION_GRANTS_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_CONVERSATION_TURN_WORKER` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_ARCHIVE_MINUTES` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_DELETE_MINUTES` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_STOP_MINUTES` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_CREATE_TIMEOUT_SECONDS` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_DOMAIN_ALLOW_LIST` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_EPHEMERAL` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_LIFECYCLE_TIMEOUT_SECONDS` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_NETWORK_ALLOW_LIST` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DAYTONA_NETWORK_BLOCK_ALL` | [Provider: Daytona (@assemblyline-agents/daytona)](#provider-daytona-assemblyline-agentsdaytona) |\n| `ASSEMBLY_LINE_DELIVERY_FILE_MAX_BYTES` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_FILE_MAX_COUNT` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_BATCH_SIZE` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_INTERVAL_MS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_LEASE_MS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_MAX_ATTEMPTS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_ATTEMPTS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_MAX_MS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_MIN_MS` | [Delivery queue](#delivery-queue) |\n| `ASSEMBLY_LINE_DELIVERY_WORKER` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_DEPLOY_ENV` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_DOCKER_COMMAND_TIMEOUT_MS` | [Provider: Docker (@assemblyline-agents/docker)](#provider-docker-assemblyline-agentsdocker) |\n| `ASSEMBLY_LINE_DOCKER_CPUS` | [Provider: Docker (@assemblyline-agents/docker)](#provider-docker-assemblyline-agentsdocker) |\n| `ASSEMBLY_LINE_DOCKER_IMAGE` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_DOCKER_MEMORY` | [Provider: Docker (@assemblyline-agents/docker)](#provider-docker-assemblyline-agentsdocker) |\n| `ASSEMBLY_LINE_DOCKER_NETWORK` | [Provider: Docker (@assemblyline-agents/docker)](#provider-docker-assemblyline-agentsdocker) |\n| `ASSEMBLY_LINE_DOCKER_PULL_POLICY` | [Provider: Docker (@assemblyline-agents/docker)](#provider-docker-assemblyline-agentsdocker) |\n| `ASSEMBLY_LINE_E2B_ALLOW_INTERNET_ACCESS` | [Provider: E2B (@assemblyline-agents/e2b)](#provider-e2b-assemblyline-agentse2b) |\n| `ASSEMBLY_LINE_E2B_PAUSE_KEEP_MEMORY` | [Provider: E2B (@assemblyline-agents/e2b)](#provider-e2b-assemblyline-agentse2b) |\n| `ASSEMBLY_LINE_E2B_REQUEST_TIMEOUT_MS` | [Provider: E2B (@assemblyline-agents/e2b)](#provider-e2b-assemblyline-agentse2b) |\n| `ASSEMBLY_LINE_E2B_RETAIN_TIMEOUT_MS` | [Provider: E2B (@assemblyline-agents/e2b)](#provider-e2b-assemblyline-agentse2b) |\n| `ASSEMBLY_LINE_E2B_TIMEOUT_MS` | [Provider: E2B (@assemblyline-agents/e2b)](#provider-e2b-assemblyline-agentse2b) |\n| `ASSEMBLY_LINE_ENABLE_API_RUNS` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_ENABLE_EVAL_RUNS` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_EVAL_JUDGE_MODEL` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_HTTP_MAX_BODY_BYTES` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_INGRESS_RATE_LIMIT` | [Concurrency and rate limiting](#concurrency-and-rate-limiting) |\n| `ASSEMBLY_LINE_LEGACY_FULL_SANDBOX_HYDRATION` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_LOG_LEVEL` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_LEARNING_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_MAX_CONCURRENT_RUNS` | [Concurrency and rate limiting](#concurrency-and-rate-limiting) |\n| `ASSEMBLY_LINE_MAX_MODEL_ITERATIONS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MAX_QUEUED_RUNS` | [Concurrency and rate limiting](#concurrency-and-rate-limiting) |\n| `ASSEMBLY_LINE_MAX_RUN_DURATION_MS` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_MEMORY_EMBEDDINGS_ENABLED` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MIGRATION_COMMAND` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_MODAL_TIMEOUT_MS` | [Provider: Modal (@assemblyline-agents/modal)](#provider-modal-assemblyline-agentsmodal) |\n| `ASSEMBLY_LINE_MODAL_WAIT_READY` | [Provider: Modal (@assemblyline-agents/modal)](#provider-modal-assemblyline-agentsmodal) |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_COUNT` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_TOTAL_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_MODEL_MAX_RETRIES` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MODEL_MAX_RETRY_DELAY_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MODEL_STREAM_IDLE_TIMEOUT_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MODEL_TIMEOUT_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_COUNT` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_TOTAL_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_OPENAI_ADMIN_KEY` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENAI_AGENT_BY_API_KEY_ID` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENAI_AGENT_BY_PROJECT_ID` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENAI_USAGE_API_KEY_IDS` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENAI_USAGE_PROJECT_IDS` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENROUTER_API_KEY_HASH` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OPENROUTER_CONTROL_AGENT_ID` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_OTLP_BATCH_MAX` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n| `ASSEMBLY_LINE_OTLP_FLUSH_MS` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n| `ASSEMBLY_LINE_OUTPUT_VALIDATION_MAX_RETRIES` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_POSTGRES_CONNECTION_ENV` | [Provider: Postgres (@assemblyline-agents/postgres)](#provider-postgres-assemblyline-agentspostgres) |\n| `ASSEMBLY_LINE_POSTGRES_SSL_REJECT_UNAUTHORIZED` | [Provider: Postgres (@assemblyline-agents/postgres)](#provider-postgres-assemblyline-agentspostgres) |\n| `ASSEMBLY_LINE_PUBLIC_URL` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_ALLOWED_KINDS` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_ALLOW_WRITABLE` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_MAX_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_MAX_FILES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_RUNS_RATE_LIMIT` | [Concurrency and rate limiting](#concurrency-and-rate-limiting) |\n| `ASSEMBLY_LINE_RUN_HEARTBEAT_MS` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_RUN_RECOVERY` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_SANDBOX_ROOT` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_MODE` | [Sandbox snapshots](#sandbox-snapshots) |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_REASON` | [Sandbox snapshots](#sandbox-snapshots) |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_RETAIN_LAST` | [Sandbox snapshots](#sandbox-snapshots) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_BATCH_SIZE` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_INLINE` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_INTERVAL_MS` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_LEASE_MS` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_MAX_ATTEMPTS` | [Sandbox sync and hydration](#sandbox-sync-and-hydration) |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_WORKER` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_SCHEDULER_ENABLED` | [Scheduler](#scheduler) |\n| `ASSEMBLY_LINE_SCHEDULER_SECRET` | [Server and auth (@assemblyline-agents/node)](#server-and-auth-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_SCHEDULES_APPROVAL` | Legacy alias for `ASSEMBLY_LINE_AUTOMATIONS_APPROVAL`. |\n| `ASSEMBLY_LINE_SCHEDULES_DYNAMIC` | Legacy alias for `ASSEMBLY_LINE_AUTOMATIONS_DYNAMIC`. |\n| `ASSEMBLY_LINE_SCHEDULES_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_SCHEDULE_MAX_FAILURES` | [Durability workers and recovery](#durability-workers-and-recovery) |\n| `ASSEMBLY_LINE_SECRET` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS` | [Graceful shutdown](#graceful-shutdown) |\n| `ASSEMBLY_LINE_SIGNAL_HANDLERS` | [Graceful shutdown](#graceful-shutdown) |\n| `ASSEMBLY_LINE_SKILLS_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_EVERY_TURNS` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_MIN_TOOL_CALLS` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_MODEL` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_SKILLS_WRITABLE` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_SKILLS_WRITE_APPROVAL` | [Self-improvement, dynamic automations, dynamic connections](#self-improvement-dynamic-automations-dynamic-connections) |\n| `ASSEMBLY_LINE_STATE_FILE` | [Secrets and local store paths (@assemblyline-agents/node)](#secrets-and-local-store-paths-assemblyline-agentsnode) |\n| `ASSEMBLY_LINE_TEAMS_ALLOWED_SERVICE_URLS` | [Provider: Microsoft Teams (@assemblyline-agents/teams)](#provider-microsoft-teams-assemblyline-agentsteams) |\n| `ASSEMBLY_LINE_TEAMS_ALLOWED_TENANTS` | [Provider: Microsoft Teams (@assemblyline-agents/teams)](#provider-microsoft-teams-assemblyline-agentsteams) |\n| `ASSEMBLY_LINE_TEAMS_OPENID_METADATA_URL` | [Provider: Microsoft Teams (@assemblyline-agents/teams)](#provider-microsoft-teams-assemblyline-agentsteams) |\n| `ASSEMBLY_LINE_TOOL_OUTPUT_MAX_CHARS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_TOOL_TIMEOUT_MS` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `ASSEMBLY_LINE_TRUST_PROXY` | [Concurrency and rate limiting](#concurrency-and-rate-limiting) |\n| `ASSEMBLY_LINE_URL` | [Build, deploy, and migrations (CLI and compiler)](#build-deploy-and-migrations-cli-and-compiler) |\n| `ASSEMBLY_LINE_USAGE_RECONCILIATION_LAG_HOURS` | [Usage accounting](#usage-accounting) |\n| `ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_BUCKET` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_REGION` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_RETENTION_DAYS` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_VPS_HOSTS_FILE` | [Provider: VPS (@assemblyline-agents/vps)](#provider-vps-assemblyline-agentsvps) |\n| `ASSEMBLY_LINE_ZIP_MAX_FILES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `ASSEMBLY_LINE_ZIP_MAX_TOTAL_BYTES` | [Attachments and resource projection](#attachments-and-resource-projection) |\n| `CODEX_ACCESS_TOKEN` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `CODEX_HOME` | [Model loop, memory, and logging](#model-loop-memory-and-logging) |\n| `OPENAI_ADMIN_KEY` | [Usage accounting](#usage-accounting) |\n| `OPENROUTER_MANAGEMENT_KEY` | [Usage accounting](#usage-accounting) |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n| `OTEL_EXPORTER_OTLP_HEADERS` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n| `OTEL_SERVICE_NAME` | [OTLP telemetry (@assemblyline-agents/otlp)](#otlp-telemetry-assemblyline-agentsotlp) |\n\n### Server and auth (`@assemblyline-agents/node`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_ADMIN_TOKEN` | unset | Bearer token for control-plane routes (`/manifest`, `/routes`, `/runs*`). Production boot fails without this or a host auth policy. |\n| `ASSEMBLY_LINE_ENABLE_API_RUNS` | disabled | `true`/`1` enables authenticated `POST /runs` in production (always on in dev mode). |\n| `ASSEMBLY_LINE_ENABLE_EVAL_RUNS` | disabled | `true`/`1` lets authenticated `POST /runs` accept the `eval` block (tool stubs, approval auto-resolve, record-only delivery). Advertised as the `eval-runs` capability on `/healthz`. Always on in dev mode; enable on test environments, not production. |\n| `ASSEMBLY_LINE_SCHEDULER_SECRET` | unset | Shared secret for `/assembly-line/automations/tick` and its deprecated scheduler alias (`Authorization: Bearer` or `x-assembly-line-scheduler-secret`, compared constant-time). Unset means dev-mode-only tick. |\n| `ASSEMBLY_LINE_HTTP_MAX_BODY_BYTES` | `10485760` (10 MiB) | Max HTTP request body size before parsing. |\n| `ASSEMBLY_LINE_PUBLIC_URL` | `http://localhost` | Public base URL; also the fallback for connection callbacks. |\n| `ASSEMBLY_LINE_CONNECTION_CALLBACK_BASE_URL` | falls back to `ASSEMBLY_LINE_PUBLIC_URL` | Base URL for OAuth/connection authorization callbacks. |\n\n### Usage accounting\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_OPENAI_ADMIN_KEY` / `OPENAI_ADMIN_KEY` | unset | OpenAI organization Admin API key used only by authenticated `POST /usage/reconcile`. Never stored in the ledger. |\n| `OPENROUTER_MANAGEMENT_KEY` | unset | Imports OpenRouter completed-day activity control totals. |\n| `ASSEMBLY_LINE_USAGE_RECONCILIATION_LAG_HOURS` | `48` | OpenAI provider-settlement delay excluded from control-total imports. |\n| `ASSEMBLY_LINE_OPENAI_USAGE_PROJECT_IDS` | all accessible | Comma-separated OpenAI project filter for provider usage and cost totals. |\n| `ASSEMBLY_LINE_OPENAI_USAGE_API_KEY_IDS` | all accessible | Comma-separated OpenAI API-key ID filter for token-usage controls. OpenAI's Costs API does not expose this filter. |\n| `ASSEMBLY_LINE_OPENAI_AGENT_BY_PROJECT_ID` | unset | JSON object mapping a dedicated OpenAI project ID to a stable Assembly Line agent ID. This is the finest supported attribution for OpenAI cash controls. |\n| `ASSEMBLY_LINE_OPENAI_AGENT_BY_API_KEY_ID` | unset | JSON object mapping a dedicated OpenAI API-key ID to a stable Assembly Line agent ID for token-usage controls. API-key mapping wins over project mapping where the provider result contains both. |\n| `ASSEMBLY_LINE_OPENROUTER_API_KEY_HASH` | unset | OpenRouter activity filter for one API-key hash. |\n| `ASSEMBLY_LINE_OPENROUTER_CONTROL_AGENT_ID` | unset | Agent attribution applied to activity totals only when the configured activity filter is dedicated to that agent. |\n\nUsage accounting is observational: it does not reserve quota, reject requests,\nor estimate missing token/cash values. Provider transactions are stored in\nexact integer micro-dollars when the provider reports cash; otherwise cash is\n`null` with `unavailable` provenance. `control_total` rows are provider\naggregate evidence and are excluded from the default transaction view so they\ncannot double count run receipts.\n\n### Concurrency and rate limiting\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_MAX_CONCURRENT_RUNS` | unlimited (`16` in production Node hosts when unset) | Max simultaneously executing brand-new runs. Accepted provider turns wait durably in their conversation mailbox; excess direct runs are rejected with `RunCapacityError` / HTTP `429`. Resumes never queue. |\n| `ASSEMBLY_LINE_MAX_QUEUED_RUNS` | `0` | Direct brand-new runs allowed to wait in the in-process semaphore before rejection. Accepted provider turns use the durable conversation mailbox instead. |\n| `ASSEMBLY_LINE_INGRESS_RATE_LIMIT` | off | Provider-channel token bucket as `capacity/refillPerSecond` (e.g. `60/10`). |\n| `ASSEMBLY_LINE_RUNS_RATE_LIMIT` | off | `POST /runs` token bucket in the same form. Also seeds the run-resume and run-control buckets unless those are configured separately through `NodeRuntimeServerOptions.rateLimit`. |\n| `ASSEMBLY_LINE_TRUST_PROXY` | off | `true` (exactly) trusts the first `X-Forwarded-For` hop as the client address for rate-limit keying. Set it when the host sits behind a reverse proxy or load balancer; without it, all proxied traffic shares one rate-limit bucket keyed by the proxy's address. |\n\n### Graceful shutdown\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS` | `30000` | Max wait for in-flight runs to drain during graceful shutdown. |\n| `ASSEMBLY_LINE_SIGNAL_HANDLERS` | on | `false`/`0` prevents `listenNodeRuntime` from installing SIGTERM/SIGINT graceful-shutdown handlers. |\n\n### OTLP telemetry (`@assemblyline-agents/otlp`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `OTEL_EXPORTER_OTLP_ENDPOINT` | required for `createOtlpSinkFromEnv` | OTLP/HTTP endpoint. |\n| `OTEL_EXPORTER_OTLP_HEADERS` | unset | Comma-separated OTLP headers such as `Authorization=Basic ...`. |\n| `OTEL_SERVICE_NAME` | instrumentation service name | Service name override for exported spans. |\n| `OTEL_EXPORTER_OTLP_TIMEOUT` | sink default | Per-export timeout in milliseconds. |\n| `ASSEMBLY_LINE_OTLP_BATCH_MAX` | sink default | Max spans batched before an OTLP flush. |\n| `ASSEMBLY_LINE_OTLP_FLUSH_MS` | sink default | Flush interval for the OTLP sink. |\n\n### Durability workers and recovery\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_DELIVERY_WORKER` | on | `false`/`0` disables the delivery queue worker. |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_WORKER` | on | `false`/`0` disables the sandbox-sync worker. |\n| `ASSEMBLY_LINE_CONVERSATION_TURN_WORKER` | on | `false`/`0` disables periodic mailbox recovery/polling. Newly accepted and terminally settled turns still request an immediate local drain. |\n| `ASSEMBLY_LINE_BACKGROUND_REVIEW_WORKER` | on | `false`/`0` disables processing queued background learning reviews. |\n| `ASSEMBLY_LINE_RUN_RECOVERY` | on | `false`/`0` disables boot recovery and the periodic orphan sweep. |\n| `ASSEMBLY_LINE_RUN_HEARTBEAT_MS` | `30000` | How often executing runs bump `updatedAt` to stay out of the orphan sweep. |\n| `ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS` | `60000` | Orphan sweep interval; runs are only candidates after `max(5min, 4x heartbeat)` staleness. |\n| `ASSEMBLY_LINE_MAX_RUN_DURATION_MS` | `3600000` | Wall-clock cap per active run execution segment, enforced by the heartbeat: on expiry the in-flight model request is aborted and the run fails with reason `run.max_duration_exceeded`. `0` disables. Paused (approval/input) runs do not consume budget. |\n| `ASSEMBLY_LINE_SCHEDULE_MAX_FAILURES` | `5` | Consecutive dynamic-schedule failures before the schedule is auto-disabled (escalating backoff between attempts: 5 min doubling, capped at 6 h). |\n\n### Delivery queue\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_ATTEMPTS` | `2` | In-process send attempts before deferring to the durable queue. |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_MIN_MS` | `250` | Min backoff between in-process retries. |\n| `ASSEMBLY_LINE_DELIVERY_RETRY_MAX_MS` | `5000` | Max backoff between in-process retries. |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_LEASE_MS` | `60000` | Lease duration for a `sending` delivery before it is requeued. The inline sender's own lease is `max(this, 120000)` so it outlives the in-process retry envelope. |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_BATCH_SIZE` | `10` | Deliveries leased per worker tick. |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_MAX_ATTEMPTS` | `5` | Total attempts before a delivery goes terminally `failed`. |\n| `ASSEMBLY_LINE_DELIVERY_QUEUE_INTERVAL_MS` | `15000` | Delivery worker tick interval. |\n| `ASSEMBLY_LINE_DELIVERY_FILE_MAX_COUNT` | `10` (max 100) | Max attachment files per delivery. |\n| `ASSEMBLY_LINE_DELIVERY_FILE_MAX_BYTES` | `52428800` (50 MiB) | Max bytes per delivery file. |\n\n### Sandbox sync and hydration\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_INLINE` | `false` | Run sandbox sync inline instead of through the background worker. |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_LEASE_MS` | `300000` | Lease duration for a claimed sync job. |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_BATCH_SIZE` | `10` | Sync jobs leased per worker tick. |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_MAX_ATTEMPTS` | `5` | Max attempts before a job is blocked for an operator. |\n| `ASSEMBLY_LINE_SANDBOX_SYNC_INTERVAL_MS` | `30000` | Sandbox-sync worker tick interval. |\n| `ASSEMBLY_LINE_LEGACY_FULL_SANDBOX_HYDRATION` | off | `true`/`1` forces legacy full hydration instead of minimal per-path hydration. |\n\n### Sandbox snapshots\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_MODE` | unset (policy default `never`) | `never`, `manual`, `on_failure`, or `always`; overrides the declared snapshot policy. |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_RETAIN_LAST` | unset | Snapshots to retain (non-negative integer); only read when a mode is set. |\n| `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_REASON` | unset | Free-form snapshot reason label. |\n\n### Model loop, memory, and logging\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `CODEX_HOME` | Codex CLI default | Location of the Codex CLI config, login cache, and persisted app-server threads used by `openai-codex/*`. Assembly Line never reads the OAuth cache directly. |\n| `CODEX_ACCESS_TOKEN` | unset | Input for `printenv CODEX_ACCESS_TOKEN | codex login --with-access-token` in eligible Enterprise automation. Assembly Line does not consume it, and it is not a general OpenAI API key. |\n| `ASSEMBLY_LINE_CHECKPOINT_EVERY_ITERATION` | `true` | Persist the harness continuation checkpoint after every model iteration (bounds crash loss to one in-flight model call). |\n| `ASSEMBLY_LINE_CHECKPOINT_MAX_ACTIVE_PER_RUN_NAME` | `2` | Latest overwrite-style active checkpoints retained per `(run_id, name)` for live/running/waiting runs. |\n| `ASSEMBLY_LINE_CHECKPOINT_TERMINAL_TTL_MS` | `604800000` (7 days) | Default TTL used by checkpoint maintenance for completed/cancelled terminal-run checkpoints. |\n| `ASSEMBLY_LINE_CHECKPOINT_FAILED_TTL_MS` | `1209600000` (14 days) | TTL used by checkpoint maintenance for failed-run checkpoints. |\n| `ASSEMBLY_LINE_CHECKPOINT_SCHEDULED_TTL_MS` | `86400000` (1 day) | Aggressive TTL used by checkpoint maintenance for terminal runs produced by schedules. |\n| `ASSEMBLY_LINE_CHECKPOINT_BLOB_THRESHOLD_BYTES` | `65536` (64 KiB) | Checkpoints at or above this serialized size are gzip-compressed into the configured blob adapter, with SQL storing a pointer/hash/size record. |\n| `ASSEMBLY_LINE_TOOL_OUTPUT_MAX_CHARS` | `8000` | Max serialized tool-output size handed back to the model before it becomes a `{ truncated, preview }` object. |\n| `ASSEMBLY_LINE_MAX_MODEL_ITERATIONS` | `25` | Default agent-loop iteration budget. A positive integer; agent/subagent `maxIterations` overrides it. |\n| `ASSEMBLY_LINE_OUTPUT_VALIDATION_MAX_RETRIES` | `2` | Corrective model retries after a final response fails `outputSchema`. Validation retries share the active execution's iteration budget. |\n| `ASSEMBLY_LINE_MODEL_MAX_RETRIES` | `2` | Retry attempts (beyond the initial one) for a model request that fails retryably, 408/429/5xx, overload, network errors, with jittered exponential backoff. Also forwarded to provider SDK clients. Fatal errors (bad key, invalid request) never retry. |\n| `ASSEMBLY_LINE_MODEL_TIMEOUT_MS` | `600000` | HTTP request timeout per model call, forwarded to the provider SDK. |\n| `ASSEMBLY_LINE_MODEL_MAX_RETRY_DELAY_MS` | `60000` | Cap on backoff delays and server-requested (`Retry-After`) waits between model retries. |\n| `ASSEMBLY_LINE_MODEL_STREAM_IDLE_TIMEOUT_MS` | `300000` | Abort a model stream when no event arrives for this long (a stalled provider stream would otherwise hang the run). The aborted attempt is retried when the retry budget allows. `0` disables. |\n| `ASSEMBLY_LINE_TOOL_TIMEOUT_MS` | `600000` | Default wall-clock deadline for tool `execute` calls; a per-tool `timeoutMs` on the tool definition overrides it. The promise is raced, not cancelled. `0` disables. |\n| `ASSEMBLY_LINE_BASH_TIMEOUT_MAX_MS` | `600000` | Upper clamp on model-supplied `bash` timeouts. `timeoutMs: 0`/negative falls back to the 30 s default instead of disabling the timeout. |\n| `ASSEMBLY_LINE_EVAL_JUDGE_MODEL` | agent model | Default `provider/model` for eval cases with `expect.judge`; `--judge-model` takes precedence. |\n| `ASSEMBLY_LINE_MEMORY_EMBEDDINGS_ENABLED` | on only when an embedding provider is configured | Enables embedding-backed semantic memory search. |\n| `ASSEMBLY_LINE_LOG_LEVEL` | `info` | Structured log verbosity: `debug`, `info`, `warn`, or `error`. |\n| `ASSEMBLY_LINE_BASH_TOOL_MODE` | `enabled` | Core `bash` tool policy: `enabled`, `approval`, or `disabled`. Hosts can gate any tool by name with `RuntimeOptions.coreToolPolicy` (e.g. `{ write: \"disabled\" }`). |\n\n### Attachments and resource projection\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_ATTACHMENT_MAX_COUNT` | `20` (max 100) | Max inbound attachments per turn. |\n| `ASSEMBLY_LINE_ATTACHMENT_MAX_BYTES` | `52428800` (50 MiB) | Max bytes per attachment. |\n| `ASSEMBLY_LINE_ATTACHMENT_FETCH_TIMEOUT_MS` | `30000` (1s-120s) | Attachment download timeout. |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_COUNT` | `20` (max 100) | Max stored images hydrated as native model input per turn. |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_BYTES` | `20971520` (20 MiB) | Max decoded bytes for one native model image input. |\n| `ASSEMBLY_LINE_MODEL_IMAGE_MAX_TOTAL_BYTES` | `52428800` (50 MiB) | Max decoded image bytes supplied to one model turn. |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_COUNT` | `4` (max 20) | Max stored videos hydrated as native model input per turn. |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_BYTES` | `52428800` (50 MiB) | Max decoded bytes for one native model video input. |\n| `ASSEMBLY_LINE_MODEL_VIDEO_MAX_TOTAL_BYTES` | `104857600` (100 MiB) | Max decoded video bytes supplied to one model turn. |\n| `ASSEMBLY_LINE_ZIP_MAX_FILES` | `500` (max 10000) | Max entries when expanding a ZIP attachment. |\n| `ASSEMBLY_LINE_ZIP_MAX_TOTAL_BYTES` | `104857600` (100 MiB) | Max total uncompressed ZIP bytes. |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_MAX_FILES` | `8` | Max resources projected into a sandbox per request. |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_MAX_BYTES` | `262144` (256 KiB) | Max bytes per projected resource. |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_ALLOWED_KINDS` | all kinds | Comma-separated allowlist of projectable resource kinds. |\n| `ASSEMBLY_LINE_RESOURCE_PROJECTION_ALLOW_WRITABLE` | `false` | Allow projecting writable resources. |\n\n### Self-improvement, dynamic automations, dynamic connections\n\nEnv overrides for the `agent.ts` blocks of the same names.\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT` | `true` | Enable durable skill writes and background review. |\n| `ASSEMBLY_LINE_SKILLS_WRITABLE` | `true` | Deprecated compatibility alias for `ASSEMBLY_LINE_SELF_IMPROVEMENT`. |\n| `ASSEMBLY_LINE_SKILLS_WRITE_APPROVAL` | `false` | Skill writes require approval. |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_EVERY_TURNS` | `10` | Review every N completed foreground turns. |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_MIN_TOOL_CALLS` | `5` | Immediately review runs with at least this many tool calls. |\n| `ASSEMBLY_LINE_SELF_IMPROVEMENT_REVIEW_MODEL` | `inherit` | Reviewer model selection. |\n| `ASSEMBLY_LINE_AUTOMATIONS_DYNAMIC` | `true` | Agent may create dynamic time-based automations. |\n| `ASSEMBLY_LINE_AUTOMATIONS_APPROVAL` | `false` | Dynamic automation changes require approval. |\n| `ASSEMBLY_LINE_SCHEDULES_DYNAMIC` | None | Deprecated alias for `ASSEMBLY_LINE_AUTOMATIONS_DYNAMIC`. |\n| `ASSEMBLY_LINE_SCHEDULES_APPROVAL` | None | Deprecated alias for `ASSEMBLY_LINE_AUTOMATIONS_APPROVAL`. |\n| `ASSEMBLY_LINE_CONNECTIONS_DYNAMIC` | `false` | Agent may persist dynamic connections. |\n| `ASSEMBLY_LINE_CONNECTIONS_APPROVAL` | `true` | Saving a dynamic connection requires approval. |\n| `ASSEMBLY_LINE_CONNECTIONS_ALLOWED_HOSTS` | manifest `allowedHosts` or empty | Comma-separated dynamic-connection host allowlist. |\n\n### Scheduler\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_SCHEDULER_ENABLED` | on | `false`/`0` disables the in-process scheduler loop. |\n\n### Secrets and local store paths (`@assemblyline-agents/node`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` | unset | Encryption secret for file-backed connection credential stores. Must be at least 32 characters outside dev mode. |\n| `ASSEMBLY_LINE_SECRET` | unset | Fallback for `ASSEMBLY_LINE_CONNECTION_STORE_SECRET`; the same 32-character minimum applies. |\n| `ASSEMBLY_LINE_CONNECTION_GRANTS_FILE` | `<artifactRoot>/connection-grants.enc.json` | Encrypted connection-grant store path. |\n| `ASSEMBLY_LINE_CONNECTION_AUTH_SESSIONS_FILE` | `<artifactRoot>/connection-auth-sessions.enc.json` | Encrypted authorization-session store path. |\n| `ASSEMBLY_LINE_CONNECTION_DEFINITIONS_FILE` | `<artifactRoot>/connection-definitions.json` | Dynamic connection definition store path. |\n| `ASSEMBLY_LINE_SKILLS_FILE` | `<artifactRoot>/skills-store.json` | Durable skill store path (file-backed state). |\n| `ASSEMBLY_LINE_LEARNING_FILE` | `<artifactRoot>/learning-store.json` | Skill revision, pending-change, and background-review queue path. |\n| `ASSEMBLY_LINE_SCHEDULES_FILE` | `<artifactRoot>/dynamic-schedules.json` | Dynamic schedule store path (file-backed state). |\n| `ASSEMBLY_LINE_STATE_FILE` | `<artifactRoot>/runtime-state.json` | File-backed runtime state path. |\n| `ASSEMBLY_LINE_BLOB_ROOT` | `<artifactRoot>/blobs` | Local blob storage root. |\n| `ASSEMBLY_LINE_SANDBOX_ROOT` | `<artifactRoot>/sandbox` | Local sandbox root. |\n\n### Build, deploy, and migrations (CLI and compiler)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_ARTIFACT_PACKAGE_MODE` | `local` | `release` makes build artifacts reference published `@assemblyline-agents/*` versions instead of vendored workspace copies. |\n| `ASSEMBLY_LINE_DOCKER_IMAGE` | `openeve:<agentRevision[0:12]>` | Docker deploy image tag (after `--docker-image`). |\n| `ASSEMBLY_LINE_DEPLOY_ENV` | `development` | Deployment environment after `--env` and before the `gateway.ts` option/default. |\n| `ASSEMBLY_LINE_MIGRATION_COMMAND` | unset | Migration runner executable for hosted deploys (after `--migration-command`). |\n| `ASSEMBLY_LINE_URL` | unset | Default deployed-agent base URL for `runs` and `agent` CLI commands when `--url` is omitted. |\n| `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION` | off | `true` (exactly) acknowledges and permits an unconfined local sandbox on a non-local deploy target; otherwise planning fails before publish. |\n\nBuilt-in deploy options use the same precedence everywhere: CLI flag,\nenvironment variable, static `gateway.ts` option, then provider default. A\nhosted plan warns when local state or local blob storage is selected because\nthose files may disappear when a container is replaced.\n\nDuring a hosted deploy the CLI sets these in the migration process\nenvironment (they are outputs, not knobs): `ASSEMBLY_LINE_ARTIFACT_ROOT`,\n`ASSEMBLY_LINE_AGENT_REVISION`, `ASSEMBLY_LINE_DEPLOY_ENV`, `ASSEMBLY_LINE_MIGRATION_FILES`.\n\n### Provider: Postgres (`@assemblyline-agents/postgres`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_POSTGRES_CONNECTION_ENV` | `DATABASE_URL` | Name of the env var holding the connection string. |\n| `ASSEMBLY_LINE_POSTGRES_SSL_REJECT_UNAUTHORIZED` | `true` (Railway preset: `false`) | Certificate verification is on by default when SSL is used; Railway's generated certificate is self-signed. |\n| `ASSEMBLY_LINE_AUTO_MIGRATE` | on | `false` (exactly) skips running Assembly Line migrations at provider construction. |\n\n`neonPostgres()`, `railwayPostgres()`, and `supabasePostgres()` all read\n`DATABASE_URL` by default and use the same Assembly Line schema and migrations.\n`railwayPostgres()` additionally accepts `databaseService` (default `Postgres`)\nand `provision` (default `true`). During a Railway deploy, those options create\nor reuse that Railway database service and wire a private `DATABASE_URL`\nreference onto the application service. Named non-default services must already\nexist and use `provision: false`. The preset also defaults\n`sslRejectUnauthorized` to `false` for Railway's generated Postgres certificate;\nTLS remains enabled. For Supabase, use the direct URL on an IPv6-capable\npersistent host or the session-pooler URL when IPv4 is required.\n\n### Provider: Docker (`@assemblyline-agents/docker`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_DOCKER_NETWORK` | `none` | Container network for sandbox containers. |\n| `ASSEMBLY_LINE_DOCKER_CPUS` | unset | CPU limit passed to `docker run`. |\n| `ASSEMBLY_LINE_DOCKER_MEMORY` | unset | Memory limit passed to `docker run`. |\n| `ASSEMBLY_LINE_DOCKER_PULL_POLICY` | unset (Docker default) | `never`, `missing`, or `always`. |\n| `ASSEMBLY_LINE_DOCKER_COMMAND_TIMEOUT_MS` | unset | Timeout for Docker CLI commands. |\n\n### Provider: VPS (`@assemblyline-agents/vps`)\n\n*Supported.*\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_VPS_HOSTS_FILE` | nearest `assembly-line.hosts.json` | Explicit versioned named-host inventory path; `--vps-hosts-file` takes precedence. |\n| Inventory `identityFileEnv` variable | required | Local path to the SSH private key for that named host. The variable name is inventory-defined. |\n| `ASSEMBLY_LINE_VPS_BACKUP_BUCKET` | required in host Postgres mode | S3-compatible database-backup bucket. |\n| `ASSEMBLY_LINE_VPS_BACKUP_REGION` | required in host Postgres mode | Backup bucket region. |\n| `ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID` | required in host Postgres mode | Backup-only S3 access key. |\n| `ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY` | required in host Postgres mode | Backup-only S3 secret key. |\n| `ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT` | provider default | Optional custom S3-compatible endpoint. |\n| `ASSEMBLY_LINE_VPS_BACKUP_RETENTION_DAYS` | `30` | Number of days retained by the daily S3-compatible backup job. |\n| `ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL` | unset | Optional webhook receiving five-minute runtime, public readiness, Postgres, backup-verification, timer, and disk alerts. Checks still run and record failures in systemd/journald when unset. |\n\nThe `vpsDeploy()` options are:\n\n| Option | Default | Effect |\n| --- | --- | --- |\n| `host` | required | Named host from the versioned inventory. |\n| `ingress: { visibility }` | host default | `\"public\"` derives a hostname from the agent ID, environment, and host namespace; `\"private\"` creates no public route. |\n| `expectedRegion` | unset | Emits a latency warning when inventory reports a different provider region. |\n| `hostsFile` | inventory discovery | Explicit inventory path. |\n| `resources: { cpus, memory, pids }` | `{ cpus: 1, memory: \"1g\", pids: 256 }` | Runtime container limits. |\n| `database: { mode }` | `\"external\"` | `\"external\"` uses `DATABASE_URL`; `\"host\"` provisions an isolated database and role in managed host Postgres. |\n| `monitoring: { enabled, diskFreeMinimumMb }` | `{ enabled: true, diskFreeMinimumMb: 5120 }` | Installs the deployment health timer. A configured alert webhook receives failures; local checks do not depend on it. |\n| `caddyImage` | `caddy:2.10.0-alpine` | Explicit non-`latest` edge image. |\n| `postgresImage` | `postgres:17.10-alpine` | Explicit numeric-major Postgres image. An image mismatch requires `state upgrade-postgres`. |\n| `awsCliImage` | `amazon/aws-cli:2.17.57` | Explicit non-`latest` backup client image. |\n\nEach host inventory entry requires\n`ingress: { baseDomain, defaultVisibility }`. Create one wildcard DNS record for\n`*.<baseDomain>` pointing at the host. The default environment receives\n`<agent-id>.<baseDomain>`; alternate environments receive\n`<agent-id>-<environment>.<baseDomain>`.\n\nCLI overrides are `--vps-host` and `--vps-hosts-file`.\nLifecycle commands add `deploy --prepare-only`, `deploy --activate`,\n`deploy --rollback`, and `deploy --ingress-only`. Operational commands are\n`hosts bootstrap`, `state migrate-postgres`, `state upgrade-postgres`,\n`secrets diff`, and `agent quiesce|resume|status`.\n\n### Provider: E2B (`@assemblyline-agents/e2b`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_E2B_TIMEOUT_MS` | SDK default | Sandbox lifetime timeout. |\n| `ASSEMBLY_LINE_E2B_RETAIN_TIMEOUT_MS` | SDK default | Retain/pause timeout for dirty sandboxes. |\n| `ASSEMBLY_LINE_E2B_REQUEST_TIMEOUT_MS` | SDK default | Per-request timeout. |\n| `ASSEMBLY_LINE_E2B_PAUSE_KEEP_MEMORY` | SDK default | Keep memory when pausing. |\n| `ASSEMBLY_LINE_E2B_ALLOW_INTERNET_ACCESS` | SDK default | Sandbox internet egress. |\n\n`sandbox/*.ts` `env` arrays are global passthrough variables. Validation and\nthe Node runtime reject host/control-plane credentials such as database URLs,\nRailway/Hetzner credentials, E2B control keys, R2 secret keys, Photon tokens,\nadmin tokens, and OTLP auth headers in this list. Supply capability\ncredentials through typed connections or the per-command `shell(..., { env })`\nscope.\n\n### Provider: Modal (`@assemblyline-agents/modal`)\n\n*Preview: this surface may change without notice.*\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_MODAL_TIMEOUT_MS` | SDK default | Sandbox timeout. |\n| `ASSEMBLY_LINE_MODAL_WAIT_READY` | SDK default | Wait for the sandbox to be ready before use. |\n\n### Provider: Daytona (`@assemblyline-agents/daytona`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_DAYTONA_CREATE_TIMEOUT_SECONDS` | SDK default | Sandbox creation timeout. |\n| `ASSEMBLY_LINE_DAYTONA_LIFECYCLE_TIMEOUT_SECONDS` | SDK default | Lifecycle operation timeout. |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_STOP_MINUTES` | SDK default | Auto-stop interval. |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_ARCHIVE_MINUTES` | SDK default | Auto-archive interval. |\n| `ASSEMBLY_LINE_DAYTONA_AUTO_DELETE_MINUTES` | SDK default | Auto-delete interval. |\n| `ASSEMBLY_LINE_DAYTONA_EPHEMERAL` | `true` | Only the literal string `false` disables ephemeral sandboxes. |\n| `ASSEMBLY_LINE_DAYTONA_NETWORK_BLOCK_ALL` | SDK default | Block all sandbox network egress. |\n| `ASSEMBLY_LINE_DAYTONA_NETWORK_ALLOW_LIST` | unset | Network allowlist. |\n| `ASSEMBLY_LINE_DAYTONA_DOMAIN_ALLOW_LIST` | unset | Domain allowlist. |\n| `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_FALLBACK` | off | `true` (exactly) allows local fallback when the Daytona client is absent (dev/test only). |\n\n### Computer Use Relay (`@assemblyline-agents/computer-use`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_COMPUTER_USE_BINDING` | required | Device-scoped encrypted binding generated when the Mac host pairs with the deployment. Keep it in the runtime secret store. |\n| `ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL` | `https://computer-use.openeve.dev/v1` | Relay base URL. Set it only for a self-hosted relay. |\n\nSee [Remote Computer Use](remote-computer-use.md) for pairing, write policy,\nand relay trust boundaries.\n\n### Provider: Microsoft Teams (`@assemblyline-agents/teams`)\n\n| Variable | Default | Effect |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_TEAMS_ALLOWED_TENANTS` | all tenants | Comma-separated tenant ID allowlist. |\n| `ASSEMBLY_LINE_TEAMS_ALLOWED_SERVICE_URLS` | all URLs | Comma-separated service-URL prefix allowlist. |\n| `ASSEMBLY_LINE_TEAMS_OPENID_METADATA_URL` | Bot Framework default | Override for the OpenID metadata URL used in JWT verification. |\n"},{"id":"contributing","sourcePath":"contributing.md","title":"Contributing","description":"Work on the Assembly Line framework itself and keep developer docs current.","url":"https://assemblyline.artificialillumination.co/docs/contributing","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/contributing.md","headings":[{"depth":1,"title":"Contributing","anchor":"contributing"},{"depth":2,"title":"Repo Setup","anchor":"repo-setup"},{"depth":2,"title":"Working Principles","anchor":"working-principles"},{"depth":2,"title":"Generated Files","anchor":"generated-files"},{"depth":2,"title":"Test Strategy","anchor":"test-strategy"},{"depth":3,"title":"Postgres Durability Suite","anchor":"postgres-durability-suite"},{"depth":2,"title":"Documentation Rule","anchor":"documentation-rule"},{"depth":2,"title":"Adding A New Agent Capability","anchor":"adding-a-new-agent-capability"},{"depth":2,"title":"Adding A Plugin Provider","anchor":"adding-a-plugin-provider"},{"depth":2,"title":"Versioning And Releases","anchor":"versioning-and-releases"},{"depth":2,"title":"Release Notes","anchor":"release-notes"}],"content":"# Contributing\n\nThis page is for developers changing Assembly Line itself. The root\n[CONTRIBUTING.md](https://github.com/jasonbadeaux/assembly-line/blob/main/CONTRIBUTING.md)\nis the canonical short entrypoint; it links here for the full guide.\n\n## Repo Setup\n\n```sh\npnpm install\npnpm build\npnpm check\npnpm test\n```\n\nThere is no separate lint step; `pnpm check` (build plus typecheck) is the\nquality gate. CI runs `pnpm check` and `pnpm test:coverage` on Node 22.19 and\n24, plus the [Postgres durability suite](#postgres-durability-suite) against a\n`postgres:16` service container. A separate weekly `Smoke` workflow\n(`.github/workflows/smoke.yml`) covers sandbox, channel, and deploy smokes,\nwith credentialed steps gated on repository secrets.\n\nAssembly Line is a TypeScript workspace: framework and plugin packages live under\n`packages/`, single-vendor capability packages under `capabilities/`, example\nagents under `examples/`, and documentation under `docs/`. The canonical\npackage-by-package layout is the\n[Repository Layout in the root README](https://github.com/jasonbadeaux/assembly-line/blob/main/README.md#repository-layout).\n\n## Working Principles\n\n- Keep framework packages product-neutral.\n- Prefer one small module per responsibility over large orchestration files.\n- Keep channel providers at the boundary: verify, normalize, preserve delivery metadata, and send replies.\n- Keep state, blob, sandbox, scheduling, approvals, and recovery in runtime or adapter contracts.\n- Keep provider helpers as sugar over stable `@assemblyline-agents/core` contracts.\n- Treat generated build output as disposable.\n\n## Generated Files\n\nDo not commit:\n\n- `.assembly-line/` (or legacy `.openeve/`)\n- package `dist/` output from local builds\n- `node_modules/`\n- package manager caches\n- local env files\n- duplicated compiled tests\n\nIf a generated artifact is needed for evidence, document the command and the relevant output instead of checking in the artifact.\n\nUse `pnpm clean:artifacts` to remove ignored `examples/**/.assembly-line`\n(and legacy `examples/**/.openeve`) directories after local build or deploy\nexperiments.\n\nAcceptance tests use small release-mode fixtures by default and reserve full\nlocal dependency vendoring for packaging assertions. Full test commands clean\ntheir workspace-scoped temp root after the suite finishes. Use\n`pnpm clean:tmp` to remove marker-owned test directories left behind by an\ninterrupted or targeted run.\n\n## Test Strategy\n\nRun the full suite before broad changes:\n\n```sh\npnpm test\n```\n\nUse `pnpm test:coverage` to run the same suite with Node's built-in coverage\nreporting, matching what CI runs.\n\nUse targeted tests during development:\n\n```sh\npnpm build\nnode --import=./tests/helpers/test-environment.mjs --test tests/openeve-compile.test.mjs\nnode --import=./tests/helpers/test-environment.mjs --test tests/openeve-runtime.test.mjs\nnode --import=./tests/helpers/test-environment.mjs --test tests/adapters.test.mjs\nnode --import=./tests/helpers/test-environment.mjs --test tests/self-improvement.test.mjs\n```\n\n### Postgres Durability Suite\n\n`tests/postgres-durability.test.mjs` exercises the production\n`PostgresStateAdapter` (migrations, skip-locked delivery/sync leasing,\nlease-token settling, idempotency reservation, and event sequencing) against a\nreal database with two adapter instances acting as two replicas. It runs with\nthe rest of `pnpm test` when a database can be provisioned and skips cleanly\notherwise; CI additionally runs it in a dedicated job with a `postgres:16`\nservice container.\n\nRun it locally with:\n\n```sh\npnpm build\nnode --test tests/postgres-durability.test.mjs\n```\n\nThe suite finds a database in this order:\n\n1. `OPENEVE_TEST_DATABASE_URL`, an existing server. The suite creates and\n   drops a throwaway `openeve_test_<hex>` database per run; if the role cannot\n   create databases it uses the given database directly and **resets its\n   `public` schema**, so never point this at a database you care about.\n2. Local `initdb`/`pg_ctl` binaries (PATH, Homebrew `postgresql*` kegs, or\n   Postgres.app), boots a temp data dir on a random port and removes it\n   afterwards.\n3. A running Docker daemon, starts a disposable `postgres:16` container\n   (override the image with `OPENEVE_TEST_POSTGRES_IMAGE`).\n\nWithout any of these the file skips cleanly with an explanatory message.\n\nChanges should include tests when they alter:\n\n- Manifest shape or validation rules.\n- Runtime lifecycle, recovery, approvals, delivery, or persistence.\n- Adapter metadata, env preflight, deploy planning, or provider helper behavior.\n- Tool execution, sandbox hydration, memory/resource behavior, or model loop integration.\n- Public package exports.\n\n## Documentation Rule\n\nDeveloper documentation must change with material code changes.\n\nWhen a change affects setup, CLI commands, agent authoring, runtime behavior, adapter behavior, provider env, deployment, public APIs, examples, or package boundaries, update the relevant docs in the same change:\n\n- [Root README](https://github.com/jasonbadeaux/assembly-line/blob/main/README.md)\n- [Docs Index](https://github.com/jasonbadeaux/assembly-line/blob/main/docs/README.md)\n- [Developer Docs Index](README.md)\n- [Getting Started](getting-started.md)\n- [Building Agents](building-agents.md)\n- [Customizing Agents](customization.md)\n- [Runtime And Deployment](runtime-and-deployment.md)\n- [Framework Guide](framework.md)\n- [Adapters](adapters.md)\n- Provider-specific docs such as [Photon](photon.md)\n- Local working notes under `docs/internal/` (gitignored, not published) when the change affects project plans, deployment evidence, or implementation rationale.\n- Example READMEs when examples change\n\nIf a material code change does not require docs, note why in the PR or commit message. Small internal refactors with no developer-visible behavior usually do not need docs updates.\n\n## Adding A New Agent Capability\n\n1. Decide the owner: core definition, compiler extraction, runtime behavior, provider adapter, or example-only code.\n2. Add the smallest public API that fits the existing `define*` and adapter patterns.\n3. Add validation and manifest output when the capability is declared from files.\n4. Add runtime behavior only where the capability is executed.\n5. Add plugin package helpers only when they keep app code smaller without hiding important contracts.\n6. Add tests at the package or acceptance level.\n7. Update the docs that teach the new behavior.\n\n## Adding A Plugin Provider\n\nPlugin provider contributions should document and test:\n\n- Required and optional environment variables.\n- Authentication or signature verification.\n- Normalized input shape.\n- Idempotency keys and retry behavior.\n- Delivery behavior.\n- Preflight requirements.\n- Local test strategy.\n\nAdd provider metadata in `@assemblyline-agents/core`, helper exports in the plugin package,\ncompiler/runtime wiring if needed, tests, and docs. Keep provider and adapter\nnames precise inside the implementation while describing the installable\npackage as a plugin in user-facing material.\n\n## Versioning And Releases\n\nAssembly Line uses [Changesets](https://github.com/changesets/changesets) for\nversioning. Every PR with a user-visible change must include a changeset:\n\n```sh\npnpm changeset\n```\n\nAll publishable packages (`@assemblyline-agents/sdk` and the other `@assemblyline-agents/*` packages) version in lockstep\npre-1.0, so bumping any one of them bumps all of them; example packages are\nignored. On pushes to `main`, the release workflow\n(`.github/workflows/release.yml`) opens or updates a \"Version Packages\" PR\nthat applies pending changesets. Merging that PR publishes to npm.\n\nPublishing requires the `NPM_TOKEN` repository secret, which is a maintainer\naction.\n\n## Release Notes\n\nAssembly Line releases as one public `@assemblyline-agents/sdk` CLI/meta package plus the\nother scoped `@assemblyline-agents/*` packages. The workspace root remains private and should never be\npublished.\n\nBefore a package release:\n\n- Run `pnpm build`.\n- Run `pnpm test`.\n- Run `pnpm --filter @assemblyline-agents/sdk pack --dry-run` for the public meta package.\n- Run `pnpm release:pack` to produce local package tarballs under\n  `.release-packs/`.\n- Run `pnpm release:dry-run` before publishing.\n\nFor each release-facing change, keep changes easy to audit:\n\n- Summarize developer-facing behavior in the PR or commit.\n- Mention migrations or required env changes explicitly.\n- Point to updated docs.\n- Include verification commands.\n"},{"id":"customization","sourcePath":"customization.md","title":"Customizing Agents","description":"Extend an agent through explicit context, runtime, policy, and provider configuration.","url":"https://assemblyline.artificialillumination.co/docs/customization","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/customization.md","headings":[{"depth":1,"title":"Customizing Agents","anchor":"customizing-agents"},{"depth":2,"title":"Context","anchor":"context"},{"depth":2,"title":"Runtime Filesystem","anchor":"runtime-filesystem"},{"depth":2,"title":"Gateway Adapters","anchor":"gateway-adapters"},{"depth":2,"title":"State Stores","anchor":"state-stores"},{"depth":2,"title":"Host Hardening (Embedders)","anchor":"host-hardening-embedders"},{"depth":2,"title":"Agent Engine","anchor":"agent-engine"},{"depth":2,"title":"Tool Discovery And Capability Metadata","anchor":"tool-discovery-and-capability-metadata"},{"depth":2,"title":"Override, Wrap, Or Disable Built-In Tools","anchor":"override-wrap-or-disable-built-in-tools"},{"depth":3,"title":"Deterministic guards","anchor":"deterministic-guards"},{"depth":3,"title":"Per-tool runtime policy (embedders)","anchor":"per-tool-runtime-policy-embedders"},{"depth":2,"title":"Agent Capability Composition","anchor":"agent-capability-composition"},{"depth":3,"title":"Persistent workflow state","anchor":"persistent-workflow-state"},{"depth":3,"title":"Event observation","anchor":"event-observation"},{"depth":2,"title":"Approvals And Safe Outputs","anchor":"approvals-and-safe-outputs"},{"depth":2,"title":"Self-Improvement","anchor":"self-improvement"},{"depth":2,"title":"Dynamic Automations","anchor":"dynamic-automations"},{"depth":2,"title":"Dynamic Connections","anchor":"dynamic-connections"},{"depth":2,"title":"Channels","anchor":"channels"},{"depth":2,"title":"Sandboxes","anchor":"sandboxes"},{"depth":2,"title":"Observability","anchor":"observability"},{"depth":3,"title":"Structured logs","anchor":"structured-logs"}],"content":"# Customizing Agents\n\nAssembly Line starts small, but every major runtime choice has an explicit customization point. Add the smallest file or config block that owns the behavior you need.\n\n## Context\n\nAgents do not need `context.ts`. Without it, Assembly Line uses `defaultContext()`.\n\nThe default context includes trusted instructions, the active event, bounded recent history, memory shape, file manifest, current attachments, compact skill and capability catalogs, visible tool summaries, channel metadata, and trust boundaries.\n\nConfigure the default:\n\n```ts\nimport { defaultContext, defineAgent, useModel } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  context: defaultContext({\n    recentHistory: { maxMessages: 12 },\n    files: { includeManifest: true }\n  }),\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\nExtend it with custom policy:\n\n```ts\n// context.ts\nimport { defaultContext, defineContext } from \"@assemblyline-agents/core\";\n\nexport const customContext = defineContext({\n  kind: \"custom\",\n  name: \"customContext\",\n  extends: defaultContext({\n    recentHistory: { maxMessages: 5 }\n  }),\n  options: {\n    includeProjectMarker: true\n  }\n});\n```\n\n```ts\n// agent.ts\nimport { defineAgent, useModel } from \"@assemblyline-agents/core\";\nimport { customContext } from \"./context\";\n\nexport default defineAgent({\n  context: customContext,\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\nContext policy is trusted runtime code. It is recorded in the manifest with source attribution.\n\n## Runtime Filesystem\n\nEvery model prompt receives the stable Assembly Line filesystem contract:\n\n```txt\n/memory   durable memory, writable through memory tools by default\n/history  read-only conversation history\n/files    read-only input context and attachments\n/workspace writable workspace for artifacts and modified copies\n/skills   durable skills when self-improvement is enabled\n```\n\n`bash` starts in the workspace cwd, so prefer relative shell paths. Use absolute\nlogical paths such as `/workspace/report.txt` with the core file tools.\n\nSandboxes are acquired lazily. Normal channel receipt, model turns without sandbox-backed tools, skill activation, memory reads/writes, and final delivery do not need a sandbox.\n\n## Gateway Adapters\n\n`gateway.ts` chooses portable runtime infrastructure:\n\n```ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\nimport { dockerDeploy, dockerSandbox } from \"@assemblyline-agents/docker\";\nimport { neonPostgres } from \"@assemblyline-agents/postgres\";\nimport { r2Blob } from \"@assemblyline-agents/s3\";\n\nexport default defineGateway({\n  deploy: dockerDeploy(),\n  runtime: adapter(\"node\"),\n  state: neonPostgres(),\n  blob: r2Blob(),\n  sandbox: dockerSandbox({ image: \"node:22-slim\", network: \"none\" }),\n  scheduler: adapter(\"local\")\n});\n```\n\nAdapter choices are independent:\n\n- Deploy target: local, Railway, and generic VPS supported; Docker and Fly preview.\n- Runtime: Node.\n- Durable state: local dev/test files or Postgres for production.\n- Blob storage: local dev/test files, R2, or generic S3-compatible storage.\n- Sandbox: local dev/test, Docker, Daytona, and E2B supported; Modal preview.\n- Scheduler: local for development and static schedule dispatch.\n\nSee [Adapters](adapters.md) for helper functions and environment variables.\n\n## State Stores\n\nDurable state is defined by capability facets rather than one monolithic\ninterface. `RuntimeOptions.state` accepts either:\n\n- a backward-compatible `StateAdapter`, which contains the historical\n  monolithic facets and is detected by duck-typing (`createRun` plus\n  `appendEvent`); or\n- a `StateStores` object that brings only the facets you can persist.\n\n```ts\nimport { AssemblyLineRuntime, type StateStores } from \"@assemblyline-agents/runtime\";\n\nconst state: StateStores = {\n  runs: myRunStore // required: runs, events, tool calls, checkpoints,\n                   // deliveries (incl. the durable queue), idempotency\n  // conversations?, conversationTurns?, schedules?, files?, usage?\n  // sandboxSessions?, memory?, settings?, agentState?\n};\nconst runtime = new AssemblyLineRuntime({ manifest, state });\n```\n\n`runs` (a `RunStore`) is required, omitting it fails the constructor.\nOptional facets are `conversations` (`ConversationStore`), `conversationTurns`\n(`ConversationTurnStore`), `schedules` (`ScheduleStateStore`), `files`\n(`FileIndexStore`), `usage` (`UsageStore`), `sandboxSessions`\n(`SandboxSessionStore`), `memory` (`MemoryStateStore`), `settings`\n(`RuntimeSettingsStore`), and `agentState` (`AgentStateStore`).\n`conversationTurns` persists the per-conversation FIFO ingress mailbox.\n`agentState` persists conversation-scoped hook state and aggregate revisions.\nThe settings facet persists agent-level operator settings and their\ncontrol-plane audit events. If it is omitted, the ingress kill switch works\nonly in process and does not survive a restart.\nAny facet you omit falls back to a non-durable in-memory implementation and\nthe runtime logs a single `state.degraded` warning at boot listing the\nmissing facets. A missing or failing usage facet is reported as degraded\nobservability but never blocks runtime boot, model requests, or response\ndelivery. Capability guards (`isRunStore`, `isConversationStore`,\n`isScheduleStateStore`, `isFileIndexStore`, `isUsageStore`,\n`isSandboxSessionStore`, `isMemoryStateStore`, `isRuntimeSettingsStore`,\n`isAgentStateStore`, `isConversationTurnStore`, and `isStateAdapter`) are\nexported for hosts that feature-detect adapters.\n\n## Host Hardening (Embedders)\n\nHosts embedding the runtime or the Node server get concurrency limits, ingress\nrate limiting, and graceful shutdown as configuration:\n\n```ts\nimport { installSignalHandlers, listenNodeRuntime, type RateLimiterStore } from \"@assemblyline-agents/node\";\n\nconst handle = await listenNodeRuntime({\n  ...runtimeOptions,\n  maxConcurrentRuns: 16,          // RunCapacityError / HTTP 429 beyond this\n  maxQueuedRuns: 8,               // optional wait queue before rejection\n  rateLimit: {\n    limits: {\n      \"provider-channel\": { capacity: 60, refillPerSecond: 10 },\n      \"run-create\": { capacity: 10, refillPerSecond: 1 }\n    }\n    // store?: RateLimiterStore, plug a shared (e.g. Redis-shaped) bucket\n    // store for multi-replica deployments; defaults to in-process memory.\n    // keyFor?(request), custom bucket key; return undefined to exempt.\n  }\n});\ninstallSignalHandlers(handle);    // SIGTERM/SIGINT -> handle.shutdown()\nawait handle.closed;\n```\n\n- `maxConcurrentRuns`/`maxQueuedRuns` (or `ASSEMBLY_LINE_MAX_CONCURRENT_RUNS`/\n  `ASSEMBLY_LINE_MAX_QUEUED_RUNS`) bound brand-new execution. Accepted provider\n  events wait in the durable per-conversation mailbox when capacity is busy;\n  direct embedders calling `runtime.run()` still use the in-process admission\n  queue and should catch `RunCapacityError` (it carries `retryAfterMs`).\n  Resumes always bypass the limit.\n- `rateLimit` is off by default (`false` disables even the env config). The\n  `RateLimiterStore` interface is a single async\n  `take(key, { capacity, refillPerSecond, cost?, now? })`, so shared stores\n  are easy to implement.\n- `handle.shutdown({ timeoutMs? })` is idempotent and drains in order:\n  `/readyz` -> 503, listener + ingress, scheduler, workers,\n  `runtime.onIdle()` (bounded by `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS`),\n  `TelemetrySink.flush?()`, then `StateAdapter.close?()`. Custom state\n  adapters and telemetry sinks can implement those optional methods to\n  participate. `runtime.hasRunCapacity()` and `runtime.onIdle()` are public\n  for hosts that build their own servers.\n\n## Agent Engine\n\nPi (`@assemblyline-agents/pi`) is the default primary engine, and `agent.ts` rejects a\n`harness:` slot at validate time. The Node host has one built-in preview route:\nan `openai-codex/*` model uses `@assemblyline-agents/codex` and the official Codex\napp-server. The runtime speaks to either engine only through the internal\n`AgentHarness` contract from `@assemblyline-agents/core`, which keeps continuations opaque\nJSON and all durability (tool execution, approvals, checkpoints, events,\nusage) runtime-owned.\n\nEmbedders and tests can still replace the engine through `RuntimeOptions`:\n\n- `agentHarness`: a single `AgentHarness` that overrides the engine for every\n  run, the seam the durability test suite uses to drive scripted engines.\n  A harness implements one method, `runTurn(input, ctx)`, and returns a\n  response plus an opaque, versioned continuation when the runtime pauses the\n  run.\n- `models`: an optional pi model catalog forwarded to\n  `piAgentHarness({ models })`.\n- `modelHarnesses`: optional host-owned `provider`-prefix routes. A matching\n  entry is selected from the persisted `modelSpec`, including on approval,\n  input, connection, and crash-recovery resumes. `agentHarness` takes\n  precedence. `@assemblyline-agents/node` installs the `openai-codex` route by default.\n\nWhen embedding a built agent, start with `loadRuntimeBundle(artifactRoot)` and\nspread the returned options into `AssemblyLineRuntime`. The bundle includes the\n`artifactRoot` module-resolution hint used for packaged dependencies. Hosts\nthat assemble `manifest` and `sourceBundle` manually should pass the same\noptional `artifactRoot` explicitly.\n\nSubagents run through Pi too. Their definitions can narrow the model,\nworkspace, tools, and connections, and follow-up messages resume from persisted\ncontinuations. An inherited `openai-codex/*` model uses the same built-in Codex\nroute and CLI OAuth session as the primary agent. There is no public primary or\nsubagent harness selector.\n\n## Tool Discovery And Capability Metadata\n\n`useTool()` selects the model-visible tools in each capability snapshot.\nLong-tail capabilities can sit behind explicitly selected discovery bridges.\nThe tool `capability:` block is one of several meanings of \"capability\" in\nAssembly Line; see the [disambiguation in Plugins](plugins.md#taxonomy).\nAuthored tools can provide capability metadata when inference is not enough:\n\n```ts\ncapability: {\n  visibility: \"deferred\",\n  execution: \"direct\",\n  namespace: \"billing\",\n  tags: [\"invoice\", \"customer\"],\n  aliases: [\"receivables\"]\n}\n```\n\nVisibility values:\n\n- `always` - visible up front.\n- `deferred` - discoverable through `tool_search`.\n- `skill` - relevant when a skill is active.\n- `hidden` - runtime/internal use.\n\nExecution values:\n\n- `direct` - trusted app runtime.\n- `sandbox` - isolated filesystem or shell work.\n- `both` - can use more than one path.\n\n## Override, Wrap, Or Disable Built-In Tools\n\nEvery built-in harness tool (`read`, `write`, `edit`, `delete`, `list`, `grep`, `bash`, `ask_question`, `load_skill`, `tool_search`, `tool_describe`, `tool_call`) is a replaceable slot. An authored file at `tools/<name>.ts` with a built-in's name replaces that built-in; the model keeps seeing a tool by that name, backed by your implementation. Subagents are exposed only through `useSubagent()`, which creates a snapshot-scoped `delegate_<name>` tool.\n\nWrap the default instead of rewriting it by spreading `builtInToolDefaults` from `@assemblyline-agents/runtime`:\n\n```ts\nimport { defineTool } from \"@assemblyline-agents/core\";\nimport { builtInToolDefaults } from \"@assemblyline-agents/runtime\";\n\nconst write = builtInToolDefaults.write;\n\nexport default defineTool({\n  ...write, // keep the default description, schema, and executor\n  async execute(input, ctx) {\n    await ctx.emit(\"audit.write_requested\", { path: (input as { path: string }).path });\n    return write.execute(input, ctx);\n  }\n});\n```\n\nThe deferred bridge tools (`load_skill`, `tool_search`, `tool_describe`, `tool_call`) are runtime-handled and have no wrappable executor; overriding them replaces the tool wholesale.\n\nRemove a built-in entirely with a `disableTool()` sentinel. The filename selects the tool, and a filename that matches no built-in fails the build instead of silently doing nothing:\n\n```ts\n// tools/bash.ts\nimport { disableTool } from \"@assemblyline-agents/core\";\n\nexport default disableTool();\n```\n\nOverrides and disables are ordinary manifest entries, so they version with the agent, contribute to `agentRevision`, and diff in evals exactly like a prompt change.\n\n### Deterministic guards\n\nA wrapper can refuse to execute until a precondition holds, turning \"the prompt asks the model to X before Y\" into a rule the harness enforces. Return a structured refusal (do not throw, authored tool throws fail the run) and the model self-corrects:\n\n```ts\nimport { defineTool } from \"@assemblyline-agents/core\";\nimport { builtInToolDefaults } from \"@assemblyline-agents/runtime\";\n\nconst write = builtInToolDefaults.write;\n\nexport default defineTool({\n  ...write,\n  async execute(input, ctx) {\n    const validated = await ctx.memory?.read({ path: `guards/${ctx.runId}/validated.json` }).catch(() => undefined);\n    if (!validated) {\n      return { error: \"Run the validate tool before writing output files.\" };\n    }\n    return write.execute(input, ctx);\n  }\n});\n```\n\nDurable state for guards can live in `ctx.memory` (persists across runs) or the sandbox filesystem; key by `ctx.runId` for per-run ordering rules.\n\n### Per-tool runtime policy (embedders)\n\nHosts can gate any tool by name without touching agent sources via `RuntimeOptions.coreToolPolicy`:\n\n```ts\nconst runtime = new AssemblyLineRuntime({\n  // ...\n  coreToolPolicy: { bash: \"approval\", write: \"disabled\" }\n});\n```\n\nModes are `enabled` (default), `approval` (forces an approval gate), and `disabled` (removed from the tool set; forced invocations fail). `ASSEMBLY_LINE_BASH_TOOL_MODE` remains the env-level shorthand for `bash`.\n\n## Agent Capability Composition\n\n`agent.ts` `setup()` is the single place that composes runtime capabilities.\nIt is synchronous and receives preloaded run and conversation state through\nbuilt-in composition functions. Keep it pure: the compiler can audit synchronous calls,\nwhile tools and adapters provide the reviewed boundaries for network,\nfilesystem, and other side effects.\n\n```ts\nimport {\n  defineAgent,\n  useInstructions,\n  useModel,\n  useRun,\n  useTool\n} from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  setup() {\n    const run = useRun();\n    if (run.metadata?.tier === \"trial\") {\n      useModel(\"openai/gpt-5.4-mini\");\n      useTool(\"search_docs\");\n      useInstructions(\"Do not access billing information.\");\n      return;\n    }\n    useModel(\"openai/gpt-5.4\");\n    useTool(\"search_docs\");\n    useTool(\"lookup_account\");\n  }\n});\n```\n\nThe compiler follows local imports, records literal composition declarations,\nand packages imported helper source. The runtime validates the declaration against the\ncompiled catalog and static policy, persists a complete checkpoint, then emits\n`run.capabilities_resolved` before the snapshot affects execution. A setup\nfailure is terminal because the runtime cannot safely guess capabilities.\n\nThe compiler rejects async `setup()`, non-literal capability names and state\nkeys, a direct branch that can exit without a model, removed static runtime\nfields, and unresolved local imports. The runtime also rejects conflicting\nmodel or sandbox choices, unknown compiled capabilities, state writes during\nsetup, and more than 50 capability snapshots in one run.\n\n### Persistent workflow state\n\n`usePersistentState(key, initial)` reads small conversation-scoped JSON state.\nThe returned setter and `ctx.agentState` perform atomic durable writes. A write\nemits `agent.state_changed` without the value and causes exactly one\nre-evaluation before the next model request.\n\n```ts\nfunction useTriageStage() {\n  const [stage] = usePersistentState(\"triage.stage\", \"reproduce\");\n  if (stage === \"reproduce\") useTool(\"submit_reproduction\");\n  else useTool(\"submit_diagnosis\");\n  return stage;\n}\n```\n\nNamed capability arguments themselves must be literals; branch around literal\ncomposition calls when selection is conditional. Persistent state is for bounded control\nstate, never credentials or long-form memory.\n\n### Event observation\n\nUse `hooks/*.ts` for application callbacks that should observe durable events:\n\n```ts\nimport { defineHook } from \"@assemblyline-agents/core\";\n\nexport default defineHook({\n  events: {\n    \"run.completed\": recordCompletion,\n    \"*\": recordAuditEvent\n  }\n});\n```\n\nHandlers run after persistence, exact events before wildcard handlers. A\nfailure emits `agent.event_handler_failed` and does not fail the run. This is\ndistinct from `instrumentation.ts`, which is the sampled/redacted telemetry\nexport pipe.\n\nThe former `useEvent()` call in `agent.ts` remains accepted with a deprecation\nwarning for compatibility.\n\n## Approvals And Safe Outputs\n\nUse `needsApproval` when a tool has durable or external side effects:\n\n```ts\nimport { approvalRequired, defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Publish a status update.\",\n  inputSchema: {\n    type: \"object\",\n    properties: { body: { type: \"string\" } },\n    required: [\"body\"]\n  },\n  needsApproval: approvalRequired(\"Publishing is externally visible.\", \"external\"),\n  async execute(input: { body: string }, ctx) {\n    await ctx.emit(\"status.publish_requested\", {\n      idempotencyKey: ctx.idempotencyKey(\"status\")\n    });\n    return { published: true };\n  }\n});\n```\n\nUse `toModelOutput` to keep the model-visible result smaller than the persisted tool result.\n\n## Self-Improvement\n\nSelf-improvement is durable learning from completed work. The runtime queues a\nbackground review after complex tool use, a recovered tool error, difficult use\nof a loaded skill, every configured number of turns, or explicit run feedback.\nThe reviewer is a separate model turn with only memory and skill management\ntools—no shell, connections, authored tools, or delivery tools.\n\n```ts\nexport default defineAgent({\n  id: \"learning-agent\",\n  selfImprovement: {\n    enabled: true,\n    writeApproval: false,\n    reviewEveryTurns: 10,\n    reviewMinToolCalls: 5,\n    reviewModel: \"inherit\"\n  },\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\nWith the default `writeApproval: false`, the reviewer writes useful memory and\nskill changes directly. Set `writeApproval: true` only when the owner wants a\nhuman gate; pending changes are stored separately, so the active skill remains\navailable until approval. The agent does not approve its own pending change—the\nnormal direct-write mode is the way to allow agent autonomy.\n\nSkills use simple full-body versioning. Every save, seed update, archive, and\nrestore appends one revision; restore copies an old body into a new current\nrevision. No Git repository, diff engine, or merge protocol is involved.\nCompiled `skills/` seed the writable durable store. `externalDirs`, when set,\nare additional read-only sources and are never rewritten automatically.\n\nSet `selfImprovement.enabled: false` for static, manifest-only skills and no\nbackground review. The old `writable` field and\n`ASSEMBLY_LINE_SKILLS_WRITABLE` env variable remain compatibility aliases.\n\nHosts can submit explicit feedback with `POST /runs/:runId/feedback` using\n`{\"rating\":\"positive|negative\",\"comment\":\"...\"}`. Operator routes under\n`/self-improvement` list review jobs, pending changes, and skill revisions, and\ncan approve/reject pending changes or restore a revision.\n\n## Dynamic Automations\n\nDynamic automations are runtime-created routines. They are separate from static files in `automations/`.\n\n```ts\nexport default defineAgent({\n  dynamicAutomations: {\n    dynamic: true,\n    approval: false\n  },\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\nA tool can use `ctx.automationManager` to create, list, update, or delete time-based automations. The runtime dispatcher leases due rows and starts one durable run per row. Automated work should be idempotent.\n\n## Dynamic Connections\n\nDynamic connections let an agent persist runtime-provided MCP, OpenAPI, or HTTP services. They are off by default and should be allowlisted.\n\n```ts\nexport default defineAgent({\n  dynamicConnections: {\n    dynamic: true,\n    approval: true,\n    allowedHosts: [\"mcp.example.com\", \"api.example.com\"]\n  },\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\nCredentials must come from host APIs, authorization flows, or encrypted grant stores. Never put secrets into skills, messages, tool inputs, agent folders, or sandbox files.\n\n## Channels\n\nUse a provider helper when one exists:\n\n```ts\nimport { defineDiscordChannel } from \"@assemblyline-agents/discord\";\n\nexport default defineDiscordChannel();\n```\n\nProvider helpers verify incoming requests, normalize provider events into `ChannelTurn`, preserve provider delivery metadata, return fast acknowledgements when appropriate, and deliver replies through provider APIs.\n\nUse `defineChannel()` when implementing a new provider boundary:\n\n```ts\nimport { defineChannel } from \"@assemblyline-agents/core\";\n\nexport default defineChannel({\n  transport: \"http\",\n  route: \"/message\",\n  methods: [\"POST\"]\n});\n```\n\nRaw custom HTTP routes are local/dev-friendly, but production requests need a\nhost auth policy or bearer auth before the runtime will use the default message\nfallback. Provider-facing channels should export `normalizeHttp()` and perform\nprovider signature, token, or tenant validation there.\n\nChannels also declare what production ingress requires and how their\nattachments download:\n\n- Set `ingress: { requiredSecretEnv: [[\"MY_WEBHOOK_SECRET\"]] }` on the channel\n  config (any-of groups of env vars). The compiler stamps it into the manifest;\n  production boot fails until at least one group is fully set, and `devMode`\n  logs a warning instead.\n- Export `resolveAttachment(attachment, ctx)` to turn a turn attachment into a\n  `{ url, headers, filename? }` download request with your provider's\n  credentials and host allowlist. The runtime downloads, size-caps, and stores\n  the bytes, and only ever consults the module of the channel that produced the\n  turn, so credentials cannot leak across channels. Return `undefined` for a\n  generic unauthenticated URL download.\n\nProvider-specific parsing belongs in channel modules. Durable state, blob\nstorage, model selection, and sandbox lifecycle belong to the runtime and\nadapters. See [Authoring Plugins: Channel Modules](authoring-adapters.md#channel-modules)\nfor the full `ChannelModule` contract, including ingress, idempotency,\ndelivery, and attachment resolution.\n\n## Sandboxes\n\nLocal sandbox:\n\n```ts\nimport { defineSandbox } from \"@assemblyline-agents/core\";\n\nexport default defineSandbox({\n  adapter: \"local\",\n  image: \"node:22\",\n  workingDirectory: \"/workspace\"\n});\n```\n\n`workingDirectory` may be omitted or set to `/workspace`; other aliases are\ninvalid because hosted shell commands see a real, physical `/workspace`.\n`/runtime` is retired. Local is a trusted temporary-directory emulation; use\nDocker when exact local shell namespace parity matters.\n\nDocker sandbox:\n\n```ts\nimport { dockerSandbox } from \"@assemblyline-agents/docker\";\n\nexport default dockerSandbox({\n  image: \"node:22-slim\",\n  network: \"none\"\n});\n```\n\nSupported hosted sandbox helpers include Daytona, E2B, and Modal. Snapshot\npolicy is opt-in:\n\n```ts\nexport default defineSandbox({\n  adapter: \"daytona\",\n  image: \"node:22\",\n  snapshot: {\n    mode: \"manual\",\n    retainLast: 3,\n    reason: \"operator-requested checkpoint\"\n  }\n});\n```\n\n## Observability\n\nAssembly Line records runs, events, checkpoints, tool calls, delivery\nobligations, usage records, and timelines by default. The Node runtime exposes\n`/runs` inspection endpoints for dashboards, tests, and operator tooling\nwithout extra configuration.\n\nTo export OpenTelemetry spans, configure telemetry in\n`agent/instrumentation.ts`. The runtime discovers this file and runs it once at\nstartup; there is no separate toggle. Return a sink from the `setup` callback.\n`@assemblyline-agents/otlp` exports OpenTelemetry GenAI spans over OTLP/HTTP to\nLangfuse, Phoenix, Grafana, Honeycomb, or another OTLP backend. The endpoint\nselects the backend, and the package does not depend on the OpenTelemetry SDK.\n\n```ts\nimport { defineInstrumentation } from \"@assemblyline-agents/core\";\nimport { createOtlpSinkFromEnv } from \"@assemblyline-agents/otlp\";\n\nexport default defineInstrumentation({\n  serviceName: \"learning-agent\",\n  captureContent: \"usage\", // \"usage\" (default) | \"content\" | \"full\" | \"off\"\n  setup: ({ env }) => createOtlpSinkFromEnv(env)\n});\n```\n\n**Capture detail.** `captureContent` decides how verbose logging is, per environment. `usage` (default) records token/cost/model/tool metadata but no message bodies; `content`/`full` add prompt/completion and tool input/output, truncated and key-redacted by the runtime before any sink sees them. Prompts can contain secrets/PII, so keep `usage` in production and reserve `full` for trusted debugging. See the [capture-detail table](config-reference.md#defineinstrumentation-instrumentationts).\n\n**Langfuse recipe.** Point the endpoint at Langfuse's OTLP ingestion and pass a Basic auth header built from your Langfuse public/secret keys:\n\n```bash\nOTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel\nOTEL_EXPORTER_OTLP_HEADERS=\"Authorization=Basic <base64(public_key:secret_key)>\"\n```\n\n`createOtlpSink({ endpoint, headers, ... })` is available too when you prefer explicit options over environment variables.\n\n### Structured logs\n\nThe runtime emits structured JSON log lines (one object per line with `level`, `time`, `msg`, and event-specific fields) for channel ingress lifecycle, HTTP requests, security warnings, and recoverable internal failures. Control verbosity with `ASSEMBLY_LINE_LOG_LEVEL` (`debug`, `info`, `warn`, or `error`; default `info`). Hosts embedding the runtime directly can replace the logger by passing `logger` (a `RuntimeLogger`) in `RuntimeOptions`, for example to route logs into an existing logging pipeline, or silence it with `noopLogger()`.\n"},{"id":"framework","sourcePath":"framework.md","title":"Framework Guide","description":"Understand Assembly Line agent folders, compiler contracts, runtime guarantees, and package boundaries.","url":"https://assemblyline.artificialillumination.co/docs/framework","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/framework.md","headings":[{"depth":1,"title":"Assembly Line Framework Guide","anchor":"assembly-line-framework-guide"},{"depth":2,"title":"Agent Folder Convention","anchor":"agent-folder-convention"},{"depth":2,"title":"`agent.ts`","anchor":"agentts"},{"depth":2,"title":"Agent Engine","anchor":"agent-engine"},{"depth":2,"title":"`context.ts`","anchor":"contextts"},{"depth":3,"title":"Prompt layout and prompt caching","anchor":"prompt-layout-and-prompt-caching"},{"depth":3,"title":"Conversation transcript resume and compaction","anchor":"conversation-transcript-resume-and-compaction"},{"depth":2,"title":"`gateway.ts`","anchor":"gatewayts"},{"depth":2,"title":"Tools","anchor":"tools"},{"depth":2,"title":"Skills And Self-Improvement","anchor":"skills-and-self-improvement"},{"depth":2,"title":"Channels","anchor":"channels"},{"depth":2,"title":"Connections","anchor":"connections"},{"depth":2,"title":"Automations","anchor":"automations"},{"depth":2,"title":"Sandbox","anchor":"sandbox"},{"depth":2,"title":"Compiler Output","anchor":"compiler-output"},{"depth":2,"title":"Durability Guarantees","anchor":"durability-guarantees"},{"depth":3,"title":"Persistence model","anchor":"persistence-model"},{"depth":3,"title":"HITL resume","anchor":"hitl-resume"},{"depth":3,"title":"Durable steps","anchor":"durable-steps"},{"depth":3,"title":"Delivery queue","anchor":"delivery-queue"},{"depth":3,"title":"Orphan recovery","anchor":"orphan-recovery"},{"depth":3,"title":"Sandbox sync","anchor":"sandbox-sync"},{"depth":3,"title":"Security boundaries","anchor":"security-boundaries"},{"depth":2,"title":"Observability","anchor":"observability"},{"depth":2,"title":"State And Blob Adapters","anchor":"state-and-blob-adapters"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# Assembly Line Framework Guide\n\nAssembly Line is a filesystem-first framework for durable AI agents. An agent folder declares what the agent is; the compiler turns that folder into a manifest and runtime artifact; the runtime executes runs durably through pluggable adapters.\n\nThis page explains the concepts and contracts. For command tables, the artifact tree, the HTTP API, and deploy targets, see [Runtime And Deployment](runtime-and-deployment.md). For every `define*` shape and `ASSEMBLY_LINE_*` variable, see the [Configuration Reference](config-reference.md).\n\nContents:\n\n- [Agent Folder Convention](#agent-folder-convention)\n- [`agent.ts`](#agentts)\n- [Agent Engine](#agent-engine)\n- [`context.ts`](#contextts)\n- [`gateway.ts`](#gatewayts)\n- [Tools](#tools)\n- [Skills And Self-Improvement](#skills-and-self-improvement)\n- [Channels](#channels)\n- [Connections](#connections)\n- [Automations](#automations)\n- [Sandbox](#sandbox)\n- [Compiler Output](#compiler-output)\n- [Durability Guarantees](#durability-guarantees)\n- [Observability](#observability)\n- [State And Blob Adapters](#state-and-blob-adapters)\n\n## Agent Folder Convention\n\nOnly `instructions.md` and `agent.ts` are required.\n\n```txt\nagent/\n  instructions.md\n  agent.ts\n  context.ts\n  gateway.ts\n  skills/\n  tools/\n  channels/\n  automations/\n  hooks/\n  connections/\n  evals/\n  sandbox/\n  subagents/\n  instrumentation.ts\n```\n\n`lib/` and `playbooks/` are not reserved Assembly Line conventions. App helpers can live wherever the app normally keeps source code.\n\n## `agent.ts`\n\n`agent.ts` exports static identity/policy and a synchronous `setup()` that\nselects runtime capabilities through composition functions.\n\n```ts\nimport { defineAgent, useModel, useReasoning, useTool } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  description: \"A portable Assembly Line agent.\",\n  maxReasoning: \"medium\",\n  setup() {\n    useModel(\"openai/gpt-5.4-mini\");\n    useReasoning(\"medium\");\n    useTool(\"echo\");\n  }\n});\n```\n\nThe compiler extracts static policy, follows the local import graph, records\nliteral composition declarations and possible models, and packages the source\nneeded to evaluate imported composition helpers. Tool implementations, secrets, databases,\ndeployment providers, and blob stores do not belong in `agent.ts`.\n\nAssembly Line reads manifest-facing config through the TypeScript AST. This applies\nto `agent.ts`, `gateway.ts`, tools, channels, automations, hooks,\nconnections, sandbox definitions, and subagent `agent.ts` files.\nManifest-affecting values must be statically readable: a default-exported\n`define*({ ... })` helper call, an exported identifier that resolves to that\ncall or object, top-level `const` indirection for literal strings, arrays, and\nobjects, shorthand properties, imported helper aliases, `as const`, and\n`satisfies` are supported. The compiler never executes config code during\nmanifest generation, so dynamic expressions cannot affect the manifest.\n\n## Agent Engine\n\nPi is the primary engine. Assembly Line adapts the\n[pi](https://github.com/badlogic/pi-mono) model loop in\n`@assemblyline-agents/pi` and constructs it directly in the runtime. Agents never\nselect a harness: `agent.ts` has no `harness:` slot (declaring one fails\nvalidation with `harness-not-configurable`). The Node host does have a built-in\npreview provider route: selecting `openai-codex/*` uses `@assemblyline-agents/codex` and\nOpenAI's official Codex app-server, authenticated by the Codex CLI. Subagents\nalso run through Pi. Their composition functions select the model and active capabilities, and\ntheir static definitions can set workspace and connection limits. They cannot\nselect another engine.\n\nThe runtime talks to Pi only through the internal `AgentHarness`\ncontract in `@assemblyline-agents/core`. This seam is not a public extension\npoint. The runtime imports no Pi types. Continuation state is opaque JSON owned\nby the engine adapter, so persisted runs never depend on Pi internals. All\ndurability (tool\nrecords, approvals, checkpoints, events, usage, spans) stays in the runtime.\nEmbedders and tests can replace the engine for every run through\n`RuntimeOptions.agentHarness`, or register host-owned provider-prefix routes\nthrough `RuntimeOptions.modelHarnesses`. The durability test suite drives\nscripted engines through exactly this seam. A sibling seam,\n`RuntimeOptions.toolStubs`, replaces named tool executions with canned\nimplementations (approval gates and tool-call recording still apply); the\neval runner's case-level `mocks` build on it. The resolved model spec is stored\nwith the continuation so paused and recovered runs resume through the same\nroute. Subagents reuse the Pi loop, including follow-ups via persisted\ncontinuations; there is no public subagent-harness extension point.\n\nTool batches run in **parallel** by default: when the model emits several\ntool calls in one response, parallel-safe tools execute concurrently.\nPause-capable tools include approval-gated tools, `ask_question`, connection\ntools, and the deferred `tool_call` bridge. They carry `execution: \"sequential\"` on\ntheir descriptors, which serializes any batch that contains one. A pause\nstops the batch wherever it happens: later sequential calls are skipped with\nan explicit `{ skipped: true }` result instead of executing after the run\nparked, and a dynamic pause inside a parallel batch (a custom tool calling\n`ctx.askQuestion`) still ends the active loop execution without another model request.\n\nThe engine also reports **token deltas** (`response_delta` events). They are\nephemeral: the runtime fans them out to live run-stream subscribers\n(`runtime.subscribeRunStream(runId, listener)`, or the Node host's\n`GET /runs/:id/stream` SSE endpoint) and never writes them to the durable\nevent log.\n\n## `context.ts`\n\nAgents do not need `context.ts`. When it is absent, Assembly Line uses `defaultContext()`.\n\n`defaultContext()` builds a Flue-shaped context bundle with trusted instructions, active event, bounded recent history, memory filesystem shape, file manifest, current attachments, a compact skill and capability catalog, always-on tool summaries, channel metadata, and trust boundaries. Deferred tool schemas and skill bodies are not injected up front. Files, screenshots, webpages, tool output, search results, memory, and attachments are context, not instructions.\n\nImage and supported video attachments are model-native by default. After\nattachment intake stores the private bytes, the runtime hydrates bounded PNG,\nJPEG, GIF, WebP, MP4, MPEG, MOV, and WebM inputs at the harness boundary. Pi\nsends native image blocks when the selected model advertises image input.\nFor OpenRouter models, Pi verifies the live model metadata and sends complete\nvideos through OpenRouter's native `video_url` content type only when\n`input_modalities` includes `video`.\n\nAssembly Line never silently substitutes sampled frames for native video. If the\nselected model or harness does not support video, the turn returns an explicit\nlimitation without making a model request or extracting frames. Frame-based\nreview remains an explicit user-approved FFmpeg operation. The Codex\napp-server input protocol accepts text and images, but not video, so\n`openai-codex/*` follows that explicit limitation path. Image-bearing MCP tool\nresults stay typed rather than being flattened into JSON text, and base64\npayloads are omitted from observability content events.\n\nEvery runtime model prompt also receives a small stable Assembly Line filesystem contract immediately after `instructions.md`. It names resource paths and their mutability: `/memory` is durable memory accessed directly through memory tools by default, `/workspace` is writable workspace for generated artifacts and modified copies when a sandbox exists, `/history` is read-only conversation history, and `/files` is read-only input context whose `manifest.json` should be inspected before reading file contents. Hosted sandboxes establish `/workspace` as a physical shell cwd, so core file tools and shell commands may use the same absolute paths. The trusted Local adapter maps those paths onto a temporary host directory; Docker is the local parity path for absolute shell semantics. These paths are projected only when sandbox-backed tools need them; the contract does not inject memory or file contents into every turn.\n\n### Prompt layout and prompt caching\n\nThe system prompt is cache-stable by construction. It contains only\nconversation-invariant content: instructions, the filesystem contract, and\ncompact JSON for the skill index, core tool summaries, capabilities, channels,\nand trust boundaries. This content stays byte-identical across the turns of a\nconversation.\n\nPer-turn data travels with the turn's user message as an `OpenEve turn context:`\nblock, followed by the user's text. This data includes the active event's\nchannel context, prompt context, automation target, and attachment metadata.\nTimestamps never enter the model-visible prompt. This keeps the invariant\nprefix eligible for provider prompt caching. The `cacheReadRatio` attribute on\neach `ai.streamText` span reports the result.\n\n### Conversation transcript resume and compaction\n\nAfter a successful conversation run, the runtime checkpoints the harness's\nfinal continuation as `conversation.transcript` and stores a pointer on the\nconversation record. The next turn resumes the full transcript, including\nassistant turns and tool calls, instead of rebuilding context from flattened\nrecent history. The pointer is an optimization, not the source of truth. A\nload, harness-version, or trim failure falls back to flattened history and\nemits `context.transcript_fallback`. Durable conversation messages and\n`/history` remain unchanged.\n\nBefore resume the harness trims the stored transcript in two layers. First, tool-result bodies outside the recent ~20k-token tail are capped with a restorable marker (re-run the tool or read the sandbox file to recover the full output; no model call). Second, when the transcript still exceeds the model context window minus a reserve, older full turns are summarized into a structured context checkpoint (pi-agent-core's summarizer), keeping the recent tail verbatim and never separating a tool result from its call; re-compactions update the previous summary instead of stacking summaries, and the run is notified to persist durable facts under `/memory`. Compactions are evented as `context.transcript_compacted` with the pre-compaction token estimate. Stored media (base64 images/video) never replays on conversation resumes. Trimming runs at resume time between runs; mid-run growth is bounded by `maxIterations`.\n\nCustom context can extend the default:\n\n```ts\nimport { defaultContext, defineContext } from \"@assemblyline-agents/core\";\n\nexport const customContext = defineContext({\n  kind: \"custom\",\n  name: \"customContext\",\n  extends: defaultContext({ recentHistory: { maxMessages: 5 } })\n});\n```\n\nContext policy is trusted app-runtime code and is recorded in the manifest with source attribution.\n\n## `gateway.ts`\n\n`gateway.ts` is the portable stack declaration. It is tiny and declarative:\n\n```ts\nimport { adapter, defineGateway } from \"@assemblyline-agents/core\";\n\nexport default defineGateway({\n  deploy: adapter(\"railway\"),\n  runtime: adapter(\"node\"),\n  state: adapter(\"postgres\"),\n  blob: adapter(\"r2\"),\n  sandbox: adapter(\"daytona\"),\n  scheduler: adapter(\"gateway\")\n});\n```\n\nRuntime host, state, blobs, sandbox, scheduler, connections, and observability are independent choices. Deploy adapters host the runtime process; they do not force a state/blob/sandbox provider.\n\nLike `agent.ts`, `gateway.ts` is parsed as TypeScript syntax rather than executed. Use statically readable `defineGateway({ ... })` declarations, `adapter(\"kind\")`, or known provider helper calls such as `railwayDeploy()`, `vpsDeploy()`, `neonPostgres()`, `railwayPostgres()`, `supabasePostgres()`, `r2Blob()`, and `dockerSandbox()`. Dynamic expressions are ignored unless they resolve to top-level literals the compiler can validate.\n\nScheduler choices are explicit. `adapter(\"local\")` starts an in-process polling loop for development or single-process hosts. `adapter(\"gateway\")` does not start a loop; a cloud scheduler, platform cron, or gateway worker calls the runtime tick endpoint or `runDueAutomations()`. `adapter(\"postgres\")` starts the same polling loop but expects Postgres state so multiple workers coordinate through shared idempotency and dynamic-automation leases.\n\nProduction state is Postgres. Neon is the default hosted path. Railway,\nSupabase, and local or custom Postgres are presets; each uses\n`adapter(\"postgres\")` because it exposes standard Postgres. On Railway,\n`railwayPostgres()` provisions or reuses a managed database and wires its\nprivate `DATABASE_URL` before publishing the runtime. Blob storage is\nS3-compatible, with R2 as a first-class preset and wrapper.\n\nRailway and generic VPS deploy providers are supported. Docker and Fly deploy\nproviders are preview. All four implement the same artifact-level persistent\nstorage and remote execution contract. Deploy choice does not imply a state,\nblob, or sandbox vendor. See [Deploy Targets](runtime-and-deployment.md#deploy-targets).\n\nSee [Adapters](adapters.md) for the current adapter list, helper functions, and provider environment variables.\n\n## Tools\n\nEach file in `tools/` becomes one model-facing tool. The filename is the tool name. Tool descriptors include name, description, input schema, optional output schema, approval policy, and model-output projection.\n\nTool code runs in the trusted app runtime by default. It only uses the sandbox through `ctx.getSandbox()`.\n\n`agent.ts` selects the exact visible tool set for each capability snapshot with `useTool()`. Core tools such as `read`, `write`, `edit`, `delete`, `list`, `grep`, `bash`, `ask_question`, `load_skill`, `tool_search`, `tool_describe`, and `tool_call` are available to select, but none becomes visible merely because it was compiled. `useSubagent(\"researcher\")` adds only `delegate_researcher`; there is no generic subagent-spawn escape hatch. Host tool policy remains the final ceiling and can remove a selected tool before the snapshot applies.\n\nEvery core harness tool is a replaceable slot: an authored `tools/<name>.ts` with a built-in's name overrides it (spread `builtInToolDefaults` from `@assemblyline-agents/runtime` to wrap instead of rewrite), and a `disableTool()` default export removes it, with unknown names failing the build. See [Customizing Agents](customization.md#override-wrap-or-disable-built-in-tools).\n\nTools can optionally declare capability metadata, but most apps should rely on inference:\n\n```ts\ncapability: {\n  visibility: \"deferred\",\n  execution: \"direct\",\n  namespace: \"billing\",\n  tags: [\"invoice\", \"customer\"],\n  aliases: [\"receivables\"]\n}\n```\n\n`visibility` can be `auto`, `always`, `deferred`, `skill`, or `hidden`; `execution` can be `auto`, `direct`, `sandbox`, or `both`. `auto` defers to inference. These fields feed deferred discovery metadata. Core harness tools are not configurable by agent authors.\n\nAll file, memory, skill, workspace, and artifact work goes through the sandbox filesystem. `/history` and `/files` are read-only; `/memory`, `/skills`, and `/workspace` are writable according to policy. The core file tools lazily acquire and hydrate the sandbox for requested paths.\n\n```ts\nimport { approvalRequired, defineTool } from \"@assemblyline-agents/core\";\n\nexport default defineTool({\n  description: \"Record a note after approval.\",\n  inputSchema: { type: \"object\", properties: { note: { type: \"string\" } }, required: [\"note\"] },\n  needsApproval: approvalRequired(\"Recording a note is a durable side effect.\"),\n  async execute(input, ctx) {\n    await ctx.emit(\"note.recorded\", { idempotencyKey: ctx.idempotencyKey(\"note\") });\n    return { recorded: true, note: input.note };\n  }\n});\n```\n\n`toModelOutput` can expose a bounded, safe projection while the rich result remains available in the durable tool log. See [tools/*.ts](config-reference.md#toolsts) for the full field reference.\n\n## Skills And Self-Improvement\n\nSkills live under `skills/<name>/SKILL.md`. Assembly Line parses skill frontmatter and builds compact skill capabilities in the manifest. `useSkill()` selects the skills present in the current snapshot. If `load_skill` is also selected with `useTool()`, the model may load the body of those selected skills only; loading never widens the skill or tool set. If a sandbox already exists, a selected skill may also be projected into `/skills/<name>/SKILL.md` for filesystem inspection.\n\n**Self-improvement means the agent reviews completed work and writes durable\nmemory or skill improvements.** Configure it with `selfImprovement` in\n`agent.ts`; it is on by default. Runtime\nautomations and connections use separate config blocks (`dynamicAutomations`\nand `dynamicConnections`) and separate tool APIs. They are not part of\nself-improvement. Turn off all three blocks for static, manifest-only behavior.\n\n```ts\nexport default defineAgent({\n  // Stable logical identity for durable learned state across revisions/deploys:\n  id: \"travel-concierge\",\n  // Background learning with direct writes:\n  selfImprovement: {\n    enabled: true,\n    writeApproval: false,\n    reviewEveryTurns: 10,\n    reviewMinToolCalls: 5,\n    reviewModel: \"inherit\"\n  },\n  // Separate, clearly-named concerns:\n  dynamicAutomations: { dynamic: true, approval: false },\n  dynamicConnections: { dynamic: false, approval: true, allowedHosts: [] },\n  setup() { useModel(\"openai/gpt-5.4-mini\"); }\n});\n```\n\n**Reviews are isolated and durable.** A post-run trigger writes a leased review\njob. Its evidence contains recent conversation messages, ordered tool calls and\nmodel-visible results/errors, terminal events, loaded skills, and explicit\nfeedback. The review run has only memory and skill tools and cannot recursively\nschedule another review. Failures retry from the durable queue.\n\n**Skills are a durable, versioned folder.** On boot, compiled `skills/` seed a\ndurable `SkillStore`, scoped by stable `agent.id` when configured and tracked by\na content hash so redeploys upgrade *pristine* seeded skills and preserve learned\nchanges. Each mutation appends a full-body history row. Delete operations archive\nthe current skill; restore copies a prior body into a new revision. Approval-gated\nwrites are separate pending rows and never replace the active skill. External\nskill directories are read-only. When `selfImprovement.enabled` is off,\n`/skills` writeback and background review are blocked (`writable` is a deprecated\nalias).\n\nTools reach these concerns through three distinct context APIs, `ctx.selfImprovement` (skills), `ctx.automationManager`, and `ctx.connectionManager`, and the runtime exposes `saveSkill`, `listSkills`, `dispatchAutomationEvent`, `runDueSchedules`, `saveConnectionDefinition`, and related methods for host-driven use. Durable stores are provided by the state adapter (Postgres) or fall back to local JSON files under the artifact root. See `examples/self-improving-agent` and [Customizing Agents](customization.md#self-improvement).\n\n## Channels\n\nChannels normalize platform entrypoints and delivery behavior. HTTP-capable channels declare a route and methods; the compiler emits a route table.\n\n```ts\nimport { defineChannel } from \"@assemblyline-agents/core\";\n\nexport default defineChannel({\n  transport: \"http\",\n  route: \"/message\",\n  methods: [\"POST\"]\n});\n```\n\nThe default raw HTTP message fallback is dev-only unless the Node host has\nauthenticated the request. Production provider routes should use a helper or\ncustom `normalizeHttp()` that verifies the provider request before accepting a\nturn.\n\nChannel files own platform event semantics, not durable state schema or sandbox lifecycle.\n\nAuthenticated channel turns carry a canonical current principal and a stable\nconversation initiator. Optional channel `resolvePrincipal()` hooks map raw\nprovider users to internal tenant, team, and role attributes before agent\n`setup()` runs. Agent composition then selects shared skills, tools,\nconnections, and subagents from that trusted identity. The runtime propagates\nthe identity into child and scheduled runs and exposes it to tools for final\nauthorization checks; it never propagates connection credentials.\n\nAssembly Line ships one-line helpers for Slack, Discord, Telegram, Microsoft Teams,\nand Photon-style agent communication channels. Provider helpers preserve the\nsame channel contract: verify the incoming event, normalize to `ChannelTurn`,\nuse provider delivery IDs for idempotency where available, and send replies\nthrough provider APIs. GitHub is available separately as an authenticated\nconnection package for repository tooling; it is not an inbound channel.\n\nFor retried webhook providers, channel modules can return `kind: \"accepted\"` to acknowledge the\nHTTP request before the model turn completes. Use the provider's stable delivery id as the\nidempotency key. For Slack Events API channels, verify the request signature, normalize the event,\nreturn a 2xx response immediately, and set `idempotencyKey` to Slack's `event_id` so retries do not\nstart duplicate turns.\n\nAccepted turns enter a durable FIFO mailbox keyed by stable agent identity and\nnormalized conversation id. Only one turn in a conversation can be running or\nparked at a time; later messages wait. Different conversations lease\nindependently and consume the ordinary global run-concurrency budget in\nparallel. A parked approval, input, connection, or suspended run keeps its\nconversation closed until it reaches a terminal state. This rule is enforced\nby the runtime after normalization, so channel modules define conversation\nboundaries but do not implement their own queues.\n\nChannel modules can also augment context after ACK and before default context\nbundle construction. Slack uses this hook to bridge a bounded set of recent\nAssembly Line Slack conversations for the same user, so a user can follow up from a\nthread in a DM without merging every Slack surface into one transcript or\nfetching workspace-wide Slack context.\n\nChannel modules can also export `startIngress(ctx, emit)` for long-lived\nprovider listeners. The Node host starts these listeners beside the scheduler,\nrestarts provider-owned listeners through adapter code, and stops them on server\nshutdown. `emit.accepted({ turn, idempotencyKey, idempotencyScope })` feeds\nGateway-style events into the same durable, idempotent run path used by accepted\nHTTP webhooks. Discord uses this for Gateway DMs, mentions, and thread messages.\n\n## Connections\n\nConnections declare required external capabilities, scopes, subject mapping, and whether they are required. Raw secrets and refresh tokens stay outside the agent folder and model context. Live MCP, OpenAPI, HTTP, and sandbox CLI connection tools are searched through `tool_search`; concrete schemas are not visible until `tool_describe` selects them. MCP supports request-policy-governed Streamable HTTP and static, directly spawned stdio processes. Sandbox CLI connections preserve the same connection policy and scoping while invoking reviewed arguments in the active run sandbox, where short-lived materialized credentials may be projected when explicitly configured. Dynamic connections are URL-only and cannot launch host or sandbox processes. Tools and channels consume connection handles from runtime context.\n\n**Dynamic connections are separate and gated.** When\n`dynamicConnections.dynamic` is on (it is off by default), a tool can use\n`ctx.connectionManager` to persist an MCP, HTTP, or OpenAPI connection in a\n`ConnectionDefinitionStore`. Stored definitions join the connection registry\nand become discoverable through `tool_search`. Credentials flow through host\nAPIs or authorization into the encrypted grant store. They never enter\nmodel-visible tool input, the agent folder, the sandbox, or model context.\nSaving requires approval and a host in `dynamicConnections.allowedHosts`.\nRemote tool descriptions remain untrusted data. Agents cannot author trusted\ntool code. Run a user-supplied CLI in the sandbox and wrap it in a skill;\nnew typed tools remain reviewed source changes.\n\n## Automations\n\nAutomations declare durable work started by either a schedule trigger or a\nnormalized provider event. Schedule triggers use the existing cron dispatcher.\nEvent triggers enter through `runtime.dispatchAutomationEvent()`, a verified\nchannel normalizer, a long-lived channel listener, or authenticated\n`POST /assembly-line/automations/events`. Both paths reserve stable idempotency keys,\nhonor run capacity, and execute the same target and lifecycle contract.\nTime-based triggers are dispatched by `runtime.runDueAutomations()` and\n`/assembly-line/automations/tick`.\n\nTrusted prepare/finalize code lives directly on its owning automation. Dynamic time-based automations use\n`dynamicAutomations` and `ctx.automationManager`; dynamic event subscriptions\nremain reviewed source because they own provider authentication and\nsubscription policy. See [automations/](agent-stack/automations.md).\n\n## Sandbox\n\nSandbox files declare the agent computer selection. The core contract supports file reads/writes, shell execution, and optional provider snapshots. The local adapter is for trusted dev/test work; Docker is the supported local isolation baseline; Daytona, E2B, and Modal are supported hosted sandbox choices. Sandboxes are acquired lazily when a sandbox-backed tool or capability asks for one.\n\nHosted sandbox paths share one physical, versioned namespace rooted at\n`/workspace`. Providers validate shell cwd and file-API agreement after create,\nconnect, and wake; runtime manifests fence older contract versions from reuse.\nProviders reject traversal and return canonical absolute paths such as\n`/workspace/report.txt` from listings. `/runtime` is retired and rejected.\nLocal sandbox execution emulates the logical namespace in a temporary host\ndirectory and is trusted\ndeveloper or self-managed execution only; use Docker, Daytona, or E2B when\nuntrusted code needs an isolation boundary.\n\nSnapshots are a scarce infrastructure checkpoint, not the normal turn persistence mechanism. Production runs should use Assembly Line state/blob sync for memory, messages, tool traces, and `/workspace` artifacts, and keep fresh run sandboxes ephemeral. The default snapshot policy is `never`, so a hosted sandbox run does not create a remote snapshot unless the agent explicitly opts in.\n\nSandbox files may declare a snapshot policy:\n\n```ts\nexport default defineSandbox({\n  adapter: \"daytona\",\n  image: \"node:22\",\n  snapshot: {\n    mode: \"manual\",\n    retainLast: 3,\n    reason: \"operator-requested checkpoint\"\n  }\n});\n```\n\nSupported modes are `never`, `manual`, `on_failure`, and `always`. `manual` only captures when run metadata includes an explicit sandbox snapshot request. `always` is intended for short-lived debugging or controlled checkpoint jobs, not chat turns. `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_MODE`, `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_RETAIN_LAST`, and `ASSEMBLY_LINE_SANDBOX_SNAPSHOT_REASON` can override policy at runtime.\n\n## Compiler Output\n\n`assembly-line build` emits the `.assembly-line/` artifact; the canonical file tree and per-file descriptions live in [Runtime And Deployment → Build Artifact](runtime-and-deployment.md#build-artifact). Two contracts matter conceptually:\n\n- `agentRevision` is a deterministic hash of source paths, source hashes, and relevant config. Rebuilding unchanged source produces the same revision; changing source changes it.\n- The manifest contains instructions, the agent definition, context policy,\n  gateway config, tools, capabilities, skills, channels, hooks, automations,\n  inline automation lifecycles, connections, sandbox declarations, subagents, instrumentation\n  source, the route table, preflight requirements, validation results, package\n  versions, and source hashes.\n\nFor the CLI commands that produce and run the artifact, see the [CLI reference](runtime-and-deployment.md#cli-commands).\n\n## Durability Guarantees\n\n### Persistence model\n\nThe runtime persists a run before execution starts. It records context creation,\nsandbox acquisition, model and tool events, approval pauses, memory sync,\ndelivery, and the final status. Replay rebuilds the run from durable events,\nrecovery checkpoints, tool calls, and delivery records.\n\nCheckpoints support recovery; they are not the permanent audit log:\n\n- Active runs keep a bounded tail of `harness.continuation` checkpoints while\n  preserving pause and durable-step data needed by the live run.\n- Events, messages, tool calls, delivery obligations, idempotency keys, files,\n  memory, schedules, and approvals remain separate durable records.\n- Large checkpoints are gzip-compressed into blob storage. SQL keeps a small\n  content-addressed record with the pointer, hash, and size. Resume hydrates\n  the payload without exposing this split to the harness.\n- `assembly-line checkpoints compact <agentRoot>` reports terminal-run cleanup\n  by default. Add `--apply` to delete rows older than the configured TTLs.\n\nSee the [model loop reference](config-reference.md#model-loop-memory-and-logging)\nfor checkpoint cadence, retention, and TTL settings.\n\n### HITL resume\n\nApprovals, `ask_question`, and suspension states are durable. Resume re-enters\nthe model loop:\n\n- `resumeApproval(runId)` executes the approved tool and returns its result to\n  the harness as the pending tool result.\n- `resumeInput(runId, answer)` returns the human answer to the agent that asked\n  the question.\n- `resumeSuspended(runId)` restores the latest compatible continuation.\n\nEach path claims a per-generation idempotency key first. Repeated approvals,\nanswers, OAuth callbacks, or suspension resumes cannot execute twice across\nreplicas. If no usable continuation exists, the runtime completes the run\ndirectly and emits `harness.resume_degraded`. Final delivery remains an\nidempotent delivery obligation.\n\n### Durable steps\n\nAuthored tools and inline automation lifecycle functions can wrap expensive or\nside-effect-adjacent substeps in `ctx.step(key, fn)`. A completed step stores\nits JSON result as a `durable_step.completed` checkpoint. If the same run\nreaches the key again, the runtime emits `durable_step.replayed` and returns\nthe stored result instead of running `fn`. Steps are scoped to one run and use\nthe existing checkpoint store. If a process dies inside `fn` before the result\nis stored, the body may run again. External writes still need\n`ctx.idempotencyKey(...)` or a destination-level deduplication key.\n\n### Delivery queue\n\nA delivery reports success only after a real channel sender runs. The two\ndocumented no-sender results are `dev-no-sender` in development and\n`local-no-sender` for a local channel with no required environment. If a\nchannel module fails to load, delivery fails as retryable and the runtime logs\nthe error.\n\nFinal delivery uses a durable queue:\n\n1. The completion path creates the delivery obligation in `sending` with a\n   lease token held by the inline sender. Another worker cannot send the same\n   obligation concurrently.\n2. The inline sender retries transient failures first. The default is two\n   retries (`ASSEMBLY_LINE_DELIVERY_RETRY_ATTEMPTS`).\n3. A remaining retryable failure returns the obligation to `pending` with\n   exponential backoff and emits `delivery.deferred`. The run completes with\n   `deliveryDeferred` metadata.\n4. `runDueDeliveries()` or `startDeliveryWorker()` recovers expired leases and\n   leases due work. Postgres uses `for update skip locked` for safe parallel\n   workers.\n5. The worker sends the persisted payload and records `delivery.sent`,\n   `delivery.retrying`, or terminal `delivery.failed`. Non-retryable errors and\n   exhausted attempts fail immediately.\n\nSee the [delivery queue reference](config-reference.md#delivery-queue) for\nlease, batch, attempt, and interval settings.\n\n### Orphan recovery\n\nEvery executing run updates `updatedAt` through the guarded `touchRun` method.\nThe default heartbeat is 30 seconds (`ASSEMBLY_LINE_RUN_HEARTBEAT_MS`). This\nwrite never changes status or revives a terminal run.\n\n`recoverIncompleteRuns()` considers only runs left in `created` or `running`\npast `max(5 minutes, 4 x heartbeat)`. `staleAfterMs` can override that window.\nThe runtime claims each candidate with a `run:recovery` idempotency key, then\napplies this policy in order:\n\n1. Complete a run that already has `delivery.sent`, without sending again.\n2. Cancel tool calls requested but not started.\n3. Mark a tool interrupted after `tool.execution_started` when no settle event\n   exists. A parked continuation receives `{ interrupted: true }` instead of\n   silently executing the tool again.\n4. Give a run with a `harness.continuation` checkpoint one in-place resume\n   attempt and emit `run.recovery_resume_attempted`.\n5. For a run with a model response but no delivery, adopt the existing\n   delivery obligation or create one with the original final-delivery key.\n6. Mark any other candidate `failed` and emit `run.failed`.\n\nThe sweep runs at host boot and through `startBackgroundWorkers()`. Its default\ninterval is 60 seconds (`ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS`).\n\nResume restarts from a checkpoint; it does not replay events deterministically.\nTool execution is therefore **at least once**. A crash after an external side\neffect but before the next continuation may lead the model to request the tool\nagain. Per-iteration checkpoints, interrupted-tool guards, and completed\n`ctx.step(...)` results reduce this window but cannot close it for arbitrary\nexternal writes. Use `ctx.idempotencyKey(...)` or a destination-level key for\nnon-idempotent writes. A crashed in-flight model request restarts from the last\ncontinuation, which may add token cost but does not lose durable state.\n\n`runtime.startBackgroundWorkers()` starts the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers, and returns a controller with `stop()`. The Node host (`listenNodeRuntime`) wires all of this automatically and each worker has an env kill-switch; see the [durability workers reference](config-reference.md#durability-workers-and-recovery).\n\n### Sandbox sync\n\nAssembly Line does not acquire a sandbox before every turn. Channel lifecycle events and final text delivery run without sandbox hydration. The first core file or shell tool lazily acquires the sandbox, then hydrates only requested paths or bounded candidate sets:\n\n```txt\n/memory\n/history\n/files\n/workspace\n```\n\nThe local adapter materializes these paths under a temporary sandbox root.\n`/memory`, `/skills`, and `/workspace` are writable; `/history` and `/files`\nbecome read-only after hydration. The runtime hydrates indexes, bounded history,\nfile manifests, runtime context, and selected resources instead of every skill\nor memory document. Write generated artifacts and modified copies under\n`/workspace`. Use Docker when a development test needs a physical absolute\n`/workspace` shell path.\n\nSandbox acquisition follows the same order for every provider:\n\n1. Connect to the current live sandbox for the agent, conversation, and project.\n2. Wake the warm sandbox recorded in Postgres.\n3. Create a new sandbox.\n\nThe runtime skips reconnect, provider lookup, and snapshot restore when the\nrecorded filesystem contract is obsolete. Versioned provider names prevent a\nreplacement from colliding with the old resource. Before a mutating side\neffect, the foreground path creates a durable session row and sync obligation.\nDelivery and the next message do not wait for filesystem scanning or writeback.\n\nSandbox sync is a durable queue. A mutating sandbox tool enqueues a\n`sandbox_sync_job` before the side effect. The worker:\n\n- leases jobs, retries with backoff, and recovers expired leases;\n- writes `/memory/**` into durable memory documents;\n- writes or deletes `/skills/*/SKILL.md` through the durable skill store;\n- stores `/workspace/**` blobs and file catalog records; and\n- records blocked failures as `blocked_requires_operator` without undoing\n  final delivery.\n\nThe runtime retains or pauses dirty sandboxes until writeback completes.\nOperators can use `sandboxSyncDiagnostics()`, `inspectSandboxSyncJob(jobId)`,\nand `retrySandboxSyncJob(jobId)` to inspect and retry jobs. See the\n[sandbox sync reference](config-reference.md#sandbox-sync-and-hydration) for\ninline mode, lease, batch, and attempt settings.\n\n### Security boundaries\n\nMemory, history, files, webpages, search results, and tool output are untrusted\ncontext, not instructions. `/files` and `/history` are read-only projections;\nwrite generated or transformed outputs under `/workspace`. Skills are trusted\ninstructions and load one selected skill at a time. A sandbox receives an\nallowlisted copy of selected resources, never ambient host filesystem access.\nSee [Architecture](architecture.md) for the full trust-boundary map.\n\n## Observability\n\nAgent-authored `hooks/*.ts` reactions run after the matching event persists;\nfailures emit `agent.event_handler_failed` and are isolated from the run.\nEvery setup evaluation is stored as a complete capability checkpoint and\nsummarized by `run.capabilities_resolved` before it applies. See\n[hooks/](agent-stack/hooks.md).\n\nRun, event, tool, checkpoint, delivery, usage, subagent, and grouped-run query\ncontracts work without `instrumentation.ts`. The Node host exposes `GET /runs`,\n`GET /runs/:id`, `GET /runs/:id/events`, `GET /runs/:id/timeline`, and\narbitrary-period `GET /usage`. Dashboards and CLI tools can inspect persisted\nruns without replaying provider calls.\n\nUsage accounting stores provider-reported or reconciled cash only. Unavailable\nvalues remain `null`, and aggregate control totals are compared with\ntransactions instead of added to them. Observation failures produce warnings\nbut do not block model execution or delivery.\n\nEvery terminal outcome produces one structured log entry. The runtime logs\n`run.completed` and `run.cancelled` at info, and `run.failed` at warn with its\nmachine-readable reason. Failed run records also carry `terminalReason` and\n`terminalError`, so operators can list failures without scanning events. Model\nretries emit `model.request_retried`; errors that escape a run emit the\nnon-terminal `run.execution_error` for crash recovery. Logs contain response\nsizes, not response content. Content capture is a separate opt-in telemetry\nsetting.\n\nOptional OpenTelemetry-shaped sinks receive a parent-child span hierarchy.\nEach completed turn emits `ai.assembly-line.turn` as the parent span, with\nchildren for model steps (`ai.streamText`), tool calls (`ai.toolCall`),\nsubagents, sandbox commands, memory sync, and delivery sends. The runtime also\nemits `assembly-line.run` and `assembly-line.tool` for compatibility. Spans\ncarry trace and span IDs plus agent revision, run, session or conversation,\nturn, channel, model, tool, sandbox, delivery, status, usage, cost, and error\nattributes where available.\n\nConfigure telemetry in `agent/instrumentation.ts`. The runtime discovers this\nfile and runs it once at startup. `@assemblyline-agents/otlp` provides\n`createOtlpSink` and `createOtlpSinkFromEnv` for OTLP/HTTP export to Langfuse,\nPhoenix, Grafana, Honeycomb, or another OTLP backend. See\n[Customizing Agents: Observability](customization.md#observability).\n\nWhen `instrumentation.ts` exports `defineInstrumentation({ setup })`, the\nruntime calls `setup({ agentName, manifest, env })` before the first turn.\n`recordInputs`, `recordOutputs`, `captureContent`, and `functionId` control\ncapture and export. The default `usage` level records tokens, cost, and model\nmetadata without message bodies. A sink returned by `setup()` is used unless\nthe host supplied one directly. An exported `telemetry` value remains a\ncompatibility fallback.\n\n## State And Blob Adapters\n\nThe state contract stores runs, run events, checkpoints, tool calls, delivery\nobligations, replay data, FIFO conversation turns, runtime settings, and\nconversation-scoped agent state. The Postgres package provides migrations plus\na driver-neutral `query(sql, params)` adapter. The Node production host wires\nthat adapter to `DATABASE_URL` through `pg`; Neon, Railway, Supabase, local\nPostgres, and custom Postgres use the same schema. The schema covers agents,\nrevisions, conversations, messages, conversation-turn mailboxes, runs, run\nevents, capability snapshots, hook state, checkpoints, tool calls, approval\ngates, delivery obligations, schedules, memory, file records, sandbox leases,\nusage receipts and aggregates, runtime controls, and idempotency keys.\n\nPostgres memory search uses indexed full-text search for keyword/exact/hybrid modes. Semantic search is optional and needs a `MemoryEmbeddingProvider`: embedding-backed search turns on automatically when a provider is configured only when the state adapter reports that its vector backend is available, and `ASSEMBLY_LINE_MEMORY_EMBEDDINGS_ENABLED` can disable the feature. Postgres enables that capability only when `optionalMigrations` includes `003_openeve_memory_embeddings_pgvector` (or is `true`); the database must provide pgvector. Run `runMemoryEmbeddingBackfill()` after enabling it to populate stale or missing embeddings. Deployments without the optional backend keep deterministic full-text and portable lexical fallback behavior without querying pgvector tables.\n\nPostgres migrations are recorded in `openeve_schema_migrations` with id, checksum, description, package version, and applied time. `PostgresStateAdapter.migrate()` is idempotent and rejects checksum drift; `planMigrations()` reports pending/applied/skipped-optional/checksum-mismatch state without applying SQL. Existing memory file indexes can be promoted into state-backed memory documents with `backfillMemoryDocumentsFromFileIndexes()` when the blob adapter can read the indexed blob keys.\n\nThe blob contract stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. The S3 package implements the blob contract against S3-compatible storage and ships R2/AWS/MinIO-style helpers. The R2 package is a compatibility wrapper and in-memory test bucket.\n\n## Related Docs\n\n- [Configuration Reference](config-reference.md): every `define*` shape and `ASSEMBLY_LINE_*` variable.\n- [Runtime And Deployment](runtime-and-deployment.md): CLI, artifact tree, HTTP API, lifecycle, deploy targets.\n- [Architecture](architecture.md): system diagram, trust boundaries, and package boundaries.\n- [Adapters](adapters.md): provider matrix and per-adapter environment.\n- [Plugins](plugins.md): the extension model and plugin catalog.\n"},{"id":"getting-started","sourcePath":"getting-started.md","title":"Getting Started","description":"Go from a fresh clone to a validated, running Assembly Line agent.","url":"https://assemblyline.artificialillumination.co/docs/getting-started","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/getting-started.md","headings":[{"depth":1,"title":"Getting Started","anchor":"getting-started"},{"depth":2,"title":"Quickstart","anchor":"quickstart"},{"depth":2,"title":"Requirements","anchor":"requirements"},{"depth":2,"title":"Install And Build","anchor":"install-and-build"},{"depth":2,"title":"Create Your First Agent","anchor":"create-your-first-agent"},{"depth":1,"title":"Edit scratch/agent/.env and set OPENAI_API_KEY.","anchor":"edit-scratchagentenv-and-set-openaiapikey"},{"depth":2,"title":"Run The Full Example Agent","anchor":"run-the-full-example-agent"},{"depth":2,"title":"Serve And Inspect","anchor":"serve-and-inspect"},{"depth":2,"title":"Development Loop","anchor":"development-loop"},{"depth":2,"title":"Use A ChatGPT Subscription Through Codex (Preview)","anchor":"use-a-chatgpt-subscription-through-codex-preview"},{"depth":2,"title":"Common Issues","anchor":"common-issues"},{"depth":2,"title":"Next Steps","anchor":"next-steps"}],"content":"# Getting Started\n\nGo from a fresh clone to a working agent run.\n\n## Quickstart\n\n```sh\ngit clone https://github.com/jasonbadeaux/assembly-line.git && cd assembly-line\npnpm install\npnpm build\npnpm assembly-line run examples/minimal-agent/agent --tool echo --message \"hello from Assembly Line\"\n```\n\nThe last command builds the example agent and executes its `echo` tool locally\nwith no model provider key. The rest of this page walks the same path in order:\nrequirements, install, your first agent, the full example, serving, and the dev\nloop.\n\n## Requirements\n\nNode.js `>=22.19.0`, pnpm `10.x`, and Git. That is everything a first run\nneeds.\n\nEverything else is per-feature:\n\n| Feature | Requirement |\n| --- | --- |\n| Model-backed runs | A provider key matching the model prefix, such as `OPENAI_API_KEY` or `OPENROUTER_API_KEY` |\n| `openai-codex/*` subscription runs (preview) | A current Codex CLI and a ChatGPT account with Codex access |\n| Docker sandbox or Docker deploys | Docker |\n| Railway deploys | Railway CLI and `RAILWAY_TOKEN` |\n| Fly deploys | Fly CLI and `FLY_API_TOKEN` |\n| Generic VPS deploys | AMD64 Ubuntu 24.04/26.04 or Debian 12 host; existing Docker host or `hcloud` for secure Hetzner bootstrap |\n| Production Postgres state | `DATABASE_URL` |\n| Production file-backed connection credential stores | `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` or `ASSEMBLY_LINE_SECRET` |\n| Production blob storage | S3 or R2 credentials |\n\n## Install And Build\n\nAssembly Line runs from a source checkout:\n\n```sh\ngit clone https://github.com/jasonbadeaux/assembly-line.git\ncd assembly-line\npnpm install\npnpm build\n```\n\nIn a checkout, invoke every CLI command through the workspace script as\n`pnpm assembly-line <command>`; this page uses that form throughout. Every other\npage uses the installed form, `assembly-line <command>`. The two forms run the\nsame command.\n\n`pnpm test` runs the test suite and `pnpm check` typechecks the packages\nwithout running tests. `pnpm assembly-line help` lists every command, and\n`pnpm assembly-line help <command>` prints one command's flags.\n\nIf Codex or Claude Code will edit agents in this checkout, install the shared,\nproject-scoped authoring guidance once:\n\n```sh\npnpm assembly-line authoring install all .\n```\n\nThe integration uses version-matched docs from the installed CLI. See\n[Coding Agents](coding-agents.md) for its progressive disclosure and optional\nMCP setup.\n\n## Create Your First Agent\n\nScaffold a new agent folder:\n\n```sh\nmkdir -p scratch\npnpm assembly-line init scratch/agent\n```\n\n```txt\nCreated Assembly Line agent at /path/to/assembly-line/scratch/agent\nNext: export OPENAI_API_KEY, then: pnpm assembly-line run /path/to/assembly-line/scratch/agent --message \"hello\"\n```\n\nThe scaffold includes:\n\n```txt\nscratch/agent/\n  instructions.md\n  agent.ts\n  gateway.ts\n  .env.example\n  tools/\n    echo.ts\n```\n\nValidate it:\n\n```sh\npnpm assembly-line validate scratch/agent\n```\n\n```txt\nValid Assembly Line agent: /path/to/assembly-line/scratch/agent\n```\n\n`validate` catches shape, export, schema, route, schedule, and adapter issues.\nIt also warns when `agent.ts` names a model outside the local model catalog\n(`pnpm assembly-line models` lists it; `--no-model-check` skips the check).\n\nRun a direct tool call. This needs no provider key: when `--tool` is provided,\nthe runtime simulates the model step and executes the named tool with inferred\nor explicit input:\n\n```sh\npnpm assembly-line run scratch/agent --tool echo --message \"hello\"\n```\n\n```json\n{\n  \"run\": {\n    \"id\": \"3f9d2b1e-…\",\n    \"status\": \"completed\",\n    ...\n  },\n  \"toolCalls\": [\n    { \"toolName\": \"echo\", \"status\": \"completed\", ... }\n  ],\n  \"response\": \"{\\\"message\\\":\\\"hello\\\"}\",\n  ...\n}\n```\n\nTo run a full model turn, omit `--tool` and provide the auth required by the\nmodel prefix in `agent.ts`: `openai/gpt-5.4-mini` requires `OPENAI_API_KEY`,\nwhile `openai-codex/gpt-5.4-mini` uses the Codex CLI session from\n`codex login` (see [the Codex preview below](#use-a-chatgpt-subscription-through-codex-preview)).\n\nExport provider values in your shell or put them in the agent-root `.env`:\n\n```sh\ncp scratch/agent/.env.example scratch/agent/.env\n# Edit scratch/agent/.env and set OPENAI_API_KEY.\npnpm assembly-line run scratch/agent --message \"hello\"\n```\n\nLocal `dev`, `run`, `serve`, and `deploy --target local --serve` commands load\nthat file into the runtime process. Existing shell values win over `.env`,\nempty values still count as missing, and local execution does not upload the\nfile or copy its values into a secret store or build artifact. See\n[Runtime And Deployment](runtime-and-deployment.md#project-environment).\n\nGrow the agent with `assembly-line add`, which installs a plugin and wires its\ncontribution into the agent (a `gateway.ts` slot, a `channels/<kind>.ts` file,\nor provider-specific guidance), then prints the environment variables to set:\n\n```sh\npnpm assembly-line add slack scratch/agent      # installs @assemblyline-agents/slack, scaffolds channels/slack.ts\npnpm assembly-line add postgres scratch/agent   # installs @assemblyline-agents/postgres, sets state: adapter(\"postgres\")\npnpm assembly-line add docker scratch/agent --role sandbox\n```\n\nThe package manager is detected from lockfiles (pnpm/yarn/npm). Pass\n`--no-install` to only wire files and print the exact install command instead\nof running it.\n\n## Run The Full Example Agent\n\nThe example at `examples/minimal-agent/agent` includes instructions, an agent\ndefinition, a local HTTP channel, tools, a skill, an automation, a connection\ndeclaration, a sandbox declaration, a subagent, and instrumentation.\n\nValidate and build it:\n\n```sh\npnpm assembly-line validate examples/minimal-agent/agent\npnpm assembly-line build examples/minimal-agent/agent\n```\n\n```txt\nValid Assembly Line agent: /path/to/assembly-line/examples/minimal-agent/agent\nBuilt Assembly Line agent revision 4b0c9a17…\nArtifact: /path/to/assembly-line/examples/minimal-agent/agent/.assembly-line\n```\n\nTo keep generated artifacts out of the example directory during experiments,\nadd `--out /private/tmp/assembly-line-minimal` to `build`, `run`, `serve`, or\n`deploy --dry-run`.\n\nRun a direct tool call:\n\n```sh\npnpm assembly-line run examples/minimal-agent/agent \\\n  --message \"hello from Assembly Line\" \\\n  --tool echo\n```\n\n## Serve And Inspect\n\nInspect the compiled manifest:\n\n```sh\npnpm assembly-line manifest examples/minimal-agent/agent\n```\n\nThis prints the full compiled manifest JSON, agent metadata, tools, channels,\nschedules, connections, and the route table.\n\nServe the runtime locally:\n\n```sh\npnpm assembly-line serve examples/minimal-agent/agent --port 3000\n```\n\n```txt\nAssembly Line runtime serving 4b0c9a17…\nhttp://127.0.0.1:3000\n```\n\nThen call the local runtime from another terminal:\n\n```sh\ncurl http://127.0.0.1:3000/health\n```\n\n```json\n{\"ok\":true,\"agentRevision\":\"4b0c9a17…\"}\n```\n\n```sh\ncurl -X POST http://127.0.0.1:3000/runs \\\n  -H \"content-type: application/json\" \\\n  -d '{\"message\":\"hello\",\"toolName\":\"echo\"}'\n```\n\n```json\n{\n  \"runId\": \"3f9d2b1e-…\",\n  \"status\": \"completed\",\n  \"response\": \"{\\\"message\\\":\\\"hello\\\"}\",\n  \"waitingForApproval\": false,\n  \"waitingForInput\": false,\n  \"waitingForConnection\": false,\n  \"eventCount\": 6,\n  \"toolCallCount\": 1\n}\n```\n\nUseful inspection endpoints:\n\n- `GET /health` and `GET /healthz`\n- `GET /manifest` (admin-authenticated in production)\n- `GET /routes` (admin-authenticated in production)\n- `GET /conversations` and `GET /conversations/:id/messages` (admin-authenticated in production)\n- `POST /conversations/:id/turns` (dev-mode only by default; opt-in and admin-authenticated in production)\n- `POST /runs` (dev-mode only by default; opt-in and admin-authenticated in production)\n- `GET /runs`: `GET /runs/:id`, `GET /runs/:id/events`, and `GET /runs/:id/timeline` (admin-authenticated in production)\n\nLocal `serve` runs in dev mode, so the inspection and API-run endpoints are\nopen on your machine. Production Node hosts require an admin auth policy or\n`ASSEMBLY_LINE_ADMIN_TOKEN` for `/manifest`, `/routes`, `/conversations`, `/runs`,\nand run detail endpoints. Production direct turns and `POST /runs` are\ndisabled unless `ASSEMBLY_LINE_ENABLE_API_RUNS=true` is set and the request is\nauthenticated with `Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>`. The full\nHTTP API table is in\n[Runtime And Deployment](runtime-and-deployment.md#node-runtime-http-api).\n\n## Development Loop\n\nStart with the watch mode, which keeps a local HTTP server running and\nrebuilds + restarts it (on the same port) whenever the agent folder changes:\n\n```sh\npnpm assembly-line dev scratch/agent --watch\n```\n\nWhile the agent is invalid, the previous server keeps running and the CLI\nprints the validation issues until the folder is valid again.\n\nFor one-off steps:\n\n1. Edit files under the agent folder.\n2. Run `validate` to catch shape, export, schema, route, schedule, and adapter issues.\n3. Run `build` to emit `.assembly-line/`.\n4. Run `run` for local one-off checks.\n5. Run `serve` when testing HTTP channels or the inspection API.\n6. Inspect `.assembly-line/manifest.json`, `.assembly-line/route-table.json`, `.assembly-line/schedules.json`, and `.assembly-line/preflight.json` when something looks surprising.\n\n## Use A ChatGPT Subscription Through Codex (Preview)\n\n*Preview: this surface may change without notice.*\n\nThe Node host can route an `openai-codex/*` model through OpenAI's official\n[Codex app-server](https://developers.openai.com/codex/app-server/). Codex owns\nthe ChatGPT sign-in and token storage; Assembly Line does not read, copy, or replay\nthe OAuth cache against undocumented endpoints.\n\nInstall a current Codex CLI, then sign in and verify the session:\n\n```sh\ncodex login\ncodex login status\n```\n\nSelect a Codex model in `scratch/agent/agent.ts`:\n\n```ts\nimport { defineAgent, useModel, useReasoning } from \"@assemblyline-agents/core\";\n\nexport default defineAgent({\n  maxReasoning: \"medium\",\n  setup() {\n    useModel(\"openai-codex/gpt-5.4-mini\");\n    useReasoning(\"medium\");\n  }\n});\n```\n\nRun it normally; `OPENAI_API_KEY` is not required for this prefix:\n\n```sh\npnpm assembly-line run scratch/agent --message \"hello from my Codex plan\"\n```\n\nUse this integration on a trusted local machine where you performed\n`codex login`; usage and limits come from the signed-in ChatGPT plan. For\ngeneral hosted API traffic, use an API-backed prefix such as `openai/*`. See\n[Runtime And Deployment](runtime-and-deployment.md#codex-preview)\nfor the hosted-auth boundary.\n\n## Common Issues\n\nCommon failures, install and build errors, a missing `assembly-line` command,\nmodel provider key errors, `.assembly-line` artifact churn, serve auth, deploy\npreflight, ingress secrets, and durability workers, are collected in\n[Troubleshooting](troubleshooting.md).\n\n## Next Steps\n\n- Learn what each file in an agent folder does in the [Agent Build Stack](agent-stack/overview.md).\n- Follow the linear tutorial in [Building Agents](building-agents.md), scaffold to gated tools, channels, schedules, and evals.\n- Learn the production path in [Runtime And Deployment](runtime-and-deployment.md).\n- Customize the runtime, context, and providers with [Customizing Agents](customization.md).\n- Look up any config field or `ASSEMBLY_LINE_*` env var in the [Configuration Reference](config-reference.md).\n- Explore the examples:\n  - [minimal-agent](https://github.com/jasonbadeaux/assembly-line/tree/main/examples/minimal-agent) - the full agent folder shape with tools, skills, channels, automations, connections, sandbox, subagents, and instrumentation.\n  - [custom-context-agent](https://github.com/jasonbadeaux/assembly-line/tree/main/examples/custom-context-agent) - a custom context policy layered on `defaultContext`.\n  - [self-improving-agent](https://github.com/jasonbadeaux/assembly-line/tree/main/examples/self-improving-agent) - durable skills, runtime-created automations, and gated connection saving.\n  - [vps-deployment-agent](https://github.com/jasonbadeaux/assembly-line/tree/main/examples/vps-deployment-agent) - an existing-VPS inventory plus the required Postgres, R2, hosted-sandbox, and `vpsDeploy()` configuration.\n"},{"id":"photon","sourcePath":"photon.md","title":"Photon iMessage Channel","description":"Photon/Spectrum channel setup, delivery modes, ingress auth, typing lifecycle, rich tools, and file handling.","url":"https://assemblyline.artificialillumination.co/docs/photon","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/photon.md","headings":[{"depth":1,"title":"Photon iMessage Channel","anchor":"photon-imessage-channel"},{"depth":2,"title":"Delivery Modes","anchor":"delivery-modes"},{"depth":2,"title":"One-Line Channel Setup","anchor":"one-line-channel-setup"},{"depth":3,"title":"Manual Wiring","anchor":"manual-wiring"},{"depth":2,"title":"Ingress Auth","anchor":"ingress-auth"},{"depth":2,"title":"Typing Lifecycle","anchor":"typing-lifecycle"},{"depth":2,"title":"Markdown Replies","anchor":"markdown-replies"},{"depth":2,"title":"Rich Feature Tools","anchor":"rich-feature-tools"},{"depth":2,"title":"Outbound Bridge Contract","anchor":"outbound-bridge-contract"},{"depth":2,"title":"Inbound Files","anchor":"inbound-files"},{"depth":2,"title":"Environment Reference","anchor":"environment-reference"},{"depth":2,"title":"Exports","anchor":"exports"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# Photon iMessage Channel\n\n`@assemblyline-agents/photon` connects an agent to iMessage through Photon/Spectrum. It keeps Photon at the channel boundary: inbound webhooks are normalized into Assembly Line turns, and outbound side effects go through a Photon transport, either Photon's Spectrum cloud directly, or a bridge you host.\n\nPhoton webhooks are at-least-once. The adapter returns a fast `2xx` accepted response and uses `X-Spectrum-Webhook-Id + message.id` as the default idempotency key, falling back to `message.id` when the webhook id header is absent.\n\n## Delivery Modes\n\nThe adapter picks the outbound transport from the environment at send time:\n\n| Mode | Selected when | Required env |\n| --- | --- | --- |\n| **Direct** (Spectrum cloud) | No transport URL is set and both project credentials are set | `PHOTON_PROJECT_ID`, `PHOTON_PROJECT_SECRET` |\n| **Bridge** (self-hosted) | `PHOTON_TRANSPORT_URL` (or `PHOTON_BRIDGE_URL`) is set | `PHOTON_TRANSPORT_URL`, `PHOTON_BRIDGE_TOKEN` |\n\n**Direct** talks to Photon's Spectrum cloud with the project's credentials and\nrequires no bridge. It loads `spectrum-ts` dynamically from the agent's\ndependencies, so bridge-only agents do not need that package. Direct mode\nsupports text and Markdown, media by URL, typing, app cards, backgrounds, and\nDM destinations. Reactions and polls depend on the installed `spectrum-ts`\nversion. The adapter skips unsupported reactions and tells you to configure a\nbridge for unsupported polls. Inbound attachment downloads always require a\nbridge; see [Inbound Files](#inbound-files).\n\n**Bridge** posts every side effect to a transport bridge you host (see [Outbound Bridge Contract](#outbound-bridge-contract)), authenticated with `Authorization: Bearer <PHOTON_BRIDGE_TOKEN>`. When a transport URL is set without a token, sends fail with `PHOTON_BRIDGE_TOKEN is required for Photon delivery.`; when neither mode is configured, sends fail with `PHOTON_TRANSPORT_URL is required for Photon delivery (or set PHOTON_PROJECT_ID/PHOTON_PROJECT_SECRET for direct delivery).`\n\n## One-Line Channel Setup\n\nCreate `channels/photon.ts` in an agent folder. `definePhotonChannel()` provides\ninbound normalization, the typing lifecycle, and outbound delivery:\n\n```ts\nimport { definePhotonChannel } from \"@assemblyline-agents/photon\";\n\nexport default definePhotonChannel();\n```\n\nThis compiles to an HTTP channel on `/photon/events`. The helper stamps the\ningress-auth requirement but no transport environment requirement. It resolves\nthe delivery mode at send time, so the same channel file works in both modes.\nPass options to override the route, methods, or description:\n`definePhotonChannel({ route: \"/imessage\" })`.\n\n### Manual Wiring\n\nIf you prefer to wire the handlers by hand (for example, to wrap one of them), re-export them explicitly instead of using the helper:\n\n```ts\nimport { defineChannel } from \"@assemblyline-agents/core\";\nimport {\n  normalizeHttp as normalizePhotonHttp,\n  send as sendPhoton,\n  startTurn as startPhotonTurn\n} from \"@assemblyline-agents/photon\";\n\nexport default defineChannel({ transport: \"http\", route: \"/photon/events\", methods: [\"POST\"] });\n// Bridge mode only: pin the transport env at preflight. Omit for direct mode.\nexport const requiredEnv = [\"PHOTON_TRANSPORT_URL\", \"PHOTON_BRIDGE_TOKEN\"];\nexport const normalizeHttp = normalizePhotonHttp;\nexport const startTurn = startPhotonTurn;\nexport const send = sendPhoton;\n```\n\nNamed exports take precedence over the default export's handlers. The explicit `requiredEnv` line pins the channel to bridge mode during preflight; `definePhotonChannel()` itself stamps no `requiredEnv`, so leave it out when the agent may run with direct delivery.\n\nIf manual wiring omits `startTurn`, the compiler emits\n`photon-channel-missing-typing` and the runtime records\n`channel.turn_lifecycle_unsupported`. The `definePhotonChannel()` form always\nwires typing.\n\n## Ingress Auth\n\nSet `PHOTON_WEBHOOK_SIGNING_SECRET` to verify `X-Spectrum-Signature` (HMAC-SHA256 over `v0:<timestamp>:<body>`, with a configurable timestamp tolerance). Alternatively set `PHOTON_INGRESS_TOKEN` and have the sender pass `Authorization: Bearer <token>`. Unsigned Photon ingress is local/dev-only; production runtime boot rejects a Photon channel when neither a signing secret nor bearer token is configured.\n\nThe production boot check accepts exactly `PHOTON_WEBHOOK_SIGNING_SECRET` or `PHOTON_INGRESS_TOKEN`; the aliases `PHOTON_SIGNING_SECRET` and `PHOTON_WEBHOOK_BEARER_TOKEN` satisfy per-request verification but not the boot check, so always set at least one canonical name in production.\n\n## Typing Lifecycle\n\nFor fast-ack HTTP ingress, the runtime starts the typing lifecycle after\ncapacity admission and idempotency reservation. It does this while runtime\ninitialization and durable run creation continue. Rejected turns and\nidempotent replays do not start a duplicate indicator. Direct runtime calls\nstart the lifecycle during run setup. Both paths start typing before\nattachment intake and model work, then stop it immediately before delivery.\n`definePhotonChannel()` maps this lifecycle to Photon typing signals.\n\nThe indicator is refreshed every `PHOTON_TYPING_REFRESH_MS` (default 4 seconds). In bridge mode, typing signals are posted to `/v1/messages/interact` with `action: \"typing\"` and `state: \"start\" | \"stop\"`; in direct mode they map onto the Spectrum typing API. Typing failures are logged as warnings and never fail the run, and when no transport or destination is available the lifecycle is a no-op.\n\n## Markdown Replies\n\nPhoton's Spectrum bridge renders full CommonMark in iMessage, so the adapter sends replies as `textFormat: \"markdown\"` whenever the response contains renderable markdown, headings, lists, tables, fenced or inline code, blockquotes, links, or bold/italic. Plain casual messages (no markdown syntax) are sent as `plain` so they read like a normal text, with casual sentence-ending punctuation softened.\n\nYou can override the detection per delivery with a `textFormat` field on the delivery payload, or call `photonTextForReply(body, \"markdown\")` directly.\n\n## Rich Feature Tools\n\nDrop these tool factories into the agent's `tools/` folder to let the agent send rich Photon side effects to the current conversation. Each resolves the transport and reply destination from the run's channel context, so the model only supplies the content:\n\n```ts\n// tools/photon_react.ts\nimport { definePhotonReactionTool } from \"@assemblyline-agents/photon\";\nexport default definePhotonReactionTool();\n```\n\nAvailable factories:\n\n- `definePhotonReactionTool`: tapback the user's latest message (`like`, `love`, `laugh`, `emphasize`, `dislike`, `question`)\n- `definePhotonPollTool`: send a native iMessage poll (title + 2–10 options)\n- `definePhotonAppCardTool`: send a styled link card (caption, subcaption, image)\n- `definePhotonBackgroundTool`: set or clear the chat background image\n\nEach factory accepts `{ description?, needsApproval? }` to customize the model-facing description or gate the side effect behind an approval. In direct mode, reactions and polls depend on the installed `spectrum-ts` version (see [Delivery Modes](#delivery-modes)).\n\n## Outbound Bridge Contract\n\nIn bridge mode, the adapter expects `PHOTON_TRANSPORT_URL` to point at a bridge exposing the Photon transport endpoints:\n\n- `POST /v1/messages/send`: final replies: text or markdown body, native reply targets, link previews, and optional media fields\n- `POST /v1/messages/interact`: reactions (`action: \"react\"`) and typing signals (`action: \"typing\"`, `state: \"start\" | \"stop\"`)\n- `POST /v1/messages/poll`\n- `POST /v1/messages/app`\n- `POST /v1/messages/background`\n\nThe direct transport additionally routes a dedicated `/v1/messages/typing` path (`{ \"action\": \"start\" | \"stop\" }`) for hosts that address typing explicitly; bridges only receive typing through `/v1/messages/interact`.\n\nOptional media fields on a send:\n\n```json\n{\n  \"mediaUrl\": \"https://example.com/image.png\",\n  \"mediaFilename\": \"image.png\",\n  \"mediaMimeType\": \"image/png\"\n}\n```\n\nEvery request carries an idempotency key. Bridge responses are parsed as JSON and read up to a fixed 512 KiB cap; non-`2xx` responses raise `Photon transport returned HTTP <status>: <payload>`.\n\n## Inbound Files\n\nPhoton attachment content is preserved as runtime-visible files, not just\nmessage metadata. When Spectrum sends attachment content with fields such as\n`name`, `mimeType`, `size`, and `downloadUrl`/`contentUrl`/`url`. The adapter\nkeeps those references in `ChannelTurn.attachments`. The runtime then downloads\nthe bytes after the webhook ACK, stores them through the configured blob\nadapter, and exposes them under `/files/original/...` with entries in\n`/files/manifest.json`. Attachment downloads are restricted to the\n`PHOTON_TRANSPORT_URL` origin, which is why inbound files require a bridge even\nwhen outbound delivery runs in direct mode.\n\nPhoton bridge multipart/form-data forwarding is also supported. When the\nbridge sends `asset_manifest_json` plus matching file parts. The Node host\nparses the file bytes and the Photon adapter converts them into inline\nattachments before runtime storage. Prefer form forwarding for uploaded files;\nJSON forwarding can describe attachments, but it cannot carry the actual file\nbytes unless it includes an explicit downloadable URL.\n\nMarkdown/text uploads read back as UTF-8 through `read`; binary\nuploads remain byte-accurate when hydrated under `/files`. ZIP uploads are kept as\ntheir original archive under `/files/original/...` and, when extraction\nsucceeds, safe entries are also exposed under\n`/files/extracted/<archive-name>/...` so the agent can open and project files\nfrom a zipped folder directly. If the download URL points at the Photon bridge\norigin. The runtime uses `PHOTON_BRIDGE_TOKEN` for the fetch without exposing\nthat token to the model context.\n\n## Environment Reference\n\n`PHOTON_*` variables are canonical on this page; runtime-wide `ASSEMBLY_LINE_*` variables live in the [Configuration Reference](config-reference.md).\n\n| Variable | Values | Default | Effect |\n| --- | --- | --- | --- |\n| `PHOTON_TRANSPORT_URL` | URL | unset | Bridge base URL; presence selects bridge mode. `PHOTON_BRIDGE_URL` is an accepted alias. |\n| `PHOTON_BRIDGE_TOKEN` | string | unset | Bearer token for bridge requests and bridge-origin attachment downloads; required in bridge mode. |\n| `PHOTON_PROJECT_ID` | string | unset | Spectrum project id for direct delivery. |\n| `PHOTON_PROJECT_SECRET` | string | unset | Spectrum project secret for direct delivery. |\n| `PHOTON_WEBHOOK_SIGNING_SECRET` | string | unset | HMAC secret for `X-Spectrum-Signature` verification. Alias: `PHOTON_SIGNING_SECRET` (request verification only, not the production boot check). |\n| `PHOTON_INGRESS_TOKEN` | string | unset | Expected webhook `Authorization: Bearer` token. Alias: `PHOTON_WEBHOOK_BEARER_TOKEN` (request verification only, not the production boot check). |\n| `PHOTON_WEBHOOK_TOLERANCE_SECONDS` | integer, 30–86400 | 300 | Maximum accepted signature timestamp age. |\n| `PHOTON_SEND_REQUEST_TIMEOUT_MS` | integer, 1000–120000 | 30000 | Timeout for sends, polls, app cards, and backgrounds. |\n| `PHOTON_TYPING_REQUEST_TIMEOUT_MS` | integer, 500–15000 | 3000 | Timeout for typing and reaction requests. |\n| `PHOTON_TYPING_REFRESH_MS` | integer, 1000–30000 | 4000 | Typing indicator refresh cadence. |\n\n## Exports\n\n`@assemblyline-agents/photon` exports, grouped by concern:\n\n- **Channel**: `definePhotonChannel`, `normalizeHttp`, `startTurn`, `send`, `resolveAttachment`\n- **Transport helpers**: `sendPhotonReply`, `sendPhotonReaction`, `sendPhotonPoll`, `sendPhotonAppCard`, `sendPhotonBackground`, `startPhotonTyping`\n- **Text formatting**: `photonTextForReply`, `photonTextFormatForReply`, `shouldSendPhotonMarkdown`, `containsRenderableMarkdown`, `softenIMessageBubbleEndings`\n- **Tools**: `definePhotonReactionTool`, `definePhotonPollTool`, `definePhotonAppCardTool`, `definePhotonBackgroundTool`, `photonTransportFromToolContext`, `photonDestinationFromToolContext`\n- **Mode and env constants**: `directPhotonEnabled`, `PHOTON_REQUIRED_ENV`, `PHOTON_DIRECT_ENV`, `PHOTON_WEBHOOK_ENV`, `PHOTON_INGRESS_SECRET_ENV`\n\n## Related Docs\n\n- [channels/](agent-stack/channels.md): the channel file contract this page plugs into.\n- [Adapters](adapters.md): the channel role matrix and the other channel providers.\n- [Configuration Reference](config-reference.md): runtime `ASSEMBLY_LINE_*` environment variables.\n- [Troubleshooting](troubleshooting.md): production ingress boot failures and webhook `401`s.\n"},{"id":"plugins","sourcePath":"plugins.md","title":"Plugins","description":"Discover, install, and wire optional Assembly Line integrations without expanding the framework core.","url":"https://assemblyline.artificialillumination.co/docs/plugins","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/plugins.md","headings":[{"depth":1,"title":"Assembly Line Plugins","anchor":"assembly-line-plugins"},{"depth":2,"title":"What A Plugin Contributes","anchor":"what-a-plugin-contributes"},{"depth":2,"title":"Taxonomy","anchor":"taxonomy"},{"depth":2,"title":"Plugin Catalog","anchor":"plugin-catalog"},{"depth":3,"title":"Channels","anchor":"channels"},{"depth":3,"title":"Substrate Providers","anchor":"substrate-providers"},{"depth":3,"title":"Connection Plugins","anchor":"connection-plugins"},{"depth":3,"title":"Capabilities Tier","anchor":"capabilities-tier"},{"depth":2,"title":"Install A Plugin With assembly-line add","anchor":"install-a-plugin-with-assembly-line-add"},{"depth":3,"title":"Role Disambiguation","anchor":"role-disambiguation"},{"depth":2,"title":"Community Plugins","anchor":"community-plugins"},{"depth":2,"title":"Auditability And Trust","anchor":"auditability-and-trust"},{"depth":2,"title":"Related Docs","anchor":"related-docs"}],"content":"# Assembly Line Plugins\n\nPlugins are installable npm packages that extend Assembly Line without changing the\nframework core. Official plugins ship integrations maintained with Assembly Line;\ncommunity plugins use the same public contracts from independent packages.\nThis page is the catalog and install guide. To implement a plugin, see\n[Authoring Plugins](authoring-adapters.md).\n\n- [What A Plugin Contributes](#what-a-plugin-contributes)\n- [Taxonomy](#taxonomy)\n- [Plugin Catalog](#plugin-catalog)\n- [Install A Plugin With assembly-line add](#install-a-plugin-with-assembly-line-add)\n- [Community Plugins](#community-plugins)\n- [Auditability And Trust](#auditability-and-trust)\n\n## What A Plugin Contributes\n\nA plugin can contribute one or more extension types:\n\n| Contribution | What it adds |\n| --- | --- |\n| Provider | A runtime implementation for an adapter role such as sandbox, state, blob, deploy, scheduler, connection, or channel. |\n| Connection helper | Agent-scoped access to an external capability through MCP, OpenAPI, HTTP, or a reviewed sandbox CLI. |\n| Channel helper | Ingress normalization and response delivery for a messaging service. |\n| Skill | On-demand operating instructions for the plugin's tools or workflows. |\n| Companion integration | A protocol and helper package for a separately installed application, such as a desktop companion. |\n\nThe package name does not need to contain `plugin`. For example,\n`@assemblyline-agents/e2b` is the official E2B sandbox plugin and `@assemblyline-agents/slack` is the\nofficial Slack channel plugin.\n\n## Taxonomy\n\nThese terms are used consistently across the Assembly Line docs:\n\n- **Plugin**: an installable npm package that extends Assembly Line. The\n  user-facing umbrella term.\n- **Contribution**: what a plugin exports: a provider registration\n  (`assemblyLineProvider`), a connection helper (`assemblyLinePlugin` plus a\n  `define<X>Connection` factory), a channel helper (`define<X>Channel`),\n  bundled skills, or a companion protocol.\n- **Provider**: an implementation registered for a role and kind through\n  `assemblyLineProvider`.\n- **Adapter**: a configured provider instance, selected in agent config with\n  `adapter(kind, options, { package })`.\n- **Connection**: an agent-scoped declaration of an external capability and\n  its credential contract: a file in `connections/`.\n- **Capability (tier)**: single-vendor feature packages under\n  `capabilities/` (currently LiveKit). A capability gives an agent something\n  new to do rather than somewhere new to run, so it has no substitutable\n  generic contract. See [Adapters](adapters.md#capabilities).\n\nThe word \"capability\" is overloaded; the meaning depends on where it appears:\n\n| Where | Meaning |\n| --- | --- |\n| Capability tier (`capabilities/`) | A single-vendor feature package such as LiveKit. |\n| Provider metadata `capabilities: []` | Feature tags a provider advertises for preflight and tooling (for example `persistent-storage`). |\n| Tool `capability:` block | Discovery metadata on an authored tool (visibility, namespace, tags). See [Customizing Agents](customization.md#tool-discovery-and-capability-metadata). |\n| Connection `capabilities: [\"issues:read\"]` | Declared capability strings on a declaration-only connection contract. |\n\n## Plugin Catalog\n\nThe framework includes a minimal zero-install baseline: local development\nstate, blob storage, scheduling, and sandbox behavior. Everything that\nconnects Assembly Line to an optional service or execution environment is a plugin,\neven when its npm package lives in the Assembly Line monorepo. The framework\npackages themselves (`@assemblyline-agents/core`, `compiler`, `runtime`, `node`, `cli`,\n`sdk`, `pi`, `codex`, `otlp`) are not plugins and do not appear here.\n\nProvider-specific deep setup, OAuth application registration, CLI installs,\naccount policy, lives in each package's README (`packages/<kind>` in the\nAssembly Line repo, or the package page on npm).\n\n### Channels\n\nChannel plugins normalize provider events into durable Assembly Line turns and\ndeliver replies. Consumption details are in\n[Adapters: Channels](adapters.md#channels).\n\n| Kind | Package | Helper | Required env | Optional env | `assembly-line add` |\n| --- | --- | --- | --- | --- | --- |\n| `slack` | `@assemblyline-agents/slack` | `defineSlackChannel` | `SLACK_SIGNING_SECRET`, `SLACK_BOT_TOKEN` | `SLACK_BOT_USER_ID`, `SLACK_ASSISTANT_ENABLED` | Scaffolds `channels/slack.ts` |\n| `discord` | `@assemblyline-agents/discord` | `defineDiscordChannel` | `DISCORD_PUBLIC_KEY`, `DISCORD_APPLICATION_ID`, `DISCORD_BOT_TOKEN` | `DISCORD_GATEWAY_ENABLED`, `DISCORD_GATEWAY_INTENTS`, `DISCORD_BOT_USER_ID` | Scaffolds `channels/discord.ts` |\n| `telegram` | `@assemblyline-agents/telegram` | `defineTelegramChannel` | `TELEGRAM_BOT_TOKEN`; `TELEGRAM_WEBHOOK_SECRET` required in production | None | Scaffolds `channels/telegram.ts` |\n| `teams` | `@assemblyline-agents/teams` | `defineTeamsChannel` | `MICROSOFT_APP_ID`, `MICROSOFT_APP_PASSWORD` | `ASSEMBLY_LINE_TEAMS_ALLOWED_TENANTS`, `ASSEMBLY_LINE_TEAMS_ALLOWED_SERVICE_URLS` | Scaffolds `channels/teams.ts` |\n| `photon` | `@assemblyline-agents/photon` | `definePhotonChannel` | `PHOTON_WEBHOOK_SIGNING_SECRET` or `PHOTON_INGRESS_TOKEN` | None | Manual: write `channels/photon.ts` yourself |\n\nFor the manual row, create the channel file yourself:\n\n```ts\n// channels/photon.ts\nimport { definePhotonChannel } from \"@assemblyline-agents/photon\";\n\nexport default definePhotonChannel();\n```\n\nSee [Photon iMessage Channel](photon.md) for the Photon bridge.\n\n### Substrate Providers\n\nSubstrate plugins fill `gateway.ts` slots and sandbox declarations.\nConfiguration, helper functions, and full env tables are in\n[Adapters](adapters.md).\n\n| Kind | Roles | Package | Required env | Notes |\n| --- | --- | --- | --- | --- |\n| `postgres` | state (also a package-less scheduler kind) | `@assemblyline-agents/postgres` | `DATABASE_URL` | `assembly-line add postgres` selects the state role. Presets: `neonPostgres()`, `railwayPostgres()`, `supabasePostgres()`, `localPostgres()` |\n| `docker` | sandbox, deploy | `@assemblyline-agents/docker` | Docker CLI/daemon | Two roles: pass `--role sandbox` or `--role deploy` |\n| `daytona` | sandbox | `@assemblyline-agents/daytona` | `DAYTONA_API_KEY` | Hosted sandboxes |\n| `e2b` | sandbox | `@assemblyline-agents/e2b` | `E2B_API_KEY` | Hosted sandboxes |\n| `modal` | sandbox | `@assemblyline-agents/modal` | `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | Hosted sandboxes |\n| `s3` | blob | `@assemblyline-agents/s3` | `S3_BUCKET`, `S3_REGION`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY` | Helpers `s3Blob()`, `minioBlob()`, `r2Blob()` |\n| `r2` | blob | `@assemblyline-agents/r2` | `R2_ACCOUNT_ID`, `R2_BUCKET`, `R2_ACCESS_KEY_ID`, `R2_SECRET_ACCESS_KEY` | Legacy wrapper; prefer `r2Blob()` from `@assemblyline-agents/s3` |\n| `railway` | deploy (a separate `railway` connection plugin also exists) | `@assemblyline-agents/railway` | `RAILWAY_TOKEN`, Railway CLI | `assembly-line add railway` selects the deploy role; use `--role connection` for the MCP connection |\n| `fly` | deploy | `@assemblyline-agents/fly` | `FLY_API_TOKEN`, `flyctl` | Generates `fly.toml` and publishes with `flyctl deploy` |\n| `vps` | deploy | `@assemblyline-agents/vps` | Named `assembly-line.hosts.json` entry, SSH key path, Docker host | Supported per-agent isolation and transactional blue/green Caddy routing; secure Hetzner create/adopt bootstrap is available |\n\n### Connection Plugins\n\nEvery connection plugin is packaged as `@assemblyline-agents/<kind>`, exports\n`assemblyLinePlugin`, is installable with `assembly-line add <kind>`, bundles at least\none skill, and scaffolds a read-only `connections/<kind>.ts`, writes stay\nhidden until you enable an approval policy. Protocol is MCP over Streamable\nHTTP unless the table says otherwise. **R** = required, **O** = optional.\n\nAssembly Line's connection packages are scaffolding, not hosted integration\naccounts: the package supplies the helper, endpoint/spec defaults, tool\nclassification, safe access defaults, skill, and preflight metadata. You\ncreate the provider application, API token, OAuth client, local process, or\nbridge. Official connection plugins are supported unless an entry explicitly\nsays otherwise.\n\nGoogle services are intentionally separate. Replace the former\n`@assemblyline-agents/google` package and `google` connection with only the grants an agent\nneeds: `@assemblyline-agents/gmail`, `@assemblyline-agents/google-calendar`, and/or\n`@assemblyline-agents/google-drive`. Connecting or revoking one does not grant or revoke\neither of the others.\n\n| Kind | Endpoint | Credential | Writes | Notes |\n| --- | --- | --- | --- | --- |\n| `gmail` | `GMAIL_MCP_URL` (R) | `GMAIL_MCP_TOKEN` (R) | Yes | Independent Gmail grant; scopes are limited to Gmail read, compose, and send |\n| `google-calendar` | `GOOGLE_CALENDAR_MCP_URL` (R) | `GOOGLE_CALENDAR_MCP_TOKEN` (R) | Yes | Independent Calendar grant; provider namespace is `google_calendar` |\n| `google-drive` | `GOOGLE_DRIVE_MCP_URL` (R) | `GOOGLE_DRIVE_MCP_TOKEN` (R) | Yes | Independent Drive grant; provider namespace is `google_drive` |\n| `github` | default `https://api.githubcopilot.com/mcp/`; `GITHUB_MCP_URL` (O) | `GITHUB_MCP_TOKEN` (R) | Yes | |\n| `slack` | default `https://mcp.slack.com/mcp`; `SLACK_MCP_URL` (O) | `SLACK_MCP_TOKEN` (R) | Yes | Pass `--role connection` (bare `slack` selects the channel) |\n| `telegram` | `TELEGRAM_MCP_URL` (R) | `TELEGRAM_MCP_TOKEN` (R) | Read-only | Pass `--role connection` (bare `telegram` selects the channel) |\n| `strava` | `STRAVA_MCP_URL` (R) | `STRAVA_MCP_TOKEN` (R) | Read-only | |\n| `polar` | `POLAR_MCP_URL` (R) | `POLAR_MCP_TOKEN` (R) | Read-only | |\n| `spotify` | `SPOTIFY_MCP_URL` (R) | `SPOTIFY_MCP_TOKEN` (R) | Yes | |\n| `xai` | `XAI_MCP_URL` (R) | `XAI_MCP_TOKEN` (R) | Read-only | |\n| `exa` | default `https://mcp.exa.ai/mcp`; `EXA_MCP_URL` (O) | `EXA_API_KEY` (O), sent as `x-api-key` | Read-only | Hosted web search and page fetch work without a key; enable advanced search through Exa's `tools` URL parameter |\n| `plaid` | `PLAID_MCP_URL` (R) | `PLAID_MCP_TOKEN` (R) | Read-only | |\n| `granola` | default `https://mcp.granola.ai/mcp`; `GRANOLA_MCP_URL` (O) | `GRANOLA_MCP_TOKEN` (R) | Read-only | |\n| `linear` | default `https://mcp.linear.app/mcp`; `LINEAR_MCP_URL` (O) | `LINEAR_MCP_TOKEN` (R) | Yes | |\n| `notion` | default `https://mcp.notion.com/mcp`; `NOTION_MCP_URL` (O) | `NOTION_MCP_TOKEN` (R) | Yes | |\n| `monday` | default `https://mcp.monday.com/mcp`; `MONDAY_MCP_URL` (O) | `MONDAY_MCP_TOKEN` (R) | Yes | |\n| `jira` | default `https://mcp.atlassian.com/v1/mcp/authv2`; `JIRA_MCP_URL` (O) | `JIRA_MCP_TOKEN` (R) | Yes | Atlassian MCP |\n| `hubspot` | default `https://mcp.hubspot.com`; `HUBSPOT_MCP_URL` (O) | `HUBSPOT_MCP_TOKEN` (R) | Read-only | |\n| `figma` | default `https://mcp.figma.com/mcp`; `FIGMA_MCP_URL` (O) | `FIGMA_MCP_TOKEN` (R) | Yes | |\n| `paper` | `PAPER_MCP_URL` (R) | `PAPER_MCP_TOKEN` (O) | Yes | Developer-exposed local bridge |\n| `sentry` | default `https://mcp.sentry.dev/mcp`; `SENTRY_MCP_URL` (O) | `SENTRY_MCP_TOKEN` (R) | Yes | |\n| `supabase` | default `https://mcp.supabase.com/mcp`; `SUPABASE_MCP_URL` (O) | `SUPABASE_MCP_TOKEN` (R) | Yes | |\n| `metabase` | `METABASE_MCP_URL` (R) | `METABASE_MCP_TOKEN` (O) | Yes | Instance MCP endpoint |\n| `cloudflare` | default `https://mcp.cloudflare.com/mcp`; `CLOUDFLARE_MCP_URL` (O) | `CLOUDFLARE_MCP_TOKEN` (R) | Yes | |\n| `vercel` | default `https://mcp.vercel.com`; `VERCEL_MCP_URL` (O) | `VERCEL_MCP_TOKEN` (R) | Yes | |\n| `railway` | default `https://mcp.railway.com`; `RAILWAY_MCP_URL` (O) | `RAILWAY_MCP_TOKEN` (R) | Yes | Pass `--role connection` (bare `railway` selects the deploy target) |\n| `refero` | default `https://api.refero.design/mcp`; `REFERO_MCP_URL` (O) | `REFERO_MCP_BEARER_TOKEN` (R) | Read-only | |\n| `resend` | default `https://mcp.resend.com/mcp`; `RESEND_MCP_URL` (O) | `RESEND_API_KEY` (R) | Yes | Official hosted MCP; API-key and webhook-secret creation/retrieval tools are blocked so credentials stay outside model context |\n| `agentcash` | `AGENTCASH_MCP_URL` (R) | `AGENTCASH_MCP_BRIDGE_TOKEN` (R) | Yes | Paid API discovery and requests |\n| `margins` | default `https://margins.artificialillumination.co/mcp`; `MARGINS_MCP_URL` (O) | One-time page/folder/workspace binding packet; agent identity overrides (O) | Yes | Host-side `margins__pair` redemption stores rotating bearer credentials outside model/sandbox context; comments and suggestions are writes; Margins independently enforces the packet's scope and suggest/edit permission |\n| `mirror` | default `https://mirror.artificialillumination.co/mcp`; `MIRROR_MCP_URL` (O) | `MIRROR_OAUTH_CLIENT_ID` (R), `MIRROR_OAUTH_REDIRECT_URI` (O) | Yes | User-scoped OAuth Authorization Code + PKCE or host-side redemption of a pre-scoped Mirror UI binding packet; exact Mirror write-like tools remain hidden until an Assembly Line approval policy is enabled |\n| `provenance` | default `https://provenance.artificialillumination.co/mcp`; `PROVENANCE_MCP_URL` (O) | `PROVENANCE_OAUTH_CLIENT_ID` (R), agent identity overrides (O) | Metadata only | Registered public-client OAuth + PKCE with `provenance:ledger`; ledger reconstruction is read-only and ambient capture is configured separately |\n| `soundcloud` | OpenAPI: bundled official spec, base `https://api.soundcloud.com` | `SOUNDCLOUD_CLIENT_ID` (R), `SOUNDCLOUD_CLIENT_SECRET` (R), `SOUNDCLOUD_REDIRECT_URI` (O) | Yes | OAuth 2.1 PKCE; the developer registers the SoundCloud app |\n| `arcads` | default `https://mcp.arcads.ai` | `ARCADS_MCP_CLIENT_ID` (R), `ARCADS_MCP_REDIRECT_URI` (O) | Yes | OAuth Authorization Code + PKCE with dynamic client registration; generation consumes credits |\n| `higgsfield` | Sandbox CLI (`protocol: \"cli\"`, `transport: \"sandbox\"`, command `higgsfield`) | None, `higgsfield auth login` inside each persistent, user-scoped sandbox | Yes | Bundles three skills (generate, Soul ID, product photoshoot); install the official CLI in the sandbox image |\n| `browser-use` | default `https://api.browser-use.com/v3/mcp` | `BROWSER_USE_API_KEY` (R), sent as `x-browser-use-api-key` header | Yes | Hosted browser sessions; account, profiles, and cost policy stay developer-owned |\n| `orgo` | stdio bridge on the runtime host | `ORGO_API_KEY` (R), `ORGO_API_BASE_URL` (O) | Yes | Cloud desktops; the bridge strips VNC passwords from results |\n| `peekaboo` | stdio, separately installed local binary | None | Yes | Same-host macOS control; host requirements `local` + `darwin`, hosted deploys are rejected |\n| `computer-use` | relay, default `https://computer-use.openeve.dev/v1`; `ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL` (O) | `ASSEMBLY_LINE_COMPUTER_USE_BINDING` (R) | Yes | End-to-end encrypted paired-Mac control; see [Remote Computer Use](remote-computer-use.md) |\n| `ffmpeg` | stdio bridge on the runtime host | None, install `ffmpeg`/`ffprobe` on the host | Yes | Typed, workspace-rooted media operations |\n| `remotion` | stdio bridge on the runtime host | None, install `remotion` + `@remotion/cli` in the project | Yes | Uses the supported Remotion CLI, not the deprecated hosted MCP |\n\nOrgo is a connection because its tools manage and control provider-owned\ndesktops by `computer_id`; it does not implement the per-run\n`SandboxSession` filesystem contract. The package can add a separate sandbox\nrole later if it binds one computer to a session and supplies the canonical\nfile and shell operations.\n\n### Capabilities Tier\n\nLiveKit voice dispatch and SIP tools live in `capabilities/livekit` as\n`@assemblyline-agents/livekit`. It exposes `defineLiveKitConnection()` and\n`defineLiveKitOutboundCallTool()` and requires `LIVEKIT_URL`,\n`LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET` (optional\n`LIVEKIT_OUTBOUND_TRUNK_ID`, `LIVEKIT_VOICE_AGENT_NAME`). It is not\ninstallable with `assembly-line add`; declare the tool and connection files\ndirectly. See [Adapters: LiveKit Voice](adapters.md#livekit-voice).\n\n## Install A Plugin With assembly-line add\n\n`assembly-line add <kind> <agentRoot>` installs the plugin package with your\ndetected package manager and wires its contribution into the agent folder.\nThe command does not hide changes in global configuration. Each result is a\nvisible file or a printed instruction.\n\n```sh\nassembly-line add notion agent\n```\n\n```\nInstalling plugin @assemblyline-agents/notion with npm in /path/to/project.\nCreated connections/notion.ts using defineNotionConnection() with explicit read-only access.\nCopied skills/notion/ from @assemblyline-agents/notion.\nSet: NOTION_MCP_TOKEN\nOptional: NOTION_MCP_URL\nSetup: Configure Notion credentials, OAuth application, or MCP bridge for this deployment.\nRun: npx assembly-line validate agent\n```\n\nWhat gets scaffolded depends on the contribution's role:\n\n- **Connection**: creates `connections/<kind>.ts` with read access enabled,\n  writes disabled, and a comment showing how to opt into approval-gated\n  writes:\n\n  ```ts\n  import { defineNotionConnection } from \"@assemblyline-agents/notion\";\n\n  export default defineNotionConnection({\n    access: {\n      read: true,\n      // Change to { approval: \"always\" } only after reviewing this plugin's write tools.\n      write: false\n    }\n  });\n  ```\n\n  It also copies the plugin's bundled skill directories into `skills/`.\n  Existing files are never overwritten: an existing connection file or skill\n  directory is left unchanged and reported.\n- **Channel**: creates `channels/<kind>.ts` calling the channel helper\n  (Slack, Discord, Telegram, and Teams have scaffolds). For other channel\n  kinds the CLI prints `No channel scaffold is known for \"<kind>\"` and tells\n  you to create the file yourself.\n- **Gateway roles** (`state`, `blob`, `sandbox`, `deploy`, `runtime`,\n  `scheduler`), edits the matching slot in `gateway.ts` to\n  `<role>: adapter(\"<kind>\")`, creating `gateway.ts` when missing. When the\n  file cannot be edited with confidence, the CLI prints the exact snippet to\n  paste instead of guessing.\n\nAfter wiring, the CLI prints the plugin's required env vars, optional env\nvars, and setup steps, then the `assembly-line validate` command to run next.\n\nPass `--no-install` to print the package-manager command and configuration\nchanges without installing the dependency. Two caveats: skills cannot be\ncopied until the package exists (the CLI prints\n`Skill <name> will be available after <package> is installed.`), and\ncommunity packages cannot be added at all without installation. Their\nmetadata is read by importing the installed package, so the CLI fails with\n`Failed to load plugin package <name> ... Install it first (or rerun without\n--no-install).`\n\n### Role Disambiguation\n\nSome kinds provide more than one role. `--role <role>` selects explicitly;\nwithout it the CLI applies one rule: when a kind has exactly one\nnon-connection contribution. That contribution wins.\n\n| Command | Result |\n| --- | --- |\n| `assembly-line add postgres agent` | State adapter in `gateway.ts` |\n| `assembly-line add railway agent` | Deploy target in `gateway.ts` |\n| `assembly-line add slack agent` | Slack channel file |\n| `assembly-line add slack agent --role connection` | Slack MCP connection file |\n| `assembly-line add telegram agent --role connection` | Telegram MCP connection file |\n| `assembly-line add docker agent --role sandbox` | Docker sandbox (docker is sandbox + deploy, so `--role` is required) |\n\n## Community Plugins\n\nInstall a community plugin by package name:\n\n```sh\nassembly-line add @acme/assembly-line-neon agent\n```\n\nThe CLI installs the package, imports it, and reads its contributions from\nthe `assemblyLinePlugin` (connections) and `assemblyLineProvider` (providers)\nexports. The same wiring and `--role` rules apply; a package providing\nmultiple roles requires `--role`. A package exporting neither symbol fails\nwith `Plugin package <name> does not export assemblyLinePlugin or\nassemblyLineProvider, so its contributions cannot be determined.`\n\nTo build such a package, see [Authoring Plugins](authoring-adapters.md).\n\n## Auditability And Trust\n\nPlugins execute trusted host code and should be reviewed like application\ndependencies. Assembly Line deliberately limits them to named extension points\ninstead of arbitrary lifecycle hooks. Agent authors should be able to audit a\nplugin's effect from the package dependency plus the explicit files and\nadapter selections in the agent folder.\n\nProvider-specific secrets stay in host environment variables, authorization\nflows, or encrypted connection grants. They must not be embedded in plugin\nskills, prompts, tool inputs, or agent source files.\n\nStdio and sandbox-CLI plugins are trusted host dependencies: Assembly Line never\nlets a model or dynamic connection choose their command, arguments, working\ndirectory, or environment. Sandbox-CLI connections additionally run only\nreviewed operations inside the active run sandbox with individually quoted\narguments. The plugin never receives an unsandboxed gateway command channel.\n\n## Related Docs\n\n- [Adapters](adapters.md): consuming substrate adapters: role matrix, gateway config, per-adapter env.\n- [connections/](agent-stack/connections.md): the connection file format, access and approval model, transports.\n- [Authoring Plugins](authoring-adapters.md): implementation contracts for every contribution type.\n- [Configuration Reference](config-reference.md): every `define*` shape and `ASSEMBLY_LINE_*` env var.\n"},{"id":"overview","sourcePath":"README.md","title":"Assembly Line","description":"Build portable, durable AI agents from ordinary files with Assembly Line.","url":"https://assemblyline.artificialillumination.co/docs","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/README.md","headings":[{"depth":1,"title":"Assembly Line Developer Documentation","anchor":"assembly-line-developer-documentation"},{"depth":2,"title":"Mental Model","anchor":"mental-model"},{"depth":2,"title":"Start Here","anchor":"start-here"},{"depth":2,"title":"Agent Build Stack","anchor":"agent-build-stack"},{"depth":2,"title":"Guides","anchor":"guides"},{"depth":2,"title":"Plugins And Providers","anchor":"plugins-and-providers"},{"depth":2,"title":"Reference","anchor":"reference"}],"content":"# Assembly Line Developer Documentation\n\nAssembly Line is a vendor-neutral, filesystem-first framework for durable AI\nagents. An agent is an ordinary folder. The compiler validates that folder and\nbuilds a deterministic manifest and runtime artifact. The runtime then executes\nthe agent through the adapters selected in `gateway.ts`.\n\nEach file has one role. A tool is one file, a channel is one file, and a\nconnection is one declaration. Official `@assemblyline-agents/*` packages use\nthe same public contracts, so agents can add channels, connections, sandboxes,\nskills, and infrastructure providers without changing the framework core.\n\n## Mental Model\n\n```txt\nagent/ folder      what the agent is\ncompiler           validates the folder and emits a manifest plus .assembly-line artifact\nruntime            executes runs durably and records events, tools, deliveries, and recovery state\ngateway.ts         chooses deploy, runtime, state, blob, sandbox, and scheduler adapters\nplugins            add optional channels, connections, sandboxes, state, blob, deploy, skills, and companions\n```\n\nA minimal agent needs only `instructions.md` and `agent.ts`. Add a file in the\nmatching folder when the agent needs a tool, skill, channel, automation,\nhook, connection, sandbox, subagent, or instrumentation setup.\n\n## Start Here\n\n1. [Getting Started](getting-started.md) - clone the repo, build it, run the example agent, create a new agent, and inspect the compiled artifact.\n2. [Agent Build Stack](agent-stack/overview.md) - navigate the agent folder by the file or directory you are editing.\n3. [Building Agents](building-agents.md) - the linear tutorial through the whole authoring path.\n4. [Coding Agents](coding-agents.md) - install version-matched Assembly Line guidance for Codex and Claude Code.\n5. [Runtime And Deployment](runtime-and-deployment.md) - understand the compiler output, durable runtime lifecycle, HTTP endpoints, preflight checks, and deploy targets.\n6. [Troubleshooting](troubleshooting.md) - diagnose install, CLI, model-key, auth, preflight, ingress, and durability-worker failures.\n\n## Agent Build Stack\n\nStart at the [Agent Build Stack overview](agent-stack/overview.md), then jump to the file you are editing:\n\n- [instructions.md](agent-stack/instructions.md) - always-on trusted instructions.\n- [agent.ts](agent-stack/agent-ts.md) - static identity/policy and synchronous runtime capability composition.\n- [context.ts](agent-stack/context-ts.md) - default or custom context bundle policy.\n- [gateway.ts](agent-stack/gateway-ts.md) - deploy, runtime, state, blob, sandbox, and scheduler adapters.\n- [tools/](agent-stack/tools.md) - typed actions the model can call.\n- [skills/](agent-stack/skills.md) - on-demand procedures and self-improvement surface.\n- [channels/](agent-stack/channels.md) - external entrypoints and provider reply delivery.\n- [automations/](agent-stack/automations.md) - schedule- and event-triggered durable work with inline preparation and finalization.\n- [hooks/](agent-stack/hooks.md) - cross-cutting reactions to persisted runtime events.\n- [connections/](agent-stack/connections.md) - external capability and credential contracts.\n- [sandbox/](agent-stack/sandbox.md) - isolated filesystem and shell backend.\n- [subagents/](agent-stack/subagents.md) - hook-composed child agents with scoped models, tools, workspaces, state, and connections.\n- [evals/](agent-stack/evals.md) - golden cases, custom evaluators, repeatable experiments, and baseline gates.\n- [instrumentation.ts](agent-stack/instrumentation.md) - telemetry sinks and capture policy.\n\n## Guides\n\n- [Building Agents](building-agents.md) - the agent folder shape, tools, skills, channels, automations, hooks, connections, sandboxes, subagents, and instrumentation in one continuous tutorial.\n- [Coding Agents](coding-agents.md) - project-scoped Codex and Claude Code skills, version-matched docs, MCP access, and structured validation.\n- [Customizing Agents](customization.md) - context policy, gateway adapters, capability metadata, approvals, self-improvement, dynamic automations and connections, and observability.\n- [Runtime And Deployment](runtime-and-deployment.md) - CLI commands, build artifacts, runtime lifecycle, HTTP endpoints, preflight, deploy targets, and production checks.\n- [Troubleshooting](troubleshooting.md) - common failures with the exact error text and the fix.\n- [Remote Computer Use](remote-computer-use.md) - pair a Mac with a hosted agent through Computer Host and an end-to-end encrypted relay.\n\n## Plugins And Providers\n\n- [Plugins](plugins.md) - install optional Assembly Line integrations and understand how plugin contributions map to providers, adapters, connections, skills, and companions.\n- [Adapters](adapters.md) - consume substrate adapters: the role matrix, gateway config, and per-adapter env reference.\n- [Photon iMessage Channel](photon.md) - Photon/Spectrum channel setup, delivery modes, typing lifecycle, rich tools, and file handling.\n- [Authoring Plugins](authoring-adapters.md) - implement new channels, connections, sandboxes, blob stores, deploy targets, and state stores as plugin provider contributions.\n\n## Reference\n\n- [Framework Guide](framework.md) - current framework contracts, manifest shape, runtime guarantees, package boundaries, and adapter boundaries.\n- [Configuration Reference](config-reference.md) - every `define*` config shape, per-file compiler contract, and `ASSEMBLY_LINE_*` environment variable.\n- [Architecture](architecture.md) - system shape, trust boundaries, and package ownership.\n- [Contributing](contributing.md) - work on the Assembly Line framework itself and keep developer docs current.\n- [Migrating from OpenEve](rename-migration.md) - what the rename changes for existing projects, what still works, and the alias removal policy.\n"},{"id":"remote-computer-use","sourcePath":"remote-computer-use.md","title":"Remote Computer Use","description":"Pair a Mac with a hosted Assembly Line agent through Computer Host and an end-to-end encrypted relay.","url":"https://assemblyline.artificialillumination.co/docs/remote-computer-use","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/remote-computer-use.md","headings":[{"depth":1,"title":"Remote Computer Use","anchor":"remote-computer-use"},{"depth":2,"title":"Security Boundary","anchor":"security-boundary"},{"depth":2,"title":"Set Up A Hosted Agent","anchor":"set-up-a-hosted-agent"},{"depth":2,"title":"Host The Relay","anchor":"host-the-relay"},{"depth":2,"title":"Operating Limits","anchor":"operating-limits"}],"content":"# Remote Computer Use\n\nUse `@assemblyline-agents/computer-use` when the agent runs on Railway, Fly, Docker, or\nanother host but must operate a specific Mac. Use `@assemblyline-agents/peekaboo` only when\nthe Assembly Line Node runtime itself runs on that Mac.\n\nThe remote path has three independently auditable pieces:\n\n1. The agent uses a static `transport: \"relay\"` connection and a deployment-\n   scoped binding.\n2. Assembly Line Computer Host, installed by Assembly Line Builder, runs Peekaboo on the Mac\n   and maintains an outbound WebSocket.\n3. A Cloudflare Worker and one Durable Object per Mac route encrypted request\n   and response envelopes.\n\n*Availability: Assembly Line Builder (the desktop app that installs Computer\nHost and manages pairing) and the `openeve-computer-relay` Workers project are\ndistributed separately from this repository and are not part of the npm\npackages. This page documents the agent-side contract they plug into.*\n\nThe middle service is needed because the hosted agent cannot normally open a\nconnection into a Mac behind NAT, sleep/wake, or a consumer firewall. Both\nends initiate outbound TLS connections to a stable public rendezvous point.\nDo not expose Peekaboo MCP, a local HTTP wrapper, or a desktop port directly to\nthe internet.\n\n## Security Boundary\n\nPairing creates a separate P-256 key pair and random bearer credential for\neach deployment binding. Assembly Line derives direction-specific AES-256-GCM keys\nwith ECDH and HKDF. Tool arguments and results are encrypted between the agent\nruntime and Computer Host; the relay sees device and binding identifiers,\nmessage timing, and ciphertext sizes, but not plaintext tool data.\n\nThe relay stores SHA-256 token digests, rejects expired or replayed envelopes,\ncaps payload and pending-request sizes, and supports immediate binding\nrevocation. Computer Host additionally enforces an exact accessibility-first\ntool allowlist, refuses actions while the Mac is locked, and applies one of\nthree local write policies:\n\n- `always`: show a native approval prompt for every write.\n- `deny`: allow inspection but reject all writes.\n- `never`: allow writes without a local prompt; reserve this for a deliberately\n  provisioned, trusted Mac account.\n\nThe Assembly Line connection policy is a second gate. New connections start with\n`write: false`; use `write: { approval: \"always\" }` until the agent's authority\nis well understood.\n\nThe binding contains the agent-side private key and is a secret. Keep it in\nAssembly Line Builder's Keychain-backed secret store and deployment environment. Never\nput it in source, a prompt, a skill, tool arguments, logs, or relay config.\n\n## Set Up A Hosted Agent\n\nAdd the plugin to the agent:\n\n```sh\nassembly-line add computer-use ./agent\n```\n\nThe generated connection is explicit and read-only:\n\n```ts\nimport { defineComputerUseConnection } from \"@assemblyline-agents/computer-use\";\n\nexport default defineComputerUseConnection({\n  access: { read: true, write: false }\n});\n```\n\nIn Assembly Line Builder:\n\n1. Add **Assembly Line Computer Use** to the agent on the Bind & Build screen.\n2. Enter the relay URL and the relay registration token for the Mac's first\n   registration.\n3. Choose the local write policy, then select **Install & pair**.\n4. Grant Accessibility and Screen Recording to the signed Assembly Line Builder/\n   Computer Host application when macOS asks.\n5. Sync or deploy the production secrets. Builder stores\n   `ASSEMBLY_LINE_COMPUTER_USE_BINDING` in Keychain and writes the local `.env` with\n   mode `0600`. A custom relay also uses `ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL`.\n\nBuilder bundles a pinned, checksum-verified universal Peekaboo release and\ninstalls a background LaunchAgent. The registration token is sent to the relay\nonly during first registration and is not part of the deployment binding.\n\nEnable connection writes only after pairing succeeds:\n\n```ts\nexport default defineComputerUseConnection({\n  access: {\n    read: true,\n    write: { approval: \"always\" }\n  }\n});\n```\n\nAgent approval and Mac approval are independent. A write runs only when both\npolicies permit it. Revoking the binding in Builder removes the local\nKeychain/`.env` material and causes the relay to reject it immediately; deploy\na new binding before the agent can use that Mac again.\n\nIf pairing fails, re-check the relay URL and registration token, confirm the\ndeployed environment carries `ASSEMBLY_LINE_COMPUTER_USE_BINDING`, and see\n[Troubleshooting](troubleshooting.md) for deploy preflight and\nconnection-secret failures.\n\n## Host The Relay\n\nThe relay is a separate, self-hostable Cloudflare Workers project named\n`openeve-computer-relay`. It uses a hibernatable WebSocket Durable Object per\ndevice and SQLite storage for token hashes, binding revocation, replay\nprotection, and metadata-only audit events.\n\nSet a long random Worker secret, deploy, and use the resulting `/v1` URL and\nsecret during first pairing:\n\n```sh\npnpm install\npnpm exec wrangler secret put ADMIN_TOKEN\npnpm test\npnpm exec wrangler deploy\n```\n\nUse HTTPS outside local development. Rotate the bootstrap secret through\nWorkers secrets; do not place it in `wrangler.jsonc`, the agent, or the Mac host\nconfiguration. The relay is deliberately not an MCP server and never receives\nthe binding's E2EE private key.\n\n## Operating Limits\n\nThe current tool surface is accessibility-tree-first. It includes inspection,\napplication/window/menu/dialog control, clicking, typing, hotkeys, scrolling,\ndragging, `set_value`, and accessibility actions. Screenshot/vision, recording,\nshell, clipboard/paste, browser-specific Peekaboo tools, configuration,\ncleanup, and nested autonomous agents are excluded.\n\nComputer Host must be online in an unlocked macOS 15+ user session. A locked,\noffline, revoked, denied, expired, or approval-rejected result is a hard policy\nboundary, not an instruction to find another route around the host.\n"},{"id":"rename-migration","sourcePath":"rename-migration.md","title":"Migrating from OpenEve","description":"What the OpenEve to Assembly Line rename changed, what still works under its old spelling, and what was removed pre-launch.","url":"https://assemblyline.artificialillumination.co/docs/rename-migration","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/rename-migration.md","headings":[{"depth":1,"title":"Migrating from OpenEve","anchor":"migrating-from-openeve"},{"depth":2,"title":"Renamed, no alias","anchor":"renamed-no-alias"},{"depth":2,"title":"The one permanent alias: environment variables","anchor":"the-one-permanent-alias-environment-variables"},{"depth":2,"title":"What never changes","anchor":"what-never-changes"}],"content":"# Migrating from OpenEve\n\nThe product formerly named OpenEve is now Assembly Line. The framework never\nshipped to external users under the OpenEve name, so the rename was a clean\ncut, not a compatibility window: every renamed surface below now accepts\n**only** its new spelling. There is one permanent exception: the\nhost environment-variable mirror.\n\nIf you have an older local checkout or a script written against the OpenEve\nnames, update it to the new spellings; the legacy ones no longer work.\n\n## Renamed, no alias\n\n| Surface | Old (no longer works) | New (only) |\n| --- | --- | --- |\n| Product name | OpenEve | Assembly Line |\n| npm scope | `@openeve/*` | `@assemblyline-agents/*` |\n| Facade package | `openeve` (unscoped) | `@assemblyline-agents/sdk` |\n| CLI binary | `openeve` | `assembly-line` |\n| Build artifact dir | `.openeve` | `.assembly-line` |\n| Hosts inventory | `openeve.hosts.json` | `assembly-line.hosts.json` |\n| Internal HTTP routes | `/openeve/*` | `/assembly-line/*` (legacy paths `404`) |\n| OAuth callback | `/openeve/connections/callback` | `/assembly-line/connections/callback` |\n| Scheduler header | `x-openeve-scheduler-secret` | `x-assembly-line-scheduler-secret` (legacy header is rejected) |\n| Plugin export | `openevePlugin` | `assemblyLinePlugin` |\n| Provider export | `openeveProvider` | `assemblyLineProvider` |\n| Telemetry namespace | `openeve.*` / `ai.openeve.*` | `assembly-line.*` / `ai.assembly-line.*` |\n| Docker/VPS labels & names (new resources) | `openeve.*` / `openeve-*` | `assembly-line.*` / `assembly-line-*` |\n\nA plugin package that still exports only `openevePlugin`/`openeveProvider` is\ntreated as exporting nothing. The CLI and compiler report it as a package\nwith no contributions. Rename the export to pick it back up.\n\n## The one permanent alias: environment variables\n\nEvery branded environment variable is still readable under **both**\nprefixes, forever:\n\n| New | Legacy | Behavior |\n| --- | --- | --- |\n| `ASSEMBLY_LINE_*` | `OPENEVE_*` | Both spellings are mirrored to the same value at env ingestion. The new name wins if both are set; setting both to the *same* value is fine; setting them to *different* values fails at startup. |\n\nThis is intentional and will not be removed. Existing deployed hosts (for\nexample a VPS deployment with `OPENEVE_ADMIN_TOKEN` in its environment) never\nneed to be touched. Prefer `ASSEMBLY_LINE_*` in new configuration, but either\nspelling works indefinitely.\n\n## What never changes\n\nThese identifiers were never part of the rename and are permanent:\n\n- Postgres `openeve_*` tables and indexes, `NNN_openeve_*` migration IDs, and\n  the `oe_` database prefix, the migration ledger is checksum-verified and\n  the physical schema lives on production databases.\n- `/var/lib/openeve`: `/home/openeve`, the `openeve` OS user, and its\n  sudoers/sshd drop-ins on existing deployed hosts.\n- Existing container, volume, and network names (`openeve-caddy`,\n  `openeve-edge`, `openeve-postgres`, per-deployment `openeve-<slug>*`).\n- The `/workspace/.openeve` reserved sandbox directory persisted inside\n  deployed sandboxes.\n- The `openeve-computer-use/v1` crypto contexts, renaming them would break\n  all existing device bindings.\n- The channel event wire value `source: \"openeve_state\"`.\n- Git history, tags, and release notes.\n\nSee `docs/rename/manifest.md` in the repository for the complete, maintained\nidentifier-by-identifier record (that file lives outside the published docs\nsite).\n"},{"id":"runtime-and-deployment","sourcePath":"runtime-and-deployment.md","title":"Runtime and Deployment","description":"Build artifacts, run agents durably, and deploy Assembly Line across supported targets.","url":"https://assemblyline.artificialillumination.co/docs/runtime-and-deployment","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/runtime-and-deployment.md","headings":[{"depth":1,"title":"Runtime And Deployment","anchor":"runtime-and-deployment"},{"depth":2,"title":"CLI Commands","anchor":"cli-commands"},{"depth":3,"title":"Project environment","anchor":"project-environment"},{"depth":2,"title":"Build Artifact","anchor":"build-artifact"},{"depth":2,"title":"Runtime Lifecycle","anchor":"runtime-lifecycle"},{"depth":2,"title":"Usage Accounting And Observability","anchor":"usage-accounting-and-observability"},{"depth":2,"title":"Node Runtime HTTP API","anchor":"node-runtime-http-api"},{"depth":3,"title":"Auth model","anchor":"auth-model"},{"depth":3,"title":"Endpoints","anchor":"endpoints"},{"depth":3,"title":"Direct conversations","anchor":"direct-conversations"},{"depth":2,"title":"Operator Controls","anchor":"operator-controls"},{"depth":2,"title":"Durability Workers","anchor":"durability-workers"},{"depth":3,"title":"Model-Call Resilience","anchor":"model-call-resilience"},{"depth":3,"title":"Run Deadline And Tool Timeouts","anchor":"run-deadline-and-tool-timeouts"},{"depth":3,"title":"Terminal Outcomes And Recovery Fidelity","anchor":"terminal-outcomes-and-recovery-fidelity"},{"depth":2,"title":"Concurrency And Rate Limiting","anchor":"concurrency-and-rate-limiting"},{"depth":2,"title":"Graceful Shutdown","anchor":"graceful-shutdown"},{"depth":2,"title":"State And Blob Storage","anchor":"state-and-blob-storage"},{"depth":2,"title":"Sandbox Sync","anchor":"sandbox-sync"},{"depth":2,"title":"Deploy Targets","anchor":"deploy-targets"},{"depth":3,"title":"Environments","anchor":"environments"},{"depth":3,"title":"Local","anchor":"local"},{"depth":3,"title":"Railway","anchor":"railway"},{"depth":4,"title":"Syncing secrets to the target","anchor":"syncing-secrets-to-the-target"},{"depth":3,"title":"Docker","anchor":"docker"},{"depth":3,"title":"Fly","anchor":"fly"},{"depth":3,"title":"Generic VPS","anchor":"generic-vps"},{"depth":1,"title":"Build the inactive slot, sync secrets, run migrations, but do not route traffic.","anchor":"build-the-inactive-slot-sync-secrets-run-migrations-but-do-not-route-traffic"},{"depth":1,"title":"Authenticate a Codex-backed prepared release if applicable.","anchor":"authenticate-a-codex-backed-prepared-release-if-applicable"},{"depth":1,"title":"Activate exactly the persisted prepared revision.","anchor":"activate-exactly-the-persisted-prepared-revision"},{"depth":1,"title":"Restore the previous runtime/route without changing durable state.","anchor":"restore-the-previous-runtimeroute-without-changing-durable-state"},{"depth":1,"title":"Reconcile public/private ingress without rebuilding.","anchor":"reconcile-publicprivate-ingress-without-rebuilding"},{"depth":1,"title":"perform the verified transfer","anchor":"perform-the-verified-transfer"},{"depth":2,"title":"Codex (Preview)","anchor":"codex-preview"},{"depth":3,"title":"Deploy and authenticate","anchor":"deploy-and-authenticate"},{"depth":3,"title":"Credential trust boundary","anchor":"credential-trust-boundary"},{"depth":2,"title":"Migrations","anchor":"migrations"},{"depth":2,"title":"Preflight","anchor":"preflight"},{"depth":2,"title":"Production Checklist","anchor":"production-checklist"}],"content":"# Runtime And Deployment\n\nAssembly Line separates authoring from execution:\n\n```txt\nagent folder -> compiler -> .assembly-line artifact -> runtime host -> durable runs\n```\n\nThe compiler validates a folder and emits a deterministic manifest. The runtime loads the compiled artifact, executes runs, records events, stores context and artifacts, manages approvals and human input pauses, and delivers final responses idempotently.\n\nContents:\n\n- [CLI Commands](#cli-commands) and [Project environment](#project-environment)\n- [Build Artifact](#build-artifact)\n- [Runtime Lifecycle](#runtime-lifecycle)\n- [Node Runtime HTTP API](#node-runtime-http-api) and [Direct conversations](#direct-conversations)\n- [Operator Controls](#operator-controls)\n- [Durability Workers](#durability-workers)\n- [Concurrency And Rate Limiting](#concurrency-and-rate-limiting)\n- [Graceful Shutdown](#graceful-shutdown)\n- [State And Blob Storage](#state-and-blob-storage) and [Sandbox Sync](#sandbox-sync)\n- [Deploy Targets](#deploy-targets)\n- [Codex (Preview)](#codex-preview)\n- [Migrations](#migrations)\n- [Preflight](#preflight)\n- [Production Checklist](#production-checklist)\n\n## CLI Commands\n\nRun the CLI against an agent root:\n\n```sh\nassembly-line <command> <agent-root>\n```\n\n(For a monorepo checkout, [Getting Started](getting-started.md) explains the in-repo `pnpm assembly-line` form.)\n\n| Command | Purpose |\n| --- | --- |\n| `init` | Scaffold a minimal agent folder. |\n| `add` | Install a plugin and wire its provider contribution into `gateway.ts` or `channels/` (`--role <role>` disambiguates multi-role plugins like `docker`; `--no-install` prints the install command instead of running it; `--pm` overrides the lockfile-detected package manager). |\n| `authoring install\\|update\\|status` | Install or inspect the project-scoped Assembly Line authoring skill for Codex and Claude Code. |\n| `docs list\\|search\\|read\\|version\\|mcp` | Query the version-matched developer docs or start their read-only stdio MCP server. |\n| `validate` | Check required files, exports, schemas, routes, automations, hooks, connections, subagents, skills, and gateway bindings. |\n| `manifest` | Print or write the compiled manifest. |\n| `build` | Emit the `.assembly-line/` runtime artifact. |\n| `dev` | Validate, build, and run a local dev check. With `--watch`, serve locally and rebuild + restart on the same port when the agent changes. |\n| `run` | Execute one local run from a message and optional tool/input. |\n| `eval` | Build once and run isolated `evals/*.json` golden cases through the production runtime path, with fingerprinted experiment artifacts, deterministic and custom assertions, optional LLM judges, repetitions/retries, and baseline regression gates. |\n| `serve` | Start the compiled Node runtime locally. |\n| `runs cancel\\|suspend\\|resume` | Cancel, suspend, or resume a run on a deployed agent over the authenticated HTTP API (`--url`, `--token`). |\n| `agent disable\\|enable\\|quiesce\\|resume\\|status` | Control ingress or pause all new runtime work for a safe state cutover. |\n| `channels wire` | Print or apply channel provider ingress URLs after deploy; Telegram webhooks are set by API when credentials are present. |\n| `checkpoints compact` | Dry-run or apply checkpoint retention cleanup for the configured state adapter. Omit `--apply` for a safe report-only run. |\n| `auth codex` | Run Codex device authentication through the selected deploy publisher and its persistent credential volume. |\n| `hosts bootstrap` | Securely create or adopt a supported VPS host and write its pinned version 2 inventory entry. |\n| `state migrate-postgres\\|upgrade-postgres` | Run verified database transfer or host-Postgres major-version upgrade workflows. |\n| `secrets diff` | Compare required and configured deploy secret names without reading remote values. |\n| `models` | List `provider/model` specs from the local model catalog (`--provider <name>` filters). |\n| `deploy --dry-run` | Build a deployment plan and report missing setup without publishing. |\n| `deploy` | Publish or prepare the target declared by `gateway.ts` or `--target`. |\n| `help [command]` | Show usage for all commands or one command (`--help` also works per command). |\n\nCommon flags:\n\n| Flag | Purpose |\n| --- | --- |\n| `--root <path>` | Agent root override. |\n| `--out <dir>` | Build artifact directory. |\n| `--message <text>` | Message for `dev` or `run`. |\n| `--tool <name>` | Force a local tool call. |\n| `--input <json>` | Tool input JSON. |\n| `--approve` | Allow or resume approval-gated tool execution. |\n| `--dry-run` | Print deploy plan or checkpoint cleanup impact without applying changes. |\n| `--apply` | For `checkpoints compact`, delete the reported checkpoint rows. |\n| `--target <name>` | Override gateway deploy target. |\n| `--once` | Boot-check long-lived commands once, then exit. |\n| `--port <number>` | Port for `serve`, `dev --watch`, or local deploy. |\n| `--watch` | Keep `dev` serving and rebuild + restart on agent changes. |\n| `--no-model-check` | Skip the `validate` model catalog check. |\n| `--json` | Emit machine-readable output for supported commands, including docs, authoring status, validation, and eval. |\n| `--latest` | For `docs`, fetch the current hosted corpus instead of using the installed version-matched corpus. |\n| `--url <baseUrl>` / `--token <token>` | Deployed agent base URL and admin token for `runs` and `agent` (fallbacks: `ASSEMBLY_LINE_URL`, `ASSEMBLY_LINE_ADMIN_TOKEN`). |\n| `--migration-command <bin>` | Run artifact migrations before hosted deploy. |\n\nRun `assembly-line help <command>` for the full per-command flag list.\nSee [Coding Agents](coding-agents.md) for the progressive authoring workflow and MCP configuration.\n\n### Project environment\n\nCLI commands load `.env` from the selected agent root before validation, build,\nlocal execution, and deployment planning. This works even when the CLI is\ninvoked from a different directory:\n\n```sh\nassembly-line run /absolute/path/to/agent --message \"hello\"\nassembly-line serve /absolute/path/to/agent --port 3000\nassembly-line deploy /absolute/path/to/agent --target local --serve\n```\n\nFor `dev`, `run`, `serve`, and a serving local deploy, non-empty `.env` values\nare added directly to the local runtime process environment. They are not\ncopied to a secret store or written into the `.assembly-line` build artifact. The\n`validate`, `manifest`, `build`, and local deploy-planning paths use the same\nresolved environment for required-variable preflight.\n\nValues already present in the invoking shell or host environment take\nprecedence over `.env`. A declared key with an empty value does not satisfy\nrequired-environment preflight. Hosted values are only copied to a provider\nsecret store when `deploy --sync-secrets` is explicitly used.\n\n## Build Artifact\n\n`assembly-line build` emits:\n\n```txt\n.assembly-line/\n  manifest.json\n  agent-revision.json\n  Dockerfile\n  package.json\n  preflight.json\n  route-table.json\n  schedules.json\n  automations.json\n  server/\n    boot.json\n    boot.js\n    sources.json\n    source-metadata.json\n    source-map.json\n  assets/\n  migrations/\n```\n\nBuilds always write `.assembly-line/`; a stale legacy `.openeve/` directory is\nnever read as a build artifact. It is still recognized by ignore/clean\ntooling (`pnpm clean:artifacts`, `.gitignore`) so old output gets swept up, delete it, or leave it for `clean:artifacts` to remove.\n\nImportant files:\n\n- `manifest.json` - complete compiled agent contract.\n- `server/sources.json` - packaged authored TypeScript, including `agent.ts`,\n  imported composition helpers, hooks, tools, channels, and automations.\n- `agent-revision.json` - deterministic source/config revision.\n- `package.json` - artifact dependency declaration and `start` script (`node server/boot.js`).\n- `preflight.json` - required env and provider setup.\n- `route-table.json` - HTTP channel routes.\n- `automations.json` - canonical schedule- and event-triggered automation registrations.\n- `schedules.json` - deprecated schedule registration compatibility metadata.\n- `server/boot.js` - production Node runtime boot entrypoint. It loads `manifest.json` and `server/sources.json`, constructs production runtime adapters from `gateway.ts`, and serves the full [HTTP API](#node-runtime-http-api), not a health-only stub.\n- `deployment.json` - created by `assembly-line deploy` after a local or provider publish/prepare operation, not by plain `build`.\n\nThe artifact is generated output and should not be committed.\n\nBuild artifacts support two package modes:\n\n- Local mode is the default **inside a monorepo checkout**. It writes\n  `file:./vendor` dependencies for Assembly Line workspace packages, copies those\n  packages once under `vendor/`, links them into `node_modules`, and copies the\n  runtime dependency closure needed for no-install local artifact smoke tests.\n  Optional native dependencies are filtered to the generated Docker target\n  (Linux x64 with glibc) plus the current build host, so the same local artifact\n  remains runnable for smoke tests without copying every platform binary.\n- Release mode is for published package deployments and is the **default when\n  the toolchain is installed from npm** (i.e. when no `packages/` workspace\n  layout is present). You can also force it with `packageMode: \"release\"` on\n  `buildAgent()` or `ASSEMBLY_LINE_ARTIFACT_PACKAGE_MODE=release`. The artifact\n  `package.json` pins the published `@assemblyline-agents/*` versions (all share one fixed\n  version) and omits local `vendor/` and `node_modules/` copies; each published\n  package's own external dependencies are resolved transitively by the package\n  manager. The generated Dockerfile then installs dependencies through normal\n  package-manager semantics.\n\n## Runtime Lifecycle\n\nFor each run, the runtime:\n\n1. Persists the run before execution starts.\n2. Resolves or creates a conversation and materializes safe attachment metadata.\n3. Loads the bounded conversation hook-state snapshot and synchronously evaluates\n   `agent.ts` `setup()`.\n4. Validates the declaration against static ceilings and the compiled catalog,\n   persists a complete capability checkpoint, then emits\n   `run.capabilities_resolved` before applying it.\n5. Checks only selected connections and builds context from permanent\n   instructions plus the resolved instructions, skills, tools, connections,\n   subagents, sandbox metadata, and output schema.\n6. Starts the model loop or forced tool call with that exact snapshot.\n7. Sends model requests without local accounting gates, then observes provider\n   responses and attempts to persist source-backed usage.\n8. Creates and sends final delivery obligations idempotently.\n9. Marks the run completed, failed, suspended, cancelled, waiting for input, or waiting for approval.\n\nAt every model-iteration boundary the harness checks the state revision.\n`ctx.agentState` or a `usePersistentState()` setter atomically increments that\nrevision and emits `agent.state_changed` without values. A dirty run re-evaluates\nbefore the next request, records the new snapshot, atomically replaces visible\ntools, and refreshes model/reasoning/prompt/connections/subagents/sandbox/schema\nselection. Nothing changes during an in-flight provider request or tool call.\n\nRecovery compares the current state revision and declaration hash with the\nlast hydrated `agent.capability_snapshot` checkpoint. It reuses an exact match\nand re-evaluates otherwise; it never reconstructs capabilities from partial\nevents. Hook-evaluation failure emits `agent.hook_evaluation_failed` and fails\nthe run. Event-observer failure emits `agent.event_handler_failed` and is\nnon-terminal.\n\nFailed final deliveries are durable. When a channel send keeps failing retryably, the obligation is deferred onto a durable delivery queue (`delivery.deferred`, status `pending` with backoff) instead of going terminal, and a delivery worker drains it later, including after a crash or in another replica (Postgres leases use `for update skip locked`). Exhausted or non-retryable deliveries end as `delivery.failed` with `failedAt`.\n\nRecovery is staleness-guarded and conservative. Executing runs heartbeat `updatedAt` (default 30s, `ASSEMBLY_LINE_RUN_HEARTBEAT_MS`); only runs stuck in `created`/`running` past `max(5min, 4x heartbeat)` are swept, and each candidate is claimed through an idempotency key so concurrent replicas never double-recover. The sweep completes already-delivered runs without re-sending, cancels tool calls before side effects start, attempts one in-place resume when a harness continuation checkpoint exists, enqueues a real pending delivery for runs that reached a model response but not delivery, and marks everything else `failed` (with `run.failed`) instead of pretending it completed. `listenNodeRuntime` runs `recoverIncompleteRuns()` once at boot and `startBackgroundWorkers()` keeps the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers running until the server closes.\n\n## Usage Accounting And Observability\n\nUsage accounting is behavior-neutral observability. It never reserves expected\ntokens or cash, estimates a request, rejects a provider call, or changes an\nagent response based on local cost state. A `response_completed` event is\nnormalized into the ledger using the provider response ID when available, so\nretries are idempotent. If ledger persistence fails, the provider response\nstill completes; the durable model event and runtime warning expose the\nobservation gap.\n\nEvery record carries the run, parent run, stable agent/revision, subagent,\niteration, provider, requested and response model, provider request ID,\nbilling mode, UTC occurrence time, native input/output/cache-write/cached/\nreasoning tokens, currency, receipt hash, and sanitized provider receipt.\nActual cash uses integer `cost_micros` and only accepts\n`provider_reported`/`provider_reconciled` provenance. Local catalog-price math\nis discarded. Unavailable cash remains `null`, never `$0` or an estimate.\n\nThe ledger has three non-additive record kinds:\n\n- `transaction`: one attributable model response; this is the default report.\n- `control_total`: a provider organization/activity bucket used to verify completeness.\n- `adjustment`: reserved for explicit accounting corrections.\n\nOpenRouter [usage accounting](https://openrouter.ai/docs/cookbook/administration/usage-accounting)\nexposes native token counts and charged credits.\nThe Pi bridge follows the generation ID to obtain the settled receipt; a\ntemporarily unavailable receipt is stored as unavailable and retried by\nreconciliation through the provider's\n[generation endpoint](https://openrouter.ai/docs/api/api-reference/generations/get-generation).\n`OPENROUTER_MANAGEMENT_KEY` additionally imports the last 30 completed UTC\ndays of activity totals.\n\nThe Codex app-server supports both ChatGPT and API-key authentication. Before\neach turn, Assembly Line asks app-server which rail is active. ChatGPT mode snapshots\nprovider rate-limit/credit state for reporting only; API-key mode is labeled\n`api_key`. Snapshot failures and depleted states do not become local gates.\nApp-server token notifications are provider-reported. It does not expose a\nper-turn dollar charge, so Assembly Line records that cash as unavailable rather\nthan zero or a price-table guess.\n\nFor API-key Codex cash and organization-wide completeness,\n`POST /usage/reconcile` calls OpenAI's\n[Admin Usage and Costs APIs](https://developers.openai.com/cookbook/examples/completions_usage_api)\nwith\n`ASSEMBLY_LINE_OPENAI_ADMIN_KEY` (or `OPENAI_ADMIN_KEY`). Completions usage and cost\nbuckets become separate provider control totals. They receive an `agentId`\nonly when an operator supplies a dedicated mapping: API-key or project for\ntoken controls, and project for cash controls because the Costs API does not\ngroup by API key. An organization invoice is never proportionally allocated to runs. This means\ntoken accounting remains exact per Codex run, while exact cash is per run only\nfor providers that issue transaction receipts and otherwise remains exact at\nthe provider control-total grain.\n\nThe reconciliation report returns transaction totals, provider control\ntotals, signed variances, separate unreconciled token/cash counts, and exact\nrun-token/run-cost coverage for any `[from,to)` USD period. Schedule the authenticated reconciliation\noperation after the provider's settlement delay (48 hours by default for\nOpenAI). Control totals are excluded from ordinary transaction summaries, so\nreconciling never doubles reported spend.\n\nA transport failure after the durable `request_started` event produces an\nunobserved transaction with zero measured tokens, `null` cash, and\n`unavailable` provenance. Crash recovery does the same for interrupted\nrequests. These records express uncertainty; they are not charged against a\nlocal quota. A missing or failing usage facet emits degradation warnings but\nnever blocks provider execution or response delivery.\n\n## Node Runtime HTTP API\n\n### Auth model\n\nIn dev mode, local inspection and API-run endpoints are open for quick\niteration. In production, the Node host fails boot unless control-plane auth is\nconfigured with a host-provided auth policy or `ASSEMBLY_LINE_ADMIN_TOKEN`.\n`/manifest`, `/routes`, `/conversations`, `/conversations/:id/messages`,\n`/runs`, `/usage`, `/runs/:id`, `/runs/:id/events`,\n`/runs/:id/timeline`, and `/runs/:id/stream` require\n`Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>` when the built-in token policy is\nused. Conversation rename/archive routes use the same authenticated\nagent-control policy. `POST /conversations/:id/turns` uses the authenticated\nrun-create policy and, like `POST /runs`, is disabled in production by\ndefault; set `ASSEMBLY_LINE_ENABLE_API_RUNS=true` only for deployments that\nintentionally expose authenticated API-triggered runs. The resume and operator\nendpoints use the same admin\nauth but are **not** gated by `ASSEMBLY_LINE_ENABLE_API_RUNS`, paused and active runs\nmust remain operable even when API-triggered run creation is off. Resumes are\nsafe to replay: each one claims the run's per-pause idempotency key, so a\ndouble-submit executes the gated tool at most once, even across replicas.\n\nCompiled channel routes, such as `/slack/events` or `/message`, are also mounted from the manifest route table. In production, first-party provider helper routes remain public so the provider can call them, but their normalizers must verify signatures or shared secrets before accepting a turn. Generic `defineChannel()` HTTP routes require a host auth policy or `Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>`; the raw `message` fallback is dev-only unless the host has authenticated the request.\n\n### Endpoints\n\n| Endpoint | Purpose |\n| --- | --- |\n| `GET /health` | Liveness plus agent revision. |\n| `GET /healthz` | Liveness plus agent revision. |\n| `GET /readyz` | Readiness: `200` normally, `503` while the runtime is draining during graceful shutdown. |\n| `GET /manifest` | Compiled manifest. |\n| `GET /routes` | Compiled route table. |\n| `GET /conversations` | List durable conversations for this stable agent. Query params: `limit` (default `40`, max `100`), `cursor` (opaque, from the response's `nextCursor`), `archived` (`true` for archived only, `all` for both; default active only), `subject`, and `channel` (default `direct`). |\n| `GET /conversations/:id/messages` | Read an ordered, paginated transcript for an agent-owned conversation. Query params: `before` (opaque message cursor) and `limit` (default `50`, max `100`). |\n| `PATCH /conversations/:id` | Rename or archive a conversation with `{\"title\":\"…\",\"archived\":true}`. |\n| `POST /conversations/:id/turns` | Send a turn through the built-in `direct` transport. Accepts JSON or multipart attachments and supports the same `\"stream\": true` SSE lifecycle as `POST /runs`. |\n| `POST /runs` | Start a local/API run. Pass `\"stream\": true` to receive the whole run as server-sent events (lifecycle events, token deltas, then a final `run_result`). |\n| `GET /runs` | Query run summaries. `limit` query param defaults to `200` (max `1000`). |\n| `GET /runs/:id` | Inspect one run timeline. |\n| `GET /runs/:id/events` | Inspect raw run events. |\n| `GET /runs/:id/timeline` | Inspect grouped timeline. |\n| `GET /runs/:id/stream` | Attach to a run's live SSE stream: replays the durable event log (SSE `id:` is the event sequence, so `Last-Event-ID` reconnects resume where they left off), then tails live events including ephemeral `model.response_delta` tokens, and ends with `stream_end` after a terminal event. Works for runs started by any channel, schedule, or client. |\n| `GET /usage` | Query source-backed usage. Defaults to transaction summaries grouped by agent/provider/model/billing mode/cost source/currency/day. `view=records` returns records. Filters: `from`, `to`, `runId`, `parentRunId`, `agentId`, `agentRevision`, `subagent`, `provider`, `model`, `responseModel`, `billingMode`, `currency`, `tokenSource`, `costSource`, and `recordKind`; `groupBy` controls summary dimensions. Currency is always an aggregate boundary so unlike currencies are never added together. |\n| `POST /usage/reconcile` | Authenticated agent-control operation that imports provider control totals and retries pending receipts. Body: `{\"provider\":\"openai\",\"from\":\"ISO\",\"to\":\"ISO?\",\"providerLabel\":\"optional\"}`; `provider` may instead be `openrouter`. Provider credentials come from runtime environment variables, never the request or ledger. |\n| `POST /runs/:id/approve` | Resume a run paused on tool approval (`waiting_for_approval`): runs the gated tool and re-enters the harness. Returns the post-resume run summary. 404 if the run is unknown, 409 if it is not waiting for approval or a resume is already in flight. |\n| `POST /runs/:id/answer` | Resume a run paused on `ask_question` (`waiting_for_input`) with `{\"answer\": \"...\"}`: splices the answer as the tool result and continues. 400 without an answer, 404/409 as above. |\n| `POST /runs/:id/cancel` | Cancel a parked run synchronously (`200`) or request cooperative cancellation of a running run (`202`). |\n| `POST /runs/:id/suspend` | Request cooperative suspension of a running run (`202`). Other statuses return `409`. |\n| `POST /runs/:id/resume` | Resume a deliberately suspended run from its latest compatible harness continuation (`200`; `404`/`409` for unknown or wrong-status runs). |\n| `GET /agent/control` | Read the durable ingress control for the stable agent identity. |\n| `POST /agent/disable` / `POST /agent/enable` | Disable or enable new channel ingress (`200`) and append a control-plane audit event. |\n| `GET/POST /assembly-line/automations/tick` | Trigger due static and dynamic time-based automations from a gateway or cloud scheduler. |\n| `POST /assembly-line/automations/events` | Submit a trusted normalized provider event for matching event automations. |\n| `GET/POST /assembly-line/connections/callback` | Complete connection authorization callbacks. |\n\nThese internal routes are served only under the `/assembly-line/*` prefix;\nthe legacy `/openeve/*` spellings return `404`. Callback-URL construction\nalways emits `/assembly-line/connections/callback`.\n\n### Direct conversations\n\n`direct` is Assembly Line's built-in, provider-neutral client transport. It is the\nright channel name for a first-party app, internal console, custom web UI, or\nmobile client chatting with an agent. `custom` remains appropriate only for a\ndeveloper-defined channel adapter with its own route, normalization, delivery,\nand trust boundary.\n\nThe direct API is control-plane HTTP, not a public provider webhook. Production\nboot requires host authentication, transcript reads require admin-read\nauthority, mutations require agent-control or run-create authority, and\nconversation ids are checked against the runtime's stable agent scope before\nmessages can be read or added. A client should send\n`Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>` (or credentials accepted by the\nhost-provided auth policy) over HTTPS and must never embed an admin token in a\npublic browser bundle.\n\nConversations persist across agent revisions when `agent.id` is stable. Their\nmessages and metadata live in the configured state adapter. Inbound attachment\nbytes are normalized into the configured private blob adapter before transcript\nmetadata is returned; generated files use the same private storage boundary.\nObject storage is not made public unless application code explicitly writes a\npublic blob.\n\nCreate and stream a turn:\n\n```sh\ncurl -N https://agent.example/conversations/01JTHREAD/turns \\\n  -H \"Authorization: Bearer $ASSEMBLY_LINE_ADMIN_TOKEN\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"message\":\"Summarize the attached brief\",\"stream\":true}'\n```\n\nThe SSE response includes durable lifecycle events, ephemeral model deltas, a\nfinal `run_result`, and `stream_end`. If the connection drops after the run id\nis known, reattach through `GET /runs/:id/stream` with `Last-Event-ID`. Use an\n`Idempotency-Key` header or a stable body `eventId` when a client may retry the\ninitial POST.\n\n## Operator Controls\n\nSuspend and cancel are cooperative at the model-request boundary. The intent\nis written atomically to the durable run row, and the executing replica's\nheartbeat observes it; cross-replica detection latency is therefore at most\n`ASSEMBLY_LINE_RUN_HEARTBEAT_MS` (30 seconds by default). A tool already executing is\nallowed to finish, preserving a consistent side-effect record. Cancel wins a\nrace with suspend.\n\nSuspension applies only to `running` runs. It persists a harness continuation,\nmoves the run to `suspended`, and keeps it outside orphan recovery while still\nbounding its active checkpoints. `POST /runs/:id/resume` claims a\nper-generation idempotency key and re-enters the normal continuation path.\nParked approval/input/connection runs can be cancelled synchronously; pending\ntool records become `cancelled`.\n\nThe agent control is an ingress kill switch, not a process kill switch.\nDisabled channel ingress returns `503` without `Retry-After` and consumes\nneither capacity nor the provider event's idempotency key. In-flight runs,\nexplicit resumes, schedules, and operator access continue. The setting is\nscoped by stable `agent.id` when present (otherwise the compiled revision).\nUse `FileStateAdapter`, `PostgresStateAdapter`, or another durable\n`RuntimeSettingsStore`; a missing settings facet falls back to memory and will\nnot survive restart.\n\nRemote CLI equivalents use the same authenticated HTTP API:\n\n```sh\nassembly-line runs suspend <runId> --url https://agent.example --token \"$ASSEMBLY_LINE_ADMIN_TOKEN\"\nassembly-line runs resume <runId> --url https://agent.example --token \"$ASSEMBLY_LINE_ADMIN_TOKEN\"\nassembly-line runs cancel <runId> --url https://agent.example --token \"$ASSEMBLY_LINE_ADMIN_TOKEN\"\nassembly-line agent disable --url https://agent.example --token \"$ASSEMBLY_LINE_ADMIN_TOKEN\"\nassembly-line agent enable --url https://agent.example --token \"$ASSEMBLY_LINE_ADMIN_TOKEN\"\n```\n\n`ASSEMBLY_LINE_URL` and `ASSEMBLY_LINE_ADMIN_TOKEN` are the flag fallbacks.\n\nRequest bodies are capped before parsing. The default limit is 10 MiB; set\n`ASSEMBLY_LINE_HTTP_MAX_BODY_BYTES` or `NodeRuntimeServerOptions.maxRequestBodyBytes`\nonly when a deployment intentionally accepts larger webhook payloads.\n\nProvider channel routes remain public HTTP routes because providers such as\nSlack, Telegram, Teams, and Discord interactions call them directly. Those\nroutes must rely on their channel adapter's signature or token verification\nbefore accepting a turn. Long-lived provider ingress such as Discord Gateway is\nstarted by the Node host through `startIngress()` and feeds the same idempotent\naccepted-turn path as HTTP channels.\n\nTrusted hosts and agent-to-agent gateways may pass `principal` and `initiator`\nto `runtime.run()`. Forward canonical identity claims only after authenticating\nthe caller; do not forward provider tokens. Runs that provide only `userId`\nreceive a compatibility principal scoped to their channel.\n\n`/assembly-line/automations/tick` (and its deprecated scheduler alias) accepts optional `now` and `lookbackMs` values in the query string or JSON body. In production, set `ASSEMBLY_LINE_SCHEDULER_SECRET` and send it as `Authorization: Bearer <secret>` or `x-assembly-line-scheduler-secret`. Without a configured secret, the endpoint only accepts dev-mode runtime requests.\n\nScheduler startup comes from `gateway.scheduler`:\n\n- `adapter(\"local\")` starts an in-process loop with the Node host.\n- `adapter(\"gateway\")` registers time-based automations but relies on the endpoint or host code calling `runDueAutomations()`.\n- `adapter(\"postgres\")` starts the polling loop and coordinates duplicate workers through Postgres-backed state/idempotency. Pair it with `state: adapter(\"postgres\")`.\n\n## Durability Workers\n\n`listenNodeRuntime` runs `recoverIncompleteRuns()` once at boot, then calls\n`runtime.startBackgroundWorkers()` to keep the delivery worker, sandbox-sync\nworker, conversation-turn mailbox worker, background-review worker, and\nperiodic orphan sweep running; all five stop when the server closes. Each worker has an env kill-switch\n(`ASSEMBLY_LINE_DELIVERY_WORKER`, `ASSEMBLY_LINE_SANDBOX_SYNC_WORKER`,\n`ASSEMBLY_LINE_CONVERSATION_TURN_WORKER`, `ASSEMBLY_LINE_BACKGROUND_REVIEW_WORKER`,\n`ASSEMBLY_LINE_RUN_RECOVERY`), the heartbeat and\nsweep cadence are tunable (`ASSEMBLY_LINE_RUN_HEARTBEAT_MS`,\n`ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS`), and the delivery queue has lease, batch,\nattempt, and interval knobs. The canonical tables are in the Configuration\nReference: [Durability workers and\nrecovery](config-reference.md#durability-workers-and-recovery) and [Delivery\nqueue](config-reference.md#delivery-queue).\n\n### Model-Call Resilience\n\nEvery model request runs inside a retry-and-deadline envelope. Retryable\nfailures (408/429/5xx, provider overload, network errors) are retried with\njittered exponential backoff, `Retry-After` hints are honored and capped, while fatal failures (invalid API key, authentication, invalid request) fail\nthe run immediately with the durable reason `model.request_failed`. A stream\nthat stops producing events is aborted by an inactivity watchdog and retried.\nRetries are visible as `model.request_retried` events and warn-level log\nlines. When the retry budget is exhausted on a *transient* error, the run is\nleft `running` with a durable `run.execution_error` event so the orphan sweep\nresumes it from the latest continuation checkpoint, one transient outage\nnever terminally fails a run. Knobs: `ASSEMBLY_LINE_MODEL_MAX_RETRIES`,\n`ASSEMBLY_LINE_MODEL_TIMEOUT_MS`, `ASSEMBLY_LINE_MODEL_MAX_RETRY_DELAY_MS`,\n`ASSEMBLY_LINE_MODEL_STREAM_IDLE_TIMEOUT_MS`.\n\n### Run Deadline And Tool Timeouts\n\nThe run heartbeat doubles as a wall-clock deadline enforcer: when an active\nexecution segment exceeds `ASSEMBLY_LINE_MAX_RUN_DURATION_MS` (default 1 h), the\nin-flight model request is aborted via `AbortSignal` and the run fails with\nreason `run.max_duration_exceeded`. Parked runs (approvals, human input) hold\nno budget, the clock only ticks while the run actively executes, and it\nresets on resume. Tool `execute` calls are raced against\n`ASSEMBLY_LINE_TOOL_TIMEOUT_MS` (per-tool `timeoutMs` on the definition overrides\nit), and model-supplied `bash` timeouts are clamped to\n`ASSEMBLY_LINE_BASH_TIMEOUT_MAX_MS`, `timeoutMs: 0` falls back to the default\nrather than disabling the timeout.\n\n### Terminal Outcomes And Recovery Fidelity\n\nEvery run that reaches `failed` records a machine-readable `terminalReason`\nand human-readable `terminalError` on the run record (queryable without\nscanning the event log; also on the `run.failed` event payload). Reasons\ninclude `model.request_failed`, `run.max_iterations_exceeded`,\n`run.max_duration_exceeded`, `output.validation_exhausted`,\n`tool.execution_failed`, `connection.tool_failed`, `trigger.*_failed`, and\n`run.orphaned`. Terminal outcomes (`run.completed`/`run.failed`/\n`run.cancelled`) are always logged at a single choke point, whichever code\npath produced them; response content is never logged, only its size.\n\nCrash recovery reads the newest continuation checkpoint when reconciling a\nrun that died after its final model response: the *actual* answer is\ndelivered (`recovered: true`, `contentRecovered: true`) and the \"response was\ninterrupted\" notice is reserved for genuinely missing checkpoints. When a\ncrash interrupted a tool batch, the resume surfaces already-completed tool\noutputs and interrupted-tool warnings to the model as a recovery report so\ncompleted side effects are not blindly re-executed. Dynamic automations that\nfail repeatedly back off exponentially and are auto-disabled after\n`ASSEMBLY_LINE_SCHEDULE_MAX_FAILURES` consecutive failures (an operator-visible\n`schedule.disabled_after_failures` control event is recorded).\n\n## Concurrency And Rate Limiting\n\nAccepted provider turns are first written to a durable per-conversation FIFO\nmailbox. One turn per `(agent, conversation)` may be running or parked; later\nturns wait, while distinct conversations can use the full global concurrency\nbudget in parallel. Provider routes therefore acknowledge valid durable work\neven when all run slots are busy instead of relying on webhook redelivery for\nbackpressure. Postgres enforces the active-turn exclusion across replicas.\n\nDirect brand-new runs still pass through the bounded semaphore before run\nstate is written. When its in-memory admission queue is full the runtime\nrejects direct work with `RunCapacityError`, and the Node host maps that to\nHTTP `429` with a whole-second `Retry-After` on `POST /runs`. Resumes\n(approvals, human input, connection callbacks, orphan-recovery continuations,\nand scheduler continuations of an existing run) never queue behind the limit, queueing a resume behind the run it unblocks would deadlock, but still\ncount toward the drain performed by graceful shutdown. A terminal resume also\nreleases the next mailbox turn for that conversation.\n\nIngress rate limiting is a token bucket applied after auth on\nprovider-channel, run-create, and scheduler routes (health and admin routes\nare exempt). It is **off by default** and enabled either through\n`NodeRuntimeServerOptions.rateLimit` (see the customization guide) or through\n`ASSEMBLY_LINE_INGRESS_RATE_LIMIT` and `ASSEMBLY_LINE_RUNS_RATE_LIMIT`\n(`capacity/refillPerSecond` form, e.g. `60/10`); `ASSEMBLY_LINE_RUNS_RATE_LIMIT` also\nseeds the run-resume and run-control buckets unless those are configured\nseparately. `ASSEMBLY_LINE_MAX_CONCURRENT_RUNS` bounds simultaneously executing\nbrand-new runs (`createProductionRuntimeOptions` defaults it to `16` outside\ndev) and `ASSEMBLY_LINE_MAX_QUEUED_RUNS` (default `0`) lets excess runs wait for a\nslot. The canonical table is [Concurrency and rate\nlimiting](config-reference.md#concurrency-and-rate-limiting).\n\nRate-limited requests receive `429 { \"error\": \"Rate limited.\" }` with a\n`Retry-After` header.\n\nRate-limit buckets are keyed by route class and client address. Behind a\nreverse proxy or load balancer, set `ASSEMBLY_LINE_TRUST_PROXY=true` so the first\n`X-Forwarded-For` hop is used as the client address; without it, every proxied\nrequest shares one bucket keyed by the proxy's address, so a single noisy\nclient can exhaust the limit for everyone.\n\n## Graceful Shutdown\n\n`listenNodeRuntime` returns a `NodeRuntimeHandle` with an idempotent\n`shutdown({ timeoutMs? })` and a `closed` promise. The shutdown sequence:\nmark draining (`/readyz` starts answering `503` so load balancers stop routing\nnew traffic; `/health`/`/healthz` stay `200` for liveness) -> close the HTTP\nlistener and stop channel ingress -> stop the scheduler -> stop background\nworkers -> wait for in-flight runs up to the timeout -> flush the telemetry\nsink -> close the state adapter (Postgres ends its pool when it created it).\nRuns still executing at the timeout are abandoned safely: orphan recovery\nrepairs them on the next boot.\n\n`assembly-line serve` and `assembly-line deploy --serve` install SIGTERM/SIGINT handlers\n(`installSignalHandlers` from `@assemblyline-agents/node`): the first signal drains\ngracefully and exits `0`; a second signal exits `1` immediately.\n\nThe drain timeout is `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS` (default `30000`), and\n`ASSEMBLY_LINE_SIGNAL_HANDLERS=false` prevents handler installation; see [Graceful\nshutdown](config-reference.md#graceful-shutdown).\n\n## State And Blob Storage\n\nLocal development uses file-backed state and local blob storage. Production state should use Postgres:\n\n```ts\nimport { defineGateway } from \"@assemblyline-agents/core\";\nimport { neonPostgres } from \"@assemblyline-agents/postgres\";\nimport { r2Blob } from \"@assemblyline-agents/s3\";\n\nexport default defineGateway({\n  state: neonPostgres(),\n  blob: r2Blob()\n});\n```\n\nPostgres stores runs and atomic control intents, events, messages,\nconversations, tool calls, approvals, delivery obligations, schedules, schedule\nrun lifecycle status, memory indexes, file catalog records, sandbox leases,\nidempotent usage receipts, exact micro-dollar/token aggregates, run queues,\nidempotency keys, learned skills,\ndynamic automations including trigger metadata, dynamic connections, runtime\nsettings, conversation-scoped agent hook state with atomic aggregate revisions,\ncapability checkpoints, and control-plane audit events. PostgreSQL provides durable\nmulti-replica usage observability; file/in-memory accounting is process-local.\n\nConnection grants and OAuth authorization sessions use the durable state\nadapter when it implements those stores. The Postgres adapter does, so\nPostgres-backed production deployments do not need a separate file encryption\nsecret for connection credentials. Tool discovery never creates authorization\nsessions; explicit authorization reuses an unexpired pending session and\nprunes expired sessions before creating a replacement.\n\nWhen a production Node deployment uses file-backed connection credential\nstores, set `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` or `ASSEMBLY_LINE_SECRET` to a stable\nsecret of at least 32 characters. The built-in local development fallback is\naccepted only with `devMode: true`; production boot rejects missing, short, or\nknown development secrets.\n\nBlob storage stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. Blob records are private by default. S3/R2 `*_PUBLIC_BASE_URL` is used only when code explicitly writes a blob with `visibility: \"public\"`; context bundles, memory files, and inbound attachments should remain private.\n\n## Sandbox Sync\n\nSandbox-backed file tools write through provider workspaces, but durable\nproduction persistence remains Assembly Line state and blob sync. When a run finishes\nwith dirty sandbox files and async sync is enabled, the runtime retains the\ndirty sandbox instead of deleting it, queues a sync job, and lets final delivery\ncomplete. The sync worker first calls adapter `connect()` and then `wake()` so\npaused, stopped, detached, or otherwise retained warm sessions can still be\nsynced. After a successful sync the session is marked clean and disposed\nthrough the adapter's clean lifecycle path.\n\nHosted sandbox adapters do not use provider snapshots as the normal persistence\npath. Snapshots are created only when the configured sandbox snapshot policy\nrequests them and the provider SDK exposes snapshot creation.\n\n## Deploy Targets\n\nEvery non-local target follows one CLI sequence: resolve the publisher, run\noptional target preflight and preparation hooks, sync secrets when requested,\nrun artifact migrations once (locally or through the publisher), publish, then\nwrite the final receipt with migration status. Built-in option\nprecedence is CLI flag, environment variable, `gateway.ts` option, then\nprovider default. Community publishers use this path too and remain compatible\nwhen they omit the newer optional hooks.\n\n### Environments\n\n`--env <name>` selects the deploy environment (default: `development`, or the\nmanifest's `gateway.deploy.options.environment`). Every environment holds an\nisolated deployment of the same agent:\n\n- **Identity.** The *default* environment keeps its legacy, unscoped provider\n  resource names, so existing deployments are never orphaned. Any other\n  environment gets environment-scoped resources: Docker suffixes container and\n  volume names (`openeve-<agent>-test`) and requires an explicit `--port` in\n  serve mode; Fly derives `<app>-<environment>` when the app name comes from\n  the manifest (an explicit `--fly-app` is always respected verbatim); VPS uses\n  the host inventory's `baseDomain` and derives `<agent-id>.<baseDomain>` for\n  the default environment or `<agent-id>-<environment>.<baseDomain>` for any\n  other environment; Railway passes the environment through natively, so the\n  named Railway environment must exist. Configure one wildcard DNS record for\n  the VPS host namespace. Explicit Docker, Fly, and Railway identity flags win\n  over derived names; VPS hostnames remain host-owned and deterministic.\n- **Receipts.** Each environment's canonical receipt is written to\n  `.assembly-line/deployments/<environment>.json`. The legacy\n  `.assembly-line/deployment.json` keeps its historical last-deploy-wins behavior for\n  existing readers; new readers should prefer the environment-scoped file.\n- **Teardown.** `assembly-line deploy <agentRoot> --env <name> --destroy` removes the\n  environment's provider resources on every built-in target (docker, fly,\n  railway, vps, local). Durable data, volumes, databases, the VPS deployment\n  directory, survives unless `--purge-data` is passed; Fly and Railway require\n  it explicitly, because destroying a Fly app or deleting a Railway environment\n  always removes the volumes and databases inside it. VPS destroy never touches\n  shared host infrastructure (the Caddy edge, the shared Postgres cluster) or\n  customer-owned external databases. Destroying the default environment\n  additionally requires `--force`.\n\nComposed with `assembly-line eval --url`, this is the ephemeral test-environment\nrecipe:\n\n```sh\nassembly-line deploy agent --env test --sync-secrets --secrets-from .env.test\nassembly-line eval agent --url https://<test-gateway> --token $ASSEMBLY_LINE_ADMIN_TOKEN\nassembly-line deploy agent --env test --destroy --purge-data\n```\n\n### Local\n\nUse local deploy for developer machines or long-lived VMs:\n\n```sh\nassembly-line deploy agent --target local --serve --port 3000\n```\n\n### Railway\n\n`adapter(\"railway\")` or `railwayDeploy()` publishes the built artifact through\nthe `@assemblyline-agents/railway` deploy publisher and Railway CLI.\n\nRequired:\n\n- `RAILWAY_TOKEN` or authenticated Railway CLI\n- Linked project/service, or `--railway-project` and `--railway-service`\n\n```sh\nassembly-line deploy agent \\\n  --target railway \\\n  --railway-project prj_x \\\n  --railway-service svc_y\n```\n\nWhen the gateway uses `state: railwayPostgres()`, deployment first inspects the\nselected Railway environment. It reuses the `Postgres` database service when\npresent, otherwise provisions one with `railway add --database postgres`, and\nsets the Assembly Line service's `DATABASE_URL` to\n`${{Postgres.DATABASE_URL}}` before `railway up`. The deploy receipt records\nwhether the database was created or reused. Use\n`railwayPostgres({ databaseService: \"name\", provision: false })` to require a\nspecific existing database service without automatic creation. Automatic\ncreation uses Railway's default `Postgres` service name. A local `DATABASE_URL`\nis not required for this auto-provisioned path; the ordinary Postgres, Neon,\nSupabase, and `provision: false` paths still require one during deployment\npreflight.\n\n#### Syncing secrets to the target\n\nBy default, `deploy` sets nothing on the remote service. You configure variables\nin the provider dashboard. Pass `--sync-secrets` to push your local secrets as\npart of the deploy: the CLI reads the project `.env` (or `--secrets-from\n<path>`), overlays declared runtime requirements from the resolved host\nenvironment, and calls the publisher's `syncSecrets` before publishing, so the\nfirst deploy boots with them. Undeclared host variables are never copied. Only\nkey **names** are logged, never values; empty keys are skipped, and removing a\nlocal key does not delete the existing remote value. Variable names must use\nportable shell identifier syntax. Values containing line breaks or null bytes\nare rejected before a provider command runs. Supported on `railway` (one\n`railway variable set KEY --stdin --skip-deploys` call per key), `fly` (one\n`flyctl secrets import` stream for all keys), `vps`\n(an atomic remote `0600` environment file), and `docker` (held in memory for\nthe deploy and handed to serve-mode `docker run` through the child process\nenvironment with value-less `--env KEY` flags, never on argv or disk;\nimage-only builds never bake secrets). The local target reports that sync is\nunsupported and leaves secrets to you.\n\nPrivate dotenv files (`.env` and `.env.*`, except `.env.example` and\n`.env.*.example`) are never included in compiled artifacts. Whenever the CLI\nloads one of these files, it repairs its permissions to owner-only (`0600`).\nRailway and Fly receive runtime values through child-process stdin; values do\nnot appear in provider argv, deployment logs, or receipts. A failed secret sync\naborts the deployment before publish.\n\nCredentials required by the selected deploy adapter remain local and are not\ncopied into the agent runtime. For example, `RAILWAY_TOKEN` authorizes the\nRailway CLI and `FLY_API_TOKEN` authorizes `flyctl`; runtime requirements such\nas model, channel, state, blob, and connection credentials are eligible for\nremote sync.\n\n```sh\nassembly-line deploy agent --target railway --sync-secrets\n```\n\nThe deploy receipt (written to `.assembly-line/deployments/<environment>.json`, with\nthe legacy `.assembly-line/deployment.json` mirroring the most recent deploy) separates\n`deploymentUrl` is the reachable service URL when the provider CLI reports\none; otherwise it is `null`. `dashboardUrl` points to the provider's management\nconsole, so the\ntwo are never conflated.\n\n### Docker\n\n*Preview: this surface may change without notice.*\n\n`adapter(\"docker\")` builds the compiled artifact as a Docker image through the\n`@assemblyline-agents/docker` deploy publisher and can run it locally.\n\nServed deployments use a stable `openeve-<agent-slug>` container name. Codex\nartifacts also use the stable `openeve-<agent-slug>-data` volume mounted at\n`/data`; the slug comes from agent `id`, then `name`, then the agent folder.\n\n```sh\nassembly-line deploy agent \\\n  --target docker \\\n  --docker-image assembly-line/my-agent \\\n  --serve \\\n  --port 3000\n```\n\n### Fly\n\n*Preview: this surface may change without notice.*\n\n`adapter(\"fly\")` writes a minimal `fly.toml` and deploys the artifact with\n`flyctl` through the `@assemblyline-agents/fly` deploy publisher.\n\nCodex artifacts provision the app-scoped `openeve_data` volume in the selected\nregion. Fly deploys are limited to single-Machine apps because Fly volumes are\nMachine-local; deploy and auth fail clearly when an existing app has more than\none Machine.\n\nRequired:\n\n- `FLY_API_TOKEN`\n- `--fly-app` or `FLY_APP_NAME`\n\n```sh\nassembly-line deploy agent \\\n  --target fly \\\n  --fly-app my-agent \\\n  --fly-region iad\n```\n\nThe deployment contract has credential-free parity checks:\n`pnpm smoke:deploy:docker` exercises a real local build, run, recreation,\nremote command, and persistent volume; `pnpm smoke:deploy:fly` parses generated\nconfiguration with the installed Fly CLI and verifies the publisher's CLI\nsurface using an intentionally invalid token. Neither check creates hosted\nresources. `pnpm smoke:deploy:fly:live` is the opt-in hosted exit gate: it\ncreates a uniquely named app, deploys through the Assembly Line publisher, verifies\nHTTP health, secret sync, SSH execution, and `/data` persistence across a\nredeploy, then destroys the app and volume and verifies their absence. It\nrequires an authenticated `flyctl` session and may incur brief provider usage.\n\n### Generic VPS\n\n*Supported.*\n\n`vpsDeploy()` from `@assemblyline-agents/vps` deploys to an AMD64 Ubuntu 24.04, Ubuntu\n26.04, or Debian 12 host. Hetzner, Hostinger, DigitalOcean, OVH, Vultr, and\nsimilar hosts use the same workload publisher.\n\nHetzner hosts can be created or adopted with `assembly-line hosts bootstrap`. The\nbootstrap verifies that the public key matches the private key selected by\n`identityFileEnv`, creates an `openeve` sudo user, installs Docker Engine and\nCompose, enables UFW, fail2ban, unattended upgrades, provider backups, delete\nand rebuild protection, and a Hetzner Firewall. The provider firewall restricts\nSSH to `--ssh-source <CIDR>` unless `--allow-global-ssh` is explicitly supplied;\nUFW admits port 22 behind that provider edge so a changing workstation address\ncannot create a second, stale allowlist. If a native deploy times out before a\nhost key is returned, Assembly Line replaces stale SSH source rules on its\nHetzner firewall with the native process's current public IPv4 `/32` and retries\nonce. It never changes firewall access after a host-key mismatch. This keeps SSH\nkey-only and host-key pinned while tolerating network or full-tunnel VPN changes.\nThe command waits for cloud-init and the security services, pins the SSH host key,\nthen writes the inventory entry. Existing servers are never rebuilt\nimplicitly. Adoption of an existing named server additionally requires\n`--host-key-sha256` obtained from the provider console or another trusted path;\nAssembly Line will not establish trust from an in-band key scan alone.\n\n```sh\nexport ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY=\"$PWD/keys/assembly-line_ed25519\"\n\nassembly-line hosts bootstrap \\\n  --provider hetzner \\\n  --host production-eu \\\n  --server assembly-line-production-eu \\\n  --inventory ./assembly-line.hosts.json \\\n  --ssh-public-key ./keys/assembly-line_ed25519.pub \\\n  --base-domain agents.example.com \\\n  --identity-file-env ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY \\\n  --location ash \\\n  --server-type cpx32 \\\n  --ssh-source 198.51.100.10/32 \\\n  --expected-region ash\n```\n\nThe server must have Docker Engine, Docker Compose, `flock`, `curl`, `ss`,\nseccomp, AppArmor, and root SSH or passwordless `sudo`. Host Postgres mode\nalso requires OpenSSL and systemd. Register it by name in\n`assembly-line.hosts.json`:\n\n```json\n{\n  \"version\": 2,\n  \"hosts\": {\n    \"production-eu\": {\n      \"address\": \"203.0.113.10\",\n      \"ingress\": {\n        \"baseDomain\": \"agents.example.com\",\n        \"defaultVisibility\": \"public\"\n      },\n      \"ssh\": {\n        \"user\": \"deploy\",\n        \"port\": 22,\n        \"identityFileEnv\": \"ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY\",\n        \"hostKeySha256\": \"SHA256:replace-with-the-pinned-fingerprint\"\n      },\n      \"provider\": {\n        \"kind\": \"hetzner\",\n        \"resourceId\": \"optional-server-id\",\n        \"region\": \"ash\"\n      }\n    }\n  }\n}\n```\n\nInventory precedence is `--vps-hosts-file`, `ASSEMBLY_LINE_VPS_HOSTS_FILE` (also\nreadable as `OPENEVE_VPS_HOSTS_FILE`, the permanent env-alias mirror), then\nthe nearest `assembly-line.hosts.json` found upward from the agent root. A\nlegacy `openeve.hosts.json` is not discovered by that upward walk. The\nSSH private key path comes from `identityFileEnv`; the key and host address\nare never written to deployment receipts. Host-key scanning must match the\npinned SHA-256 fingerprint before strict SSH is allowed.\n\n```ts\nimport { defineGateway, adapter } from \"@assemblyline-agents/core\";\nimport { vpsDeploy } from \"@assemblyline-agents/vps\";\n\nexport default defineGateway({\n  deploy: vpsDeploy({\n    host: \"production-eu\",\n    environment: \"production\",\n    expectedRegion: \"ash\",\n    resources: { cpus: 1, memory: \"1g\", pids: 256 },\n    database: { mode: \"host\" },\n    monitoring: { enabled: true, diskFreeMinimumMb: 5120 }\n  }),\n  runtime: adapter(\"node\"),\n  state: adapter(\"postgres\"),\n  blob: adapter(\"r2\"),\n  sandbox: adapter(\"e2b\")\n});\n```\n\nVPS deployment requires a stable `agent.id`, Node runtime, Postgres state,\nS3/R2 blobs, a hosted sandbox, `ASSEMBLY_LINE_ADMIN_TOKEN`, and one wildcard DNS\nrecord for the host inventory's ingress base domain. Local state/blob storage and local or Docker-socket\nsandboxes are hard preflight failures. `ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL` is an\noptional notification destination; health checks continue to run and record\nfailures in systemd/journald when it is unset. `expectedRegion` compares the\nconfigured intent with inventory and warns about likely user, Photon, database,\nor sandbox latency.\n\nUse `--sync-secrets` on the first deploy. Repeat deploys can reuse the complete\nremote `0600` environment without copying runtime credentials back to the\noperator machine; the publisher validates required keys remotely before\ndatabase setup, migrations, and activation. For public agents, the VPS\npublisher sets `ASSEMBLY_LINE_PUBLIC_URL` to the derived HTTPS hostname on every\nsecret sync so callbacks and generated public links cannot retain a prior\nprovider's hostname. Private agents receive no public URL.\n\n`assembly-line secrets diff agent --target vps --env production` compares required\nand configured key names without returning remote values. If `.env` is absent,\n`--sync-secrets` still reads declared runtime variables from the command\nenvironment. An explicitly requested missing `--secrets-from` file is an\nerror.\n\n```sh\nassembly-line deploy agent \\\n  --target vps \\\n  --vps-host production-eu \\\n  --sync-secrets \\\n  --env production\n```\n\nThe host owns `agents.example.com`. The default environment for agent ID\n`support` receives `support.agents.example.com`; an alternate `staging`\nenvironment receives `support-staging.agents.example.com`. This requires one\n`*.agents.example.com` DNS record, not per-agent DNS configuration.\n\nEach agent receives a dedicated hardened non-root container, ingress network,\ndata network, `/data` volume, secret file, hostname ownership record, and\ndatabase identity. Runtime containers are read-only, drop all capabilities,\nset `no-new-privileges`, carry CPU/memory/PID/log limits, and never mount the\nDocker socket. A bounded, non-executable `/app/.assembly-line/module-cache`\ntmpfs holds generated runtime module-cache files without making the application\nroot or the rest of the Assembly Line artifact namespace writable. Runtime\nstartup probes that exact cache path and fails readiness with\n`runtime_module_cache_unwritable` when the deployment did not provide it. A\ntrusted shared Caddy container joins each ingress network but agents do not\njoin one another's networks.\n\nFor an internal-only agent, declare\n`ingress: { visibility: \"private\" }`. Private activation does not bootstrap or\nattach Caddy, request a certificate, publish a hostname, or run public\nreadiness checks. Changing visibility removes or attaches the route and\nhostname claim transactionally.\n\nReleases use immutable revision-labelled images and inactive blue/green slots.\nPreparation, activation, rollback, and ingress reconciliation are separate operations:\n\n```sh\n# Build the inactive slot, sync secrets, run migrations, but do not route traffic.\nassembly-line deploy agent --target vps --env production --sync-secrets --prepare-only\n\n# Authenticate a Codex-backed prepared release if applicable.\nassembly-line auth codex agent --target vps --env production --prepared\n\n# Activate exactly the persisted prepared revision.\nassembly-line deploy agent --target vps --env production --activate\n\n# Restore the previous runtime/route without changing durable state.\nassembly-line deploy agent --target vps --env production --rollback\n\n# Reconcile public/private ingress without rebuilding.\nassembly-line deploy agent --target vps --env production --ingress-only\n```\n\nActivation rejects stale prepared metadata, waits for container `/readyz`,\ntransactionally claims the derived hostname for public agents, reloads Caddy,\nverifies public readiness, records the prior slot, and only then removes the\nold runtime. Private activation transactionally removes any old route and\nhostname claim. Failures restore the prior ingress state. The host\nretains the active and previous revision directories and prunes older release\ndirectories and images.\n\nBefore a state cutover, use the durable maintenance fence:\n\n```sh\nassembly-line agent quiesce --url https://agent.example.com\nassembly-line agent status --url https://agent.example.com\n# perform the verified transfer\nassembly-line agent resume --url https://agent.example.com\n```\n\nQuiescence stops new ingress, schedules, delivery work, sandbox-sync work, and\nrun recovery, then waits for the reported in-flight run count to reach zero.\nThe state survives process restarts.\n\n`database.mode: \"external\"` requires `DATABASE_URL` and writes a deployment\nownership marker before migrations, refusing reuse by another agent.\n`database.mode: \"host\"` runs one private Postgres cluster and creates a\nseparate database/login role per agent. Host mode requires the\n`ASSEMBLY_LINE_VPS_BACKUP_BUCKET`, `ASSEMBLY_LINE_VPS_BACKUP_REGION`,\n`ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID`, and\n`ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY` secrets; optional\n`ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT` selects a custom S3-compatible endpoint and\n`ASSEMBLY_LINE_VPS_BACKUP_RETENTION_DAYS` defaults to 30. A daily systemd timer\ncreates a compressed dump, verifies the uploaded object, and enforces\nretention. A weekly timer downloads the newest backup and restores it into a\nscratch database. A deployment-scoped restore script remains on the VPS; it\nrequires the literal `--replace-confirmed` argument, takes a fresh backup, and\nautomatically restores the pre-restore database if the requested restore\nfails.\nMigration commands connect to the private cluster\nthrough a temporary fingerprint-pinned SSH tunnel; the tunnel closes as soon\nas the migration command finishes.\n\nTo move an external database such as Neon into host mode, quiesce the source\nfirst and keep its URL in an environment variable:\n\n```sh\nexport NEON_DATABASE_URL='postgresql://...'\nassembly-line state migrate-postgres agent \\\n  --env production \\\n  --source-url-env NEON_DATABASE_URL \\\n  --source-quiesced \\\n  --replace-target\n```\n\nThe transfer uses version-matched containerized clients, rejects an older\ntarget major, takes an offsite and local pre-transfer backup, verifies the\nuploaded dump checksum, restores into the isolated role/database, and compares\nnormalized schema plus exact per-table row counts. Any restore or verification\nfailure automatically restores the pre-transfer target. Receipts contain the\nsource environment-variable name and dump hash, never the URL.\n\nPostgres images require explicit numeric tags and default to\n`postgres:17.10-alpine`. PostgreSQL 18+ uses the official image's\n`/var/lib/postgresql` volume layout; 17 and earlier use\n`/var/lib/postgresql/data`. Change the configured image only through:\n\n```sh\nassembly-line state upgrade-postgres agent --env production --confirm-upgrade\n```\n\nThe upgrade pulls and verifies the image major, creates logical globals and\nper-database custom dumps, restores into a new versioned volume, compares exact\nrow counts, and retains the stopped previous container and volume for rollback.\nOrdinary deploys refuse an image mismatch instead of silently upgrading.\n\nThe host monitor runs every five minutes and checks the active container,\npublic `/readyz`, Postgres, backup/restore-verification units and timers, and\nfree disk space. Failures are recorded by systemd/journald and are also posted\nto `ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL` when it is configured.\n\nThis is trusted-owner process isolation, not hostile multi-tenant isolation.\nUse separate VMs or microVMs for mutually untrusted tenants.\nV1 schedules one AMD64 replica per agent; ARM64, high availability,\nmulti-replica scheduling, automatic workload deletion, and non-Hetzner\nprovider bootstrapping are deferred.\n\n\n## Codex (Preview)\n\n*Preview: this surface may change without notice.*\n\nAn agent whose primary model starts with `openai-codex/` uses the official\nCodex CLI session. That session may be authenticated through ChatGPT\nentitlements/credits or an OpenAI API key; Assembly Line reads app-server's\ncredential-free account mode and labels every receipt with the active billing\nrail.\nThe Node host starts the official Codex app-server and lets the Codex CLI\nresolve its own authentication. Assembly Line never reads or exports the OAuth\ncache, and the Codex built-in shell, filesystem, patch, web, app, MCP, and\nsubagent tools are disabled; app-server dynamic tools call back through\nAssembly Line's durable tool and approval path.\n\nThe generated container includes a pinned compatible `@openai/codex` CLI,\nsets `CODEX_HOME=/data/codex`, and puts the artifact's local npm binaries on\n`PATH`. The compiler stamps two requirements into every Codex artifact:\npersistent `/data` storage and remote command execution. Planning rejects\ndeploy targets that do not advertise both `persistent-storage` and\n`remote-exec`, and auth also requires the publisher to implement\n`runRemoteCommand`.\n\n### Deploy and authenticate\n\nDeploy and authenticate in two explicit steps:\n\n```sh\nassembly-line deploy agent --target railway --env production\nassembly-line auth codex agent --target railway --env production\nassembly-line auth codex agent --target railway --env production --status\n```\n\n`assembly-line auth codex` runs `codex login --device-auth`, then verifies it with\n`codex login status`. The remote Codex CLI performs the browser/device exchange\nand writes its own cache below `/data/codex`; Assembly Line never reads, copies,\nlogs, syncs, or records the OAuth token in environment variables, provider\nsecret stores, artifacts, or deployment receipts.\n\nRailway, Docker, Fly, and VPS satisfy the same contract:\n\n- Railway creates or reuses a service volume mounted at `/data` and executes\n  the Codex commands with Railway SSH.\n- Docker creates or reuses `openeve-<agent-slug>-data`, mounts it at `/data`\n  for the stable served container, and uses a one-shot interactive container\n  with the same image and volume for login or status. Build-only deploys can\n  therefore authenticate without starting the HTTP service.\n- Fly creates or reuses the app-scoped `openeve_data` volume in the effective\n  region, writes the `/data` mount to `fly.toml`, and uses Fly SSH. Codex auth\n  requires a deployed single-Machine app because Fly volumes are Machine-local.\n- VPS uses the active immutable image plus the agent's dedicated `/data`\n  volume, environment, and data network for one-shot device login/status\n  containers.\n\n### Credential trust boundary\n\nThe runtime service and its persistent volume are part of the credential trust\nboundary. Restrict provider and shell access, treat snapshots/backups as\ncredential-bearing, and revoke the session when the deployment is retired. Do\nnot upload a developer-machine `auth.json`; direct device authentication avoids\nduplicating that local credential.\n\nFor a personal ChatGPT subscription, use this route only on a trusted runtime\nwhose persistent cache was authenticated with `assembly-line auth codex`. Do not copy\na personal `CODEX_HOME` into a shared image or expose it as a service\ncredential. The Codex CLI persists the app-server thread there so an Assembly Line\napproval, human-input, connection, or crash-recovery resume can continue the\nsame model thread.\n\nFor headless automation, OpenAI documents managed `CODEX_ACCESS_TOKEN` use for\neligible Enterprise workspaces. For general hosted application\ntraffic, use `openai/*` with `OPENAI_API_KEY` instead. See OpenAI's\n[Codex authentication guide](https://developers.openai.com/codex/auth/) and\n[plan availability](https://help.openai.com/en/articles/11369540-using-codex-with-your-chat).\n\n```sh\nprintenv CODEX_ACCESS_TOKEN | codex login --with-access-token\n```\n\nThe deployment image must include a current `codex` executable; an outdated\nCLI fails with an update hint.\n\nChatGPT rate-limit and credit snapshots are captured only as observational\nprovider receipts; they never gate a request. API-key mode is billed through\nthe OpenAI API account and can be reconciled to the OpenAI organization\nUsage/Costs APIs. Codex app-server reports per-turn tokens but not a per-turn\ndollar charge, so exact cash remains unavailable at run grain unless the\nprovider adds such a receipt. See\n[Usage Accounting And Observability](#usage-accounting-and-observability).\n\n## Migrations\n\nIf the build artifact contains migration files under `.assembly-line/migrations`, hosted deploys require a migration runner:\n\n```sh\nassembly-line deploy agent \\\n  --migration-command ./scripts/run-assembly-line-migrations\n```\n\nThe migration process receives:\n\n- `ASSEMBLY_LINE_ARTIFACT_ROOT`\n- `ASSEMBLY_LINE_AGENT_REVISION`\n- `ASSEMBLY_LINE_DEPLOY_ENV`\n- `ASSEMBLY_LINE_MIGRATION_FILES`\n\nThe Postgres adapter records schema migrations with id, checksum, description, package version, and applied time.\n\n## Preflight\n\nUse dry-run deploys before publishing:\n\n```sh\nassembly-line deploy agent --target railway --dry-run\n```\n\nPreflight requirements are inferred from:\n\n- `gateway.ts` adapters.\n- Static connection files.\n- Static channel files.\n- The model provider prefix in `agent.ts`.\n- Artifact deployment requirements such as persistent directories and remote\n  execution.\n\nFor every non-local target, planning refuses `sandbox: adapter(\"local\")`\nunless `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true` explicitly acknowledges\nthe unconfined execution risk. Local state and local blob storage are allowed,\nbut the plan warns that container replacement can lose them.\n\nStatic stdio MCP connections launch their configured binary on the Node runtime\nhost and are closed during graceful runtime shutdown. The command is never\nrouted through a shell. Ensure the binary, working directory, and OS permissions\nexist on every replica. In particular, `@assemblyline-agents/peekaboo` only operates when\nthe Node host itself is a permitted macOS 15+ machine; deploying that agent to a\nLinux container does not create remote access back to the developer's Mac.\nPeekaboo declares `local` and `darwin` host requirements, so an incompatible\ndeployment plan is rejected before publishing and a non-macOS runtime rejects\nthe connection before process launch. Remote computer access uses the separate\n`@assemblyline-agents/computer-use` connection, Assembly Line Builder's Mac Computer Host, and an\nend-to-end encrypted relay; it is not a mode of this stdio plugin. The hosted\nruntime requires `ASSEMBLY_LINE_COMPUTER_USE_BINDING`; a self-hosted relay can also\nset `ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL`. See\n[Remote Computer Use](remote-computer-use.md).\n\nModel provider env keys:\n\n| Prefix | Env |\n| --- | --- |\n| `anthropic/` | `ANTHROPIC_API_KEY` |\n| `cerebras/` | `CEREBRAS_API_KEY` |\n| `deepseek/` | `DEEPSEEK_API_KEY` |\n| `fireworks/` | `FIREWORKS_API_KEY` |\n| `google/` | `GOOGLE_API_KEY` |\n| `groq/` | `GROQ_API_KEY` |\n| `mistral/` | `MISTRAL_API_KEY` |\n| `openai-codex/` | Codex CLI login; no model API-key env requirement |\n| `openai/` | `OPENAI_API_KEY` |\n| `openrouter/` | `OPENROUTER_API_KEY` |\n| `together/` | `TOGETHER_API_KEY` |\n| `xai/` | `XAI_API_KEY` |\n\nLiveKit voice-call tools and connections use `LIVEKIT_URL`,\n`LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`. Outbound phone-call defaults can\nalso use `LIVEKIT_OUTBOUND_TRUNK_ID` and `LIVEKIT_VOICE_AGENT_NAME`.\n`defineLiveKitConnection()` contributes the required LiveKit env to preflight.\n\nProvider setup details are in [Adapters](adapters.md).\n\n## Production Checklist\n\n- Use a stable `agent.id` for agents with learned skills or durable state.\n- Use Postgres for hosted durable state.\n- Set `ASSEMBLY_LINE_CONNECTION_STORE_SECRET` or `ASSEMBLY_LINE_SECRET` if production uses\n  file-backed connection credential stores instead of Postgres-backed stores.\n- Use S3-compatible blob storage or R2 for attachments and artifacts.\n- Use Docker or a hosted sandbox for untrusted code and shell work. The production Node helper refuses the local sandbox unless `ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true`.\n- Use `deploy --dry-run` and resolve all required preflight items.\n- Keep model provider, channel, database, blob, sandbox, and deploy credentials out of the agent folder.\n- For `openai-codex/*`, install the Codex CLI in the runtime image and follow\n  the subscription/managed-token boundary above; never bake a personal\n  `CODEX_HOME` into a shared image.\n- Set `ASSEMBLY_LINE_ADMIN_TOKEN` or provide a host auth policy before production boot.\n- Set `ASSEMBLY_LINE_ENABLE_API_RUNS=true` only when authenticated API-created runs are intended.\n- Set `ASSEMBLY_LINE_BASH_TOOL_MODE=approval` or `disabled` for agents that should not get direct shell access. The default is `enabled` in every runtime mode. Embedders can gate any tool by name via `RuntimeOptions.coreToolPolicy` (e.g. `{ write: \"disabled\" }`).\n- Set `TELEGRAM_WEBHOOK_SECRET` for Telegram channels and either `PHOTON_WEBHOOK_SIGNING_SECRET`/`PHOTON_SIGNING_SECRET` or `PHOTON_INGRESS_TOKEN`/`PHOTON_WEBHOOK_BEARER_TOKEN` for Photon channels before production boot.\n- Bound run concurrency (`ASSEMBLY_LINE_MAX_CONCURRENT_RUNS`; `createProductionRuntimeOptions` defaults to 16 outside dev) and enable ingress rate limiting (`ASSEMBLY_LINE_INGRESS_RATE_LIMIT`, `ASSEMBLY_LINE_RUNS_RATE_LIMIT`) on internet-facing hosts.\n- Set `ASSEMBLY_LINE_TRUST_PROXY=true` when the host sits behind a reverse proxy or\n  load balancer so rate limits key on the real client address.\n- Point load-balancer readiness at `GET /readyz` (drains to `503` during shutdown) and liveness at `/health`; deliver `SIGTERM` for deploys so in-flight runs drain within `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS`.\n- Verify `/health`, `/readyz`, authenticated `/manifest`, channel routes, and authenticated `/runs` after deploy.\n- Make side-effect tools idempotent and approval-gated where appropriate.\n"},{"id":"troubleshooting","sourcePath":"troubleshooting.md","title":"Troubleshooting","description":"Common Assembly Line failures, what they mean, and how to fix them.","url":"https://assemblyline.artificialillumination.co/docs/troubleshooting","rawUrl":"https://raw.githubusercontent.com/jasonbadeaux/assembly-line/main/docs/developers/troubleshooting.md","headings":[{"depth":1,"title":"Troubleshooting","anchor":"troubleshooting"},{"depth":2,"title":"Quick Index","anchor":"quick-index"},{"depth":2,"title":"Install And Build Failures","anchor":"install-and-build-failures"},{"depth":2,"title":"CLI Errors","anchor":"cli-errors"},{"depth":2,"title":"Model And Provider Key Errors","anchor":"model-and-provider-key-errors"},{"depth":2,"title":"Agent Hook Errors","anchor":"agent-hook-errors"},{"depth":2,"title":"Build Artifacts And `.assembly-line` Churn","anchor":"build-artifacts-and-assembly-line-churn"},{"depth":2,"title":"401, 403, And 429 From The HTTP API","anchor":"401-403-and-429-from-the-http-api"},{"depth":2,"title":"Connection Store Secrets","anchor":"connection-store-secrets"},{"depth":2,"title":"Deploy Preflight Failures","anchor":"deploy-preflight-failures"},{"depth":2,"title":"Webhooks And Scheduled Runs Rejected","anchor":"webhooks-and-scheduled-runs-rejected"},{"depth":2,"title":"Runs Stuck, Deliveries Missing, Sandbox Writes Lost","anchor":"runs-stuck-deliveries-missing-sandbox-writes-lost"},{"depth":2,"title":"Still Stuck?","anchor":"still-stuck"}],"content":"# Troubleshooting\n\nCommon failures, what they mean, and how to fix them. Setup steps live in\n[Getting Started](getting-started.md); every environment variable referenced\nhere is defined in the [Configuration Reference](config-reference.md).\n\n## Quick Index\n\n| Error | Section |\n| --- | --- |\n| `assembly-line: command not found` | [CLI Errors](#cli-errors) |\n| `Unknown command: <cmd>. Run \"assembly-line help\" for usage.` | [CLI Errors](#cli-errors) |\n| `Agent root not found: <path>` | [CLI Errors](#cli-errors) |\n| `Hint: the model prefix in agent.ts decides the required env var ...` | [Model And Provider Key Errors](#model-and-provider-key-errors) |\n| `Model catalog unavailable.` | [Model And Provider Key Errors](#model-and-provider-key-errors) |\n| `invalid-agent-hooks` | [Agent Hook Errors](#agent-hook-errors) |\n| `agent.hook_evaluation_failed` | [Agent Hook Errors](#agent-hook-errors) |\n| `state.degraded` | [Agent Hook Errors](#agent-hook-errors) |\n| `Production OpenEve Node runtime requires OPENEVE_ADMIN_TOKEN ...` | [401, 403, And 429 From The HTTP API](#401-403-and-429-from-the-http-api) |\n| `401 admin_auth_required` / `403 admin_auth_invalid` | [401, 403, And 429 From The HTTP API](#401-403-and-429-from-the-http-api) |\n| `403 API run creation is disabled.` | [401, 403, And 429 From The HTTP API](#401-403-and-429-from-the-http-api) |\n| `429 Rate limited.` / `429 Run capacity exhausted. Retry later.` | [401, 403, And 429 From The HTTP API](#401-403-and-429-from-the-http-api) |\n| `File-backed connection credential stores require OPENEVE_CONNECTION_STORE_SECRET ...` | [Connection Store Secrets](#connection-store-secrets) |\n| `<bin> not found. Install it, or pass the matching --*-bin flag` | [Deploy Preflight Failures](#deploy-preflight-failures) |\n| `Migration files are present, but no migration runner was configured.` | [Deploy Preflight Failures](#deploy-preflight-failures) |\n| `Failed to load plugin package ...` | [Deploy Preflight Failures](#deploy-preflight-failures) |\n| `403 scheduler_unauthorized` | [Webhooks And Scheduled Runs Rejected](#webhooks-and-scheduled-runs-rejected) |\n| `Production provider ingress requires webhook authentication: ...` | [Webhooks And Scheduled Runs Rejected](#webhooks-and-scheduled-runs-rejected) |\n\n## Install And Build Failures\n\n- **`pnpm install` fails or picks the wrong pnpm.** The repo pins\n  `pnpm@10.x` through the `packageManager` field. Use Corepack instead of a\n  globally installed pnpm: `corepack enable`, then `corepack pnpm install`.\n- **Engine errors during install.** Assembly Line requires Node `>=22.19.0`\n  (declared in `engines`). Check `node --version` and upgrade before\n  reinstalling.\n- **`pnpm assembly-line` fails with a module-not-found error.** The CLI runs from\n  built output; run `pnpm build` first. This also applies after pulling\n  changes that touch any `packages/*/src`.\n- **Typecheck or tests fail on a fresh clone.** Run in order:\n  `pnpm install`, `pnpm build`, `pnpm check`, `pnpm test`.\n\n## CLI Errors\n\n- **`assembly-line: command not found`.** The global `assembly-line` binary comes from the\n  published `@assemblyline-agents/sdk` npm package. When it is not installed, including\n  whenever you work from a source checkout, run the same commands through the\n  repo workspace instead: `pnpm assembly-line <command>` resolves the built\n  `@assemblyline-agents/cli` binary (run `pnpm build` first).\n- **`pnpm assembly-line` prints nothing useful.** Run `pnpm assembly-line help` for the\n  full command list or `pnpm assembly-line help <command>` (or\n  `pnpm assembly-line <command> --help`) for per-command flags.\n- **`Unknown command: <cmd>. Run \"assembly-line help\" for usage.`** Check the\n  spelling against `pnpm assembly-line help`; commands are `init`, `add`, `validate`,\n  `manifest`, `build`, `dev`, `run`, `eval`, `serve`, `channels`,\n  `checkpoints`, `auth`, `hosts`, `state`, `secrets`, `runs`, `agent`, `deploy`,\n  `models`, and `help`.\n- **`Agent root not found: <path>`.** The positional argument (or `--root`)\n  must point at an agent folder. Run `assembly-line init <path>` to scaffold one.\n\n## Model And Provider Key Errors\n\n- **A model run fails before the first response.** The model prefix in\n  `agent.ts` decides the required env var: `openai/*` needs `OPENAI_API_KEY`,\n  `anthropic/*` needs `ANTHROPIC_API_KEY`, and so on (full table in\n  [Runtime And Deployment](runtime-and-deployment.md#preflight)). The CLI\n  prints the same hint when a provider auth error reaches the top level:\n  `Hint: the model prefix in agent.ts decides the required env var (e.g.\n  openai/* needs OPENAI_API_KEY).`\n- **`openai-codex/*` says the Codex CLI is missing.** Install a current Codex\n  CLI in the same environment that runs the Node host, then run `codex login`.\n- **`openai-codex/*` reports an authentication failure.** Run\n  `codex login status`, then `codex login` if needed. This prefix does not use\n  `OPENAI_API_KEY`; it uses the CLI-owned ChatGPT session. Assembly Line does not\n  read or copy the OAuth cache.\n- **`openai-codex/*` says dynamic tools or the experimental API are not\n  supported.** Update the Codex CLI. The preview bridge requires app-server's\n  dynamic-tool protocol and deliberately does not fall back to a private HTTP\n  endpoint.\n- **`validate` warns `unknown-model`.** The model in `agent.ts` is not in the\n  local model catalog. It is a warning, never an error. Custom providers\n  still work at runtime, so first check for typos. `pnpm assembly-line models`\n  lists the catalog (`--provider openai` filters); `--no-model-check` skips\n  the check entirely.\n- **`assembly-line models` prints `Model catalog unavailable.`** The catalog comes\n  from `@assemblyline-agents/pi`; make sure the workspace is built (`pnpm build`). When\n  the catalog cannot load, `validate` silently skips the model check rather\n  than failing.\n\n## Agent Hook Errors\n\n- **`validate` reports `invalid-agent-hooks`.** Make `setup()` synchronous and\n  ensure every baseline path calls `useModel()`. Calls to `useModel()`,\n  `usePersistentState()`, `useReasoning()`, `useTool()`, `useSkill()`,\n  `useConnection()`, `useSubagent()`, and `useSandbox()` must use\n  string literals where the compiler requires them. The issue includes the\n  source file and line to fix.\n- **A run fails with `agent.hook_evaluation_failed`.** Inspect\n  `GET /runs/:id/events` for the error and evaluation reason. Common causes\n  include a setup path without a model, conflicting singular hooks, writing\n  persistent state during `setup()`, and exceeding the 50-snapshot limit.\n  Keep `setup()` pure. Move I/O and state writes into tools or event handlers.\n- **The host logs `state.degraded` with `agentState` or `conversationTurns`.**\n  The configured state backend omitted those optional facets, so the runtime\n  is using in-memory fallbacks. Runs still work, but hook state or queued\n  conversation turns will not survive a restart. Implement those facets or\n  use the file or Postgres state adapter before production.\n\n## Build Artifacts And `.assembly-line` Churn\n\n- **A command fails while rebuilding an example's `.assembly-line/` directory.**\n  Point the artifact somewhere disposable with\n  `--out /private/tmp/assembly-line-minimal` (supported by `build`, `run`, `dev`,\n  `serve`, and `deploy`), or delete the generated `.assembly-line/` directory and\n  rebuild.\n- **Example artifacts pile up.** `pnpm clean:artifacts` removes ignored\n  `examples/**/.assembly-line` (and legacy `examples/**/.openeve`) directories;\n  `pnpm clean` removes package `dist/`\n  output; `pnpm clean:tmp` removes `openeve-*` temp directories.\n- **Generated files show up in `git status`.** `.assembly-line/`, `dist/`,\n  `node_modules/`, and env files are local build output and should stay out\n  of commits.\n- **A rebuild did not restart `dev --watch`.** Restarts only happen when the\n  agent revision changes; the CLI prints\n  `Rebuilt: no manifest change; keeping the running server.` for cosmetic\n  edits. While the folder is invalid, the previous server keeps serving and\n  the CLI prints the validation issues until the folder is valid again.\n\n## 401, 403, And 429 From The HTTP API\n\n- **Production boot fails with\n  `Production OpenEve Node runtime requires OPENEVE_ADMIN_TOKEN or a\n  host-provided auth policy to protect control-plane routes.`** Set\n  `ASSEMBLY_LINE_ADMIN_TOKEN` (or pass an `auth` policy from host code). Local\n  `assembly-line serve` runs in dev mode, where inspection endpoints are open; a\n  deployed artifact boots in production mode.\n- **`401 admin_auth_required` / `403 admin_auth_invalid`.** `/manifest`,\n  `/routes`, and the `/runs*` inspection endpoints require\n  `Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>` in production.\n- **`403 API run creation is disabled.`** Production `POST /runs` is off by\n  default. Set `ASSEMBLY_LINE_ENABLE_API_RUNS=true` and authenticate with the admin\n  token; leave it off unless API-triggered runs are intended.\n- **`429 { \"error\": \"Rate limited.\" }`.** Ingress rate limiting is enabled\n  (`ASSEMBLY_LINE_INGRESS_RATE_LIMIT` / `ASSEMBLY_LINE_RUNS_RATE_LIMIT` or the host\n  `rateLimit` option). The `Retry-After` header says when to retry.\n- **`429 Run capacity exhausted. Retry later.`** The run concurrency cap\n  (`ASSEMBLY_LINE_MAX_CONCURRENT_RUNS`, production default 16) rejected a direct\n  brand-new run such as `POST /runs` or a legacy synchronous custom channel.\n  Accepted provider webhooks are durably queued per conversation and do not\n  return this capacity error.\n- **Load balancer keeps routing during deploys.** Point readiness at\n  `GET /readyz`. It returns `503 { \"draining\": true }` during graceful\n  shutdown. Use `/health` or `/healthz` for liveness; both stay `200`.\n  Deliver `SIGTERM` so in-flight runs drain within\n  `ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS` (default 30s); a second signal exits\n  immediately.\n\n## Connection Store Secrets\n\nFile-backed connection credential stores are encrypted, and production boot\nvalidates the secret:\n\n- **`File-backed connection credential stores require\n  OPENEVE_CONNECTION_STORE_SECRET or OPENEVE_SECRET outside dev mode.`** Set\n  one of the two, or use Postgres state (the Postgres adapter implements the\n  grant stores directly and needs no file secret).\n- **`... must be at least 32 characters outside dev mode.`** The secret has a\n  hard 32-character minimum in production.\n- **`... cannot use the local development secret outside dev mode.`** The\n  built-in dev fallback value is rejected in production; generate a real\n  secret.\n\nRotating the secret makes previously encrypted grant files unreadable, plan\nrotation as a re-authorization event.\n\n## Deploy Preflight Failures\n\n- **`deploy --dry-run` reports missing setup.** Each preflight item names a\n  required env var or provider setup step inferred from `gateway.ts`,\n  channels, connections, and the model prefix. Satisfy every required item\n  before publishing.\n- **Railway:** requires `RAILWAY_TOKEN` (or an authenticated Railway CLI) and\n  a linked project/service or explicit `--railway-project` and\n  `--railway-service`.\n- **Docker:** requires Docker locally; the image tag comes from\n  `--docker-image` or `ASSEMBLY_LINE_DOCKER_IMAGE`.\n- **Fly:** requires `FLY_API_TOKEN` and `--fly-app` or\n  `FLY_APP_NAME` (`Fly deploys require --fly-app or FLY_APP_NAME.`).\n- **VPS:** requires a named `assembly-line.hosts.json` entry, a verified\n  `SHA256:` SSH host-key fingerprint, the inventory-selected identity-file\n  environment variable, supported AMD64 Ubuntu 24.04/26.04 or Debian 12,\n  Docker Compose, and available ports 80/443. Create or adopt a Hetzner host\n  with `assembly-line hosts bootstrap`; it does not report success until cloud-init,\n  Docker, UFW, fail2ban, and unattended upgrades are active. First deploys\n  need `--sync-secrets` unless a complete remote runtime environment already\n  exists.\n- **`SSH host-key fingerprint mismatch`:** verify the new fingerprint through\n  the provider console or another trusted path. Do not replace the pin based\n  only on the key returned by the same network connection.\n- **A VPS release rolls back after readiness:** inspect the blue/green runtime\n  containers and `openeve-caddy` on the host. Assembly Line restores the previous\n  route when container health, Caddy validation/reload, or public HTTPS\n  `/readyz` fails.\n- **A derived hostname is already owned:** another deployment has the\n  host-local ownership claim. Confirm that agent IDs and environments are\n  unique within the host namespace, then run `deploy --ingress-only`; never\n  delete ownership files merely to bypass the collision. Ingress reconciliation\n  retains the old claim until the new Caddy route passes public readiness and\n  rolls the new claim back on failure.\n- **A prepared release cannot activate:** `--activate` is revision-fenced. Run\n  it from the same agent revision that produced `--prepare-only`. If another\n  prepare superseded it, prepare the intended revision again.\n- **Postgres image mismatch:** ordinary deployment refuses to recreate the\n  cluster under a different image. Review downtime and backups, then run\n  `assembly-line state upgrade-postgres <agentRoot> --confirm-upgrade`.\n- **Postgres transfer verification failed:** Assembly Line compares normalized\n  schema and exact per-table counts and restores the pre-transfer target on\n  failure. Keep the source quiesced, inspect the reported failure, and do not\n  resume traffic until a later verified transfer succeeds.\n- **`<bin> not found. Install it, or pass the matching --*-bin flag`.** The\n  deploy path shells out to `railway`, `docker`, `flyctl`, or the VPS SSH\n  client; install the\n  binary or point `--railway-bin`/`--docker-bin`/`--fly-bin` at it.\n- **`Migration files are present, but no migration runner was configured.`**\n  The artifact contains `.assembly-line/migrations`; pass `--migration-command` or\n  set `ASSEMBLY_LINE_MIGRATION_COMMAND`.\n- **Community plugin providers:** a `provider-package-unresolved` validation\n  warning means the compiler could not import the `packageName` given to\n  `adapter(kind, opts, { package })` from the agent root, so preflight falls\n  back to a generic requirement. Install the package where the agent builds.\n  At boot, the Node host re-resolves it and fails with a specific error when\n  the package is missing (`Failed to load plugin package ...`), does not\n  export `assemblyLineProvider`, or has no registration for the role/kind. See\n  [Authoring Plugin Providers](authoring-adapters.md).\n\n## Webhooks And Scheduled Runs Rejected\n\n- **`403 scheduler_unauthorized` on `/assembly-line/automations/tick` (or its deprecated scheduler alias).** Set\n  `ASSEMBLY_LINE_SCHEDULER_SECRET` and send it as `Authorization: Bearer <secret>`\n  or `x-assembly-line-scheduler-secret`. Without a configured secret the endpoint\n  only accepts dev-mode requests.\n- **Production boot fails with `Production provider ingress requires webhook\n  authentication: ...`.** A channel declared ingress secrets\n  (`ingress.requiredSecretEnv`, any-of groups) and none of its groups is\n  fully set, for example Telegram needs `TELEGRAM_WEBHOOK_SECRET`, and\n  Photon needs a signing secret or a bearer token. Dev mode logs a warning\n  instead of failing. Set the secrets named in the error.\n- **Webhooks return `401` even though boot succeeded.** Boot checks that the\n  secrets exist; each request is still verified by the channel module (for\n  example Telegram compares `x-telegram-bot-api-secret-token` constant-time).\n  Make sure the provider-side webhook config sends the same secret.\n\n## Runs Stuck, Deliveries Missing, Sandbox Writes Lost\n\n- **Deliveries sit in `pending` and never send.** The delivery worker drains\n  the durable queue; check that `ASSEMBLY_LINE_DELIVERY_WORKER` is not set to\n  `false`/`0` and that the host called `startBackgroundWorkers()`\n  (`listenNodeRuntime` does this automatically).\n- **A run completed but the user got no message yet.** A retryable send\n  failure defers the delivery instead of failing the run: the run event log\n  shows `delivery.deferred` with the error, attempt count, and\n  `nextAttemptAt`, and the worker retries with backoff up to\n  `ASSEMBLY_LINE_DELIVERY_QUEUE_MAX_ATTEMPTS` (default 5) before a terminal\n  `delivery.failed`.\n- **Runs stuck in `running` after a crash or redeploy.** Orphan recovery\n  sweeps runs stale past `max(5min, 4x heartbeat)`: already-delivered runs\n  complete, runs with a continuation checkpoint get one resume attempt, runs\n  with a model response get a real pending delivery, and everything else is\n  marked `failed`. It runs at boot and every\n  `ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS`; the kill-switch is\n  `ASSEMBLY_LINE_RUN_RECOVERY=false`.\n- **Sandbox writes (memory, skills, `/workspace` files) not persisting.** The\n  sandbox-sync worker (`ASSEMBLY_LINE_SANDBOX_SYNC_WORKER`) performs the writeback;\n  use `sandboxSyncDiagnostics()` and `inspectSandboxSyncJob(jobId)` to see\n  due/leased/expired/blocked jobs, and `retrySandboxSyncJob(jobId)` after\n  fixing a blocked one.\n- **A sandbox reports `/home/user`, `/root`, or another cwd instead of\n  `/workspace`.** Current built-in hosted adapters reject that session during\n  initialization; do not rewrite command strings or add a symlink in agent\n  code. Confirm the deployment contains the current adapter packages and\n  inspect its provider metadata for `openeve.filesystemContractVersion`.\n  Sandboxes without the current version are deliberately not reconnected.\n- **`workingDirectory must be /workspace`.** Remove the alias or set it to\n  `/workspace`. The field cannot remap absolute paths embedded in shell\n  commands. For local testing of those absolute paths, use Docker instead of\n  the Local adapter.\n- **Kill-switches for debugging:** `ASSEMBLY_LINE_DELIVERY_WORKER`,\n  `ASSEMBLY_LINE_SANDBOX_SYNC_WORKER`, and `ASSEMBLY_LINE_RUN_RECOVERY` each accept\n  `false`/`0`. Every background behavior has one; see the tables in\n  [Runtime And Deployment](runtime-and-deployment.md#durability-workers).\n\n## Still Stuck?\n\nInspect the durable record: `.assembly-line/manifest.json`,\n`.assembly-line/preflight.json`, and `.assembly-line/route-table.json` for compile-time\nsurprises. Use `GET /runs/:id/events` (or the state file in development) for\nruntime behavior. Every model step, tool call, pause, delivery attempt, and\nrecovery action is recorded as an event.\n"}]}