@action-llama/skill 0.23.8 → 0.24.2

package/skills/al/debugging.md

@@ -0,0 +1,1075 @@
+# Agent Commands
+
+Agents running in Docker containers have access to shell commands for persisting environment variables, signaling the scheduler, calling other agents, and coordinating with resource locks. These commands are installed at `/tmp/bin/` and taught to agents via a preamble injected before `SKILL.md`.
+
+## Environment Commands
+
+### `setenv`
+
+Persist an environment variable across bash commands. Each bash command the agent runs starts in a fresh shell, so variables set with `export` are lost between commands. `setenv` makes them stick.
+
+```bash
+setenv <NAME> <value>
+```
+
+**First bash command — set variables:**
+
+```bash
+setenv REPO "acme/app"
+setenv ISSUE_NUMBER 42
+```
+
+**Later bash command — variables are still available:**
+
+```bash
+gh issue view $ISSUE_NUMBER --repo $REPO
+```
+
+### How it works
+
+`setenv` writes each variable to `/tmp/env.sh`, which is automatically sourced at the start of every bash command. The variable is also exported immediately in the current shell, so it's available right away in the same command.
+
+## Signal Commands
+
+Signal commands write signal files that the scheduler reads after the session ends.
+
+### `al-rerun`
+
+Request an immediate rerun to drain remaining backlog. Without this, the scheduler treats the run as complete and waits for the next scheduled tick.
+
+```bash
+al-rerun
+```
+
+- Only applies to **scheduled** runs. Webhook-triggered and agent-called runs do not re-run.
+- Reruns continue until the agent completes without calling `al-rerun`, hits an error, or reaches the `maxReruns` limit (default: 10).
+
+### `al-status "<text>"`
+
+Update the status text shown in the TUI and web dashboard.
+
+```bash
+al-status "reviewing PR #42"
+al-status "found 3 issues to work on"
+```
+
+### `al-return "<value>"`
+
+Return a value to the calling agent. Used when this agent was invoked via `al-subagent`.
+
+```bash
+al-return "PR looks good. Approved with minor suggestions."
+al-return '{"approved": true, "comments": 2}'
+```
+
+The calling agent receives this value when it calls `al-subagent-wait`.
+
+### `al-exit [code]`
+
+Terminate the agent with an exit code indicating an unrecoverable error. Defaults to exit code 15.
+
+```bash
+al-exit          # exit code 15
+al-exit 1        # exit code 1
+```
+
+## Call Commands
+
+Agent-to-agent calls allow agents to delegate work and collect results. These commands require the gateway (`GATEWAY_URL` must be set).
+
+### `al-subagent <agent>`
+
+Call another agent. Pass context via stdin. Returns a JSON response with a `callId`.
+
+```bash
+echo "Review PR #42 on acme/app" | al-subagent reviewer
+```
+
+**Response:**
+
+```json
+{"ok": true, "callId": "abc123"}
+```
+
+**Errors:**
+
+```json
+{"ok": false, "error": "self-call not allowed"}
+{"ok": false, "error": "queue full"}
+```
+
+### `al-subagent-check <callId>`
+
+Non-blocking status check on a call. Never blocks.
+
+```bash
+al-subagent-check abc123
+```
+
+**Response:**
+
+```json
+{"status": "pending"}
+{"status": "running"}
+{"status": "completed", "returnValue": "PR approved."}
+{"status": "error", "error": "timeout"}
+```
+
+### `al-subagent-wait <callId> [...] [--timeout N]`
+
+Wait for one or more calls to complete. Polls every 5 seconds. Default timeout: 900 seconds.
+
+```bash
+al-subagent-wait abc123 --timeout 600
+al-subagent-wait abc123 def456 --timeout 300
+```
+
+**Response:**
+
+```json
+{
+  "abc123": {"status": "completed", "returnValue": "PR approved."},
+  "def456": {"status": "completed", "returnValue": "Tests pass."}
+}
+```
+
+### Complete call example
+
+```bash
+# Fire multiple calls
+REVIEW_ID=$(echo "Review PR #42 on acme/app" | al-subagent reviewer | jq -r .callId)
+TEST_ID=$(echo "Run full test suite for acme/app" | al-subagent tester | jq -r .callId)
+
+# ... do other work ...
+
+# Collect results
+RESULTS=$(al-subagent-wait "$REVIEW_ID" "$TEST_ID" --timeout 600)
+echo "$RESULTS" | jq ".\"$REVIEW_ID\".returnValue"
+echo "$RESULTS" | jq ".\"$TEST_ID\".returnValue"
+```
+
+### Call rules
+
+- An agent cannot call itself (self-calls are rejected)
+- If all runners for the target agent are busy, the call is queued (up to `workQueueSize`, default: 100)
+- Call chains are allowed (A calls B, B calls C) up to `maxCallDepth` (default: 3)
+- Called runs do not re-run — they respond to the single call
+- The called agent receives a `<skill-subagent>` block with the caller name and context
+- To return a value, the called agent uses `al-return`
+
+## Lock Commands
+
+Resource locks prevent multiple agent instances from working on the same resource. Lock keys use URI format (e.g. `github://acme/app/issues/42`).
+
+### `rlock`
+
+Acquire an exclusive lock on a resource.
+
+```bash
+rlock "github://acme/app/issues/42"
+```
+
+**Success:**
+
+```json
+{"ok": true}
+```
+
+**Already held:**
+
+```json
+{"ok": false, "holder": "dev-abc123", "heldSince": "2025-01-15T10:30:00Z"}
+```
+
+**Deadlock detected:**
+
+```json
+{"ok": false, "reason": "possible deadlock detected", "cycle": ["dev-abc", "github://acme/app/pr/10", "dev-def", "deploy://api-prod"]}
+```
+
+### `runlock`
+
+Release a lock. Only the holder can release.
+
+```bash
+runlock "github://acme/app/issues/42"
+```
+
+**Success:**
+
+```json
+{"ok": true}
+```
+
+**Not holder:**
+
+```json
+{"ok": false, "reason": "not the lock holder"}
+```
+
+### `rlock-heartbeat`
+
+Reset the TTL on a held lock. Use during long-running work to prevent the lock from expiring.
+
+```bash
+rlock-heartbeat "github://acme/app/issues/42"
+```
+
+**Success:**
+
+```json
+{"ok": true, "expiresAt": "2025-01-15T11:00:00Z"}
+```
+
+### Example in SKILL.md
+
+Reference lock commands directly in your `SKILL.md` workflow:
+
+```markdown
+## Workflow
+
+1. List open issues labeled "agent" in repos from `<skill-config>`
+2. For each issue:
+   - rlock "github://owner/repo/issues/123"
+   - If the lock fails, skip this issue — another instance is handling it
+   - Clone the repo, create a branch, implement the fix
+   - Open a PR and link it to the issue
+   - runlock "github://owner/repo/issues/123"
+3. If you completed work and there may be more issues, run `al-rerun`
+```
+
+The preamble teaches the agent the lock commands, their responses, and the URI key format.
+
+### Lock authentication
+
+Each container gets a unique per-run secret. Lock requests are authenticated with this secret, so only the container that acquired a lock can release or heartbeat it. There is no way for one agent instance to release another's lock.
+
+### Auto-release on exit
+
+When a container exits — whether it finishes successfully, hits an error, or times out — all of its locks are released automatically by the scheduler.
+
+See [Resource Locks](/concepts/resource-locks) for a complete description of the locking system.
+
+---
+
+# Runtime Context
+
+When your agent runs, Action Llama assembles a prompt from several sources and passes it to the LLM as a single user message. Your `SKILL.md` body becomes the system prompt; everything below is the **user prompt** your agent receives alongside it.
+
+Understanding this structure helps you write better `SKILL.md` instructions — you can reference the injected blocks by name, avoid duplicating information that's already provided, and tailor your instructions to complement the runtime context.
+
+## Prompt Structure
+
+Here's the full user prompt for a webhook-triggered agent with a GitHub token credential:
+
+```xml
+<agent-config>
+{"repo":"acme/widgets","labels":["bug","triage"]}
+</agent-config>
+
+<credential-context>
+Credential files are mounted at `/credentials/` (read-only).
+
+Environment variables already set from credentials:
+- `GITHUB_TOKEN` / `GH_TOKEN` — use `gh` CLI and `git` directly
+
+Use standard tools directly: `gh` CLI, `git`, `curl`.
+
+Git clone protocol: Always clone repos via SSH...
+
+Anti-exfiltration policy:
+[security instructions omitted]
+</credential-context>
+
+<environment>
+Filesystem: The root filesystem is read-only. `/tmp` is the only writable directory.
+Use `/tmp` for cloning repos, writing scratch files, and any other disk I/O.
+Your working directory is `/app/static` which contains your agent files.
+
+Environment variables: Use `setenv NAME value` to persist variables across bash commands.
+See the agent commands reference for details.
+</environment>
+
+<webhook-trigger>
+{"source":"github","event":"issues","action":"opened","repo":"acme/widgets",
+ "number":42,"title":"Login button broken on Safari","body":"Steps to reproduce...",
+ "url":"https://github.com/acme/widgets/issues/42","author":"jdoe",
+ "labels":["bug"],"sender":"jdoe","timestamp":"2026-03-24T14:30:00Z",
+ "receiptId":"wh_abc123"}
+</webhook-trigger>
+
+A webhook event just fired. Review the trigger context above and take appropriate action.
+```
+
+Let's walk through each section.
+
+## Agent Config
+
+```xml
+<agent-config>
+{"repo":"acme/widgets","labels":["bug","triage"]}
+</agent-config>
+```
+
+This is the JSON serialization of the `params` field from your agent's `config.toml`. Use it to pass configuration values that your agent's instructions can reference — repo names, label filters, thresholds, or any structured data your agent needs.
+
+Your `SKILL.md` instructions can reference this directly, e.g.: *"Read the repo and labels from `<agent-config>` to determine which issues to process."*
+
+See [Agent Config Reference](/reference/agent-config#params) for details on the `params` field.
+
+## Credential Context
+
+```xml
+<credential-context>
+Credential files are mounted at `/credentials/` (read-only).
+
+Environment variables already set from credentials:
+- `GITHUB_TOKEN` / `GH_TOKEN` — use `gh` CLI and `git` directly
+
+Use standard tools directly: `gh` CLI, `git`, `curl`.
+...
+</credential-context>
+```
+
+This block tells the agent which credentials are available and how to use them. Each credential type defines its own context line — for example, a GitHub token credential explains that `GITHUB_TOKEN` is set and the `gh` CLI is ready to use.
+
+The block also includes SSH clone instructions and a **security policy** that instructs the agent never to leak credentials in logs, comments, or API calls.
+
+You don't need to repeat any of this in your `SKILL.md`. The agent already knows it can use `gh` and `git` — your instructions just need to say *what* to do, not *how* to authenticate.
+
+See [Credentials Reference](/reference/credentials) for all credential types and their injected context.
+
+## Environment
+
+```xml
+<environment>
+Filesystem: The root filesystem is read-only. `/tmp` is the only writable directory.
+...
+Environment variables: Use `setenv NAME value` to persist variables across bash commands.
+</environment>
+```
+
+Describes the filesystem constraints. In Docker mode, the agent learns that `/tmp` is writable and the root filesystem is read-only. In [host-user mode](/reference/agent-config#runtime), the agent's working directory is `/tmp/al-runs/<instance-id>/`. The `setenv` command persists environment variables in both modes.
+
+See [Environment Commands](/reference/agent-commands#environment-commands) for `setenv` details, and [Container Filesystem](/concepts/agents#container-filesystem) for the full mount table.
+
+## Trigger Context
+
+The final section varies by how the agent was triggered. This is the only part of the prompt that changes between runs.
+
+### Webhook
+
+```xml
+<webhook-trigger>
+{"source":"github","event":"issues","action":"opened","repo":"acme/widgets",...}
+</webhook-trigger>
+
+A webhook event just fired. Review the trigger context above and take appropriate action.
+```
+
+Contains the full webhook payload as JSON — source, event type, action, repo, issue/PR details, sender, timestamp, and a receipt ID for replay. Your `SKILL.md` instructions should describe how to handle the events your agent subscribes to.
+
+See [Webhooks Reference](/reference/webhooks) for the full payload schema.
+
+### Scheduled
+
+```
+You are running on a schedule. Check for new work and act on anything you find.
+```
+
+No structured data — the agent is expected to go find work on its own (poll for open issues, check a queue, etc.).
+
+### Manual
+
+```
+You have been triggered manually. Check for new work and act on anything you find.
+```
+
+Same as scheduled. If you pass a prompt to `al run`, the agent instead receives:
+
+```xml
+<user-prompt>
+Your prompt text here
+</user-prompt>
+
+You have been given a specific task. Complete the task described above.
+```
+
+### Agent Call
+
+```xml
+<agent-call>
+{"caller":"orchestrator","context":"Find competitors for Acme in the CRM space"}
+</agent-call>
+
+You were called by the "orchestrator" agent. Review the call context above,
+do the requested work, and use `al-return` to send back your result.
+```
+
+Contains the calling agent's name and the context string it passed. See the [Subagents Guide](/guides/subagents) for details on agent-to-agent calls.
+
+## Skills
+
+If your agent enables [skills](/reference/agent-config#skills) like `lock` or `subagent`, additional instruction blocks are injected between the `<environment>` block and the trigger context. These teach the agent the commands it can use — `rlock`/`runlock` for [resource locks](/concepts/resource-locks), or `al-subagent`/`al-subagent-wait` for [subagent calls](/guides/subagents).
+
+Skill blocks only appear when explicitly enabled in your `SKILL.md` frontmatter.
+
+## Dynamic Context Injection
+
+Beyond the assembled prompt, you can inject runtime data into your `SKILL.md` body using the `` !`command` `` syntax. This runs shell commands during container startup and replaces the markers with their output — useful for fetching live data before the LLM session begins.
+
+See the [Dynamic Context Guide](/guides/dynamic-context) for details.
+
+## Writing Better Instructions
+
+Now that you know what the agent receives automatically, here are some tips:
+
+- **Don't repeat what's injected.** You don't need to tell the agent about `GITHUB_TOKEN` or filesystem constraints — it already knows.
+- **Reference injected blocks by name.** Say *"Read the config from `<agent-config>`"* rather than hardcoding values.
+- **Handle your trigger types.** If your agent subscribes to both cron and webhooks, your instructions should cover both paths — the trigger context tells the agent which one fired.
+- **Keep instructions focused on behavior.** The runtime context handles the "how" (credentials, environment, tools). Your `SKILL.md` should focus on the "what" and "why."
+
+---
+
+# Resource Locks
+
+When you set `scale > 1` on an agent, multiple instances run concurrently. Without coordination, two instances might pick up the same GitHub issue, review the same PR, or deploy the same service at the same time. Resource locks prevent this.
+
+## Why Locks Exist
+
+Locks let concurrent agent instances claim exclusive ownership of a resource before working on it. If another instance already holds the lock, the agent skips that resource and moves on.
+
+## How It Works
+
+1. Before working on a shared resource, the agent runs `rlock "github://acme/app/issues/42"`.
+2. If the lock is free, the agent gets it and proceeds.
+3. If another instance already holds the lock, the agent gets back the holder's name and skips that resource.
+4. When done, the agent runs `runlock "github://acme/app/issues/42"`.
+
+The agent learns the lock commands from a preamble injected before the session starts. Agent authors just reference the commands in their `SKILL.md` workflow — no need to think about HTTP endpoints or authentication.
+
+## Commands
+
+| Command | Description |
+|---------|-------------|
+| `rlock "<uri>"` | Acquire an exclusive lock. Fails if another instance holds it. |
+| `runlock "<uri>"` | Release a lock. Only the holder can release. |
+| `rlock-heartbeat "<uri>"` | Reset the TTL on a held lock. |
+
+See [Agent Commands — Locks](/reference/agent-commands#lock-commands) for the full command reference with response JSON.
+
+## Resource Key URIs
+
+Lock keys use URI format. Use a scheme that identifies the resource type, and a path that uniquely identifies the instance:
+
+| Pattern | Example |
+|---------|---------|
+| `github://owner/repo/issues/number` | `rlock "github://acme/app/issues/42"` |
+| `github://owner/repo/pr/number` | `rlock "github://acme/app/pr/17"` |
+| `deploy://service-name` | `rlock "deploy://api-prod"` |
+
+## TTL and Expiry
+
+Locks expire automatically after **30 minutes** by default. This prevents deadlocks if an agent crashes or hangs without releasing its lock. The timeout is configurable via `resourceLockTimeout` in `config.toml` (value in seconds).
+
+For work that takes longer than the timeout, use `rlock-heartbeat` to extend the TTL. Each heartbeat resets the clock to another full TTL period. If the agent forgets to heartbeat and the lock expires, another instance can claim it.
+
+## Heartbeat
+
+During long-running work, periodically run `rlock-heartbeat` to keep the lock alive:
+
+```markdown
+## Workflow
+
+1. rlock "deploy://api-prod"
+2. Run the deployment (may take 45+ minutes)
+   - Every 10 minutes, run rlock-heartbeat "deploy://api-prod"
+3. runlock "deploy://api-prod"
+```
+
+Each heartbeat resets the expiry to a full TTL period from the current time.
+
+## Multiple Locks and Deadlock Detection
+
+An agent instance can hold multiple locks simultaneously when working across related resources. However, this introduces the possibility of circular waits — agent A holds lock X and waits for lock Y, while agent B holds lock Y and waits for lock X.
+
+The gateway detects these cycles automatically. When an `rlock` request would create a circular wait in the wait-for graph, it returns a `possible deadlock` error with the cycle path instead of blocking forever. The agent can then release its held locks and retry.
+
+```
+# Example deadlock cycle:
+# Agent A holds "github://acme/app/pr/10", wants "deploy://api-prod"
+# Agent B holds "deploy://api-prod", wants "github://acme/app/pr/10"
+# → rlock "deploy://api-prod" returns: possible deadlock detected
+```
+
+Note: The agent preamble constrains agents to one lock at a time for simplicity. Multi-lock is available for advanced use cases where the agent is explicitly instructed to hold multiple locks.
+
+## Authentication
+
+Each container gets a unique per-run secret (the same one used for the shutdown API). Lock requests are authenticated with this secret, so only the container that acquired a lock can release or heartbeat it. There is no way for one agent instance to release another's lock — it must wait for the TTL to expire.
+
+## Auto-release on Exit
+
+When a container exits — whether it finishes successfully, hits an error, or times out — all of its locks are released automatically by the scheduler. You don't need to worry about cleanup in error paths.
+
+## Example in SKILL.md
+
+```markdown
+## Workflow
+
+1. List open issues labeled "agent" in repos from `<agent-config>`
+2. For each issue:
+   - rlock "github://owner/repo/issues/123"
+   - If the lock fails, skip this issue — another instance is handling it
+   - Clone the repo, create a branch, implement the fix
+   - Open a PR and link it to the issue
+   - runlock "github://owner/repo/issues/123"
+3. If you completed work and there may be more issues, run `al-rerun`
+```
+
+## Configuration
+
+| Setting | Location | Default | Description |
+|---------|----------|---------|-------------|
+| `resourceLockTimeout` | `config.toml` | `1800` (30 min) | Default TTL for locks in seconds |
+
+## See Also
+
+- [Agent Commands — Locks](/reference/agent-commands#lock-commands) — full command syntax and response JSON
+- [Scaling Agents](/guides/scaling-agents) — guide on scaling with locks
+
+---
+
+# Dynamic Context
+
+Agents spend tokens every time they fetch context at runtime — cloning repos, listing issues, calling APIs. Hooks let you stage this data before the LLM session starts, so the agent begins with everything it needs.
+
+## The Problem
+
+Without hooks, a typical agent run looks like:
+
+1. LLM starts
+2. LLM runs `git clone` (waits, uses tokens to read output)
+3. LLM runs `gh issue list` (waits, uses tokens to parse JSON)
+4. LLM starts actual work
+
+Steps 2-3 are mechanical — the agent always needs to do them, and they don't benefit from LLM reasoning.
+
+## The Solution: Hooks
+
+Pre-hooks run **after credentials are loaded** but **before the LLM session starts**. They execute inside the container with full access to credentials and environment variables. Define them in the agent's `config.toml`:
+
+```toml
+# agents/<name>/config.toml
+[hooks]
+pre = [
+  "gh repo clone acme/app /tmp/repo --depth 1",
+  "gh issue list --repo acme/app --label bug --json number,title,body --limit 20 > /tmp/context/issues.json",
+]
+```
+
+Then reference the staged files in the body of your `SKILL.md`:
+
+```markdown
+## Context
+
+- The repo is cloned at `/tmp/repo`
+- Open bug issues are at `/tmp/context/issues.json`
+```
+
+## Example: Git clone
+
+The most common hook — clone the repo the agent will work on:
+
+```toml
+[hooks]
+pre = ["gh repo clone acme/app /tmp/repo --depth 1"]
+```
+
+## Example: Shell command
+
+Run any shell command. `GITHUB_TOKEN`, `GH_TOKEN`, and other credential env vars are already set:
+
+```toml
+[hooks]
+pre = ["gh issue list --repo acme/app --label P1 --json number,title,body --limit 20 > /tmp/context/issues.json"]
+```
+
+## Example: HTTP fetch
+
+Fetch data from an API endpoint:
+
+```toml
+[hooks]
+pre = ["curl -sf -H 'Authorization: Bearer ${INTERNAL_TOKEN}' https://api.internal/v1/feature-flags -o /tmp/context/flags.json"]
+```
+
+Environment variable interpolation (`${VAR_NAME}`) is supported since commands run via `/bin/sh`.
+
+## Post-hooks
+
+Post-hooks run after the LLM session completes. Use them for cleanup, artifact upload, or reporting:
+
+```toml
+[hooks]
+pre = ["gh repo clone acme/app /tmp/repo --depth 1"]
+post = [
+  "upload-artifacts.sh",
+  "curl -X POST https://hooks.slack.com/... -d '{\"text\": \"Agent run complete\"}'",
+]
+```
+
+## Referencing Staged Files in SKILL.md
+
+After hooks run, tell the agent what's available in the body of your `SKILL.md`:
+
+```markdown
+## Context
+
+- The repo is cloned at `/tmp/repo`
+- Open P1 issues are at `/tmp/context/issues.json`
+- Feature flags (if available) are at `/tmp/context/flags.json`
+```
+
+## Direct Context Injection
+
+For simple, inline data that needs to be embedded directly in your SKILL.md instructions, use direct context injection with the `` !`command` `` syntax. Commands are executed after pre-hooks but before the LLM session starts, and their output replaces the expression inline.
+
+### Syntax
+
+```markdown
+The current time is !`date`.
+There are !`ls /tmp/repo/src | wc -l` source files in the repo.
+```
+
+Becomes:
+
+```
+The current time is Mon Mar 22 21:30:45 UTC 2026.
+There are 42 source files in the repo.
+```
+
+### When to use
+
+- **Hooks**: For setup tasks like cloning repos or downloading data files
+- **Direct injection**: For inline values the agent needs to reference in instructions
+
+### Examples
+
+**Basic usage**:
+```markdown
+You are analyzing code at !`date +"%Y-%m-%d %H:%M"`.
+```
+
+**With pre-staged data**:
+```toml
+[hooks]
+pre = ["gh issue list --repo acme/app --label bug --json number --limit 20 > /tmp/issues.json"]
+```
+
+```markdown
+There are !`cat /tmp/issues.json | jq length` open bug issues to work on.
+```
+
+**Error handling**:
+If a command fails, it's replaced with `[Error: <message>]`:
+
+```markdown
+Config value: !`cat /nonexistent/file`
+```
+
+Becomes:
+```
+Config value: [Error: cat: can't open '/nonexistent/file': No such file or directory]
+```
+
+### Limitations
+
+- Commands have a 60-second timeout
+- Output is limited to prevent prompt explosion
+- Errors are inline — use hooks for critical setup that should fail the run
+
+## Tips
+
+- **Hooks run sequentially** in the order defined in `config.toml`
+- **Each hook has a 5-minute timeout** — hooks are also bounded by the container-level timeout
+- **If a command fails** (non-zero exit), the run aborts with an error
+- **Environment variables** set inside hook commands do not propagate back to the agent's `process.env`
+- **Use hooks for setup, direct injection for values** — hooks for cloning repos or staging files, direct context injection for inline dynamic values the agent needs to reference
+
+## Next steps
+
+- [Agent Config — Hooks](/reference/agent-config#hooks) — full field reference
+- [Agents (concepts)](/concepts/agents) — full runtime lifecycle
+
+---
+
+# Shared Context
+
+Agents often need the same context — coding conventions, repo layout, team policies. The `shared/` directory lets you maintain this context in one place and reference it from any agent's `SKILL.md`.
+
+## How it works
+
+Place files in a `shared/` directory at your project root. At image build time, these files are baked into every agent's container at `/app/static/shared/`.
+
+```
+my-project/
+├── config.toml
+├── shared/
+│   ├── conventions.md
+│   ├── repo-layout.md
+│   └── team/
+│       └── review-policy.md
+├── agents/
+│   ├── dev/SKILL.md
+│   └── reviewer/SKILL.md
+```
+
+## Referencing shared files in SKILL.md
+
+Use direct context injection to include shared files in your agent's prompt:
+
+```markdown
+## Context
+
+!`cat /app/static/shared/conventions.md`
+!`cat /app/static/shared/repo-layout.md`
+```
+
+Each agent chooses which shared files to include. There is no automatic injection — you control exactly what context each agent receives.
+
+## Example: coding conventions
+
+Create `shared/conventions.md`:
+
+```markdown
+# Coding Conventions
+
+- TypeScript strict mode, no `any`
+- Use `vitest` for tests, mirror `src/` structure in `test/`
+- Prefer named exports over default exports
+- Error messages should include enough context to debug without a stack trace
+```
+
+Reference it in `agents/dev/SKILL.md`:
+
+```markdown
+# Dev Agent
+
+You solve GitHub issues by writing code.
+
+## Context
+
+!`cat /app/static/shared/conventions.md`
+
+## Workflow
+
+1. Read the issue
+2. Write the fix following the conventions above
+3. Run tests
+4. Open a PR
+```
+
+And in `agents/reviewer/SKILL.md`:
+
+```markdown
+# Reviewer Agent
+
+You review pull requests for correctness and style.
+
+## Context
+
+!`cat /app/static/shared/conventions.md`
+
+## Workflow
+
+1. Read the PR diff
+2. Check against conventions
+3. Approve or request changes
+```
+
+Both agents now share the same conventions without duplication.
+
+## Subdirectories
+
+The `shared/` directory supports subdirectories. A file at `shared/team/review-policy.md` is available at `/app/static/shared/team/review-policy.md` inside the container.
+
+## Cloud deployment
+
+When deploying with `al push`, the `shared/` directory is included automatically — `rsync` syncs the entire project directory.
+
+---
+
+# Subagents
+
+Agents can call other agents to delegate work and collect results. This enables multi-agent workflows like planner → developer → reviewer pipelines.
+
+## Use Case
+
+A planner agent triages an issue and creates an implementation plan. It calls a dev agent to implement it, then calls a reviewer agent to review the PR.
+
+## `al-subagent`: Fire a call
+
+Pass context to another agent via stdin:
+
+```bash
+echo "Implement the fix for issue #42 on acme/app" | al-subagent dev
+```
+
+**Response:**
+
+```json
+{"ok": true, "callId": "abc123"}
+```
+
+The call is non-blocking — the calling agent continues working immediately.
+
+## `al-subagent-check`: Non-blocking status
+
+Check if a call has finished without waiting:
+
+```bash
+al-subagent-check abc123
+```
+
+**Response:**
+
+```json
+{"status": "pending"}
+{"status": "running"}
+{"status": "completed", "returnValue": "PR #17 opened."}
+{"status": "error", "error": "timeout"}
+```
+
+## `al-subagent-wait`: Block until done
+
+Wait for one or more calls to complete:
+
+```bash
+al-subagent-wait abc123 --timeout 600
+al-subagent-wait abc123 def456 --timeout 300
+```
+
+**Response:**
+
+```json
+{
+  "abc123": {"status": "completed", "returnValue": "PR #17 opened."},
+  "def456": {"status": "completed", "returnValue": "All tests pass."}
+}
+```
+
+Default timeout: 900 seconds. Polls every 5 seconds.
+
+## `al-return`: Send back a result
+
+The called agent uses `al-return` to send a value back to the caller:
+
+```bash
+al-return "PR #17 opened. Ready for review."
+```
+
+## Multi-call Pattern
+
+Fire several calls, continue working, then collect all results:
+
+```bash
+# Fire calls
+DEV_ID=$(echo "Implement fix for #42" | al-subagent dev | jq -r .callId)
+REVIEW_ID=$(echo "Review PR #17" | al-subagent reviewer | jq -r .callId)
+
+# ... do other work while they run ...
+
+# Collect results
+RESULTS=$(al-subagent-wait "$DEV_ID" "$REVIEW_ID" --timeout 600)
+echo "$RESULTS" | jq ".\"$DEV_ID\".returnValue"
+echo "$RESULTS" | jq ".\"$REVIEW_ID\".returnValue"
+```
+
+## Complete Example: SKILL.md
+
+Here's a planner agent that delegates to dev and reviewer:
+
+```markdown
+# Planner Agent
+
+You orchestrate development workflows. When triggered, you assess the issue,
+create an implementation plan, and delegate to other agents.
+
+## Workflow
+
+1. Read the issue from the webhook trigger or search for labeled issues
+2. Assess the issue — is it clear enough for development?
+3. If not, comment asking for clarification and stop
+4. Write an implementation plan as a comment on the issue
+5. Call the dev agent:
+   ```
+   echo "Implement the plan in comment #N on issue #M in owner/repo" | al-subagent dev
+   ```
+6. Wait for dev to finish:
+   ```
+   RESULT=$(al-subagent-wait "$CALL_ID" --timeout 1800)
+   ```
+7. If dev succeeded, call the reviewer:
+   ```
+   echo "Review PR #P on owner/repo" | al-subagent reviewer
+   ```
+8. Comment on the issue with the final status
+```
+
+## Rules
+
+- **No self-calls** — an agent cannot call itself (the call is rejected)
+- **Call depth limit** — chains like A → B → C are allowed up to `maxCallDepth` (default: 3)
+- **Queuing** — if all runners for the target agent are busy, the call is queued (up to `workQueueSize`, default: 100)
+- **No reruns** — called agents do not re-run. They respond to the single call.
+- **Gateway required** — call commands require the gateway. They return errors if `GATEWAY_URL` is not set.
+
+## What the Called Agent Sees
+
+The called agent receives a `<skill-subagent>` block in its prompt with:
+
+- The name of the calling agent
+- The context string passed via stdin
+
+The called agent's `SKILL.md` should handle this trigger type:
+
+```markdown
+## Trigger handling
+
+- **Agent call**: The `<skill-subagent>` block contains context from the calling agent.
+  Do what was requested and use `al-return` to send back results.
+```
+
+## Next steps
+
+- [Agent Commands](/reference/agent-commands) — full call command syntax and responses
+- [Agents (concepts)](/concepts/agents) — runtime lifecycle and trigger types
+
+---
+
+# Scaling Agents
+
+By default, each agent runs one instance at a time. This guide shows how to scale up and use [resource locks](/concepts/resource-locks) to prevent duplicate work.
+
+## The Problem
+
+With `scale = 1`, a single agent instance handles all work sequentially. If 5 GitHub issues arrive via webhook while the agent is working on one, those 5 events queue up and wait. For high-volume workloads, this creates a bottleneck.
+
+## Increase Scale
+
+In the agent's `config.toml`:
+
+```toml
+# agents/dev/config.toml
+scale = 3    # Run up to 3 instances concurrently
+```
+
+Now when 5 issues arrive, up to 3 are processed simultaneously. The remaining 2 wait in the work queue.
+
+## Add Locking
+
+With multiple instances, two agents might try to work on the same issue. Add a [lock/skip/work/unlock](/concepts/resource-locks) pattern to your `SKILL.md`:
+
+```markdown
+## Workflow
+
+1. List open issues labeled "agent" in repos from `<agent-config>`
+2. For each issue:
+   - rlock "github://owner/repo/issues/123"
+   - If the lock fails, skip this issue — another instance is handling it
+   - Clone the repo, create a branch, implement the fix
+   - Open a PR and link it to the issue
+   - runlock "github://owner/repo/issues/123"
+3. If you completed work and there may be more issues, run `al-rerun`
+```
+
+### How lock commands work
+
+When the agent runs `rlock "github://owner/repo/issues/123"`:
+
+- **Lock acquired:** `{"ok": true}` — proceed with work
+- **Already held:** `{"ok": false, "holder": "dev-abc123", ...}` — skip this resource
+
+When done: `runlock "github://owner/repo/issues/123"` releases the lock.
+
+If the agent crashes or times out, locks are [auto-released](/concepts/resource-locks#auto-release-on-exit).
+
+## Monitor with `al stat`
+
+Check queue depth and running instances:
+
+```bash
+al stat
+al stat -E production
+```
+
+The `queue` column shows how many events are waiting. If it's consistently high, consider increasing `scale`.
+
+## Resource Considerations
+
+Each parallel instance:
+
+- Uses a separate Docker container
+- Consumes memory (`local.memory` per container, default 4GB)
+- Consumes CPU (`local.cpus` per container, default 2)
+- Makes independent LLM API calls (watch your rate limits and quota)
+
+### Tune work queue size
+
+If events arrive faster than agents can process them, the queue buffers them:
+
+```toml
+# config.toml
+workQueueSize = 200    # default: 100 per agent
+```
+
+When the queue is full, the oldest items are dropped.
+
+### Default agent scale
+
+Set the default scale for all agents that don't have an explicit `scale` in their `config.toml`:
+
+```toml
+# config.toml
+defaultAgentScale = 3    # each agent gets 3 runners unless overridden
+```
+
+Without this setting, agents default to 1 runner each.
+
+### Project-wide scale cap
+
+Limit total concurrent runners across all agents:
+
+```toml
+# config.toml
+scale = 10    # max 10 runners total across all agents
+```
+
+If `defaultAgentScale * agentCount` exceeds `scale`, agents are throttled at startup and a warning is shown.
+
+## Example Configuration
+
+Agent runtime config in `agents/dev/config.toml`:
+
+```toml
+credentials = ["github_token", "git_ssh"]
+schedule = "*/5 * * * *"
+models = ["sonnet"]
+scale = 3
+
+[[webhooks]]
+source = "my-github"
+events = ["issues"]
+actions = ["labeled"]
+labels = ["agent"]
+
+[params]
+repos = ["acme/app", "acme/api"]
+triggerLabel = "agent"
+```
+
+## Next steps
+
+- [Resource Locks (concepts)](/concepts/resource-locks) — TTL, heartbeat, deadlock detection
+- [Agent Commands — Locks](/reference/agent-commands#lock-commands) — full command syntax and responses