@action-llama/skill 0.23.4 → 0.24.1

package/skills/al/agent-authoring.md

@@ -0,0 +1,1436 @@
+# Agents
+
+An agent is a directory inside your project that contains instructions and configuration for an autonomous LLM session. Each run is self-contained: the agent wakes up (on a schedule or webhook), executes its task, and shuts down.
+
+## Skills vs Agents
+
+A **skill** is a portable artifact — a `SKILL.md` file (and optionally a `Dockerfile`) that defines what an agent does. Skills can be shared, published, and installed from git repos.
+
+An **agent** is a skill instantiated in your project with local runtime configuration. When you run `al add` to install a skill, it becomes an agent with its own `config.toml` for project-specific settings like credentials, schedule, and model.
+
+## Agent Structure
+
+An agent is a directory with at least two files:
+
+```
+agents/<name>/
+├── SKILL.md        # Portable metadata + instructions (the skill)
+├── config.toml     # Project-local runtime config
+└── Dockerfile      # Optional — custom container image
+```
+
+- **`SKILL.md`** contains portable metadata (name, description, license, compatibility) in its YAML frontmatter, and the agent's instructions in its markdown body.
+- **`config.toml`** contains project-specific runtime configuration: credentials, models, schedule, webhooks, hooks, params, scale, and timeout.
+- **`Dockerfile`** is optional. Defines custom container dependencies. May be provided by the skill author or customized per-project.
+
+The directory name becomes the agent name. No registration is needed — the scheduler discovers agents by scanning for directories that contain a `SKILL.md`.
+
+## How Context is Assembled
+
+At runtime, the agent's LLM session receives two inputs:
+
+### System prompt
+
+The markdown body of `SKILL.md`, prepended with a **preamble** that teaches the agent its [language skills](#language-skills).
+
+### User prompt
+
+Assembled from several blocks:
+
+1. **`<agent-config>`** — JSON of the `params` field from `config.toml`
+2. **`<credential-context>`** — describes which environment variables and tools are available (e.g. `GITHUB_TOKEN`, `git`, `gh`, SSH config)
+3. **`<environment>`** — filesystem constraints and working directory info
+4. **Trigger context** (one of):
+   - *Scheduled run:* "You are running on a schedule. Check for new work and act on anything you find."
+   - *Manual run:* "You have been triggered manually. Check for new work and act on anything you find."
+   - *Webhook:* `<webhook-trigger>` block with the full event payload (source, event, action, repo, etc.)
+   - *Subagent call:* `<skill-subagent>` block with the caller agent name and context
+
+Your `SKILL.md` instructions should reference `<agent-config>` for parameter values and handle both scheduled and webhook triggers if the agent uses both.
+
+## Language Skills
+
+Before the `SKILL.md` instructions run, the agent receives a preamble that teaches it a set of **language skills** — shorthand operations the skill can reference naturally. The preamble explains the underlying mechanics (curl commands, env vars) so agent authors never need to think about them.
+
+| Category | Skills | Description |
+|----------|--------|-------------|
+| **Signals** | `al-rerun`, `al-status`, `al-return`, `al-exit` | Shell commands for signaling the scheduler |
+| **Calls** | `al-subagent`, `al-subagent-check`, `al-subagent-wait` | Agent-to-agent calls with return values |
+| **Locks** | `LOCK(...)`, `UNLOCK(...)`, `HEARTBEAT(...)` | Resource locking for parallel coordination |
+| **Credentials** | `GITHUB_TOKEN`, `gh`, `git`, etc. | Credential access and tool usage |
+
+Agent authors write the shorthand naturally (e.g. `LOCK("github issue acme/app#42")`). The agent learns what it means from the preamble.
+
+See [Agent Commands](/reference/agent-commands) for the complete command reference.
+
+## Runtime Lifecycle
+
+Each agent run is an isolated, short-lived process. By default agents run in Docker containers, but agents can also be configured to run on the host machine under a separate OS user (see [Runtime](/reference/agent-config#runtime)). Here's the full sequence from trigger to exit:
+
+1. **Trigger fires** — a cron tick, webhook event, manual `al run`, or `al-subagent` from another agent.
+2. **Work is queued** — if all runners for the target agent are busy, the trigger is placed in a SQLite-backed work queue until a runner becomes available.
+3. **Process launches** — a fresh container (or host-user process) starts with credentials and config passed via environment variables and volume mounts (or temp directories).
+4. **Credentials are loaded** — the entry point reads credential files from the credentials path (`/credentials/` in containers, or the `AL_CREDENTIALS_PATH` temp directory in host-user mode). Key credentials are injected as env vars the LLM can use directly: `GITHUB_TOKEN`, `GH_TOKEN`, `SENTRY_AUTH_TOKEN`, `GIT_SSH_COMMAND`, git author identity, etc.
+5. **Hooks run** — if `hooks.pre` steps are defined in `config.toml`, they execute sequentially (clone repos, fetch data, run shell commands) to stage context before the LLM starts. See [Dynamic Context](/guides/dynamic-context).
+6. **LLM session starts** — the model receives the `SKILL.md` instructions as system prompt and the assembled user prompt.
+7. **Agent runs autonomously** — the LLM executes tools (bash, file I/O, API calls) until it finishes or hits an error. Rate-limited API calls are retried automatically (up to 5 attempts with exponential backoff).
+8. **Error detection** — the runtime watches for repeated auth/permission failures (e.g. "bad credentials", "permission denied"). After 3 such errors, it aborts early.
+9. **Signals are processed** — the agent uses `al-rerun`, `al-status`, `al-return`, and `al-exit` commands to write signal files. The scheduler reads them after the session ends.
+10. **Process exits** — exit code 0 (success), 1 (error), or 124 (timeout). Any held locks are released automatically. The scheduler logs the result and the container is removed (or the working directory is cleaned up in host-user mode).
+
+## Timeout
+
+Each agent process has a self-termination timer controlled by `timeout` in the agent's `config.toml` (falls back to `local.timeout` in project `config.toml`, then 900 seconds). If the timer fires, the process exits with code 124. This is a hard kill — there is no graceful shutdown.
+
+See [Agent Config — Timeout](/reference/agent-config#timeout) for configuration.
+
+## Reruns
+
+When a scheduled agent runs `al-rerun`, the scheduler immediately re-runs it. This continues until the agent completes without `al-rerun` (no more work), hits an error, or reaches the `maxReruns` limit (default: 10, configurable in `config.toml`). This lets an agent drain its work queue without waiting for the next cron tick.
+
+Webhook-triggered and agent-called runs do not re-run — they respond to a single event.
+
+## Work Queue
+
+When a trigger fires (webhook event or agent call) but all runner instances for the target agent are busy, the event is placed in a **work queue** instead of being dropped. Items are dequeued and executed as runners become available.
+
+The queue is backed by SQLite (`.al/work-queue.db`), so pending items survive scheduler restarts. Each agent has its own queue. If the queue is full, the oldest items are dropped.
+
+You can see queue depth per agent in `al stat` output (the `queue` column).
+
+| Setting | Location | Default | Description |
+|---------|----------|---------|-------------|
+| `workQueueSize` | `config.toml` | `100` | Maximum queued work items per agent |
+
+## Container Filesystem
+
+When running in the default container runtime:
+
+| Path | Mode | Contents |
+|------|------|----------|
+| `/app` | read-only | Action Llama application + node_modules |
+| `/credentials` | read-only | Mounted credential files (`/<type>/<instance>/<field>`) |
+| `/tmp` | read-write (tmpfs, 2GB) | Agent working directory — repos, scratch files, SSH keys |
+| `/workspace` | read-write (2GB) | Persistent workspace |
+| `/home/node` | read-write (64MB) | Home directory |
+
+The root filesystem is read-only. All agent work should happen in `/tmp`.
+
+### Host-User Filesystem
+
+When running in [host-user mode](/reference/agent-config#runtime), the agent runs directly on the host:
+
+| Path | Contents |
+|------|----------|
+| `/tmp/al-runs/<instance-id>/` | Working directory (chowned to agent user) |
+| `AL_CREDENTIALS_PATH` (temp dir) | Staged credential files (`/<type>/<instance>/<field>`) |
+
+The agent has access to the host filesystem but runs as a separate OS user, so it cannot access other users' files or credentials.
+
+## See Also
+
+- [Getting Started](/first-steps/getting-started) — create your first agent
+- [Agent Commands](/reference/agent-commands) — signals, calls, and lock commands
+- [Agent Config Reference](/reference/agent-config) — SKILL.md and config.toml fields
+- [Agent Docs](/reference/agent-docs) — SKILL.md, AGENTS.md, CLAUDE.md
+- [Credentials](/reference/credentials) — credential types and storage
+- [Web Dashboard](/reference/web-dashboard) — monitoring agents in your browser
+
+---
+
+# Agent Configuration
+
+Each agent lives in a directory under `agents/<name>/`. Configuration is split across two files:
+
+| File | Purpose | Portable? |
+|------|---------|-----------|
+| `SKILL.md` | Portable metadata (name, description, license, compatibility) + markdown instructions | Yes — travels with the skill |
+| `config.toml` | Runtime config (credentials, models, schedule, webhooks, hooks, params, scale, timeout) | No — project-local |
+
+A **skill** is the portable artifact (SKILL.md + optionally a Dockerfile). An **agent** is a skill instantiated in your project with local runtime config. When you `al add` a skill, it becomes an agent.
+
+An optional `Dockerfile` can also live in the agent directory for custom container images. It may be provided by the skill author or customized per-project.
+
+```
+agents/<name>/
+├── SKILL.md        # Portable metadata + instructions
+├── config.toml     # Project-local runtime config
+└── Dockerfile      # Optional — custom container image (container runtime only)
+```
+
+## SKILL.md
+
+The YAML frontmatter contains portable metadata. The markdown body contains the agent's instructions.
+
+```yaml
+---
+name: dev-agent
+description: Solves GitHub issues by writing and testing code
+license: MIT
+compatibility: ">=0.5.0"
+---
+
+# Instructions
+
+You are a dev agent. Check for open issues labeled "agent" and fix them.
+
+## Workflow
+
+1. List open issues labeled "agent" in repos from `<agent-config>`
+2. For each issue, clone the repo, create a branch, implement the fix
+3. Open a PR and link it to the issue
+```
+
+### SKILL.md Frontmatter Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `name` | string | No | Human-readable name (defaults to directory name) |
+| `description` | string | No | Short description of what the agent does |
+| `license` | string | No | License identifier (e.g. `"MIT"`) |
+| `compatibility` | string | No | Semver range for Action Llama compatibility |
+
+## config.toml
+
+The per-agent `config.toml` contains project-specific runtime configuration. This file is created by `al add`, `al agent new`, or `al config`.
+
+```toml
+# Install origin — used by `al update` to pull upstream SKILL.md changes
+source = "acme/dev-skills"
+
+# Required: named model references from project config.toml [models.*]
+# First in list is primary; rest are fallbacks tried on rate limits
+models = ["sonnet", "haiku"]
+
+# Required: credential types the agent needs at runtime
+# Use "type" for default instance, "type:instance" for named instance
+credentials = ["github_token", "git_ssh", "sentry_token"]
+
+# Optional: cron schedule (standard cron syntax)
+# Agent must have at least one of: schedule, webhooks
+schedule = "*/5 * * * *"
+
+# Optional: number of concurrent runs allowed (default: 1)
+# When scale > 1, use LOCK/UNLOCK in your actions to coordinate
+scale = 2
+
+# Optional: max runtime in seconds (default: falls back to [local].timeout, then 900)
+timeout = 600
+
+# Optional: max queued work items for this agent (default: global workQueueSize)
+# When all runners are busy, incoming events are queued up to this limit.
+# Oldest events are dropped to make room for newer ones.
+maxWorkQueueSize = 50
+
+# Optional: webhook triggers (instead of or in addition to schedule)
+[[webhooks]]
+source = "my-github"
+repos = ["acme/app"]
+events = ["issues"]
+actions = ["labeled"]
+labels = ["agent"]
+
+[[webhooks]]
+source = "my-sentry"
+resources = ["error", "event_alert"]
+
+[[webhooks]]
+source = "my-linear"
+events = ["issues"]
+actions = ["create", "update"]
+labels = ["bug"]
+
+[[webhooks]]
+source = "my-mintlify"
+events = ["build"]
+actions = ["failed"]
+
+# Optional: hooks — shell commands that run before or after the LLM session
+[hooks]
+pre = [
+  "gh repo clone acme/app /tmp/repo --depth 1",
+  "curl -o /tmp/context/flags.json https://api.internal/v1/flags",
+  "gh issue list --repo acme/app --label P1 --json number,title,body --limit 20 > /tmp/context/issues.json",
+]
+post = ["upload-artifacts.sh"]
+
+# Optional: custom parameters injected into the agent prompt
+[params]
+repos = ["acme/app", "acme/api"]
+triggerLabel = "agent"
+assignee = "bot-user"
+sentryOrg = "acme"
+sentryProjects = ["web-app", "api"]
+
+# Optional: runtime mode — "container" (default) or "host-user"
+# Host-user runs the agent as a separate OS user via sudo, without Docker.
+# Useful when the agent needs to run Docker commands itself.
+[runtime]
+type = "host-user"
+run_as = "al-agent"           # OS user to run as (default: "al-agent")
+```
+
+### config.toml Field Reference
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `source` | string | No | Git URL or GitHub shorthand for `al update`. Set automatically by `al add`. |
+| `models` | string[] | Yes | Named model references from `config.toml [models.*]`. First is primary; rest are fallbacks tried automatically on rate limits. |
+| `credentials` | string[] | Yes | Credential refs: `"type"` for default instance, `"type:instance"` for named instance. See [Credentials](/reference/credentials). |
+| `schedule` | string | No* | Cron expression for polling |
+| `scale` | number | No | Number of concurrent runs allowed (default: 1). Set to `0` to disable the agent. Use lock skills in your actions to coordinate instances. See [Resource Locks](/concepts/resource-locks). |
+| `timeout` | number | No | Max runtime in seconds. Falls back to `[local].timeout` in project config, then `900`. See [Timeout](#timeout). |
+| `maxWorkQueueSize` | number | No | Maximum queued work items when all runners are busy. Falls back to global `workQueueSize` (default: 20). Oldest events are dropped to make room for newer ones. |
+| `webhooks` | array | No* | Array of webhook trigger objects. See [Webhooks](/reference/webhooks). |
+| `hooks` | table | No | Pre/post hooks that run around the LLM session. See [Hooks](#hooks). |
+| `params` | table | No | Custom key-value params for the agent prompt |
+| `runtime` | table | No | Runtime mode configuration. See [Runtime](#runtime). |
+
+*At least one of `schedule` or `webhooks` is required (unless `scale = 0`).
+
+## Scale
+
+The `scale` field controls how many instances of an agent can run concurrently.
+
+- **Default**: 1 (only one instance can run at a time)
+- **Minimum**: 0 (disables the agent — no runners, cron jobs, or webhook bindings are created)
+- **Maximum**: No hard limit, but consider system resources and model API rate limits
+
+### How it works
+
+1. **Scheduled runs**: If a cron trigger fires but all agent instances are busy, the scheduled run is skipped with a warning
+2. **Webhook events**: If a webhook arrives but all instances are busy, the event is queued (up to `workQueueSize` limit in global config, default: 100)
+3. **Agent calls**: If one agent calls another but all target instances are busy, the call is queued in the same work queue
+
+### Example use cases
+
+- **Dev agent** with `scale = 3`: Handle multiple GitHub issues simultaneously
+- **Review agent** with `scale = 2`: Review multiple PRs in parallel
+- **Monitoring agent** with `scale = 1`: Ensure only one instance processes alerts at a time
+- **Disabled agent** with `scale = 0`: Keep the config in the project but don't run it
+
+### Resource considerations
+
+Each parallel instance:
+- Uses a separate Docker container (or OS process in host-user mode)
+- Has independent logging streams
+- May consume LLM API quota concurrently
+- Uses system memory and CPU
+
+See [Scaling Agents](/guides/scaling-agents) for a guide on scaling with resource locks.
+
+## Timeout
+
+The `timeout` field controls the maximum runtime for an agent invocation. When the timeout expires, the process is terminated with exit code 124.
+
+**Resolution order:** `config.toml timeout` -> `project config.toml [local].timeout` -> `900` (default)
+
+This means you can set a project-wide default in `[local].timeout` and override it per-agent.
+
+### Examples
+
+```toml
+# Fast webhook responder
+timeout = 300       # 5 minutes
+
+# Medium-length task
+timeout = 900       # 15 minutes
+
+# Long-running agent
+timeout = 3600      # 1 hour
+
+# Omit timeout — uses [local].timeout or defaults to 900s
+```
+
+## Hooks
+
+Hooks run shell commands before and after the LLM session. Pre-hooks (`hooks.pre`) run after credentials are loaded but before the LLM session starts — use them for cloning repos, fetching data, or staging files. Post-hooks (`hooks.post`) run after the session completes — use them for cleanup, artifact upload, or reporting.
+
+See [Dynamic Context](/guides/dynamic-context) for a guide on using hooks effectively.
+
+### How it works
+
+1. Commands run **sequentially** in the order they appear in `config.toml`
+2. Commands run inside the agent's execution environment (container or host-user process) after credential/env setup
+3. Each command runs via `/bin/sh -c "..."`
+4. If any command exits non-zero, the run aborts with an error
+5. Credential env vars (`GITHUB_TOKEN`, `GH_TOKEN`, etc.) are available to hook commands
+
+### Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `hooks.pre` | string[] | No | Shell commands to run before the LLM session |
+| `hooks.post` | string[] | No | Shell commands to run after the LLM session |
+
+### Examples
+
+```toml
+[hooks]
+pre = [
+  "gh repo clone acme/app /tmp/repo --depth 1",
+  "gh issue list --repo acme/app --label P1 --json number,title,body --limit 20 > /tmp/context/issues.json",
+  "curl -o /tmp/context/flags.json https://api.internal/v1/flags",
+]
+post = ["upload-artifacts.sh"]
+```
+
+### Notes
+
+- Each hook has a 5-minute timeout
+- Hooks are bounded by the container-level timeout
+- Environment variables set inside hook commands do not propagate back to the agent's `process.env`
+
+## Runtime
+
+The `[runtime]` table controls how the agent process is launched. By default, agents run in Docker containers. The `host-user` runtime runs agents as a separate OS user on the host machine via `sudo -u`, without Docker.
+
+Host-user mode is useful when agents need to run Docker commands themselves (Docker-in-Docker is insecure), or when you want lightweight isolation without container overhead.
+
+```toml
+[runtime]
+type = "host-user"
+run_as = "al-agent"
+```
+
+### Fields
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `type` | string | `"container"` | Runtime mode: `"container"` (Docker) or `"host-user"` (OS user isolation) |
+| `run_as` | string | `"al-agent"` | OS username to run the agent as. Only used when `type = "host-user"`. |
+
+### How host-user mode works
+
+1. The scheduler spawns `sudo -u <run_as> al _run-agent <agent> --project <dir>`
+2. Credentials are staged to a temp directory and chowned to the agent user
+3. Each run gets an isolated working directory at `/tmp/al-runs/<instance-id>/`
+4. Logs are written to `/tmp/al-runs/<instance-id>.log` (owned by the scheduler, not the agent)
+5. No Docker images are built for host-user agents
+
+### Setup
+
+The agent OS user must exist and sudoers must be configured. Run `al doctor` to validate and auto-configure (Linux only):
+
+```bash
+al doctor
+```
+
+On Linux, `al doctor` will:
+- Create the OS user if it doesn't exist (`useradd --system --shell /usr/sbin/nologin <run_as>`)
+- Add a sudoers rule allowing passwordless execution
+
+On macOS, `al doctor` prints manual setup instructions.
+
+### Limitations
+
+- No custom Dockerfiles — `Dockerfile` in the agent directory is ignored
+- No container filesystem isolation — the agent runs on the host filesystem
+- The `[local]` config section (`memory`, `cpus`, `image`) does not apply to host-user agents
+- `needsGateway` is false — the gateway is not started for host-user-only projects
+
+## Webhook Trigger Fields
+
+Each entry in the `webhooks` array has the following fields:
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `source` | string | Yes | Name of a webhook source from the project's `config.toml` (e.g. `"my-github"`) |
+
+All filter fields below are optional. Omit all of them to trigger on everything from that source. See [Webhooks](/reference/webhooks) for complete filter field tables per provider.
+
+### GitHub filter fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `repos` | string[] | Filter to specific repos |
+| `orgs` | string[] | Filter to specific organizations |
+| `org` | string | Filter to a single organization |
+| `events` | string[] | Event types: issues, pull_request, push, etc. |
+| `actions` | string[] | Event actions: opened, labeled, closed, etc. |
+| `labels` | string[] | Only trigger when issue/PR has these labels |
+| `assignee` | string | Only trigger when assigned to this user |
+| `author` | string | Only trigger for this author |
+| `branches` | string[] | Only trigger for these branches |
+| `conclusions` | string[] | Only for workflow_run events with these conclusions |
+
+### Sentry filter fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `resources` | string[] | Resource types: event_alert, metric_alert, issue, error, comment |
+
+### Linear filter fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `organizations` | string[] | Filter to specific Linear organizations |
+| `events` | string[] | Linear event types: issues, issue_comment, etc. |
+| `actions` | string[] | Event actions: create, update, delete, etc. |
+| `labels` | string[] | Only when issue has these labels |
+| `assignee` | string | Only when assigned to this user (email) |
+| `author` | string | Only for this author (email) |
+
+### Mintlify filter fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `projects` | string[] | Filter to specific Mintlify projects |
+| `events` | string[] | Mintlify event types: build, etc. |
+| `actions` | string[] | Event actions: failed, succeeded, etc. |
+| `branches` | string[] | Only for these branches |
+
+## Model Configuration
+
+The `models` field references named models defined in `config.toml` under `[models.<name>]`. List one or more model names — the first is the primary model, and subsequent entries are fallbacks tried automatically when the primary is rate-limited or unavailable.
+
+```toml
+models = ["sonnet", "haiku", "gpt4o"]
+```
+
+See [Models](/reference/models) for all supported providers, model IDs, auth types, thinking levels, and credential setup.
+
+---
+
+# Agent Docs
+
+Action Llama uses three markdown files with distinct purposes. Only `SKILL.md` is required.
+
+## SKILL.md
+
+The portable part of an agent. YAML frontmatter at the top contains portable metadata (name, description, license, compatibility), and the markdown body is the system prompt that drives the agent's behavior. Runtime configuration (credentials, schedule, models, webhooks, hooks, params) lives in the separate `config.toml` file.
+
+### What it is
+
+`SKILL.md` lives inside each agent's directory (e.g. `agents/dev/SKILL.md`). The frontmatter is parsed for portable metadata, and the markdown body is injected as the LLM's system prompt at runtime. Everything the agent does is driven by these instructions.
+
+### How it's used
+
+The scheduler reads `SKILL.md` and sends the markdown body as the system prompt. Runtime configuration is loaded from `config.toml`. A preamble is prepended that teaches the agent its [language skills](/reference/agent-commands) (signals, calls, locks, credentials). The LLM then receives a user-message prompt with:
+
+1. **`<skill-config>`** — JSON of the `params` table from the agent's `config.toml`
+2. **`<credential-context>`** — describes available environment variables and tools
+3. **`<environment>`** — filesystem constraints (read-only root, `/tmp` for work)
+4. **Trigger context** — schedule, webhook payload, manual, or subagent call context
+
+### Writing tips
+
+- **Write as direct instructions to an LLM.** "You are an automation agent. Your job is to..."
+- **Be specific about workflow.** Numbered steps are better than vague descriptions.
+- **Reference `<skill-config>` for parameter values.** Don't hardcode repo names or labels — put them in `params` and reference them.
+- **Handle both trigger types** if the agent uses both schedule and webhooks.
+- **Use signal commands** like `al-rerun` for backlog draining and `al-status` for TUI updates.
+- **Keep it concise.** The system prompt consumes tokens every run. Avoid lengthy explanations of things the LLM already knows.
+
+### Example
+
+```markdown
+# Dev Agent
+
+You are an automation agent. Your job is to pick up GitHub issues and implement the requested changes.
+
+Your configuration is in the `<skill-config>` block at the start of your prompt.
+`GITHUB_TOKEN` is already set in your environment. Use `gh` CLI and `git` directly.
+
+## Workflow
+
+1. **Find work** — If triggered by webhook, work on the issue from the trigger. If scheduled, search for open issues labeled with the label from `<skill-config>`.triggerLabel.
+2. **Lock the issue** — LOCK("github issue owner/repo#N"). If locked, skip it.
+3. **Implement** — Clone the repo, create a branch, make changes, run tests.
+4. **Open a PR** — Push and open a PR linking to the issue.
+5. **Unlock** — UNLOCK("github issue owner/repo#N").
+6. **Continue** — If you completed work and there may be more issues, run `al-rerun`.
+
+## Rules
+
+- Never force-push
+- Run tests before opening a PR
+- If tests fail after 3 attempts, comment on the issue and move on
+```
+
+### Handling trigger context
+
+Your `SKILL.md` should handle different trigger types if the agent uses both:
+
+```markdown
+## Trigger handling
+
+- **Webhook trigger**: The `<webhook-trigger>` block contains the full event payload. Work on the specific issue/PR from the event.
+- **Scheduled trigger**: Search for open work matching your configured filters.
+- **Subagent call**: The `<skill-subagent>` block contains context from the calling agent. Do what was requested and use `al-return` to send back results.
+```
+
+## AGENTS.md
+
+Project-level shared instructions that are loaded by `al chat`.
+
+### What it is
+
+`AGENTS.md` lives at the project root (e.g. `my-project/AGENTS.md`). It provides shared context about the project that is relevant when interacting with agents via the chat console.
+
+### How it's used
+
+When you run `al chat` (project-level console), the contents of `AGENTS.md` are loaded as reference context. It is **not** injected into automated agent runs — only into the interactive chat session.
+
+### When to use it
+
+Use `AGENTS.md` to document:
+
+- Project conventions that are relevant when working with agents interactively
+- How agents in the project relate to each other
+- Common tasks and how to accomplish them via chat
+
+## CLAUDE.md
+
+Instructions for AI development tools (like Claude Code). Not used by Action Llama at runtime.
+
+### What it is
+
+`CLAUDE.md` lives at the project root and provides instructions to AI coding assistants that may be editing the project's source code. Do not confuse it with `SKILL.md`, which is the agent instructions file.
+
+### How it's used
+
+`CLAUDE.md` is **not** read by Action Llama. It has no effect on agent runs, the scheduler, or the chat console. It exists purely for developer tooling context.
+
+### When to use it
+
+If you use Claude Code or similar AI development tools to work on your Action Llama project, put project conventions, build commands, and coding guidelines in `CLAUDE.md`. See [Claude Integration](/integrations/claude) for how Claude Code uses this file alongside the MCP server.
+
+---
+
+# Credentials
+
+Credentials are stored in `~/.action-llama/credentials/<type>/<instance>/<field>`. Each credential type is a directory containing one file per field. Reference them in your agent's `config.toml` by type name (e.g. `"github_token"`) for the `default` instance, or use `"type:instance"` for a named instance (e.g. `"git_ssh:botty"`).
+
+## Built-in Credentials
+
+### Agent runtime credentials
+
+| Type | Fields | Description | Runtime Injection |
+|------|--------|-------------|-------------------|
+| `github_token` | `token` | GitHub PAT with repo and workflow scopes | `GITHUB_TOKEN` and `GH_TOKEN` env vars |
+| `anthropic_key` | `token` | Anthropic API key, OAuth token, or pi auth | _(read by SDK)_ |
+| `openai_key` | `token` | OpenAI API key | _(read by SDK)_ |
+| `groq_key` | `token` | Groq API key | _(read by SDK)_ |
+| `google_key` | `token` | Google Gemini API key | _(read by SDK)_ |
+| `xai_key` | `token` | xAI API key | _(read by SDK)_ |
+| `mistral_key` | `token` | Mistral API key | _(read by SDK)_ |
+| `openrouter_key` | `token` | OpenRouter API key | _(read by SDK)_ |
+| `custom_key` | `token` | Custom provider API key | _(read by SDK)_ |
+| `sentry_token` | `token` | Sentry auth token for error monitoring | `SENTRY_AUTH_TOKEN` env var |
+| `linear_token` | `token` | Linear personal API token | `LINEAR_API_TOKEN` env var |
+| `linear_oauth` | `client_id`, `client_secret`, `access_token`, `refresh_token` | Linear OAuth2 credentials | `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET`, `LINEAR_ACCESS_TOKEN`, `LINEAR_REFRESH_TOKEN` env vars |
+| `bugsnag_token` | `token` | Bugsnag auth token | `BUGSNAG_AUTH_TOKEN` env var |
+| `netlify_token` | `token` | Netlify Personal Access Token | `NETLIFY_AUTH_TOKEN` env var |
+| `mintlify_token` | `token` | Mintlify API token | `MINTLIFY_API_TOKEN` env var |
+| `slack_bot_token` | `token` | Slack bot user OAuth token | `SLACK_BOT_TOKEN` env var |
+| `git_ssh` | `id_rsa`, `username`, `email` | SSH private key + git author identity | SSH key mounted as file; `GIT_AUTHOR_NAME`/`GIT_AUTHOR_EMAIL`/`GIT_COMMITTER_NAME`/`GIT_COMMITTER_EMAIL` set from `username`/`email` |
+| `x_twitter_api` | `api_key`, `api_secret`, `bearer_token`, `access_token`, `access_token_secret` | X (Twitter) API credentials | `X_API_KEY`, `X_API_SECRET`, `X_BEARER_TOKEN`, `X_ACCESS_TOKEN`, `X_ACCESS_TOKEN_SECRET` env vars |
+| `aws` | `access_key_id`, `secret_access_key`, `default_region` | AWS credentials | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_DEFAULT_REGION` env vars |
+| `reddit_oauth` | `client_id`, `client_secret`, `username`, `password`, `user_agent` | Reddit OAuth2 credentials for script apps | `REDDIT_CLIENT_ID`, `REDDIT_CLIENT_SECRET`, `REDDIT_USERNAME`, `REDDIT_PASSWORD`, `REDDIT_USER_AGENT` env vars |
+
+### Webhook secrets
+
+| Type | Fields | Description |
+|------|--------|-------------|
+| `github_webhook_secret` | `secret` | Shared secret for GitHub webhook HMAC verification |
+| `sentry_client_secret` | `secret` | Client secret for Sentry webhook verification |
+| `linear_webhook_secret` | `secret` | Shared secret for Linear webhook verification |
+| `mintlify_webhook_secret` | `secret` | Shared secret for Mintlify webhook verification |
+| `discord_bot` | `application_id`, `public_key`, `bot_token` | Discord bot credentials for Ed25519 webhook verification and API access. Injected as `DISCORD_APPLICATION_ID`, `DISCORD_PUBLIC_KEY`, `DISCORD_BOT_TOKEN`. |
+| `slack_signing_secret` | `secret` | Signing secret for Slack webhook verification |
+
+### Infrastructure credentials
+
+These are used by CLI commands (provisioning, deployment) and are not injected into agent containers.
+
+| Type | Fields | Description |
+|------|--------|-------------|
+| `gateway_api_key` | `key` | API key for dashboard and CLI access to the gateway |
+| `vultr_api_key` | `api_key` | Vultr API key for VPS provisioning |
+| `hetzner_api_key` | `api_key` | Hetzner API key for VPS provisioning |
+| `cloudflare_api_token` | `token` | Cloudflare API token for DNS and TLS setup during provisioning |
+| `vps_ssh` | `id_rsa` | SSH private key for VPS access (generated or selected during provisioning) |
+| `gcp_service_account` | `key_json` | GCP service account JSON key for Cloud Run Jobs runtime (requires `roles/run.admin`, `roles/secretmanager.admin`, `roles/artifactregistry.admin`) |
+
+## How Credentials Work
+
+1. **Configuration**: List credential types in your agent's `config.toml`:
+   ```toml
+   credentials = ["github_token", "git_ssh"]
+   ```
+
+2. **Storage**: Credential values live in `~/.action-llama/credentials/<type>/<instance>/<field>`. Each field is a plain text file.
+
+3. **Injection**: When an agent runs, the credentials it requires are made available at a credentials path and key values are injected as environment variables. In Docker mode, credentials are mounted at `/credentials/<type>/<instance>/<field>`. In [host-user mode](/reference/agent-config#runtime), they are staged to a temp directory (path set via `AL_CREDENTIALS_PATH`) and chowned to the agent user.
+
+4. **Git identity**: The `git_ssh` credential includes `username` and `email` fields (prompted during `al new`/`al doctor`). These are injected as `GIT_AUTHOR_NAME`/`GIT_AUTHOR_EMAIL` and `GIT_COMMITTER_NAME`/`GIT_COMMITTER_EMAIL` env vars at runtime, so `git commit` works without requiring `git config`.
+
+5. **LLM credentials**: The LLM credential (e.g. `anthropic_key`) does not need to be listed in the agent's `credentials` array — it is loaded automatically based on the `[models.*]` config.
+
+## Named Instances
+
+Each credential type supports named instances. For example, you could have webhook secrets for multiple GitHub orgs:
+
+```
+~/.action-llama/credentials/github_webhook_secret/MyOrg/secret
+~/.action-llama/credentials/github_webhook_secret/OtherOrg/secret
+```
+
+Or multiple SSH keys:
+
+```
+~/.action-llama/credentials/git_ssh/default/id_rsa
+~/.action-llama/credentials/git_ssh/default/username
+~/.action-llama/credentials/git_ssh/botty/id_rsa
+~/.action-llama/credentials/git_ssh/botty/username
+```
+
+By default, just reference `"git_ssh"` — this resolves to the `default` instance. To use a named instance, use colon syntax: `"git_ssh:botty"`.
+
+## Managing Credentials
+
+### `al creds add`
+
+Add or update a credential interactively. Runs validation for the credential type (e.g. API key format, GitHub API check):
+
+```bash
+al creds add github_token              # default instance
+al creds add github_webhook_secret:myapp
+al creds add git_ssh:prod
+```
+
+### `al creds rm`
+
+Remove a credential:
+
+```bash
+al creds rm github_token               # default instance
+al creds rm github_webhook_secret:myapp
+```
+
+### `al creds ls`
+
+List all stored credentials grouped by type:
+
+```bash
+al creds ls
+```
+
+### `al creds types`
+
+Browse available credential types interactively. Shows all 26 built-in types with their fields, environment variables, and descriptions. Offers to add the selected credential immediately.
+
+```bash
+al creds types
+```
+
+### `al doctor`
+
+Scan all agents in a project and prompt for any missing credentials:
+
+```bash
+al doctor
+```
+
+### During `al new`
+
+The `al new` command prompts for the Anthropic credential during initial setup. Other credentials are configured per-agent by `al doctor` or `al creds add`.
+
+### Manually
+
+Write credential files directly:
+
+```bash
+mkdir -p ~/.action-llama/credentials/github_token/default
+echo "ghp_your_token_here" > ~/.action-llama/credentials/github_token/default/token
+
+mkdir -p ~/.action-llama/credentials/anthropic_key/default
+echo "sk-ant-api-your_key_here" > ~/.action-llama/credentials/anthropic_key/default/token
+```
+
+## Anthropic Auth Methods
+
+Three auth methods are supported:
+
+- **`api_key`** — Standard API key (`sk-ant-api-...`). Set `authType = "api_key"` in model config.
+- **`oauth_token`** — OAuth token (`sk-ant-oat-...`). Set `authType = "oauth_token"`.
+- **`pi_auth`** — Use existing pi auth credentials (`~/.pi/agent/auth.json`). Set `authType = "pi_auth"`. No credential file needed.
+
+## Webhook Secrets
+
+Webhook secrets use named credential instances. For example, to set up a GitHub webhook secret for your org:
+
+```bash
+al creds add github_webhook_secret:MyOrg
+```
+
+Or manually:
+
+```bash
+mkdir -p ~/.action-llama/credentials/github_webhook_secret/MyOrg
+echo "your-webhook-secret" > ~/.action-llama/credentials/github_webhook_secret/MyOrg/secret
+```
+
+The gateway automatically loads secrets from all credential instances (e.g. `github_webhook_secret:MyOrg`, `sentry_client_secret:MyOrg`) and uses them to verify incoming webhook payloads. No global configuration is needed.
+
+## VPS Credential Sync
+
+When deploying to a VPS, credentials are transferred to the remote server via SSH. The remote layout mirrors the local one: `~/.action-llama/credentials/{type}/{instance}/{field}`.
+
+No external secrets manager is needed — same trust model as SSH access.
+
+## Troubleshooting
+
+### "Bad credentials" or "401 Unauthorized"
+
+```bash
+al doctor    # Re-prompts for missing or invalid credentials
+al creds ls       # Verify stored credentials
+```
+
+For GitHub tokens, ensure the token has the required scopes (`repo`, `read:org`, `workflow`).
+
+### Credential not found at runtime
+
+Agents only receive credentials listed in their `config.toml`:
+
+```toml
+credentials = ["github_token", "git_ssh"]
+```
+
+If a credential is missing from this list, the agent won't have access to it. Add it and re-run.
+
+---
+
+# Models
+
+Action Llama supports 8 LLM providers. Define named models in `config.toml` under `[models.<name>]`, then reference them by name in each agent's `config.toml`. Agents list models in priority order — the first is the primary, the rest are fallbacks tried automatically on rate limits.
+
+## `[models.<name>]` Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `provider` | string | Yes | Provider name (see table below) |
+| `model` | string | Yes | Model ID |
+| `authType` | string | Yes | `"api_key"`, `"oauth_token"`, or `"pi_auth"` |
+| `thinkingLevel` | string | No | Reasoning budget (Anthropic only) |
+
+## Providers
+
+### Anthropic
+
+Claude models with optional extended thinking.
+
+```toml
+[models.sonnet]
+provider = "anthropic"
+model = "claude-sonnet-4-20250514"
+thinkingLevel = "medium"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `claude-opus-4-20250514` | Most capable, best for complex multi-step tasks |
+| `claude-sonnet-4-20250514` | Balanced performance and cost (recommended) |
+| `claude-haiku-3-5-20241022` | Fastest and cheapest |
+
+**Credential:** `anthropic_key` (field: `token`)
+
+**Auth types:**
+
+| `authType` | Token format | Description |
+|------------|-------------|-------------|
+| `api_key` | `sk-ant-api-...` | Standard Anthropic API key |
+| `oauth_token` | `sk-ant-oat-...` | OAuth token from `claude setup-token` |
+| `pi_auth` | _(none)_ | Uses existing pi auth credentials (`~/.pi/agent/auth.json`). No credential file needed. |
+
+**Note:** `pi_auth` is not supported in Docker mode. Switch to `api_key` or `oauth_token` for containerized runs.
+
+**Thinking level:** Anthropic is the only provider that supports `thinkingLevel`. Valid values:
+
+| Level | Description |
+|-------|-------------|
+| `off` | No extended thinking |
+| `minimal` | Minimal reasoning |
+| `low` | Light reasoning |
+| `medium` | Balanced (recommended) |
+| `high` | Deep reasoning |
+| `xhigh` | Maximum reasoning budget |
+
+If omitted, thinking is not explicitly configured. For other providers, `thinkingLevel` is ignored.
+
+### OpenAI
+
+```toml
+[models.gpt4o]
+provider = "openai"
+model = "gpt-4o"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `gpt-4o` | Flagship multimodal model (recommended) |
+| `gpt-4o-mini` | Smaller, faster, cheaper |
+| `gpt-4-turbo` | Previous generation |
+| `o1-preview` | Reasoning model |
+| `o1-mini` | Smaller reasoning model |
+
+**Credential:** `openai_key` (field: `token`)
+
+### Groq
+
+```toml
+[models.groq-llama]
+provider = "groq"
+model = "llama-3.3-70b-versatile"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `llama-3.3-70b-versatile` | Llama 3.3 70B on Groq inference |
+
+Groq runs open-source models at high speed. Check Groq's docs for the full list of available model IDs.
+
+**Credential:** `groq_key` (field: `token`)
+
+### Google Gemini
+
+```toml
+[models.gemini]
+provider = "google"
+model = "gemini-2.0-flash-exp"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `gemini-2.0-flash-exp` | Fast experimental model |
+
+Check Google AI Studio for the full list of available model IDs.
+
+**Credential:** `google_key` (field: `token`)
+
+### xAI
+
+```toml
+[models.grok]
+provider = "xai"
+model = "grok-beta"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `grok-beta` | Grok beta |
+
+**Credential:** `xai_key` (field: `token`)
+
+### Mistral
+
+```toml
+[models.mistral]
+provider = "mistral"
+model = "mistral-large-2411"
+authType = "api_key"
+```
+
+| Model | Description |
+|-------|-------------|
+| `mistral-large-2411` | Mistral Large (November 2024) |
+
+Check Mistral's docs for the full list of available model IDs.
+
+**Credential:** `mistral_key` (field: `token`)
+
+### OpenRouter
+
+OpenRouter provides access to models from many providers through a single API.
+
+```toml
+[models.or-sonnet]
+provider = "openrouter"
+model = "anthropic/claude-3.5-sonnet"
+authType = "api_key"
+```
+
+Model IDs use the `provider/model` format. See OpenRouter's model list for all available models.
+
+**Credential:** `openrouter_key` (field: `token`)
+
+### Custom
+
+For any provider not listed above. The model ID and API routing are handled by the underlying pi.dev agent harness.
+
+```toml
+[models.my-model]
+provider = "custom"
+model = "your-model-name"
+authType = "api_key"
+```
+
+**Credential:** `custom_key` (field: `token`)
+
+## Mixing Models
+
+Each agent can use a different model. Define all models in the project's `config.toml`, then reference them by name in each agent's `config.toml`:
+
+```
+config.toml              → [models.sonnet], [models.gpt4o], [models.groq-llama]
+agents/dev/config.toml   → models = ["sonnet"]
+agents/reviewer/config.toml → models = ["gpt4o"]
+agents/devops/config.toml   → models = ["groq-llama"]
+```
+
+## Model Fallback
+
+Agents can list multiple models to create a fallback chain. When the primary model is rate-limited or unavailable, Action Llama automatically tries the next model in the list:
+
+```toml
+# agents/<name>/config.toml
+models = ["sonnet", "haiku", "gpt4o"]
+```
+
+Fallback switching is instant — there is no delay when moving to the next model. Exponential backoff only kicks in after all models in the chain have been exhausted.
+
+A circuit breaker tracks model availability in memory. When a model returns a rate limit or overload error, it is marked unavailable for 60 seconds. After the cooldown expires, the model is retried.
+
+## Credential Setup
+
+Each provider requires a corresponding credential in `~/.action-llama/credentials/`. Run `al doctor` to configure them interactively.
+
+LLM credentials are loaded automatically based on the models referenced in the agent's `models` list in `config.toml` — they do not need to be listed in the agent's `credentials` array. The `credentials` array is for runtime credentials the agent uses during execution (GitHub tokens, SSH keys, etc.).
+
+See [Credentials](/reference/credentials) for the full credential reference.
+
+---
+
+# Webhooks
+
+Action Llama agents can be triggered by webhooks in addition to (or instead of) cron schedules. Seven providers are supported: GitHub, Sentry, Linear, Mintlify, Slack, Discord, and X (Twitter).
+
+## Supported Providers
+
+| Provider | Type | Description |
+|----------|------|-------------|
+| [GitHub](/reference/webhooks/github) | `"github"` | Repository events: issues, pull requests, pushes, workflow runs, and more |
+| [Sentry](/reference/webhooks/sentry) | `"sentry"` | Error monitoring events: alerts, issues, errors, comments |
+| [Linear](/reference/webhooks/linear) | `"linear"` | Project management events: issues, comments, labels |
+| [Mintlify](/reference/webhooks/mintlify) | `"mintlify"` | Documentation events: build successes and failures |
+| [Slack](/reference/webhooks/slack) | `"slack"` | Workspace events: messages, app mentions, reactions |
+| [Discord](/reference/webhooks/discord) | `"discord"` | Discord Interactions Endpoint: slash commands, message components, modals, autocomplete |
+| [X (Twitter)](/reference/webhooks/twitter) | `"twitter"` | Account activity events: tweets, likes, follows, direct messages |
+
+See each provider's page for filter fields, setup instructions, and example configurations.
+
+## Defining Webhook Sources
+
+Webhook sources are defined once in the project's `config.toml`. Each source has a name, a provider type, and an optional credential for signature validation:
+
+```toml
+[webhooks.my-github]
+type = "github"
+credential = "MyOrg"          # credential instance name (github_webhook_secret:MyOrg)
+
+[webhooks.my-sentry]
+type = "sentry"
+credential = "SentryProd"     # credential instance name (sentry_client_secret:SentryProd)
+
+[webhooks.my-linear]
+type = "linear"
+credential = "LinearMain"     # credential instance name (linear_webhook_secret:LinearMain)
+
+[webhooks.my-mintlify]
+type = "mintlify"
+credential = "MintlifyMain"   # credential instance name (mintlify_webhook_secret:MintlifyMain)
+
+[webhooks.my-slack]
+type = "slack"
+credential = "MainWorkspace"  # credential instance name (slack_signing_secret:MainWorkspace)
+
+[webhooks.my-twitter]
+type = "twitter"
+credential = "MyXApp"         # credential instance name (x_twitter_api:MyXApp)
+```
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `type` | string | Yes | Provider type: `"github"`, `"sentry"`, `"linear"`, `"mintlify"`, `"slack"`, `"discord"`, or `"twitter"` |
+| `credential` | string | No | Credential instance name for HMAC signature validation. Omit for unsigned webhooks. |
+
+## Agent Webhook Triggers
+
+Agents reference a webhook source by name and add filters in their `config.toml`:
+
+```toml
+# agents/<name>/config.toml
+[[webhooks]]
+source = "my-github"
+repos = ["acme/app"]
+events = ["issues"]
+actions = ["labeled"]
+labels = ["agent"]
+```
+
+Each `[[webhooks]]` entry is a trigger. The `source` field (referencing a name from the project's `config.toml`) is required. All filter fields are optional — omit all of them to trigger on everything from that source.
+
+An agent must have at least one of `schedule` or `webhooks` (or both).
+
+## How Webhooks Work at Runtime
+
+1. The gateway receives a webhook POST request at `/webhooks/<type>` (e.g. `/webhooks/github`)
+2. It verifies the payload signature using secrets loaded from the credential instances defined in `config.toml` webhook sources
+3. It parses the event into a `WebhookContext` (source, event, action, repo, etc.)
+4. It matches the context against each agent's webhook triggers
+5. Matching agents are triggered with the webhook context injected into their prompt
+
+## Hybrid Agents
+
+Agents can have both `schedule` and `webhooks`. Scheduled runs poll for work proactively; webhook runs respond to events immediately.
+
+## Troubleshooting
+
+### Webhooks not firing
+
+1. **Check the webhook URL** — GitHub/Sentry/Linear must be able to reach your gateway. For local development, use a tunnel (e.g. ngrok, Cloudflare Tunnel).
+2. **Check the webhook secret** — if using HMAC validation, the secret in `al creds` must match the one configured in the external service.
+3. **Check event filters** — verify the `events`, `actions`, and other filter fields in the agent's `config.toml` match the incoming event.
+
+```bash
+al doctor    # Validates webhook trigger field names and types
+```
+
+### Webhook events being dropped
+
+If runners are busy, events are queued (up to `workQueueSize`, default 100). If the queue is full, old events are dropped. Check queue depth with `al stat`.
+
+To handle more concurrent events, increase `scale` in the agent's `config.toml`:
+
+```toml
+scale = 3    # Run up to 3 instances concurrently
+```
+
+## Discord Webhooks
+
+Discord support uses the **Interactions Endpoint** (HTTP-based), which covers slash commands, message components, modals, and autocomplete. This requires a `discord_bot` credential with your application's public key for Ed25519 signature verification.
+
+| Filter Field | Type | Description |
+|---|---|---|
+| `events` | string[] | Interaction types: `application_command`, `message_component`, `modal_submit`, `autocomplete` |
+| `guilds` | string[] | Discord guild (server) IDs to filter on |
+| `channels` | string[] | Channel IDs to filter on |
+| `commands` | string[] | Slash command names (e.g. `ask`, `deploy`) — only applies to `application_command` and `autocomplete` events |
+
+### Setup
+
+1. In the [Discord Developer Portal](https://discord.com/developers/applications), create or select your application
+2. Under **General Information**, copy your **Application ID** and **Public Key**
+3. Under **Bot**, copy or reset your **Bot Token**
+4. Add these to your credentials: `al creds add discord_bot`
+5. Set your Interactions Endpoint URL to `https://your-server:8080/webhooks/discord`
+6. Discord will send a PING verification request — Action Llama responds automatically after validating the signature
+
+### Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.my-discord]
+type = "discord"
+credential = "MyBot"     # discord_bot credential instance name
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "my-discord"
+events = ["application_command"]
+commands = ["ask", "status"]
+guilds = ["1234567890"]
+```
+
+---
+
+# GitHub Webhooks
+
+GitHub webhooks let agents respond to repository events like issues, pull requests, and pushes.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `repos` | string[] | Only trigger for these repos |
+| `orgs` | string[] | Only trigger for these organizations |
+| `org` | string | Only trigger for this organization (singular form) |
+| `events` | string[] | GitHub event types (issues, pull_request, push, etc.) |
+| `actions` | string[] | Event actions (opened, labeled, closed, etc.) |
+| `labels` | string[] | Only when issue/PR has these labels |
+| `assignee` | string | Only when assigned to this user |
+| `author` | string | Only for this author |
+| `branches` | string[] | Only for these branches |
+| `conclusions` | string[] | Only for workflow_run events with these conclusions (success, failure, cancelled, skipped, timed_out, action_required) |
+
+## Setup
+
+1. In your GitHub repo, go to **Settings > Webhooks > Add webhook**
+2. Set the payload URL to your Action Llama gateway (e.g. `https://your-server:8080/webhooks/github`)
+3. Set content type to `application/json`
+4. Set the secret to match the `github_webhook_secret` credential instance referenced by the webhook source in `config.toml`
+5. Select the events you want to receive
+
+## Using ngrok for Local Development
+
+```bash
+ngrok http 8080
+```
+
+Use the ngrok URL as your webhook payload URL in GitHub. See [Using Webhooks](/first-steps/using-webhooks) for a full tutorial.
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.my-github]
+type = "github"
+credential = "MyOrg"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "my-github"
+repos = ["acme/app"]
+events = ["issues"]
+actions = ["labeled"]
+labels = ["agent"]
+```
+
+---
+
+# Sentry Webhooks
+
+Sentry webhooks let agents respond to error alerts, metric alerts, and issue events.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `resources` | string[] | Resource types: event_alert, metric_alert, issue, error, comment |
+
+## Setup
+
+1. In Sentry, go to **Settings > Developer Settings > New Internal Integration**
+2. Set the webhook URL to your gateway (e.g. `https://your-server:8080/webhooks/sentry`)
+3. Copy the client secret to `~/.action-llama/credentials/sentry_client_secret/<instance>/secret`
+4. Select the resource types you want to receive
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.my-sentry]
+type = "sentry"
+credential = "SentryProd"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "my-sentry"
+resources = ["event_alert", "issue"]
+```
+
+---
+
+# Linear Webhooks
+
+Linear webhooks let agents respond to issue and comment events in your Linear workspace.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `organizations` | string[] | Only trigger for these Linear organizations |
+| `events` | string[] | Linear event types (issues, issue_comment, etc.) |
+| `actions` | string[] | Event actions (create, update, delete, etc.) |
+| `labels` | string[] | Only when issue has these labels |
+| `assignee` | string | Only when assigned to this user (email) |
+| `author` | string | Only for this author (email) |
+
+## Setup
+
+1. In Linear, go to **Settings > Workspace > Webhooks**
+2. Click **Create webhook**
+3. Set the URL to your Action Llama gateway (e.g. `https://your-server:8080/webhooks/linear`)
+4. Set the secret to match the `linear_webhook_secret` credential instance referenced by the webhook source in `config.toml`
+5. Select the resource types you want to receive (Issues, Comments, etc.)
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.linear-main]
+type = "linear"
+credential = "main-workspace"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "linear-main"
+events = ["issues", "issue_comment"]
+actions = ["create", "update"]
+organizations = ["your-org-id"]
+labels = ["bug", "ready-for-dev"]
+```
+
+---
+
+# Mintlify Webhooks
+
+Mintlify webhooks let agents respond to documentation build events.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `projects` | string[] | Only trigger for these Mintlify projects |
+| `events` | string[] | Mintlify event types (build, etc.) |
+| `actions` | string[] | Event actions (failed, succeeded, etc.) |
+| `branches` | string[] | Only for these branches |
+
+## Setup
+
+1. In Mintlify, go to your project settings
+2. Navigate to **Webhooks** or **Integrations**
+3. Add a webhook with URL: `https://your-server:8080/webhooks/mintlify`
+4. Set the secret to match the `mintlify_webhook_secret` credential instance referenced by the webhook source in `config.toml`
+5. Select build events you want to receive (build failures, successes, etc.)
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.mintlify-docs]
+type = "mintlify"
+credential = "docs-project"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "mintlify-docs"
+events = ["build"]
+actions = ["failed"]
+projects = ["my-docs"]
+```
+
+---
+
+# Slack Webhooks
+
+Slack webhooks let agents respond to workspace events like messages, app mentions, and reactions via the Slack Events API.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `events` | string[] | Slack event types: `message`, `app_mention`, `reaction_added`, `reaction_removed`, `channel_created`, `member_joined_channel` |
+| `channels` | string[] | Only trigger for these Slack channel IDs |
+| `team_ids` | string[] | Only trigger for these Slack workspace/team IDs |
+
+## Setup
+
+1. In the Slack API dashboard (https://api.slack.com/apps), create or select your app
+2. Go to **Event Subscriptions** and enable events
+3. Set the Request URL to `https://your-server:8080/webhooks/slack`
+4. The gateway automatically responds to Slack's URL verification challenge
+5. Subscribe to the bot events you need (e.g., `message.channels`, `app_mention`)
+6. Go to **Basic Information → App Credentials** and copy the **Signing Secret**
+7. Set the signing secret credential to match `slack_signing_secret` referenced by the webhook source in `config.toml`
+
+## URL Verification
+
+The Slack provider automatically handles URL verification challenges. When Slack sends a `url_verification` request, the gateway validates the signature and responds with the challenge token. No manual setup is needed.
+
+## Replay Protection
+
+The provider rejects requests with timestamps older than 5 minutes, protecting against replay attacks.
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.my-slack]
+type = "slack"
+credential = "MainWorkspace"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "my-slack"
+events = ["message", "app_mention"]
+channels = ["C0123456789"]
+team_ids = ["T0123456789"]
+```
+
+---
+
+# Twitter Webhooks
+
+X (Twitter) webhooks let agents respond to account activity events like tweets, likes, follows, and direct messages via the Account Activity API.
+
+## Filter Fields (all optional)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `events` | string[] | Twitter event types: `tweet_create_events`, `tweet_delete_events`, `favorite_events`, `follow_events`, `unfollow_events`, `block_events`, `unblock_events`, `mute_events`, `unmute_events`, `direct_message_events`, `direct_message_indicate_typing_events`, `direct_message_mark_read_events` |
+| `users` | string[] | Only trigger for these subscribed user IDs (`for_user_id` values) |
+
+## Setup
+
+1. In the X Developer Portal, create or open your app
+2. Under **Keys and tokens**, note your **Consumer Key** and **Consumer Secret**
+3. Generate an **Access Token** and **Access Token Secret** for your bot user account
+4. Run `al doctor` — it will prompt for all four values plus your **Bearer Token**
+5. Register your webhook URL with the Account Activity API: `https://your-server:8080/webhooks/twitter`
+6. The gateway handles the CRC challenge-response handshake automatically using the Consumer Secret
+7. On startup, the gateway automatically subscribes your bot user if Access Token credentials are configured
+
+## Credentials
+
+The `x_twitter_api` credential has five fields:
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `consumer_key` | Yes | OAuth 1.0a Consumer Key |
+| `consumer_secret` | Yes | OAuth 1.0a Consumer Secret (used for CRC and HMAC validation) |
+| `bearer_token` | Yes | App-Only Bearer Token (used for API v2 validation) |
+| `access_token` | No | OAuth 1.0a Access Token for the bot user (enables auto-subscribe) |
+| `access_token_secret` | No | OAuth 1.0a Access Token Secret for the bot user (enables auto-subscribe) |
+
+When `access_token` and `access_token_secret` are provided, the gateway will automatically check the webhook registration status and subscribe the bot user to the Account Activity API on startup.
+
+## CRC Challenge
+
+The Twitter provider automatically handles CRC (Challenge-Response Check) validation. When Twitter sends a GET request with a `crc_token` query parameter, the gateway responds with the correct HMAC-SHA256 response token. No manual setup is needed.
+
+## Example Configuration
+
+```toml
+# In project config.toml
+[webhooks.my-twitter]
+type = "twitter"
+credential = "MyXApp"
+```
+
+```toml
+# In agents/<name>/config.toml
+[[webhooks]]
+source = "my-twitter"
+events = ["tweet_create_events", "favorite_events", "direct_message_events"]
+users = ["123456789"]
+```