archal 0.9.18 → 0.9.20

package/skills/eval/SKILL.md

@@ -90,6 +90,40 @@ Exit codes: `0` pass, `1` fail or score < threshold, `2` validation error. For G
 
 Workspace API keys are runtime and CI credentials bound to one workspace. They can run clones, upload and read traces, and read usage for that workspace. They cannot manage audit events or workspace API keys. Use an owner/admin user credential, either `archal login` or a dashboard-issued user API key, for workspace administration.
 
+## Pre-production autonomous loop
+
+Use `archal preprod start` when the user wants a coding agent to run a bounded
+pack of scenarios before shipping, remediate failures, rerun, validate, and
+open a draft PR. This is different from post-production `archal autoloop`: it
+starts from repo scenarios and clone runs, not imported production traces.
+
+First do a safe dry run:
+
+```bash
+archal preprod start --scenario-count 20 --dry-run --artifacts .archal/preprod
+```
+
+Then, only after the dry-run artifacts look like real agent/scenario failures,
+allow the managed remediation path:
+
+```bash
+archal preprod start \
+  --scenario-count 20 \
+  --allow-external-execution \
+  --remediation-agent codex \
+  --validation-command 'pnpm test' \
+  --open-pr \
+  --pr-command 'gh pr create --draft --fill' \
+  --artifacts .archal/preprod
+```
+
+Read `.archal/preprod/preprod-result.json`,
+`.archal/preprod/preprod-failures.json`, and the remediation context before
+summarizing results. Treat runs without validation evidence as local
+remediation passes, not release proof. If a run stops after `initial-runs`,
+`fix`, or `validation`, resume with `archal preprod start --resume
+.archal/preprod --artifacts .archal/preprod`.
+
 ## Artifacts + dashboard
 
 - **Local (always written):** `.archal/cache/last-run.json` (summary), `.archal/cache/runs/*.json` (full redacted trace).
@@ -108,6 +142,6 @@ Don't tell users they need `-o json` to save artifacts locally - that's only for
 ## Docs
 
 - Running with an agent: https://docs.archal.ai/guides/run-with-agent
-- Existing repo playbook: https://docs.archal.ai/guides/existing-agent-repo
+- Existing repo playbook: https://docs.archal.ai/guides/run-with-agent
 - Scenario authoring: hand off to the `scenario` skill
 - Clone sessions: https://docs.archal.ai/guides/clone-sessions

package/skills/install-agent/SKILL.md

@@ -0,0 +1,221 @@
+---
+name: install-agent
+description: Connect an agent's repo and its production observability to Archal so its traces get captured and graded. Detects an existing observability stack (LangSmith, Langfuse, Datadog, OpenTelemetry, Braintrust), connects the GitHub App, opens an observability setup PR, and wires an existing trace vendor through `archal trace-source`. USE THIS whenever the user says "connect my agent", "install the Archal agent", "set up observability", "capture my agent's traces", "hook up my production traces", "where do my traces go", or asks how to get an already-running agent into Archal. Reach for it before telling anyone a capability is missing — read the honest limits below first.
+user-invocable: true
+argument-hint: "[repo + where its traces live]"
+---
+
+# Archal Install Agent
+
+You are connecting a real, already-running agent to Archal so its production
+behavior gets captured and graded. Two things have to land: (1) Archal can read
+the **repo** (GitHub App), and (2) Archal can read the agent's **traces** —
+either by adding instrumentation through a setup PR, or by ingesting an existing
+observability vendor. Once traces flow, grading and the autoloop take over.
+
+Be honest about what this is. The "install agent" is **not** a sandboxed coding
+agent that edits arbitrary code in the repo. It is deterministic repo inspection
+plus a deterministic, templated setup PR, plus an optional one-shot managed
+planner (an LLM call) that only relocates the bootstrap file and writes advisory
+PR-body text. Set that expectation up front so nobody waits for an autonomous
+coder that does not exist yet. The honest limit is spelled out below — do not
+oversell it.
+
+## Why this exists
+
+Archal grades agent behavior from traces. An agent that already runs in
+production has traces somewhere — your own logs, or a vendor like Langfuse or
+Braintrust. The install path's whole job is to get those traces into Archal's
+normalized shape without you hand-writing exporters or copying secrets around.
+Capturing the trace is the precondition for everything downstream: grading,
+reproduction, and the autoloop that turns reproduced failures into PRs.
+
+## Discover first
+
+Before changing anything, read the repo and find out where traces live:
+
+1. `package.json` / `pyproject.toml` / `requirements.txt`: language and
+   framework. Language matters — the planner and `@archal/state-capture` are
+   TypeScript-only today (see limits).
+2. Existing observability dependencies. Archal's detector recognizes exactly
+   five vendors by dependency name:
+   - `langsmith` -> LangSmith
+   - `langfuse`, `langfuse-node` (TS) / `langfuse` (py) -> Langfuse
+   - `dd-trace` (TS) / `ddtrace` (py) -> Datadog
+   - `@opentelemetry/sdk-node`, `@opentelemetry/sdk-trace-node` (TS) /
+     `opentelemetry-sdk` (py) -> OpenTelemetry
+   - `braintrust` -> Braintrust
+   A repo with any of these is a candidate for **augment**; a repo with none is
+   **greenfield**.
+3. GitHub remote — augment/greenfield setup PRs and the autoloop need a GitHub
+   remote that resolves to `github.com/<owner>/<repo>`:
+   ```bash
+   git remote get-url origin
+   ```
+4. Where do the traces actually go? Ask the user. The answer routes you:
+   - already in a hosted vendor (Langfuse, Braintrust) or a Postgres/Supabase
+     table -> ingest path (`archal trace-source`, delegate detail to `autoloop`)
+   - exported files on disk -> `archal trace-source import`
+   - nowhere yet / only app logs -> the observability setup PR
+
+Never print secrets while inspecting. Show env var names or secret references,
+never plaintext keys or database URLs.
+
+## Preconditions
+
+- Archal CLI installed in the repo or reachable with `npx archal`
+- authenticated user (`archal login`) or `ARCHAL_TOKEN=archal_ws_...` (a
+  workspace key for CI)
+- the **Archal GitHub App** installed on the target repo (required for the setup
+  PR and for autoloop fix PRs)
+- a GitHub remote resolving to `github.com/<owner>/<repo>`
+- for the ingest path: a read-only credential for the trace vendor
+
+If a precondition is missing, make the smallest safe change and name what is
+still required. Do not fake a connection.
+
+## Step 1 — connect the GitHub App
+
+The repo connection is the GitHub App, not a token paste. Confirm the **Archal
+GitHub App** is installed on the target repository and that the org granted it
+access. Without it, the setup PR cannot be opened and the autoloop cannot open
+fix PRs. If it is not installed, send the user to the dashboard's integration
+flow to install it, then continue.
+
+## Step 2 — open the observability setup PR
+
+When traces are not yet exported anywhere (or the user wants Archal's own
+capture), open the **observability setup PR**. It is a deterministic, templated
+patch — every file's contents are pre-generated; nothing is freely authored by
+an LLM. The patch resolves into one of two install modes:
+
+### Greenfield (no existing observability detected)
+
+Adds standard OpenTelemetry instrumentation pointed at Archal's OTLP endpoint:
+
+- `archal-otel.ts` (TS) or `archal_otel.py` (Python) — an OpenTelemetry init
+  bootstrap (OTLP HTTP exporter + the node/python SDK), **not** an Archal-only
+  exporter
+- `archal-replay-capsule.ts` / `archal_replay_capsule.py` — a replay helper
+  template
+- OpenTelemetry SDK + framework instrumentation added to `package.json` (TS) or
+  `requirements.txt` (Python)
+- an `.env.example` entry for the workspace key and an `ARCHAL_OBSERVABILITY.md`
+  guide
+
+### Augment (existing observability detected, TypeScript only)
+
+When the repo already has one of the five vendors above **and** is TypeScript,
+the PR instead adds Archal's state capture alongside the existing stack rather
+than replacing it:
+
+- `archal-state-capture.ts` importing from `@archal/state-capture`
+- `@archal/state-capture` and `@opentelemetry/api` added to `package.json`
+- the same `.env.example` entry and `ARCHAL_OBSERVABILITY.md` guide
+
+Python repos with existing observability fall back to the greenfield
+OpenTelemetry install — `@archal/state-capture` has no Python build yet, and the
+PR body says so. Tell the user that explicitly rather than implying parity.
+
+### The optional install planner (managed LLM)
+
+On Pro/Enterprise workspaces, for a TypeScript greenfield/augment install with
+repo detection available, one managed LLM call (intent `observability-install`,
+public label **Archal install planner**, routed through the managed eval model
+lane — gemini-class — and metered as `cogs_only` spend) adapts the deterministic
+patch to the repo's real layout. It is strictly additive and **fail-open**: any
+availability, auth, plan-gate, or validation problem ships the deterministic
+patch unchanged. The planner can only:
+
+- relocate the bootstrap file to a better directory (path only — file contents
+  are never edited)
+- append advisory text to the PR body (where to wire startup, which functions to
+  wrap)
+
+It never edits application code, never modifies existing instrumentation, and
+never runs free-form codegen. Disable it with `ARCHAL_INSTALL_PLANNER_DISABLED=1`
+to force the deterministic install.
+
+## Step 3 — ingest an existing observability vendor
+
+If the agent already emits traces to a vendor, you usually do **not** need the
+setup PR — you normalize the existing traces with `archal trace-source`. This is
+the genuine ingest path. It maps a vendor's payloads into Archal's trace upload
+envelopes and uploads them to hosted autoloop when workspace auth is present.
+
+Supported providers: `langfuse`, `braintrust`, `otel`, `http`, `supabase`,
+`postgres`, `file`, `custom`. Pull/sync vendors (`langfuse`, `braintrust`,
+Postgres/Supabase) are fetched on a cursor; push sources (`otel`, `http`,
+`custom`) receive traces continuously through `serve`.
+
+The command surface:
+
+```bash
+archal trace-source connect <provider>   # register a source (e.g. langfuse, braintrust, otel, custom)
+archal trace-source test [source]        # validate credentials and reachability
+archal trace-source sync [source]        # pull-fetch new traces (langfuse/braintrust/db)
+archal trace-source watch [source]       # continuous pull loop
+archal trace-source serve [source]       # receiver for push sources (otel/http/custom)
+archal trace-source import <path>        # normalize exported trace files on disk
+archal trace-source status [source]      # registry validation, cursor, last-sync state
+archal trace-source list                 # registered sources
+archal trace-source use|disable <source> # select / disable a source
+```
+
+**Delegate the deep mapping to the `autoloop` skill.** It owns the per-vendor
+flags (`--base-url`, `--api-key-env`, `--out`, `--upload`, `--repository`,
+schema/cursor/filter mapping for database sources) and the full
+import/grade/reproduce/fix loop. Quote the command names here; point the user
+there for the flag detail and the autoloop wiring.
+
+## The honest limit
+
+There is **no sandboxed coding agent** that reads the whole repo and edits
+arbitrary code to wire up instrumentation. For a small or conventional repo, the
+setup PR drops in and works. For a large or unusual repo, the setup PR is a
+**generic bootstrap the user finishes** — they still import the bootstrap at
+startup and wrap the call sites the planner's advisory section points at. Say
+this plainly. The typed model lanes `remediation_agent` and
+`observability_install_agent` exist in the contract but are
+`agent-executor-contract-only`: the lane is declared, but no real executor
+consumes it yet. Do not describe either as a working autonomous coder.
+
+## Failure taxonomy
+
+Classify precisely; do not paper over a missing precondition:
+
+- **GitHub App not connected** — setup PR and fix PRs cannot open. Install the
+  Archal GitHub App on the repo.
+- **No GitHub remote** — augment/greenfield PRs need `github.com/<owner>/<repo>`.
+- **Language unsupported by augment** — Python with existing observability falls
+  back to greenfield OTel; `@archal/state-capture` is TS-only.
+- **Planner skipped** — wrong plan (needs Pro/Enterprise), non-TypeScript,
+  `augment-existing-vendor` legacy mode, detection unavailable, or
+  `ARCHAL_INSTALL_PLANNER_DISABLED=1`. The deterministic patch still ships; this
+  is not a failure, just a narrower install.
+- **Setup PR is a stub for this repo** — large/unusual layout; the user finishes
+  wiring. Not a bug; the honest limit.
+- **Trace ingest failure** — `trace-source` adapter mismatch, bad credential,
+  rejected upload, or missing workspace auth. Use `archal trace-source test` and
+  `status` to localize it.
+- **No usable trace evidence** — once ingested, grading or reproduction can still
+  block if the trace lacks task context or state. Hand off to the `autoloop`
+  skill's evidence diagnosis.
+
+## What to report back
+
+After install or debugging, give the user:
+
+- repo full name and whether the GitHub App is connected
+- chosen path: setup PR (greenfield vs augment) or `trace-source` ingest
+- if a setup PR: the install mode, the files it adds, and whether the planner ran
+- if ingest: provider, source id, and the next `archal trace-source` command
+- whether traces are flowing into Archal yet, or the exact blocker
+- next command or next owner
+
+## Docs
+
+- Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces
+- Autonomous loops: https://docs.archal.ai/guides/autoloop-production-traces
+- CLI reference: https://docs.archal.ai/cli/autoloop
+- Quickstart: https://docs.archal.ai/quickstart

package/skills/onboard/SKILL.md

@@ -87,6 +87,16 @@ Confirm detected clones, then ask which of these the user wants. Each delegates
 
 If the user doesn't have a harness yet, prefer `npx archal init`; it creates `./.archal/harness.mjs`, points `.archal.json` at it, and adds a starter scenario without overwriting existing files. The generated harness is a guarded stub: Archal refuses to score it until the user edits it to call their Cursor, Codex, Claude Code, or custom agent. A custom harness should read `AGENT_TASK` from env, call the agent runtime, print `{ "text": "..." }` to stdout, and call `reportAgentMetrics()` from `archal/harness` with accumulated `{ inputTokens, outputTokens, llmCallCount }` before exit. Service clients need one explicit routing mode: use sandbox/Docker routing when the harness calls normal service URLs such as `https://api.github.com`, or configure SDK base URLs from `AGENT_CLONE_URLS` and add the JSON headers from `AGENT_ROUTE_HEADERS` to those clone requests. Alternative: skip `agent` in `.archal.json` and pass `--harness <path>` per-run.
 
+### Or run a packaged agent (no harness to write)
+
+If the user just wants to evaluate a real, ready-made agent, point them at a packaged agent instead of writing a harness. A packaged agent runs unmodified in Docker while Archal's TLS-intercept sidecar routes its calls to seeded clones and injects the host's model API key on its model calls. The bundled agents live under `examples/agents/<name>` (`openclaw`, `hermes`, `github-octokit`).
+
+- `archal run <scenario>.md --sandbox` - run the bundled OpenClaw agent (needs Docker). Pick the model with `--agent-model <provider/model>`; export the matching key in the shell first (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`).
+- `archal run <scenario>.md --harness examples/agents/hermes --dockerfile examples/agents/hermes/Dockerfile` - run any other bundled agent through the Docker harness flags (swap in `github-octokit` for the other one).
+- `archal run <scenario>.md --harness ./<dir> --dockerfile ./<dir>/Dockerfile` - run your own packaged agent. A packaged agent is just a directory with a `Dockerfile`, a drive script (reads `AGENT_TASK`, prints the answer to stdout), and an `.archal.json`.
+
+See the "Run a packaged agent" guide: https://docs.archal.ai/guides/packaged-agents
+
 ### Option A - Evaluate an agent with scenarios
 
 Write markdown scenario files that describe setup, prompt, and success criteria; `archal run` executes them against clones.
@@ -122,6 +132,74 @@ Do not paste a sample config here. The right shape depends on what's already in
 
 Run: `archal clone start <detected clones>` - gives live clone URLs the user's SDK clients can point at. `archal clone status` shows the active session; `archal clone stop` tears down.
 
+### Option E - Bounded pre-prod autonomous loop
+
+Use this when the repo already has scenarios or can safely generate starter
+pre-prod scenarios, and the user wants a coding agent to run checks, classify
+failures, optionally patch, validate, and open a draft PR.
+
+Start with:
+
+```bash
+archal preprod plan --repo . --write-scenarios --write-config --out .archal/preprod-plan.json
+archal preprod start --scenario-count 20 --dry-run --artifacts .archal/preprod
+```
+
+`--write-scenarios` writes generated scenario markdown under `archal/` by
+default, and `--write-config` writes `.archal.json` only when it can do so
+without overwriting an existing config. `preprod start` creates or reuses
+`.archal/preprod-pack.json`, writes generated scenarios under
+`archal/generated/` by default, runs the pack, and leaves resumable artifacts.
+If the repo already has `.archal.json`, read `.archal/preprod-plan.json` and
+confirm the detected clone/harness surface before starting the loop.
+
+Only enable local fix or PR commands after the dry-run artifacts have been
+reviewed:
+
+```bash
+archal preprod start \
+  --scenario-count 20 \
+  --allow-external-execution \
+  --remediation-agent codex \
+  --validation-command '<test command>' \
+  --open-pr \
+  --pr-command '<draft-pr command>' \
+  --artifacts .archal/preprod
+```
+
+`--open-pr` requires both `--validation-command` and `--pr-command`; PR
+publishing still stays disabled unless `--allow-external-execution` is present.
+`preprod start` uses the managed preprod remediation path by default. It writes
+a repo-local remediation context, invokes the selected coding agent, reruns the
+scenario pack, and validates before PR creation. The remediation command
+receives `ARCHAL_PREPROD_FAILURES_JSON`, `ARCHAL_PREPROD_ATTEMPT`,
+`ARCHAL_PREPROD_REMEDIATION_CONTEXT_PATH`, and `ARCHAL_PREPROD_USAGE_PATH`.
+If the coding agent can report its own model usage, write JSON to
+`ARCHAL_PREPROD_USAGE_PATH` with fields such as `inputTokens`, `outputTokens`,
+`provider`, `model`, `isByok`, and `costUsd`.
+
+Tell the user to inspect `.archal/preprod/preprod-result.json` and
+`.archal/preprod/preprod-failures.json` for status, stop reason, attempts,
+scenario run ids, validation, and PR summary. If the run was stopped with
+`--stop-after` or interrupted, resume with `archal preprod start --resume
+.archal/preprod --artifacts .archal/preprod`.
+
+### Option F - Autoloop real trace sources
+
+Use this when the repo already has agent traces from pre-production or
+production and the user wants Archal to import, grade, reproduce, and turn
+reproduced failures into GitHub issues or PRs.
+
+**Delegate to the `autoloop` skill.** It owns the trace-source mapping,
+`archal/harness.json`, `archal/scenario.md`, seed templates, `archal autoloop`
+commands, dashboard expectations, and failure taxonomy. Do not inline the
+Autoloop flow here; it changes faster than starter scenario setup.
+
+Set the expectation carefully: Autoloop is not arbitrary production trace replay.
+It can reproduce only failures with enough trace evidence plus repo-owned
+scenario/seed context to reconstruct realistic clone state. Missing evidence
+should block with a clear artifact instead of being guessed.
+
 ## Verify
 
 Run the first scenario or task. For Options A and B, hand off to the `eval` skill to interpret the satisfaction score and diagnose failures - that skill owns the runtime mental model (`[D]` vs `[P]` criteria, trace inspection, harness execution diagnostics).
@@ -144,3 +222,5 @@ Run the first scenario or task. For Options A and B, hand off to the `eval` skil
 
 - Quickstart: https://docs.archal.ai/quickstart
 - Full docs: https://docs.archal.ai
+- Autonomous loops: https://docs.archal.ai/guides/autoloop-production-traces
+- Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces

package/skills/scenario/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: scenario
-description: Write, edit, and validate Archal scenario files. Knows the markdown format, success criteria syntax, and config options.
+description: Write, edit, and validate Archal scenario markdown — the format, success criteria syntax, and config. USE THIS whenever the user wants to "write a scenario", "add a test for my agent", "fix/edit my scenario", asks "what's the success criteria syntax" or about `[D]`/`[P]` criteria, needs a multi-clone scenario, or is validating scenario files. Reach for it on any mention of authoring or fixing Archal scenarios.
 user-invocable: true
 argument-hint: "[scenario description or file path]"
 ---
@@ -15,7 +15,7 @@ You write and edit Archal scenario files. Scenarios are markdown files that defi
 # Scenario Title
 
 ## Setup
-Starting state described in plain English. Drives seed generation.
+Starting state in plain English. Context Archal reconstructs and the agent + evaluator read. Does NOT generate seed state.
 
 ## Prompt
 The task instruction given to the agent.
@@ -95,14 +95,25 @@ and `archal seed list` over maintaining a separate list in this skill.
 | Clone | Seeds |
 |------|-------|
 | `apify` | `empty` |
+| `calcom` | `empty`, `demo` |
+| `clickup` | `empty`, `demo` |
+| `customerio` | `empty` |
+| `datadog` | `empty` |
 | `github` | `empty`, `small-project`, `enterprise-repo`, `ci-cd-pipeline`, `stale-issues`, `large-backlog` |
+| `gitlab` | `empty`, `demo` |
+| `hubspot` | `empty`, `demo`, `stale-data` |
 | `slack` | `empty`, `engineering-team`, `busy-workspace`, `incident-active` |
 | `stripe` | `empty`, `small-business`, `checkout-flow`, `subscription-lifecycle`, `subscription-heavy` |
 | `jira` | `empty`, `small-project`, `enterprise`, `sprint-active`, `large-backlog` |
 | `linear` | `empty`, `small-team`, `engineering-org`, `multi-team`, `busy-backlog` |
 | `supabase` | `empty`, `small-project`, `saas-starter`, `ecommerce` |
 | `google-workspace` | `empty`, `assistant-baseline`, `gmail-busy-inbox`, `calendar-packed-week` |
+| `ownerrez` | `empty` |
+| `pricelabs` | `empty` |
+| `sentry` | `empty`, `demo` |
 | `tavily` | `empty` |
+| `unipile` | `empty` |
+| `webflow` | `empty` |
 | `ramp` | `empty`, `default` |
 | `discord` | `empty`, `small-server`, `harvested` |
 
@@ -131,7 +142,11 @@ Use multiple clones by listing them in config:
 clones: github, slack
 ```
 
-The Setup section can describe state across both services. Each clone gets its own seed.
+The Setup section can describe context across both services. Attach explicit seed state per clone via `seed:` or `## Seed State` (see Seed state below).
+
+## Seed state
+
+Seeding is deterministic — explicit committed state, no LLM. Scenarios attach it via the `seed:` config key or a `## Seed State` section. To author or load explicit JSON/SQL/catalog state into a clone, delegate to the sibling `seed` skill (`packages/archal/skills/seed`) rather than handling the seeding mechanics here.
 
 ## Validation
 
@@ -147,7 +162,7 @@ Run `archal scenario list` to verify scenarios parse correctly. A valid scenario
 1. Writing `[D]` criteria that require subjective judgment
 2. Writing `[P]` criteria that could be checked deterministically
 3. Forgetting to specify which clone the scenario uses
-4. Writing Setup descriptions that are too vague for seed generation
+4. Writing Setup descriptions too vague to ground the agent and evaluator
 5. Using seed names that don't exist (check the seed table above)
 
 ## Documentation