archal 0.9.19 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (83) hide show
  1. package/README.md +62 -154
  2. package/agents/openclaw/.archal.json +8 -0
  3. package/agents/openclaw/Dockerfile +97 -0
  4. package/agents/openclaw/README.md +113 -0
  5. package/agents/openclaw/drive.mjs +301 -0
  6. package/agents/openclaw/extract-openclaw-response-text.mjs +49 -0
  7. package/agents/openclaw/package.json +9 -0
  8. package/agents/openclaw/scenarios/acceptance/github-escalation-and-cleanup.md +43 -0
  9. package/agents/openclaw/scenarios/acceptance/jira-linear-release-triage.md +39 -0
  10. package/agents/openclaw/scenarios/acceptance/multi-clone-customer-risk-review.md +45 -0
  11. package/agents/openclaw/scenarios/acceptance/slack-incident-handoff.md +42 -0
  12. package/agents/openclaw/scenarios/acceptance/stripe-dunning-remediation.md +40 -0
  13. package/agents/openclaw/scenarios/close-stale-issues.md +35 -0
  14. package/agents/openclaw/scenarios/github-issue-triage-read-only.md +44 -0
  15. package/agents/openclaw/workspace/AGENTS.md +23 -0
  16. package/agents/openclaw/workspace/IDENTITY.md +8 -0
  17. package/agents/openclaw/workspace/SOUL.md +14 -0
  18. package/agents/openclaw/workspace/TOOLS.md +54 -0
  19. package/clone-assets/apify/tools.json +256 -22
  20. package/clone-assets/calcom/tools.json +2062 -0
  21. package/clone-assets/clickup/tools.json +3245 -0
  22. package/clone-assets/customerio/tools.json +2350 -0
  23. package/clone-assets/datadog/tools.json +734 -0
  24. package/clone-assets/github/tools.json +307 -27
  25. package/clone-assets/gitlab/tools.json +10688 -0
  26. package/clone-assets/google-workspace/tools.json +18 -6
  27. package/clone-assets/hubspot/tools.json +1604 -0
  28. package/clone-assets/jira/fidelity.json +1 -1
  29. package/clone-assets/jira/tools.json +266 -543
  30. package/clone-assets/linear/tools.json +278 -40
  31. package/clone-assets/ownerrez/tools.json +548 -0
  32. package/clone-assets/pricelabs/tools.json +343 -0
  33. package/clone-assets/sentry/tools.json +2821 -0
  34. package/clone-assets/slack/tools.json +1 -2
  35. package/clone-assets/stripe/tools.json +299 -46
  36. package/clone-assets/supabase/tools.json +437 -0
  37. package/clone-assets/unipile/tools.json +408 -0
  38. package/clone-assets/webflow/tools.json +2185 -0
  39. package/dist/autoloop-worker-types-CHaclqtD.d.cts +160 -0
  40. package/dist/cli.cjs +127896 -84811
  41. package/dist/{vitest/chunk-L36NXAU6.js → commands/autoloop-hosted-control-plane-client.cjs} +16344 -18845
  42. package/dist/commands/autoloop-hosted-control-plane-client.d.cts +133 -0
  43. package/dist/commands/autoloop-pr-verification.cjs +4312 -0
  44. package/dist/commands/autoloop-pr-verification.d.cts +19 -0
  45. package/dist/commands/autoloop-result-parser.cjs +27058 -0
  46. package/dist/commands/autoloop-result-parser.d.cts +196 -0
  47. package/dist/commands/autoloop-worker.cjs +38488 -0
  48. package/dist/commands/autoloop-worker.d.cts +102 -0
  49. package/dist/eval-shim.mjs +216 -0
  50. package/dist/index.cjs +15 -2
  51. package/dist/index.d.cts +3 -1
  52. package/dist/layer-1-ziaLpfLr.d.cts +62 -0
  53. package/dist/metafile-cjs.json +1 -0
  54. package/dist/reporter-entry.js +3 -0
  55. package/dist/sdk/index.cjs +51630 -0
  56. package/dist/sdk/index.d.cts +626 -0
  57. package/dist/vitest/chunk-2PDHTPZC.js +4867 -0
  58. package/dist/vitest/chunk-7L4BBB6M.js +2613 -0
  59. package/dist/vitest/index.cjs +6152 -75718
  60. package/dist/vitest/index.d.ts +22 -86
  61. package/dist/vitest/index.js +45 -414
  62. package/dist/vitest/runtime/hosted-session-reaper.cjs +682 -34399
  63. package/dist/vitest/runtime/hosted-session-reaper.js +1 -1
  64. package/dist/vitest/runtime/setup-files.js +2 -2
  65. package/manifest.json +9 -0
  66. package/package.json +20 -14
  67. package/skills/archal-agent/SKILL.md +86 -0
  68. package/skills/autoloop/SKILL.md +391 -0
  69. package/skills/autoloop/references/hosted-sources.md +94 -0
  70. package/skills/autoloop/references/trace-schema-mapping.md +104 -0
  71. package/skills/eval/SKILL.md +17 -15
  72. package/skills/free-account/SKILL.md +103 -0
  73. package/skills/install-agent/SKILL.md +202 -0
  74. package/skills/onboard/SKILL.md +29 -18
  75. package/skills/scenario/SKILL.md +36 -18
  76. package/skills/seed/SKILL.md +238 -0
  77. package/skills/vitest/SKILL.md +3 -2
  78. package/dist/harness.cjs +0 -62
  79. package/dist/harness.d.cts +0 -20
  80. package/dist/seed/dynamic-generator.cjs +0 -45687
  81. package/dist/seed/dynamic-generator.d.cts +0 -106
  82. package/dist/vitest/chunk-WZ7SA4CK.js +0 -47369
  83. package/skills/attach/SKILL.md +0 -402
@@ -0,0 +1,104 @@
1
+ # Trace schema mapping
2
+
3
+ Config reference for hosted Postgres/Supabase trace sources whose trace and span
4
+ tables do not match the defaults below. Schema mapping, cursor mode, and filters
5
+ live in the `.archal.json` `autoloop` block under `autoloop.source`, not in flags.
6
+
7
+ ## Contents
8
+
9
+ - Defaults
10
+ - Custom schema
11
+ - Append-only sources
12
+ - Filters for noisy sources
13
+
14
+ ## Defaults
15
+
16
+ | Concept | Config field | Default |
17
+ |---------|--------------|---------|
18
+ | trace table | `source.mapping.traces.table` | `ai_traces` |
19
+ | span table | `source.mapping.spans.table` | `ai_spans` |
20
+ | trace id | `source.mapping.traces.idColumn` | `id` |
21
+ | span id | `source.mapping.spans.idColumn` | `id` |
22
+ | span trace id | `source.mapping.spans.traceIdColumn` | `trace_id` |
23
+ | span parent id | `source.mapping.spans.parentSpanIdColumn` | `parent_span_id` |
24
+ | trace updated cursor | `source.mapping.traces.updatedAtColumn` | `updated_at` |
25
+ | span updated cursor | `source.mapping.spans.updatedAtColumn` | `updated_at` |
26
+ | cursor mode | `source.cursor.mode` | `updated_at_id` |
27
+
28
+ ## Custom schema
29
+
30
+ For a custom schema, set the mapping in `.archal.json`. Only the fields that
31
+ differ from the defaults are required:
32
+
33
+ ```jsonc
34
+ // .archal.json
35
+ {
36
+ "autoloop": {
37
+ "source": {
38
+ "mapping": {
39
+ "traces": {
40
+ "table": "public.agent_traces",
41
+ "idColumn": "trace_id",
42
+ "updatedAtColumn": "updated_at"
43
+ },
44
+ "spans": {
45
+ "table": "public.agent_spans",
46
+ "idColumn": "span_id",
47
+ "traceIdColumn": "trace_id",
48
+ "parentSpanIdColumn": "parent_span_id",
49
+ "updatedAtColumn": "updated_at"
50
+ }
51
+ }
52
+ }
53
+ }
54
+ }
55
+ ```
56
+
57
+ ## Append-only sources
58
+
59
+ For append-only sources, switch the cursor to `created_at_id` and point it at the
60
+ created-at columns:
61
+
62
+ ```jsonc
63
+ // .archal.json
64
+ {
65
+ "autoloop": {
66
+ "source": {
67
+ "cursor": { "mode": "created_at_id" },
68
+ "mapping": {
69
+ "traces": { "createdAtColumn": "created_at" },
70
+ "spans": { "createdAtColumn": "created_at" }
71
+ }
72
+ }
73
+ }
74
+ }
75
+ ```
76
+
77
+ `source.cursor.watermarks` is a repair escape hatch: set
78
+ `{ traces: { at, id }, spans: { at, id } }` (any of which may be `null`) only when
79
+ you need to re-seat the cursor after a bad import.
80
+
81
+ ## Filters for noisy sources
82
+
83
+ Use filters to narrow a noisy source. All filter fields are optional:
84
+
85
+ ```jsonc
86
+ // .archal.json
87
+ {
88
+ "autoloop": {
89
+ "source": {
90
+ "filters": {
91
+ "workspaceIds": ["workspace_123"],
92
+ "agentIds": ["support-agent"],
93
+ "statuses": ["failed", "error"],
94
+ "traceGroups": ["billing-support"],
95
+ "limit": 250
96
+ }
97
+ }
98
+ }
99
+ }
100
+ ```
101
+
102
+ Arbitrary source metadata for the registration goes under `source.metadata` (a
103
+ map of short-identifier keys to string values). Regression scenarios export is a top-level
104
+ `autoloop.benchmarkExport: true` (the config key keeps the legacy `benchmark` name).
@@ -2,7 +2,7 @@
2
2
  name: eval
3
3
  description: Run Archal scenarios or inline tasks against hosted clones, diagnose failed runs, and interpret satisfaction scores. Triggers on "run my scenario", "evaluate my agent", "archal run X", "debug this failing run", "what does this satisfaction score mean".
4
4
  user-invocable: true
5
- argument-hint: "[scenario.md or task description]"
5
+ argument-hint: '[scenario.md or task description]'
6
6
  ---
7
7
 
8
8
  # Archal Eval Runner
@@ -35,14 +35,14 @@ If a local harness returns text but records zero clone-observed calls, check for
35
35
  Scenario from a file:
36
36
 
37
37
  ```bash
38
- archal run scenario.md
39
- archal run scenario.md --runs 5 --seed enterprise-repo # N runs -> satisfaction score
38
+ archal run scenario.md --docker
39
+ archal run scenario.md --docker --runs 5 --seed enterprise-repo # N runs -> satisfaction score
40
40
  ```
41
41
 
42
42
  Inline task (no scenario file):
43
43
 
44
44
  ```bash
45
- archal run --task "Create an issue titled hello" --harness ./.archal/harness.mjs --clone github
45
+ archal run --task "Create an issue titled hello" --harness ./.archal/harness.mjs --docker --clone github
46
46
  ```
47
47
 
48
48
  `--task` only replaces the scenario file - it still needs a runnable agent path. `--clone` is required with `--task`; repeat or comma-separate for multiple clones.
@@ -52,11 +52,13 @@ When `.archal.json` exists in cwd, bare `archal run` uses it. If the user doesn'
52
52
  ## Interpret results
53
53
 
54
54
  Score breakdown:
55
+
55
56
  - `100%` = every run passed every criterion
56
57
  - `80%` = 4/5 runs passed
57
58
  - `0%` = none passed
58
59
 
59
60
  Criterion types:
61
+
60
62
  - `[D]` - deterministic state check. A failure is real; never a model variance artifact.
61
63
  - `[P]` - LLM judge reads trace + final state. A single failure can be variance; re-run with `--runs 3+` to confirm before acting on it.
62
64
 
@@ -65,30 +67,30 @@ Criterion types:
65
67
  Re-run with `-v` for the full trace, then classify with these signals:
66
68
 
67
69
  - **Agent bug** - wrong tool called, wrong arguments, stopped early.
68
- *Signals:* trace shows the correct tool was available but the agent chose another; or arguments are malformed.
69
- *Fix:* agent prompt, tool wiring, or underlying model.
70
+ _Signals:_ trace shows the correct tool was available but the agent chose another; or arguments are malformed.
71
+ _Fix:_ agent prompt, tool wiring, or underlying model.
70
72
 
71
73
  - **Scenario bug** - criteria are too strict, ambiguous, or contradict the Setup.
72
- *Signals:* agent clearly did the right thing but a `[D]` criterion expects an exact count the Setup didn't guarantee; or two criteria contradict each other.
73
- *Fix:* make Setup more specific, or relax the criterion. Use the `scenario` skill.
74
+ _Signals:_ agent clearly did the right thing but a `[D]` criterion expects an exact count the Setup didn't guarantee; or two criteria contradict each other.
75
+ _Fix:_ make Setup more specific, or relax the criterion. Use the `scenario` skill.
74
76
 
75
77
  - **Seed mismatch** - clone state doesn't match what Setup describes.
76
- *Signals:* agent's first introspection tool call returns unexpected state (e.g. Setup says "4 stale issues" but the seed has 3).
77
- *Fix:* different seed, or adjust Setup to match. `archal seed list <clone>` to browse.
78
+ _Signals:_ agent's first introspection tool call returns unexpected state (e.g. Setup says "4 stale issues" but the seed has 3).
79
+ _Fix:_ different seed, or adjust Setup to match. `archal seed list <clone>` to browse.
78
80
 
79
81
  - **Harness bug** - agent process never started, crashed immediately, or hung.
80
- *Signals:* no tool calls in the trace, stderr shows a boot error, or the run times out at the configured `--timeout`.
81
- *Fix:* smoke-test the harness directly with `AGENT_TASK="Reply with OK." node ./.archal/harness.mjs`, then look for an untouched starter stub, UI-only imports, missing provider keys, or interactive auth.
82
+ _Signals:_ no tool calls in the trace, stderr shows a boot error, or the run times out at the configured `--timeout`.
83
+ _Fix:_ smoke-test the harness directly with `AGENT_TASK="Reply with OK." node ./.archal/harness.mjs`, then look for an untouched starter stub, UI-only imports, missing provider keys, or interactive auth.
82
84
 
83
85
  ## CI mode
84
86
 
85
87
  ```bash
86
- archal run scenario.md --runs 3 --pass-threshold 80 -o json -q
88
+ archal run scenario.md --docker --runs 3 --pass-threshold 80 -o json -q
87
89
  ```
88
90
 
89
91
  Exit codes: `0` pass, `1` fail or score < threshold, `2` validation error. For GitHub Actions, inject `ARCHAL_TOKEN` as a secret. Use a workspace API key (`archal_ws_...`) for CI, not a personal token.
90
92
 
91
- Workspace API keys are runtime and CI credentials bound to one workspace. They can run clones, upload and read traces, and read usage for that workspace. They cannot manage audit events or workspace API keys. Use an owner/admin user credential, either `archal login` or a dashboard-issued user API key, for workspace administration.
93
+ Workspace API keys are runtime and CI credentials bound to one workspace. They can run clones, upload and read traces, and read usage for that workspace. They cannot manage audit events or workspace API keys. Create/list/revoke them with `archal workspace api-key create <label>`, `archal workspace api-keys`, and `archal workspace api-key revoke <key-id>` from an owner/admin user session. Use an owner/admin user credential, either `archal login` or a dashboard-issued user API key, for workspace administration.
92
94
 
93
95
  ## Artifacts + dashboard
94
96
 
@@ -108,6 +110,6 @@ Don't tell users they need `-o json` to save artifacts locally - that's only for
108
110
  ## Docs
109
111
 
110
112
  - Running with an agent: https://docs.archal.ai/guides/run-with-agent
111
- - Existing repo playbook: https://docs.archal.ai/guides/existing-agent-repo
113
+ - Existing repo playbook: https://docs.archal.ai/guides/run-with-agent
112
114
  - Scenario authoring: hand off to the `scenario` skill
113
115
  - Clone sessions: https://docs.archal.ai/guides/clone-sessions
@@ -0,0 +1,103 @@
1
+ ---
2
+ name: free-account
3
+ description: B2C free-tier guide — run scenarios and start clones within free entitlements. Use when an orchestrating or coding agent should use Archal under a free user account to "run my agent", evaluate a task against clones, or spin up clones without paid features.
4
+ user-invocable: true
5
+ argument-hint: "[repo, task, or 'run my agent']"
6
+ ---
7
+
8
+ # Archal Free Account
9
+
10
+ You are driving Archal under a free user account. Archal tests AI agents against
11
+ clones of real services (GitHub, Slack, Stripe, etc.). On free, you can run
12
+ scenarios and start clones within the free entitlements. You cannot start any
13
+ autonomous loop, remediation, or observability-install flow. Stay inside the
14
+ allowed commands below and never attempt a blocked one.
15
+
16
+ ## Authenticate first
17
+
18
+ Do this before any other command.
19
+
20
+ ```bash
21
+ archal login # OAuth browser flow -> ~/.archal/credentials.json
22
+ archal usage # verify auth + plan
23
+ ```
24
+
25
+ In CI, skip the browser flow and set a token:
26
+
27
+ ```bash
28
+ export ARCHAL_TOKEN=... # workspace or user token
29
+ archal usage
30
+ ```
31
+
32
+ Sessions are Archal-hosted, so provider credentials may be placeholders. You do
33
+ not need real GitHub, Slack, or Stripe keys to start a clone or run a scenario.
34
+
35
+ ## Free limits
36
+
37
+ State these upfront so the user is not surprised mid-run.
38
+
39
+ | Limit | Free value |
40
+ |-------|-----------|
41
+ | Cost | $0 |
42
+ | Workspace | free workspace entitlements only |
43
+ | Concurrent sessions | 3 |
44
+ | Session TTL | 30 minutes |
45
+ | Credits | 5,000 per month, recurring |
46
+
47
+ If the user asks for paid workspace capabilities, point them to
48
+ https://archal.ai/contact. Do not attempt blocked actions.
49
+
50
+ ## Allowed commands only
51
+
52
+ Use only these on free. Nothing else.
53
+
54
+ ```bash
55
+ archal run <scenario.md> # run a scenario against clones
56
+ archal clone start [clones...] # start one or more hosted clones
57
+ archal clone status # inspect the active session
58
+ archal clone stop # tear the session down
59
+ archal clone # list available clones
60
+ archal clone seed <clone> <name> # apply a named seed to a running clone
61
+ archal scenario list # list scenarios
62
+ archal seed list # list seeds
63
+ archal usage # check plan, credits, and quota
64
+ ```
65
+
66
+ ## Prohibited on free
67
+
68
+ Never instruct any of these. They are Pro features and fail closed: the web
69
+ returns HTTP 402 and the CLI fails closed with an entitlement error. Do not run
70
+ them, do not retry them, and do not describe them as available.
71
+
72
+ - autonomous remediation loops (`archal autoloop ...`)
73
+ - remediation / fix runs
74
+ - `archal autoloop detach`
75
+ - observability-install flow
76
+
77
+ This mirrors the server-side blocked set: `start_loop`, `reproduce_failure`,
78
+ `retry_loop`, `connect_observability`. If the user wants any of these, point
79
+ them to https://archal.ai/contact.
80
+
81
+ ## Quota awareness
82
+
83
+ Check quota with `archal usage` before a run, and watch for quota-exceeded
84
+ results during a run. On quota-exceeded, stop and direct the user to
85
+ https://archal.ai/contact. Do not retry a blocked or over-quota command, and do
86
+ not try to work around the limit.
87
+
88
+ ## What to report back
89
+
90
+ After a run or clone session, give the user:
91
+
92
+ - command run
93
+ - satisfaction score or clone session id
94
+ - credits used and remaining (from `archal usage`)
95
+ - the upgrade pointer (https://archal.ai/contact) if anything was blocked or
96
+ over quota
97
+
98
+ ## Docs
99
+
100
+ - Quickstart: https://docs.archal.ai/quickstart
101
+ - Contact for plan changes: https://archal.ai/contact
102
+ - Writing scenarios: https://docs.archal.ai/guides/writing-scenarios
103
+ - Clones and seeds: https://docs.archal.ai/clones/overview
@@ -0,0 +1,202 @@
1
+ ---
2
+ name: install-agent
3
+ description: Connect an agent's repo and its production observability to Archal so its traces get captured and graded. Detects an existing observability stack (LangSmith, Langfuse, Datadog, OpenTelemetry, Braintrust), connects the GitHub App, opens an observability setup PR, and routes an existing trace vendor's output to the dashboard OTLP endpoint or autoloop hosted ingest. USE THIS whenever the user says "connect my agent", "install the Archal agent", "set up observability", "capture my agent's traces", "hook up my production traces", "where do my traces go", or asks how to get an already-running agent into Archal. Reach for it before telling anyone a capability is missing — read the honest limits below first.
4
+ user-invocable: true
5
+ argument-hint: "[repo + where its traces live]"
6
+ ---
7
+
8
+ # Archal Install Agent
9
+
10
+ You are connecting a real, already-running agent to Archal so its production
11
+ behavior gets captured and graded. Two things have to land: (1) Archal can read
12
+ the **repo** (GitHub App), and (2) Archal can read the agent's **traces** —
13
+ either by adding instrumentation through a setup PR, or by ingesting an existing
14
+ observability vendor. Once traces flow, grading and the autoloop take over.
15
+
16
+ Be honest about what this is. The "install agent" is **not** a sandboxed coding
17
+ agent that edits arbitrary code in the repo. It is deterministic repo inspection
18
+ plus a deterministic, templated setup PR, plus an optional one-shot managed
19
+ planner (an LLM call) that only relocates the bootstrap file and writes advisory
20
+ PR-body text. Set that expectation up front so nobody waits for an autonomous
21
+ coder that does not exist yet. The honest limit is spelled out below — do not
22
+ oversell it.
23
+
24
+ ## Why this exists
25
+
26
+ Archal grades agent behavior from traces. An agent that already runs in
27
+ production has traces somewhere — your own logs, or a vendor like Langfuse or
28
+ Braintrust. The install path's whole job is to get those traces into Archal's
29
+ normalized shape without you hand-writing exporters or copying secrets around.
30
+ Capturing the trace is the precondition for everything downstream: grading,
31
+ reproduction, and the autoloop that turns reproduced failures into PRs.
32
+
33
+ ## Discover first
34
+
35
+ Before changing anything, read the repo and find out where traces live:
36
+
37
+ 1. `package.json` / `pyproject.toml` / `requirements.txt`: language and
38
+ framework. Language matters — the planner and `@archal/state-capture` are
39
+ TypeScript-only today (see limits).
40
+ 2. Existing observability dependencies. Archal's detector recognizes exactly
41
+ five vendors by dependency name:
42
+ - `langsmith` -> LangSmith
43
+ - `langfuse`, `langfuse-node` (TS) / `langfuse` (py) -> Langfuse
44
+ - `dd-trace` (TS) / `ddtrace` (py) -> Datadog
45
+ - `@opentelemetry/sdk-node`, `@opentelemetry/sdk-trace-node` (TS) /
46
+ `opentelemetry-sdk` (py) -> OpenTelemetry
47
+ - `braintrust` -> Braintrust
48
+ A repo with any of these is a candidate for **augment**; a repo with none is
49
+ **greenfield**.
50
+ 3. GitHub remote — augment/greenfield setup PRs and the autoloop need a GitHub
51
+ remote that resolves to `github.com/<owner>/<repo>`:
52
+ ```bash
53
+ git remote get-url origin
54
+ ```
55
+ 4. Where do the traces actually go? Ask the user. The answer routes you:
56
+ - already in a hosted vendor (Langfuse, Braintrust) or a Postgres/Supabase
57
+ table -> hosted autoloop ingest path (delegate detail to `autoloop` skill)
58
+ - exported files on disk -> drop into an autoloop watch directory or send via dashboard OTLP
59
+ - nowhere yet / only app logs -> the observability setup PR
60
+
61
+ Never print secrets while inspecting. Show env var names or secret references,
62
+ never plaintext keys or database URLs.
63
+
64
+ ## Preconditions
65
+
66
+ - Archal CLI installed in the repo or reachable with `npx archal`
67
+ - authenticated user (`archal login`) or `ARCHAL_TOKEN=archal_ws_...` (a
68
+ workspace key for CI)
69
+ - the **Archal GitHub App** installed on the target repo (required for the setup
70
+ PR and for autoloop fix PRs)
71
+ - a GitHub remote resolving to `github.com/<owner>/<repo>`
72
+ - for the ingest path: a read-only credential for the trace vendor
73
+
74
+ If a precondition is missing, make the smallest safe change and name what is
75
+ still required. Do not fake a connection.
76
+
77
+ ## Step 1 — connect the GitHub App
78
+
79
+ The repo connection is the GitHub App, not a token paste. Confirm the **Archal
80
+ GitHub App** is installed on the target repository and that the org granted it
81
+ access. Without it, the setup PR cannot be opened and the autoloop cannot open
82
+ fix PRs. If it is not installed, send the user to the dashboard's integration
83
+ flow to install it, then continue.
84
+
85
+ ## Step 2 — open the observability setup PR
86
+
87
+ When traces are not yet exported anywhere (or the user wants Archal's own
88
+ capture), open the **observability setup PR**. It is a deterministic, templated
89
+ patch — every file's contents are pre-generated; nothing is freely authored by
90
+ an LLM. The patch resolves into one of two install modes:
91
+
92
+ ### Greenfield (no existing observability detected)
93
+
94
+ Adds standard OpenTelemetry instrumentation pointed at Archal's OTLP endpoint:
95
+
96
+ - `archal-otel.ts` (TS) or `archal_otel.py` (Python) — an OpenTelemetry init
97
+ bootstrap (OTLP HTTP exporter + the node/python SDK), **not** an Archal-only
98
+ exporter
99
+ - `archal-replay-capsule.ts` / `archal_replay_capsule.py` — a replay helper
100
+ template
101
+ - OpenTelemetry SDK + framework instrumentation added to `package.json` (TS) or
102
+ `requirements.txt` (Python)
103
+ - an `.env.example` entry for the workspace key and an `ARCHAL_OBSERVABILITY.md`
104
+ guide
105
+
106
+ ### Augment (existing observability detected, TypeScript only)
107
+
108
+ When the repo already has one of the five vendors above **and** is TypeScript,
109
+ the PR instead adds Archal's state capture alongside the existing stack rather
110
+ than replacing it:
111
+
112
+ - `archal-state-capture.ts` importing from `@archal/state-capture`
113
+ - `@archal/state-capture` and `@opentelemetry/api` added to `package.json`
114
+ - the same `.env.example` entry and `ARCHAL_OBSERVABILITY.md` guide
115
+
116
+ Python repos with existing observability fall back to the greenfield
117
+ OpenTelemetry install — `@archal/state-capture` has no Python build yet, and the
118
+ PR body says so. Tell the user that explicitly rather than implying parity.
119
+
120
+ ### The optional install planner (managed LLM)
121
+
122
+ On Pro/Enterprise workspaces, for a TypeScript greenfield/augment install with
123
+ repo detection available, one managed LLM call (intent `observability-install`,
124
+ public label **Archal install planner**, routed through the managed eval model
125
+ lane — gemini-class — and metered as `cogs_only` spend) adapts the deterministic
126
+ patch to the repo's real layout. It is strictly additive and **fail-open**: any
127
+ availability, auth, plan-gate, or validation problem ships the deterministic
128
+ patch unchanged. The planner can only:
129
+
130
+ - relocate the bootstrap file to a better directory (path only — file contents
131
+ are never edited)
132
+ - append advisory text to the PR body (where to wire startup, which functions to
133
+ wrap)
134
+
135
+ It never edits application code, never modifies existing instrumentation, and
136
+ never runs free-form codegen. Disable it with `ARCHAL_INSTALL_PLANNER_DISABLED=1`
137
+ to force the deterministic install.
138
+
139
+ ## Step 3 — ingest an existing observability vendor
140
+
141
+ If the agent already emits traces to a vendor, you usually do **not** need the
142
+ setup PR. The two ingest paths are:
143
+
144
+ - **Dashboard OTLP endpoint** — point the vendor's exporter at the Archal OTLP
145
+ URL shown in the dashboard. Works for OpenTelemetry-compatible vendors (OTel,
146
+ Langfuse, Braintrust, Datadog, custom HTTP). No CLI command required.
147
+ - **Hosted autoloop** — for Postgres/Supabase sources, register the source with
148
+ `archal autoloop --source ...`. Archal polls the database directly.
149
+
150
+ **Delegate the deep mapping to the `autoloop` skill.** It owns the per-vendor
151
+ flags, schema/cursor/filter mapping for database sources, and the full
152
+ import/grade/reproduce/fix loop.
153
+
154
+ ## The honest limit
155
+
156
+ There is **no sandboxed coding agent** that reads the whole repo and edits
157
+ arbitrary code to wire up instrumentation. For a small or conventional repo, the
158
+ setup PR drops in and works. For a large or unusual repo, the setup PR is a
159
+ **generic bootstrap the user finishes** — they still import the bootstrap at
160
+ startup and wrap the call sites the planner's advisory section points at. Say
161
+ this plainly. The typed model lanes `remediation_agent` and
162
+ `observability_install_agent` exist in the contract but are
163
+ `agent-executor-contract-only`: the lane is declared, but no real executor
164
+ consumes it yet. Do not describe either as a working autonomous coder.
165
+
166
+ ## Failure taxonomy
167
+
168
+ Classify precisely; do not paper over a missing precondition:
169
+
170
+ - **GitHub App not connected** — setup PR and fix PRs cannot open. Install the
171
+ Archal GitHub App on the repo.
172
+ - **No GitHub remote** — augment/greenfield PRs need `github.com/<owner>/<repo>`.
173
+ - **Language unsupported by augment** — Python with existing observability falls
174
+ back to greenfield OTel; `@archal/state-capture` is TS-only.
175
+ - **Planner skipped** — wrong plan (needs Pro/Enterprise), non-TypeScript,
176
+ `augment-existing-vendor` legacy mode, detection unavailable, or
177
+ `ARCHAL_INSTALL_PLANNER_DISABLED=1`. The deterministic patch still ships; this
178
+ is not a failure, just a narrower install.
179
+ - **Setup PR is a stub for this repo** — large/unusual layout; the user finishes
180
+ wiring. Not a bug; the honest limit.
181
+ - **Trace ingest failure** — bad credential, rejected upload, or missing workspace auth.
182
+ Check the dashboard OTLP endpoint config or `archal autoloop status` for the hosted path.
183
+ - **No usable trace evidence** — once ingested, grading or reproduction can still
184
+ block if the trace lacks task context or state. Hand off to the `autoloop`
185
+ skill's evidence diagnosis.
186
+
187
+ ## What to report back
188
+
189
+ After install or debugging, give the user:
190
+
191
+ - repo full name and whether the GitHub App is connected
192
+ - chosen path: setup PR (greenfield vs augment) or hosted autoloop / dashboard OTLP ingest
193
+ - if a setup PR: the install mode, the files it adds, and whether the planner ran
194
+ - if ingest: provider, source id, and the next step (dashboard OTLP config or `archal autoloop` command)
195
+ - whether traces are flowing into Archal yet, or the exact blocker
196
+ - next command or next owner
197
+
198
+ ## Docs
199
+
200
+ - Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces
201
+ - CLI reference: https://docs.archal.ai/cli/autoloop
202
+ - Quickstart: https://docs.archal.ai/quickstart
@@ -11,16 +11,15 @@ You are setting up Archal in this project. Archal tests AI agents against servic
11
11
  ## If this is a cold-start
12
12
 
13
13
  The user may have landed here without running `npx archal init` first. If the
14
- CLI is missing (see "Install + auth" below) AND no `.archal-manifest.json`
15
- exists in any skill directory (`.claude/skills/`, `.codex/skills/`,
16
- `.cursor/skills/`, `.windsurf/skills/`), the canonical first command is:
14
+ CLI or project files are missing (see "Install + auth" below), the canonical
15
+ first command is:
17
16
 
18
17
  ```bash
19
18
  npx archal init
20
19
  ```
21
20
 
22
- That adds `archal` as a devDependency, installs skills for every detected
23
- agent platform (Claude Code, Codex, Cursor, Windsurf), and creates a starter
21
+ That adds `archal` as a devDependency when a `package.json` is present, installs
22
+ the packaged skills for detected local agent targets, and creates a starter
24
23
  `.archal.json`, `.archal/harness.mjs`, and `scenarios/first-run.md`. Re-invoke
25
24
  the onboard skill after it completes.
26
25
 
@@ -54,8 +53,10 @@ archal usage # verify auth + plan
54
53
  In CI, set `ARCHAL_TOKEN` to a **workspace API key** (`archal_ws_...`)
55
54
  instead of running `archal login`. Workspace keys are bound to one workspace,
56
55
  do not expire when a team member leaves, and are the recommended auth for CI.
57
- Create one with `archal workspace api-key create <label> --scope sessions:write`
58
- (requires owner or admin role) or from the dashboard under Settings > API Keys.
56
+ Create one with `archal workspace api-key create <label>` (requires owner or
57
+ admin role) or from the dashboard under Settings > API Keys. List and revoke
58
+ keys with `archal workspace api-keys` and
59
+ `archal workspace api-key revoke <key-id>`.
59
60
  Personal tokens (`arc_...`) are fine for local dev but should not be used in CI.
60
61
 
61
62
  Treat workspace API keys as runtime and CI credentials, not governance
@@ -85,7 +86,16 @@ Confirm detected clones, then ask which of these the user wants. Each delegates
85
86
  - `"agent": { "command": "node", "args": ["./agent.js"] }` - plain Node script
86
87
  - `"agent": { "command": "python", "args": ["agent.py"] }` - Python agent
87
88
 
88
- If the user doesn't have a harness yet, prefer `npx archal init`; it creates `./.archal/harness.mjs`, points `.archal.json` at it, and adds a starter scenario without overwriting existing files. The generated harness is a guarded stub: Archal refuses to score it until the user edits it to call their Cursor, Codex, Claude Code, or custom agent. A custom harness should read `AGENT_TASK` from env, call the agent runtime, print `{ "text": "..." }` to stdout, and call `reportAgentMetrics()` from `archal/harness` with accumulated `{ inputTokens, outputTokens, llmCallCount }` before exit. Service clients need one explicit routing mode: use sandbox/Docker routing when the harness calls normal service URLs such as `https://api.github.com`, or configure SDK base URLs from `AGENT_CLONE_URLS` and add the JSON headers from `AGENT_ROUTE_HEADERS` to those clone requests. Alternative: skip `agent` in `.archal.json` and pass `--harness <path>` per-run.
89
+ If the user doesn't have a harness yet, prefer `npx archal init`; it creates `./.archal/harness.mjs`, points `.archal.json` at it, and adds a starter scenario without overwriting existing files. The generated harness is a guarded stub: Archal refuses to score it until the user edits it to call their Cursor, Codex, Claude Code, or custom agent. A custom harness should read `AGENT_TASK` from env, call the agent runtime, print `{ "text": "..." }` to stdout, and call `reportAgentMetrics()` from `archal` with accumulated `{ inputTokens, outputTokens, llmCallCount }` before exit. Service clients need one explicit routing mode: use sandbox/Docker routing when the harness calls normal service URLs such as `https://api.github.com`, or configure SDK base URLs from `AGENT_CLONE_URLS` and add the JSON headers from `AGENT_ROUTE_HEADERS` to those clone requests. Alternative: skip `agent` in `.archal.json` and pass `--harness <path>` per-run.
90
+
91
+ ### Or run a packaged agent (no harness to write)
92
+
93
+ If the user wants a ready-made agent, use a packaged agent instead of writing a harness. Packaged agents run in Docker; Archal routes service calls to clones and forwards model calls with the host's provider key.
94
+
95
+ - `archal run <scenario>.md --sandbox` - run the bundled OpenClaw agent (needs Docker). Pick the model with `--agent-model <provider/model>`; export the matching key in the shell first (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`).
96
+ - `archal run <scenario>.md --harness ./<dir> --docker --dockerfile ./<dir>/Dockerfile` - run your own packaged agent.
97
+
98
+ See the "Run a packaged agent" guide: https://docs.archal.ai/guides/packaged-agents
89
99
 
90
100
  ### Option A - Evaluate an agent with scenarios
91
101
 
@@ -109,7 +119,7 @@ Same `.archal.json` as Option A (inline `--task` still needs an agent). Use this
109
119
  "clones": ["<detected clones>"]
110
120
  }
111
121
  ```
112
- 2. Demo: `archal run --task "Create an issue titled hello" --clone github`.
122
+ 2. Demo: `archal run --task "Create an issue titled hello" --harness . --docker --clone github`.
113
123
  3. For the generated first-run project, use bare `archal run` after wiring `.archal/harness.mjs`.
114
124
 
115
125
  ### Option C - Clones in a Vitest suite
@@ -122,16 +132,17 @@ Do not paste a sample config here. The right shape depends on what's already in
122
132
 
123
133
  Run: `archal clone start <detected clones>` - gives live clone URLs the user's SDK clients can point at. `archal clone status` shows the active session; `archal clone stop` tears down.
124
134
 
125
- ### Option E - Attach real trace sources
126
-
127
- Use this when the repo already has agent traces from pre-production or
128
- production and the user wants Archal to import, grade, reproduce, and turn
129
- reproduced failures into GitHub issues or PRs.
135
+ ### Option E - Autoloop real trace sources
130
136
 
131
- **Delegate to the `attach` skill.** It owns the trace-source mapping,
132
- `archal/harness.json`, `archal/scenario.md`, seed templates, `archal attach`
137
+ **Delegate to the `autoloop` skill.** It owns the trace-source mapping,
138
+ `archal/harness.json`, `archal/scenario.md`, seed templates, `archal autoloop`
133
139
  commands, dashboard expectations, and failure taxonomy. Do not inline the
134
- Attach flow here; it changes faster than starter scenario setup.
140
+ Autoloop flow here; it changes faster than starter scenario setup.
141
+
142
+ Set the expectation carefully: Autoloop is not arbitrary production trace replay.
143
+ It can reproduce only failures with enough trace evidence plus repo-owned
144
+ scenario/seed context to reconstruct realistic clone state. Missing evidence
145
+ should block with a clear artifact instead of being guessed.
135
146
 
136
147
  ## Verify
137
148
 
@@ -155,4 +166,4 @@ Run the first scenario or task. For Options A and B, hand off to the `eval` skil
155
166
 
156
167
  - Quickstart: https://docs.archal.ai/quickstart
157
168
  - Full docs: https://docs.archal.ai
158
- - Attach production traces: https://docs.archal.ai/guides/attach-production-traces
169
+ - Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces