archal 0.9.19 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +62 -154
- package/agents/openclaw/.archal.json +8 -0
- package/agents/openclaw/Dockerfile +97 -0
- package/agents/openclaw/README.md +113 -0
- package/agents/openclaw/drive.mjs +301 -0
- package/agents/openclaw/extract-openclaw-response-text.mjs +49 -0
- package/agents/openclaw/package.json +9 -0
- package/agents/openclaw/scenarios/acceptance/github-escalation-and-cleanup.md +43 -0
- package/agents/openclaw/scenarios/acceptance/jira-linear-release-triage.md +39 -0
- package/agents/openclaw/scenarios/acceptance/multi-clone-customer-risk-review.md +45 -0
- package/agents/openclaw/scenarios/acceptance/slack-incident-handoff.md +42 -0
- package/agents/openclaw/scenarios/acceptance/stripe-dunning-remediation.md +40 -0
- package/agents/openclaw/scenarios/close-stale-issues.md +35 -0
- package/agents/openclaw/scenarios/github-issue-triage-read-only.md +44 -0
- package/agents/openclaw/workspace/AGENTS.md +23 -0
- package/agents/openclaw/workspace/IDENTITY.md +8 -0
- package/agents/openclaw/workspace/SOUL.md +14 -0
- package/agents/openclaw/workspace/TOOLS.md +54 -0
- package/clone-assets/apify/tools.json +256 -22
- package/clone-assets/calcom/tools.json +2062 -0
- package/clone-assets/clickup/tools.json +3245 -0
- package/clone-assets/customerio/tools.json +2350 -0
- package/clone-assets/datadog/tools.json +734 -0
- package/clone-assets/github/tools.json +307 -27
- package/clone-assets/gitlab/tools.json +10688 -0
- package/clone-assets/google-workspace/tools.json +18 -6
- package/clone-assets/hubspot/tools.json +1604 -0
- package/clone-assets/jira/fidelity.json +1 -1
- package/clone-assets/jira/tools.json +266 -543
- package/clone-assets/linear/tools.json +278 -40
- package/clone-assets/ownerrez/tools.json +548 -0
- package/clone-assets/pricelabs/tools.json +343 -0
- package/clone-assets/sentry/tools.json +2821 -0
- package/clone-assets/slack/tools.json +1 -2
- package/clone-assets/stripe/tools.json +299 -46
- package/clone-assets/supabase/tools.json +437 -0
- package/clone-assets/unipile/tools.json +408 -0
- package/clone-assets/webflow/tools.json +2185 -0
- package/dist/autoloop-worker-types-CHaclqtD.d.cts +160 -0
- package/dist/cli.cjs +127896 -84811
- package/dist/{vitest/chunk-L36NXAU6.js → commands/autoloop-hosted-control-plane-client.cjs} +16344 -18845
- package/dist/commands/autoloop-hosted-control-plane-client.d.cts +133 -0
- package/dist/commands/autoloop-pr-verification.cjs +4312 -0
- package/dist/commands/autoloop-pr-verification.d.cts +19 -0
- package/dist/commands/autoloop-result-parser.cjs +27058 -0
- package/dist/commands/autoloop-result-parser.d.cts +196 -0
- package/dist/commands/autoloop-worker.cjs +38488 -0
- package/dist/commands/autoloop-worker.d.cts +102 -0
- package/dist/eval-shim.mjs +216 -0
- package/dist/index.cjs +15 -2
- package/dist/index.d.cts +3 -1
- package/dist/layer-1-ziaLpfLr.d.cts +62 -0
- package/dist/metafile-cjs.json +1 -0
- package/dist/reporter-entry.js +3 -0
- package/dist/sdk/index.cjs +51630 -0
- package/dist/sdk/index.d.cts +626 -0
- package/dist/vitest/chunk-2PDHTPZC.js +4867 -0
- package/dist/vitest/chunk-7L4BBB6M.js +2613 -0
- package/dist/vitest/index.cjs +6152 -75718
- package/dist/vitest/index.d.ts +22 -86
- package/dist/vitest/index.js +45 -414
- package/dist/vitest/runtime/hosted-session-reaper.cjs +682 -34399
- package/dist/vitest/runtime/hosted-session-reaper.js +1 -1
- package/dist/vitest/runtime/setup-files.js +2 -2
- package/manifest.json +9 -0
- package/package.json +20 -14
- package/skills/archal-agent/SKILL.md +86 -0
- package/skills/autoloop/SKILL.md +391 -0
- package/skills/autoloop/references/hosted-sources.md +94 -0
- package/skills/autoloop/references/trace-schema-mapping.md +104 -0
- package/skills/eval/SKILL.md +17 -15
- package/skills/free-account/SKILL.md +103 -0
- package/skills/install-agent/SKILL.md +202 -0
- package/skills/onboard/SKILL.md +29 -18
- package/skills/scenario/SKILL.md +36 -18
- package/skills/seed/SKILL.md +238 -0
- package/skills/vitest/SKILL.md +3 -2
- package/dist/harness.cjs +0 -62
- package/dist/harness.d.cts +0 -20
- package/dist/seed/dynamic-generator.cjs +0 -45687
- package/dist/seed/dynamic-generator.d.cts +0 -106
- package/dist/vitest/chunk-WZ7SA4CK.js +0 -47369
- package/skills/attach/SKILL.md +0 -402
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
# Trace schema mapping
|
|
2
|
+
|
|
3
|
+
Config reference for hosted Postgres/Supabase trace sources whose trace and span
|
|
4
|
+
tables do not match the defaults below. Schema mapping, cursor mode, and filters
|
|
5
|
+
live in the `.archal.json` `autoloop` block under `autoloop.source`, not in flags.
|
|
6
|
+
|
|
7
|
+
## Contents
|
|
8
|
+
|
|
9
|
+
- Defaults
|
|
10
|
+
- Custom schema
|
|
11
|
+
- Append-only sources
|
|
12
|
+
- Filters for noisy sources
|
|
13
|
+
|
|
14
|
+
## Defaults
|
|
15
|
+
|
|
16
|
+
| Concept | Config field | Default |
|
|
17
|
+
|---------|--------------|---------|
|
|
18
|
+
| trace table | `source.mapping.traces.table` | `ai_traces` |
|
|
19
|
+
| span table | `source.mapping.spans.table` | `ai_spans` |
|
|
20
|
+
| trace id | `source.mapping.traces.idColumn` | `id` |
|
|
21
|
+
| span id | `source.mapping.spans.idColumn` | `id` |
|
|
22
|
+
| span trace id | `source.mapping.spans.traceIdColumn` | `trace_id` |
|
|
23
|
+
| span parent id | `source.mapping.spans.parentSpanIdColumn` | `parent_span_id` |
|
|
24
|
+
| trace updated cursor | `source.mapping.traces.updatedAtColumn` | `updated_at` |
|
|
25
|
+
| span updated cursor | `source.mapping.spans.updatedAtColumn` | `updated_at` |
|
|
26
|
+
| cursor mode | `source.cursor.mode` | `updated_at_id` |
|
|
27
|
+
|
|
28
|
+
## Custom schema
|
|
29
|
+
|
|
30
|
+
For a custom schema, set the mapping in `.archal.json`. Only the fields that
|
|
31
|
+
differ from the defaults are required:
|
|
32
|
+
|
|
33
|
+
```jsonc
|
|
34
|
+
// .archal.json
|
|
35
|
+
{
|
|
36
|
+
"autoloop": {
|
|
37
|
+
"source": {
|
|
38
|
+
"mapping": {
|
|
39
|
+
"traces": {
|
|
40
|
+
"table": "public.agent_traces",
|
|
41
|
+
"idColumn": "trace_id",
|
|
42
|
+
"updatedAtColumn": "updated_at"
|
|
43
|
+
},
|
|
44
|
+
"spans": {
|
|
45
|
+
"table": "public.agent_spans",
|
|
46
|
+
"idColumn": "span_id",
|
|
47
|
+
"traceIdColumn": "trace_id",
|
|
48
|
+
"parentSpanIdColumn": "parent_span_id",
|
|
49
|
+
"updatedAtColumn": "updated_at"
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
## Append-only sources
|
|
58
|
+
|
|
59
|
+
For append-only sources, switch the cursor to `created_at_id` and point it at the
|
|
60
|
+
created-at columns:
|
|
61
|
+
|
|
62
|
+
```jsonc
|
|
63
|
+
// .archal.json
|
|
64
|
+
{
|
|
65
|
+
"autoloop": {
|
|
66
|
+
"source": {
|
|
67
|
+
"cursor": { "mode": "created_at_id" },
|
|
68
|
+
"mapping": {
|
|
69
|
+
"traces": { "createdAtColumn": "created_at" },
|
|
70
|
+
"spans": { "createdAtColumn": "created_at" }
|
|
71
|
+
}
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
`source.cursor.watermarks` is a repair escape hatch: set
|
|
78
|
+
`{ traces: { at, id }, spans: { at, id } }` (any of which may be `null`) only when
|
|
79
|
+
you need to re-seat the cursor after a bad import.
|
|
80
|
+
|
|
81
|
+
## Filters for noisy sources
|
|
82
|
+
|
|
83
|
+
Use filters to narrow a noisy source. All filter fields are optional:
|
|
84
|
+
|
|
85
|
+
```jsonc
|
|
86
|
+
// .archal.json
|
|
87
|
+
{
|
|
88
|
+
"autoloop": {
|
|
89
|
+
"source": {
|
|
90
|
+
"filters": {
|
|
91
|
+
"workspaceIds": ["workspace_123"],
|
|
92
|
+
"agentIds": ["support-agent"],
|
|
93
|
+
"statuses": ["failed", "error"],
|
|
94
|
+
"traceGroups": ["billing-support"],
|
|
95
|
+
"limit": 250
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Arbitrary source metadata for the registration goes under `source.metadata` (a
|
|
103
|
+
map of short-identifier keys to string values). Regression scenarios export is a top-level
|
|
104
|
+
`autoloop.benchmarkExport: true` (the config key keeps the legacy `benchmark` name).
|
package/skills/eval/SKILL.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: eval
|
|
3
3
|
description: Run Archal scenarios or inline tasks against hosted clones, diagnose failed runs, and interpret satisfaction scores. Triggers on "run my scenario", "evaluate my agent", "archal run X", "debug this failing run", "what does this satisfaction score mean".
|
|
4
4
|
user-invocable: true
|
|
5
|
-
argument-hint:
|
|
5
|
+
argument-hint: '[scenario.md or task description]'
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
# Archal Eval Runner
|
|
@@ -35,14 +35,14 @@ If a local harness returns text but records zero clone-observed calls, check for
|
|
|
35
35
|
Scenario from a file:
|
|
36
36
|
|
|
37
37
|
```bash
|
|
38
|
-
archal run scenario.md
|
|
39
|
-
archal run scenario.md --runs 5 --seed enterprise-repo # N runs -> satisfaction score
|
|
38
|
+
archal run scenario.md --docker
|
|
39
|
+
archal run scenario.md --docker --runs 5 --seed enterprise-repo # N runs -> satisfaction score
|
|
40
40
|
```
|
|
41
41
|
|
|
42
42
|
Inline task (no scenario file):
|
|
43
43
|
|
|
44
44
|
```bash
|
|
45
|
-
archal run --task "Create an issue titled hello" --harness ./.archal/harness.mjs --clone github
|
|
45
|
+
archal run --task "Create an issue titled hello" --harness ./.archal/harness.mjs --docker --clone github
|
|
46
46
|
```
|
|
47
47
|
|
|
48
48
|
`--task` only replaces the scenario file - it still needs a runnable agent path. `--clone` is required with `--task`; repeat or comma-separate for multiple clones.
|
|
@@ -52,11 +52,13 @@ When `.archal.json` exists in cwd, bare `archal run` uses it. If the user doesn'
|
|
|
52
52
|
## Interpret results
|
|
53
53
|
|
|
54
54
|
Score breakdown:
|
|
55
|
+
|
|
55
56
|
- `100%` = every run passed every criterion
|
|
56
57
|
- `80%` = 4/5 runs passed
|
|
57
58
|
- `0%` = none passed
|
|
58
59
|
|
|
59
60
|
Criterion types:
|
|
61
|
+
|
|
60
62
|
- `[D]` - deterministic state check. A failure is real; never a model variance artifact.
|
|
61
63
|
- `[P]` - LLM judge reads trace + final state. A single failure can be variance; re-run with `--runs 3+` to confirm before acting on it.
|
|
62
64
|
|
|
@@ -65,30 +67,30 @@ Criterion types:
|
|
|
65
67
|
Re-run with `-v` for the full trace, then classify with these signals:
|
|
66
68
|
|
|
67
69
|
- **Agent bug** - wrong tool called, wrong arguments, stopped early.
|
|
68
|
-
|
|
69
|
-
|
|
70
|
+
_Signals:_ trace shows the correct tool was available but the agent chose another; or arguments are malformed.
|
|
71
|
+
_Fix:_ agent prompt, tool wiring, or underlying model.
|
|
70
72
|
|
|
71
73
|
- **Scenario bug** - criteria are too strict, ambiguous, or contradict the Setup.
|
|
72
|
-
|
|
73
|
-
|
|
74
|
+
_Signals:_ agent clearly did the right thing but a `[D]` criterion expects an exact count the Setup didn't guarantee; or two criteria contradict each other.
|
|
75
|
+
_Fix:_ make Setup more specific, or relax the criterion. Use the `scenario` skill.
|
|
74
76
|
|
|
75
77
|
- **Seed mismatch** - clone state doesn't match what Setup describes.
|
|
76
|
-
|
|
77
|
-
|
|
78
|
+
_Signals:_ agent's first introspection tool call returns unexpected state (e.g. Setup says "4 stale issues" but the seed has 3).
|
|
79
|
+
_Fix:_ different seed, or adjust Setup to match. `archal seed list <clone>` to browse.
|
|
78
80
|
|
|
79
81
|
- **Harness bug** - agent process never started, crashed immediately, or hung.
|
|
80
|
-
|
|
81
|
-
|
|
82
|
+
_Signals:_ no tool calls in the trace, stderr shows a boot error, or the run times out at the configured `--timeout`.
|
|
83
|
+
_Fix:_ smoke-test the harness directly with `AGENT_TASK="Reply with OK." node ./.archal/harness.mjs`, then look for an untouched starter stub, UI-only imports, missing provider keys, or interactive auth.
|
|
82
84
|
|
|
83
85
|
## CI mode
|
|
84
86
|
|
|
85
87
|
```bash
|
|
86
|
-
archal run scenario.md --runs 3 --pass-threshold 80 -o json -q
|
|
88
|
+
archal run scenario.md --docker --runs 3 --pass-threshold 80 -o json -q
|
|
87
89
|
```
|
|
88
90
|
|
|
89
91
|
Exit codes: `0` pass, `1` fail or score < threshold, `2` validation error. For GitHub Actions, inject `ARCHAL_TOKEN` as a secret. Use a workspace API key (`archal_ws_...`) for CI, not a personal token.
|
|
90
92
|
|
|
91
|
-
Workspace API keys are runtime and CI credentials bound to one workspace. They can run clones, upload and read traces, and read usage for that workspace. They cannot manage audit events or workspace API keys. Use an owner/admin user credential, either `archal login` or a dashboard-issued user API key, for workspace administration.
|
|
93
|
+
Workspace API keys are runtime and CI credentials bound to one workspace. They can run clones, upload and read traces, and read usage for that workspace. They cannot manage audit events or workspace API keys. Create/list/revoke them with `archal workspace api-key create <label>`, `archal workspace api-keys`, and `archal workspace api-key revoke <key-id>` from an owner/admin user session. Use an owner/admin user credential, either `archal login` or a dashboard-issued user API key, for workspace administration.
|
|
92
94
|
|
|
93
95
|
## Artifacts + dashboard
|
|
94
96
|
|
|
@@ -108,6 +110,6 @@ Don't tell users they need `-o json` to save artifacts locally - that's only for
|
|
|
108
110
|
## Docs
|
|
109
111
|
|
|
110
112
|
- Running with an agent: https://docs.archal.ai/guides/run-with-agent
|
|
111
|
-
- Existing repo playbook: https://docs.archal.ai/guides/
|
|
113
|
+
- Existing repo playbook: https://docs.archal.ai/guides/run-with-agent
|
|
112
114
|
- Scenario authoring: hand off to the `scenario` skill
|
|
113
115
|
- Clone sessions: https://docs.archal.ai/guides/clone-sessions
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: free-account
|
|
3
|
+
description: B2C free-tier guide — run scenarios and start clones within free entitlements. Use when an orchestrating or coding agent should use Archal under a free user account to "run my agent", evaluate a task against clones, or spin up clones without paid features.
|
|
4
|
+
user-invocable: true
|
|
5
|
+
argument-hint: "[repo, task, or 'run my agent']"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Archal Free Account
|
|
9
|
+
|
|
10
|
+
You are driving Archal under a free user account. Archal tests AI agents against
|
|
11
|
+
clones of real services (GitHub, Slack, Stripe, etc.). On free, you can run
|
|
12
|
+
scenarios and start clones within the free entitlements. You cannot start any
|
|
13
|
+
autonomous loop, remediation, or observability-install flow. Stay inside the
|
|
14
|
+
allowed commands below and never attempt a blocked one.
|
|
15
|
+
|
|
16
|
+
## Authenticate first
|
|
17
|
+
|
|
18
|
+
Do this before any other command.
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
archal login # OAuth browser flow -> ~/.archal/credentials.json
|
|
22
|
+
archal usage # verify auth + plan
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
In CI, skip the browser flow and set a token:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
export ARCHAL_TOKEN=... # workspace or user token
|
|
29
|
+
archal usage
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Sessions are Archal-hosted, so provider credentials may be placeholders. You do
|
|
33
|
+
not need real GitHub, Slack, or Stripe keys to start a clone or run a scenario.
|
|
34
|
+
|
|
35
|
+
## Free limits
|
|
36
|
+
|
|
37
|
+
State these upfront so the user is not surprised mid-run.
|
|
38
|
+
|
|
39
|
+
| Limit | Free value |
|
|
40
|
+
|-------|-----------|
|
|
41
|
+
| Cost | $0 |
|
|
42
|
+
| Workspace | free workspace entitlements only |
|
|
43
|
+
| Concurrent sessions | 3 |
|
|
44
|
+
| Session TTL | 30 minutes |
|
|
45
|
+
| Credits | 5,000 per month, recurring |
|
|
46
|
+
|
|
47
|
+
If the user asks for paid workspace capabilities, point them to
|
|
48
|
+
https://archal.ai/contact. Do not attempt blocked actions.
|
|
49
|
+
|
|
50
|
+
## Allowed commands only
|
|
51
|
+
|
|
52
|
+
Use only these on free. Nothing else.
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
archal run <scenario.md> # run a scenario against clones
|
|
56
|
+
archal clone start [clones...] # start one or more hosted clones
|
|
57
|
+
archal clone status # inspect the active session
|
|
58
|
+
archal clone stop # tear the session down
|
|
59
|
+
archal clone # list available clones
|
|
60
|
+
archal clone seed <clone> <name> # apply a named seed to a running clone
|
|
61
|
+
archal scenario list # list scenarios
|
|
62
|
+
archal seed list # list seeds
|
|
63
|
+
archal usage # check plan, credits, and quota
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Prohibited on free
|
|
67
|
+
|
|
68
|
+
Never instruct any of these. They are Pro features and fail closed: the web
|
|
69
|
+
returns HTTP 402 and the CLI fails closed with an entitlement error. Do not run
|
|
70
|
+
them, do not retry them, and do not describe them as available.
|
|
71
|
+
|
|
72
|
+
- autonomous remediation loops (`archal autoloop ...`)
|
|
73
|
+
- remediation / fix runs
|
|
74
|
+
- `archal autoloop detach`
|
|
75
|
+
- observability-install flow
|
|
76
|
+
|
|
77
|
+
This mirrors the server-side blocked set: `start_loop`, `reproduce_failure`,
|
|
78
|
+
`retry_loop`, `connect_observability`. If the user wants any of these, point
|
|
79
|
+
them to https://archal.ai/contact.
|
|
80
|
+
|
|
81
|
+
## Quota awareness
|
|
82
|
+
|
|
83
|
+
Check quota with `archal usage` before a run, and watch for quota-exceeded
|
|
84
|
+
results during a run. On quota-exceeded, stop and direct the user to
|
|
85
|
+
https://archal.ai/contact. Do not retry a blocked or over-quota command, and do
|
|
86
|
+
not try to work around the limit.
|
|
87
|
+
|
|
88
|
+
## What to report back
|
|
89
|
+
|
|
90
|
+
After a run or clone session, give the user:
|
|
91
|
+
|
|
92
|
+
- command run
|
|
93
|
+
- satisfaction score or clone session id
|
|
94
|
+
- credits used and remaining (from `archal usage`)
|
|
95
|
+
- the upgrade pointer (https://archal.ai/contact) if anything was blocked or
|
|
96
|
+
over quota
|
|
97
|
+
|
|
98
|
+
## Docs
|
|
99
|
+
|
|
100
|
+
- Quickstart: https://docs.archal.ai/quickstart
|
|
101
|
+
- Contact for plan changes: https://archal.ai/contact
|
|
102
|
+
- Writing scenarios: https://docs.archal.ai/guides/writing-scenarios
|
|
103
|
+
- Clones and seeds: https://docs.archal.ai/clones/overview
|
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: install-agent
|
|
3
|
+
description: Connect an agent's repo and its production observability to Archal so its traces get captured and graded. Detects an existing observability stack (LangSmith, Langfuse, Datadog, OpenTelemetry, Braintrust), connects the GitHub App, opens an observability setup PR, and routes an existing trace vendor's output to the dashboard OTLP endpoint or autoloop hosted ingest. USE THIS whenever the user says "connect my agent", "install the Archal agent", "set up observability", "capture my agent's traces", "hook up my production traces", "where do my traces go", or asks how to get an already-running agent into Archal. Reach for it before telling anyone a capability is missing — read the honest limits below first.
|
|
4
|
+
user-invocable: true
|
|
5
|
+
argument-hint: "[repo + where its traces live]"
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Archal Install Agent
|
|
9
|
+
|
|
10
|
+
You are connecting a real, already-running agent to Archal so its production
|
|
11
|
+
behavior gets captured and graded. Two things have to land: (1) Archal can read
|
|
12
|
+
the **repo** (GitHub App), and (2) Archal can read the agent's **traces** —
|
|
13
|
+
either by adding instrumentation through a setup PR, or by ingesting an existing
|
|
14
|
+
observability vendor. Once traces flow, grading and the autoloop take over.
|
|
15
|
+
|
|
16
|
+
Be honest about what this is. The "install agent" is **not** a sandboxed coding
|
|
17
|
+
agent that edits arbitrary code in the repo. It is deterministic repo inspection
|
|
18
|
+
plus a deterministic, templated setup PR, plus an optional one-shot managed
|
|
19
|
+
planner (an LLM call) that only relocates the bootstrap file and writes advisory
|
|
20
|
+
PR-body text. Set that expectation up front so nobody waits for an autonomous
|
|
21
|
+
coder that does not exist yet. The honest limit is spelled out below — do not
|
|
22
|
+
oversell it.
|
|
23
|
+
|
|
24
|
+
## Why this exists
|
|
25
|
+
|
|
26
|
+
Archal grades agent behavior from traces. An agent that already runs in
|
|
27
|
+
production has traces somewhere — your own logs, or a vendor like Langfuse or
|
|
28
|
+
Braintrust. The install path's whole job is to get those traces into Archal's
|
|
29
|
+
normalized shape without you hand-writing exporters or copying secrets around.
|
|
30
|
+
Capturing the trace is the precondition for everything downstream: grading,
|
|
31
|
+
reproduction, and the autoloop that turns reproduced failures into PRs.
|
|
32
|
+
|
|
33
|
+
## Discover first
|
|
34
|
+
|
|
35
|
+
Before changing anything, read the repo and find out where traces live:
|
|
36
|
+
|
|
37
|
+
1. `package.json` / `pyproject.toml` / `requirements.txt`: language and
|
|
38
|
+
framework. Language matters — the planner and `@archal/state-capture` are
|
|
39
|
+
TypeScript-only today (see limits).
|
|
40
|
+
2. Existing observability dependencies. Archal's detector recognizes exactly
|
|
41
|
+
five vendors by dependency name:
|
|
42
|
+
- `langsmith` -> LangSmith
|
|
43
|
+
- `langfuse`, `langfuse-node` (TS) / `langfuse` (py) -> Langfuse
|
|
44
|
+
- `dd-trace` (TS) / `ddtrace` (py) -> Datadog
|
|
45
|
+
- `@opentelemetry/sdk-node`, `@opentelemetry/sdk-trace-node` (TS) /
|
|
46
|
+
`opentelemetry-sdk` (py) -> OpenTelemetry
|
|
47
|
+
- `braintrust` -> Braintrust
|
|
48
|
+
A repo with any of these is a candidate for **augment**; a repo with none is
|
|
49
|
+
**greenfield**.
|
|
50
|
+
3. GitHub remote — augment/greenfield setup PRs and the autoloop need a GitHub
|
|
51
|
+
remote that resolves to `github.com/<owner>/<repo>`:
|
|
52
|
+
```bash
|
|
53
|
+
git remote get-url origin
|
|
54
|
+
```
|
|
55
|
+
4. Where do the traces actually go? Ask the user. The answer routes you:
|
|
56
|
+
- already in a hosted vendor (Langfuse, Braintrust) or a Postgres/Supabase
|
|
57
|
+
table -> hosted autoloop ingest path (delegate detail to `autoloop` skill)
|
|
58
|
+
- exported files on disk -> drop into an autoloop watch directory or send via dashboard OTLP
|
|
59
|
+
- nowhere yet / only app logs -> the observability setup PR
|
|
60
|
+
|
|
61
|
+
Never print secrets while inspecting. Show env var names or secret references,
|
|
62
|
+
never plaintext keys or database URLs.
|
|
63
|
+
|
|
64
|
+
## Preconditions
|
|
65
|
+
|
|
66
|
+
- Archal CLI installed in the repo or reachable with `npx archal`
|
|
67
|
+
- authenticated user (`archal login`) or `ARCHAL_TOKEN=archal_ws_...` (a
|
|
68
|
+
workspace key for CI)
|
|
69
|
+
- the **Archal GitHub App** installed on the target repo (required for the setup
|
|
70
|
+
PR and for autoloop fix PRs)
|
|
71
|
+
- a GitHub remote resolving to `github.com/<owner>/<repo>`
|
|
72
|
+
- for the ingest path: a read-only credential for the trace vendor
|
|
73
|
+
|
|
74
|
+
If a precondition is missing, make the smallest safe change and name what is
|
|
75
|
+
still required. Do not fake a connection.
|
|
76
|
+
|
|
77
|
+
## Step 1 — connect the GitHub App
|
|
78
|
+
|
|
79
|
+
The repo connection is the GitHub App, not a token paste. Confirm the **Archal
|
|
80
|
+
GitHub App** is installed on the target repository and that the org granted it
|
|
81
|
+
access. Without it, the setup PR cannot be opened and the autoloop cannot open
|
|
82
|
+
fix PRs. If it is not installed, send the user to the dashboard's integration
|
|
83
|
+
flow to install it, then continue.
|
|
84
|
+
|
|
85
|
+
## Step 2 — open the observability setup PR
|
|
86
|
+
|
|
87
|
+
When traces are not yet exported anywhere (or the user wants Archal's own
|
|
88
|
+
capture), open the **observability setup PR**. It is a deterministic, templated
|
|
89
|
+
patch — every file's contents are pre-generated; nothing is freely authored by
|
|
90
|
+
an LLM. The patch resolves into one of two install modes:
|
|
91
|
+
|
|
92
|
+
### Greenfield (no existing observability detected)
|
|
93
|
+
|
|
94
|
+
Adds standard OpenTelemetry instrumentation pointed at Archal's OTLP endpoint:
|
|
95
|
+
|
|
96
|
+
- `archal-otel.ts` (TS) or `archal_otel.py` (Python) — an OpenTelemetry init
|
|
97
|
+
bootstrap (OTLP HTTP exporter + the node/python SDK), **not** an Archal-only
|
|
98
|
+
exporter
|
|
99
|
+
- `archal-replay-capsule.ts` / `archal_replay_capsule.py` — a replay helper
|
|
100
|
+
template
|
|
101
|
+
- OpenTelemetry SDK + framework instrumentation added to `package.json` (TS) or
|
|
102
|
+
`requirements.txt` (Python)
|
|
103
|
+
- an `.env.example` entry for the workspace key and an `ARCHAL_OBSERVABILITY.md`
|
|
104
|
+
guide
|
|
105
|
+
|
|
106
|
+
### Augment (existing observability detected, TypeScript only)
|
|
107
|
+
|
|
108
|
+
When the repo already has one of the five vendors above **and** is TypeScript,
|
|
109
|
+
the PR instead adds Archal's state capture alongside the existing stack rather
|
|
110
|
+
than replacing it:
|
|
111
|
+
|
|
112
|
+
- `archal-state-capture.ts` importing from `@archal/state-capture`
|
|
113
|
+
- `@archal/state-capture` and `@opentelemetry/api` added to `package.json`
|
|
114
|
+
- the same `.env.example` entry and `ARCHAL_OBSERVABILITY.md` guide
|
|
115
|
+
|
|
116
|
+
Python repos with existing observability fall back to the greenfield
|
|
117
|
+
OpenTelemetry install — `@archal/state-capture` has no Python build yet, and the
|
|
118
|
+
PR body says so. Tell the user that explicitly rather than implying parity.
|
|
119
|
+
|
|
120
|
+
### The optional install planner (managed LLM)
|
|
121
|
+
|
|
122
|
+
On Pro/Enterprise workspaces, for a TypeScript greenfield/augment install with
|
|
123
|
+
repo detection available, one managed LLM call (intent `observability-install`,
|
|
124
|
+
public label **Archal install planner**, routed through the managed eval model
|
|
125
|
+
lane — gemini-class — and metered as `cogs_only` spend) adapts the deterministic
|
|
126
|
+
patch to the repo's real layout. It is strictly additive and **fail-open**: any
|
|
127
|
+
availability, auth, plan-gate, or validation problem ships the deterministic
|
|
128
|
+
patch unchanged. The planner can only:
|
|
129
|
+
|
|
130
|
+
- relocate the bootstrap file to a better directory (path only — file contents
|
|
131
|
+
are never edited)
|
|
132
|
+
- append advisory text to the PR body (where to wire startup, which functions to
|
|
133
|
+
wrap)
|
|
134
|
+
|
|
135
|
+
It never edits application code, never modifies existing instrumentation, and
|
|
136
|
+
never runs free-form codegen. Disable it with `ARCHAL_INSTALL_PLANNER_DISABLED=1`
|
|
137
|
+
to force the deterministic install.
|
|
138
|
+
|
|
139
|
+
## Step 3 — ingest an existing observability vendor
|
|
140
|
+
|
|
141
|
+
If the agent already emits traces to a vendor, you usually do **not** need the
|
|
142
|
+
setup PR. The two ingest paths are:
|
|
143
|
+
|
|
144
|
+
- **Dashboard OTLP endpoint** — point the vendor's exporter at the Archal OTLP
|
|
145
|
+
URL shown in the dashboard. Works for OpenTelemetry-compatible vendors (OTel,
|
|
146
|
+
Langfuse, Braintrust, Datadog, custom HTTP). No CLI command required.
|
|
147
|
+
- **Hosted autoloop** — for Postgres/Supabase sources, register the source with
|
|
148
|
+
`archal autoloop --source ...`. Archal polls the database directly.
|
|
149
|
+
|
|
150
|
+
**Delegate the deep mapping to the `autoloop` skill.** It owns the per-vendor
|
|
151
|
+
flags, schema/cursor/filter mapping for database sources, and the full
|
|
152
|
+
import/grade/reproduce/fix loop.
|
|
153
|
+
|
|
154
|
+
## The honest limit
|
|
155
|
+
|
|
156
|
+
There is **no sandboxed coding agent** that reads the whole repo and edits
|
|
157
|
+
arbitrary code to wire up instrumentation. For a small or conventional repo, the
|
|
158
|
+
setup PR drops in and works. For a large or unusual repo, the setup PR is a
|
|
159
|
+
**generic bootstrap the user finishes** — they still import the bootstrap at
|
|
160
|
+
startup and wrap the call sites the planner's advisory section points at. Say
|
|
161
|
+
this plainly. The typed model lanes `remediation_agent` and
|
|
162
|
+
`observability_install_agent` exist in the contract but are
|
|
163
|
+
`agent-executor-contract-only`: the lane is declared, but no real executor
|
|
164
|
+
consumes it yet. Do not describe either as a working autonomous coder.
|
|
165
|
+
|
|
166
|
+
## Failure taxonomy
|
|
167
|
+
|
|
168
|
+
Classify precisely; do not paper over a missing precondition:
|
|
169
|
+
|
|
170
|
+
- **GitHub App not connected** — setup PR and fix PRs cannot open. Install the
|
|
171
|
+
Archal GitHub App on the repo.
|
|
172
|
+
- **No GitHub remote** — augment/greenfield PRs need `github.com/<owner>/<repo>`.
|
|
173
|
+
- **Language unsupported by augment** — Python with existing observability falls
|
|
174
|
+
back to greenfield OTel; `@archal/state-capture` is TS-only.
|
|
175
|
+
- **Planner skipped** — wrong plan (needs Pro/Enterprise), non-TypeScript,
|
|
176
|
+
`augment-existing-vendor` legacy mode, detection unavailable, or
|
|
177
|
+
`ARCHAL_INSTALL_PLANNER_DISABLED=1`. The deterministic patch still ships; this
|
|
178
|
+
is not a failure, just a narrower install.
|
|
179
|
+
- **Setup PR is a stub for this repo** — large/unusual layout; the user finishes
|
|
180
|
+
wiring. Not a bug; the honest limit.
|
|
181
|
+
- **Trace ingest failure** — bad credential, rejected upload, or missing workspace auth.
|
|
182
|
+
Check the dashboard OTLP endpoint config or `archal autoloop status` for the hosted path.
|
|
183
|
+
- **No usable trace evidence** — once ingested, grading or reproduction can still
|
|
184
|
+
block if the trace lacks task context or state. Hand off to the `autoloop`
|
|
185
|
+
skill's evidence diagnosis.
|
|
186
|
+
|
|
187
|
+
## What to report back
|
|
188
|
+
|
|
189
|
+
After install or debugging, give the user:
|
|
190
|
+
|
|
191
|
+
- repo full name and whether the GitHub App is connected
|
|
192
|
+
- chosen path: setup PR (greenfield vs augment) or hosted autoloop / dashboard OTLP ingest
|
|
193
|
+
- if a setup PR: the install mode, the files it adds, and whether the planner ran
|
|
194
|
+
- if ingest: provider, source id, and the next step (dashboard OTLP config or `archal autoloop` command)
|
|
195
|
+
- whether traces are flowing into Archal yet, or the exact blocker
|
|
196
|
+
- next command or next owner
|
|
197
|
+
|
|
198
|
+
## Docs
|
|
199
|
+
|
|
200
|
+
- Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces
|
|
201
|
+
- CLI reference: https://docs.archal.ai/cli/autoloop
|
|
202
|
+
- Quickstart: https://docs.archal.ai/quickstart
|
package/skills/onboard/SKILL.md
CHANGED
|
@@ -11,16 +11,15 @@ You are setting up Archal in this project. Archal tests AI agents against servic
|
|
|
11
11
|
## If this is a cold-start
|
|
12
12
|
|
|
13
13
|
The user may have landed here without running `npx archal init` first. If the
|
|
14
|
-
CLI
|
|
15
|
-
|
|
16
|
-
`.cursor/skills/`, `.windsurf/skills/`), the canonical first command is:
|
|
14
|
+
CLI or project files are missing (see "Install + auth" below), the canonical
|
|
15
|
+
first command is:
|
|
17
16
|
|
|
18
17
|
```bash
|
|
19
18
|
npx archal init
|
|
20
19
|
```
|
|
21
20
|
|
|
22
|
-
That adds `archal` as a devDependency
|
|
23
|
-
|
|
21
|
+
That adds `archal` as a devDependency when a `package.json` is present, installs
|
|
22
|
+
the packaged skills for detected local agent targets, and creates a starter
|
|
24
23
|
`.archal.json`, `.archal/harness.mjs`, and `scenarios/first-run.md`. Re-invoke
|
|
25
24
|
the onboard skill after it completes.
|
|
26
25
|
|
|
@@ -54,8 +53,10 @@ archal usage # verify auth + plan
|
|
|
54
53
|
In CI, set `ARCHAL_TOKEN` to a **workspace API key** (`archal_ws_...`)
|
|
55
54
|
instead of running `archal login`. Workspace keys are bound to one workspace,
|
|
56
55
|
do not expire when a team member leaves, and are the recommended auth for CI.
|
|
57
|
-
Create one with `archal workspace api-key create <label
|
|
58
|
-
|
|
56
|
+
Create one with `archal workspace api-key create <label>` (requires owner or
|
|
57
|
+
admin role) or from the dashboard under Settings > API Keys. List and revoke
|
|
58
|
+
keys with `archal workspace api-keys` and
|
|
59
|
+
`archal workspace api-key revoke <key-id>`.
|
|
59
60
|
Personal tokens (`arc_...`) are fine for local dev but should not be used in CI.
|
|
60
61
|
|
|
61
62
|
Treat workspace API keys as runtime and CI credentials, not governance
|
|
@@ -85,7 +86,16 @@ Confirm detected clones, then ask which of these the user wants. Each delegates
|
|
|
85
86
|
- `"agent": { "command": "node", "args": ["./agent.js"] }` - plain Node script
|
|
86
87
|
- `"agent": { "command": "python", "args": ["agent.py"] }` - Python agent
|
|
87
88
|
|
|
88
|
-
If the user doesn't have a harness yet, prefer `npx archal init`; it creates `./.archal/harness.mjs`, points `.archal.json` at it, and adds a starter scenario without overwriting existing files. The generated harness is a guarded stub: Archal refuses to score it until the user edits it to call their Cursor, Codex, Claude Code, or custom agent. A custom harness should read `AGENT_TASK` from env, call the agent runtime, print `{ "text": "..." }` to stdout, and call `reportAgentMetrics()` from `archal
|
|
89
|
+
If the user doesn't have a harness yet, prefer `npx archal init`; it creates `./.archal/harness.mjs`, points `.archal.json` at it, and adds a starter scenario without overwriting existing files. The generated harness is a guarded stub: Archal refuses to score it until the user edits it to call their Cursor, Codex, Claude Code, or custom agent. A custom harness should read `AGENT_TASK` from env, call the agent runtime, print `{ "text": "..." }` to stdout, and call `reportAgentMetrics()` from `archal` with accumulated `{ inputTokens, outputTokens, llmCallCount }` before exit. Service clients need one explicit routing mode: use sandbox/Docker routing when the harness calls normal service URLs such as `https://api.github.com`, or configure SDK base URLs from `AGENT_CLONE_URLS` and add the JSON headers from `AGENT_ROUTE_HEADERS` to those clone requests. Alternative: skip `agent` in `.archal.json` and pass `--harness <path>` per-run.
|
|
90
|
+
|
|
91
|
+
### Or run a packaged agent (no harness to write)
|
|
92
|
+
|
|
93
|
+
If the user wants a ready-made agent, use a packaged agent instead of writing a harness. Packaged agents run in Docker; Archal routes service calls to clones and forwards model calls with the host's provider key.
|
|
94
|
+
|
|
95
|
+
- `archal run <scenario>.md --sandbox` - run the bundled OpenClaw agent (needs Docker). Pick the model with `--agent-model <provider/model>`; export the matching key in the shell first (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` / `GEMINI_API_KEY`).
|
|
96
|
+
- `archal run <scenario>.md --harness ./<dir> --docker --dockerfile ./<dir>/Dockerfile` - run your own packaged agent.
|
|
97
|
+
|
|
98
|
+
See the "Run a packaged agent" guide: https://docs.archal.ai/guides/packaged-agents
|
|
89
99
|
|
|
90
100
|
### Option A - Evaluate an agent with scenarios
|
|
91
101
|
|
|
@@ -109,7 +119,7 @@ Same `.archal.json` as Option A (inline `--task` still needs an agent). Use this
|
|
|
109
119
|
"clones": ["<detected clones>"]
|
|
110
120
|
}
|
|
111
121
|
```
|
|
112
|
-
2. Demo: `archal run --task "Create an issue titled hello" --clone github`.
|
|
122
|
+
2. Demo: `archal run --task "Create an issue titled hello" --harness . --docker --clone github`.
|
|
113
123
|
3. For the generated first-run project, use bare `archal run` after wiring `.archal/harness.mjs`.
|
|
114
124
|
|
|
115
125
|
### Option C - Clones in a Vitest suite
|
|
@@ -122,16 +132,17 @@ Do not paste a sample config here. The right shape depends on what's already in
|
|
|
122
132
|
|
|
123
133
|
Run: `archal clone start <detected clones>` - gives live clone URLs the user's SDK clients can point at. `archal clone status` shows the active session; `archal clone stop` tears down.
|
|
124
134
|
|
|
125
|
-
### Option E -
|
|
126
|
-
|
|
127
|
-
Use this when the repo already has agent traces from pre-production or
|
|
128
|
-
production and the user wants Archal to import, grade, reproduce, and turn
|
|
129
|
-
reproduced failures into GitHub issues or PRs.
|
|
135
|
+
### Option E - Autoloop real trace sources
|
|
130
136
|
|
|
131
|
-
**Delegate to the `
|
|
132
|
-
`archal/harness.json`, `archal/scenario.md`, seed templates, `archal
|
|
137
|
+
**Delegate to the `autoloop` skill.** It owns the trace-source mapping,
|
|
138
|
+
`archal/harness.json`, `archal/scenario.md`, seed templates, `archal autoloop`
|
|
133
139
|
commands, dashboard expectations, and failure taxonomy. Do not inline the
|
|
134
|
-
|
|
140
|
+
Autoloop flow here; it changes faster than starter scenario setup.
|
|
141
|
+
|
|
142
|
+
Set the expectation carefully: Autoloop is not arbitrary production trace replay.
|
|
143
|
+
It can reproduce only failures with enough trace evidence plus repo-owned
|
|
144
|
+
scenario/seed context to reconstruct realistic clone state. Missing evidence
|
|
145
|
+
should block with a clear artifact instead of being guessed.
|
|
135
146
|
|
|
136
147
|
## Verify
|
|
137
148
|
|
|
@@ -155,4 +166,4 @@ Run the first scenario or task. For Options A and B, hand off to the `eval` skil
|
|
|
155
166
|
|
|
156
167
|
- Quickstart: https://docs.archal.ai/quickstart
|
|
157
168
|
- Full docs: https://docs.archal.ai
|
|
158
|
-
-
|
|
169
|
+
- Autoloop production traces: https://docs.archal.ai/guides/autoloop-production-traces
|