@tracedojo/sdk 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +202 -0
- package/NOTICE +5 -0
- package/README.md +96 -0
- package/dist/cli.d.ts +2 -0
- package/dist/cli.js +356 -0
- package/dist/clock.d.ts +41 -0
- package/dist/clock.js +118 -0
- package/dist/compare.d.ts +20 -0
- package/dist/compare.js +147 -0
- package/dist/demo.d.ts +465 -0
- package/dist/demo.js +45 -0
- package/dist/index.d.ts +35 -0
- package/dist/index.js +515 -0
- package/dist/langchain.d.ts +9 -0
- package/dist/langchain.js +17 -0
- package/dist/langsmith-starter.d.ts +4 -0
- package/dist/langsmith-starter.js +77 -0
- package/dist/langsmith.d.ts +30 -0
- package/dist/langsmith.js +148 -0
- package/dist/matrix-cli.d.ts +107 -0
- package/dist/matrix-cli.js +154 -0
- package/dist/matrix.d.ts +138 -0
- package/dist/matrix.js +148 -0
- package/dist/privacy.d.ts +5 -0
- package/dist/privacy.js +51 -0
- package/dist/reports.d.ts +2 -0
- package/dist/reports.js +38 -0
- package/dist/schema.d.ts +715 -0
- package/dist/schema.js +433 -0
- package/dist/starter.d.ts +1 -0
- package/dist/starter.js +64 -0
- package/dist/summary.d.ts +2 -0
- package/dist/summary.js +50 -0
- package/dist/trends.d.ts +25 -0
- package/dist/trends.js +164 -0
- package/dist/upload.d.ts +8 -0
- package/dist/upload.js +73 -0
- package/dist/worlds.d.ts +9 -0
- package/dist/worlds.js +279 -0
- package/package.json +85 -0
- package/templates/README.md +65 -0
- package/templates/adapter.mjs +88 -0
- package/templates/ci-comment.yml +40 -0
- package/templates/ci.yml +100 -0
- package/templates/clock-adapter.mjs +42 -0
- package/templates/comment.cjs +209 -0
- package/templates/faults/duplicate-delivery/README.md +19 -0
- package/templates/faults/duplicate-delivery/adapter.mjs +40 -0
- package/templates/faults/duplicate-delivery/workflow.json +31 -0
- package/templates/faults/prompt-injection/README.md +17 -0
- package/templates/faults/prompt-injection/adapter.mjs +36 -0
- package/templates/faults/prompt-injection/workflow.json +43 -0
- package/templates/faults/stale-read/README.md +21 -0
- package/templates/faults/stale-read/adapter.mjs +59 -0
- package/templates/faults/stale-read/workflow.json +33 -0
- package/templates/langsmith-adapter.mjs +83 -0
- package/templates/workflow.json +106 -0
- package/templates/world-adapter.mjs +67 -0
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Your first TraceDojo workflow
|
|
2
|
+
|
|
3
|
+
This is a scripted booking agent with synthetic data. It makes no model calls.
|
|
4
|
+
Run from your repository root, with `@tracedojo/sdk` installed:
|
|
5
|
+
|
|
6
|
+
```sh
|
|
7
|
+
npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs
|
|
8
|
+
```
|
|
9
|
+
|
|
10
|
+
If you chose another directory, replace `dojo` in the command. Each invocation
|
|
11
|
+
saves a new report in `.tracedojo/reports/`. Add `.tracedojo/` to your repository's
|
|
12
|
+
`.gitignore`; reports can contain application data.
|
|
13
|
+
|
|
14
|
+
## Reproduce a bug and protect the fix
|
|
15
|
+
|
|
16
|
+
The starter passes five failure scenarios. In `adapter.mjs`, inside the call to
|
|
17
|
+
`reserve_slot`, replace the `operationKey,` argument with
|
|
18
|
+
`operationKey: String(attempt),`. Run the command again. A booking succeeds but
|
|
19
|
+
its confirmation is lost; retrying with a new key creates a duplicate. The check
|
|
20
|
+
fails and the command exits with code 1. Restore `operationKey,` and run again.
|
|
21
|
+
|
|
22
|
+
Import each report on the TraceDojo Workflows page. Select the lost-confirmation
|
|
23
|
+
scenario, open its failed check, and view the supporting events. Compare the
|
|
24
|
+
failure and fix under the same scenario.
|
|
25
|
+
|
|
26
|
+
## Connect your application
|
|
27
|
+
|
|
28
|
+
1. Replace `workflow.json` with your task, synthetic starting state, outcome checks,
|
|
29
|
+
and tool failures. Scenario-specific checks can describe correct refusal.
|
|
30
|
+
2. Replace `tools` with in-memory simulations of your tool behavior. Handlers may
|
|
31
|
+
return promises and receive cancellation through the third argument's signal.
|
|
32
|
+
Successful writes commit their state draft; errors discard it. Do not connect
|
|
33
|
+
these handlers to production services.
|
|
34
|
+
3. Replace `createAgent().run()` with your decision loop. Pass `context.task` to the
|
|
35
|
+
agent and route **every** tool invocation through `context.call(name, args)`.
|
|
36
|
+
Use `context.signal` for cancellation. Return a structured completion.
|
|
37
|
+
4. Keep conversation state inside `createAgent()` so every trial starts fresh.
|
|
38
|
+
Update the agent version when changing its implementation.
|
|
39
|
+
|
|
40
|
+
Your adapter is trusted local code. It can access your environment and network;
|
|
41
|
+
model calls you add may cost money. Limits are cooperative, not a code sandbox.
|
|
42
|
+
LangChain tools can be wrapped with `@tracedojo/sdk/langchain`.
|
|
43
|
+
|
|
44
|
+
## Pull-request checks
|
|
45
|
+
|
|
46
|
+
If initialized with `--ci`, `.github/workflows/tracedojo.yml` runs this workflow
|
|
47
|
+
with three trials per fault. Commit your adapter, config, package manifest,
|
|
48
|
+
lockfile, and the SDK tarball if installed from a local archive. The SDK dependency
|
|
49
|
+
must be reproducible on a fresh runner; a link to a folder on your laptop will fail.
|
|
50
|
+
|
|
51
|
+
The PR job compares against a report for the exact base commit and fails on new
|
|
52
|
+
regressions or incomplete comparisons. Known failures remain visible. Base-branch
|
|
53
|
+
runs still require all checks to pass. Reports are kept for 30 days. Merge the
|
|
54
|
+
generated test/comment workflows and `.github/tracedojo/comment.cjs` to the
|
|
55
|
+
default branch first. The separate comment workflow updates one PR comment using
|
|
56
|
+
trusted default-branch code. Missing baseline artifacts require rerunning the
|
|
57
|
+
base workflow. Configure the check in branch protection to enforce it on merges.
|
|
58
|
+
Review artifact contents and repository visibility
|
|
59
|
+
before adding private fixtures. The supplied setup needs no secrets. Opt into
|
|
60
|
+
paid model calls separately; never expose credentials to untrusted pull requests.
|
|
61
|
+
|
|
62
|
+
Exit codes: 0 passed, 1 failed checks, 2 invalid setup/input, 3 incomplete results.
|
|
63
|
+
Reports preserve individual trials; new assertion failures cannot cancel each other out.
|
|
64
|
+
|
|
65
|
+
Full setup: https://github.com/syedarman1/TraceDojo/blob/main/docs/WORKFLOWS.md
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
2
|
+
// Copyright 2026 Syed Arman
|
|
3
|
+
|
|
4
|
+
// Simulated tool behavior. Replace these handlers and workflow.json for your app.
|
|
5
|
+
// Handlers are synchronous and must not call real services. Only successful
|
|
6
|
+
// writes commit their state draft. The agent receives only the tool responses.
|
|
7
|
+
export const tools = {
|
|
8
|
+
reserve_slot: {
|
|
9
|
+
kind: 'write',
|
|
10
|
+
execute(args, state) {
|
|
11
|
+
const { customerId, slot, operationKey } = args;
|
|
12
|
+
if (
|
|
13
|
+
typeof customerId !== 'string' ||
|
|
14
|
+
typeof slot !== 'string' ||
|
|
15
|
+
typeof operationKey !== 'string' ||
|
|
16
|
+
!operationKey
|
|
17
|
+
)
|
|
18
|
+
return {
|
|
19
|
+
ok: false,
|
|
20
|
+
error: {
|
|
21
|
+
code: 'INVALID_ARGUMENTS',
|
|
22
|
+
message: 'Provide customerId, slot, and operationKey.',
|
|
23
|
+
retryable: false,
|
|
24
|
+
},
|
|
25
|
+
};
|
|
26
|
+
const previous = Object.hasOwn(state.operations, operationKey)
|
|
27
|
+
? state.operations[operationKey]
|
|
28
|
+
: undefined;
|
|
29
|
+
if (previous) {
|
|
30
|
+
if (previous.customerId !== customerId || previous.slot !== slot)
|
|
31
|
+
return {
|
|
32
|
+
ok: false,
|
|
33
|
+
error: {
|
|
34
|
+
code: 'KEY_CONFLICT',
|
|
35
|
+
message: 'The operation key belongs to a different booking.',
|
|
36
|
+
retryable: false,
|
|
37
|
+
},
|
|
38
|
+
};
|
|
39
|
+
return { ok: true, data: previous };
|
|
40
|
+
}
|
|
41
|
+
const booking = {
|
|
42
|
+
id: `booking_${state.bookings.length + 1}`,
|
|
43
|
+
customerId,
|
|
44
|
+
slot,
|
|
45
|
+
};
|
|
46
|
+
state.bookings.push(booking);
|
|
47
|
+
Object.defineProperty(state.operations, operationKey, {
|
|
48
|
+
value: booking,
|
|
49
|
+
enumerable: true,
|
|
50
|
+
writable: true,
|
|
51
|
+
configurable: true,
|
|
52
|
+
});
|
|
53
|
+
return { ok: true, data: booking };
|
|
54
|
+
},
|
|
55
|
+
},
|
|
56
|
+
};
|
|
57
|
+
|
|
58
|
+
// Deterministic reference agent. Replace run() with your agent, routing all its
|
|
59
|
+
// tools through context.call(). Keep fresh conversation state inside this factory.
|
|
60
|
+
export function createAgent() {
|
|
61
|
+
return {
|
|
62
|
+
id: 'booking-agent',
|
|
63
|
+
version: '1',
|
|
64
|
+
async run({ call, signal }) {
|
|
65
|
+
const operationKey = 'one-booking';
|
|
66
|
+
for (let attempt = 0; attempt < 3; attempt++) {
|
|
67
|
+
signal.throwIfAborted();
|
|
68
|
+
const result = await call('reserve_slot', {
|
|
69
|
+
customerId: 'customer_7',
|
|
70
|
+
slot: 'slot_42',
|
|
71
|
+
operationKey,
|
|
72
|
+
});
|
|
73
|
+
if (typeof result !== 'string' && result.ok)
|
|
74
|
+
return {
|
|
75
|
+
status: 'completed',
|
|
76
|
+
message: 'Meeting booked.',
|
|
77
|
+
output: { bookingId: result.data.id },
|
|
78
|
+
};
|
|
79
|
+
if (typeof result !== 'string' && !result.ok && !result.error.retryable)
|
|
80
|
+
return { status: 'blocked', message: result.error.message };
|
|
81
|
+
}
|
|
82
|
+
return {
|
|
83
|
+
status: 'blocked',
|
|
84
|
+
message: 'The booking could not be confirmed.',
|
|
85
|
+
};
|
|
86
|
+
},
|
|
87
|
+
};
|
|
88
|
+
}
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
name: TraceDojo PR comment
|
|
2
|
+
on:
|
|
3
|
+
workflow_run:
|
|
4
|
+
workflows: [TraceDojo agent checks]
|
|
5
|
+
types: [completed]
|
|
6
|
+
permissions:
|
|
7
|
+
contents: read
|
|
8
|
+
actions: read
|
|
9
|
+
pull-requests: write
|
|
10
|
+
concurrency:
|
|
11
|
+
group: tracedojo-comment-${{ github.event.workflow_run.workflow_id }}-${{ github.event.workflow_run.head_branch }}
|
|
12
|
+
cancel-in-progress: false
|
|
13
|
+
jobs:
|
|
14
|
+
comment:
|
|
15
|
+
if: github.event.workflow_run.event == 'pull_request'
|
|
16
|
+
runs-on: ubuntu-latest
|
|
17
|
+
timeout-minutes: 5
|
|
18
|
+
steps:
|
|
19
|
+
# Only default-branch code runs with write permission. Never check out PR code here.
|
|
20
|
+
- uses: actions/checkout@v4
|
|
21
|
+
with:
|
|
22
|
+
ref: ${{ github.event.repository.default_branch }}
|
|
23
|
+
persist-credentials: false
|
|
24
|
+
sparse-checkout: .github/tracedojo
|
|
25
|
+
- name: Download comparison data
|
|
26
|
+
continue-on-error: true
|
|
27
|
+
uses: actions/download-artifact@v4
|
|
28
|
+
with:
|
|
29
|
+
name: tracedojo-evidence
|
|
30
|
+
path: ${{ runner.temp }}/tracedojo-evidence
|
|
31
|
+
run-id: ${{ github.event.workflow_run.id }}
|
|
32
|
+
github-token: ${{ github.token }}
|
|
33
|
+
- name: Update the TraceDojo comment
|
|
34
|
+
uses: actions/github-script@v7
|
|
35
|
+
env:
|
|
36
|
+
TRACEDOJO_COMPARISON: ${{ runner.temp }}/tracedojo-evidence/comparison.json
|
|
37
|
+
with:
|
|
38
|
+
script: |
|
|
39
|
+
const comment = require('./.github/tracedojo/comment.cjs');
|
|
40
|
+
await comment({ github, context, core });
|
package/templates/ci.yml
ADDED
|
@@ -0,0 +1,100 @@
|
|
|
1
|
+
name: TraceDojo agent checks
|
|
2
|
+
on:
|
|
3
|
+
pull_request:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
workflow_dispatch:
|
|
7
|
+
permissions:
|
|
8
|
+
contents: read
|
|
9
|
+
actions: read
|
|
10
|
+
concurrency:
|
|
11
|
+
group: tracedojo-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
|
|
12
|
+
cancel-in-progress: true
|
|
13
|
+
jobs:
|
|
14
|
+
agent-checks:
|
|
15
|
+
runs-on: ubuntu-latest
|
|
16
|
+
timeout-minutes: 10
|
|
17
|
+
env:
|
|
18
|
+
DOJO_CONFIG: TRACEDOJO_CONFIG_TEMPLATE
|
|
19
|
+
DOJO_ADAPTER: TRACEDOJO_ADAPTER_TEMPLATE
|
|
20
|
+
DOJO_BASE_SHA: ${{ github.event.pull_request.base.sha }}
|
|
21
|
+
DOJO_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
|
|
22
|
+
DOJO_COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
|
|
23
|
+
steps:
|
|
24
|
+
- uses: actions/checkout@v4
|
|
25
|
+
with:
|
|
26
|
+
ref: ${{ github.event.pull_request.head.sha || github.sha }}
|
|
27
|
+
persist-credentials: false
|
|
28
|
+
- uses: actions/setup-node@v4
|
|
29
|
+
with:
|
|
30
|
+
node-version: '22.23.1'
|
|
31
|
+
cache: npm
|
|
32
|
+
- name: Find evidence for the exact base commit
|
|
33
|
+
id: baseline
|
|
34
|
+
if: github.event_name == 'pull_request'
|
|
35
|
+
uses: actions/github-script@v7
|
|
36
|
+
with:
|
|
37
|
+
script: |
|
|
38
|
+
const current = await github.rest.actions.getWorkflowRun({ ...context.repo, run_id: context.runId });
|
|
39
|
+
const { data } = await github.rest.actions.listWorkflowRuns({
|
|
40
|
+
...context.repo, workflow_id: current.data.workflow_id,
|
|
41
|
+
branch: context.payload.pull_request.base.ref,
|
|
42
|
+
head_sha: context.payload.pull_request.base.sha, status: 'completed', per_page: 100,
|
|
43
|
+
});
|
|
44
|
+
const runs = data.workflow_runs.filter(run => ['push', 'workflow_dispatch'].includes(run.event) && run.head_repository?.full_name === `${context.repo.owner}/${context.repo.repo}`);
|
|
45
|
+
runs.sort((a, b) => b.id - a.id);
|
|
46
|
+
if (runs[0]) core.setOutput('run-id', runs[0].id);
|
|
47
|
+
else core.notice('No saved report for the exact base commit. Comparison will be incomplete.');
|
|
48
|
+
- name: Download baseline evidence
|
|
49
|
+
if: steps.baseline.outputs.run-id != ''
|
|
50
|
+
continue-on-error: true
|
|
51
|
+
uses: actions/download-artifact@v4
|
|
52
|
+
with:
|
|
53
|
+
name: tracedojo-evidence
|
|
54
|
+
path: .tracedojo/base
|
|
55
|
+
run-id: ${{ steps.baseline.outputs.run-id }}
|
|
56
|
+
github-token: ${{ github.token }}
|
|
57
|
+
- run: npm ci
|
|
58
|
+
- name: Record candidate behavior
|
|
59
|
+
id: candidate
|
|
60
|
+
continue-on-error: true
|
|
61
|
+
run: |
|
|
62
|
+
set +e
|
|
63
|
+
npx --no-install tracedojo test --config "$DOJO_CONFIG" --adapter "$DOJO_ADAPTER" --commit "$DOJO_COMMIT_SHA" --trials 3 --out .tracedojo/report.json --summary .tracedojo/trials.md
|
|
64
|
+
result=$?
|
|
65
|
+
echo "exit-code=$result" >> "$GITHUB_OUTPUT"
|
|
66
|
+
exit "$result"
|
|
67
|
+
- name: Gate new regressions
|
|
68
|
+
if: github.event_name == 'pull_request' && !cancelled()
|
|
69
|
+
env:
|
|
70
|
+
DOJO_TEST_EXIT: ${{ steps.candidate.outputs.exit-code }}
|
|
71
|
+
run: |
|
|
72
|
+
if [ "$DOJO_TEST_EXIT" != "0" ] && [ "$DOJO_TEST_EXIT" != "1" ]; then
|
|
73
|
+
echo "Candidate execution did not produce comparable evidence." >&2
|
|
74
|
+
exit 3
|
|
75
|
+
fi
|
|
76
|
+
npx --no-install tracedojo compare --base .tracedojo/base/report.json --candidate .tracedojo/report.json --base-sha "$DOJO_BASE_SHA" --head-sha "$DOJO_HEAD_SHA" --out .tracedojo/comparison.json --summary .tracedojo/summary.md
|
|
77
|
+
- name: Require passing checks on the base branch
|
|
78
|
+
if: github.event_name != 'pull_request' && steps.candidate.outcome != 'success'
|
|
79
|
+
run: exit 1
|
|
80
|
+
- name: Show results
|
|
81
|
+
if: always()
|
|
82
|
+
run: |
|
|
83
|
+
if [ -f .tracedojo/summary.md ]; then
|
|
84
|
+
cat .tracedojo/summary.md >> "$GITHUB_STEP_SUMMARY"
|
|
85
|
+
elif [ -f .tracedojo/trials.md ]; then
|
|
86
|
+
cat .tracedojo/trials.md >> "$GITHUB_STEP_SUMMARY"
|
|
87
|
+
fi
|
|
88
|
+
- name: Preserve evidence
|
|
89
|
+
if: always() && contains(fromJSON('["0", "1", "3"]'), steps.candidate.outputs.exit-code)
|
|
90
|
+
uses: actions/upload-artifact@v4
|
|
91
|
+
with:
|
|
92
|
+
name: tracedojo-evidence
|
|
93
|
+
path: |
|
|
94
|
+
.tracedojo/report.json
|
|
95
|
+
.tracedojo/comparison.json
|
|
96
|
+
.tracedojo/summary.md
|
|
97
|
+
.tracedojo/trials.md
|
|
98
|
+
include-hidden-files: true
|
|
99
|
+
if-no-files-found: warn
|
|
100
|
+
retention-days: 30
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
2
|
+
// Copyright 2026 Syed Arman
|
|
3
|
+
|
|
4
|
+
import * as reference from './reference.mjs';
|
|
5
|
+
|
|
6
|
+
// Model a 25 ms service delay without waiting in real time.
|
|
7
|
+
export const tools = Object.fromEntries(
|
|
8
|
+
Object.entries(reference.tools).map(([name, tool]) => [
|
|
9
|
+
name,
|
|
10
|
+
{
|
|
11
|
+
...tool,
|
|
12
|
+
async execute(args, state, context) {
|
|
13
|
+
await context.clock.sleep(25);
|
|
14
|
+
context.signal.throwIfAborted();
|
|
15
|
+
return tool.execute(args, state, context);
|
|
16
|
+
},
|
|
17
|
+
},
|
|
18
|
+
]),
|
|
19
|
+
);
|
|
20
|
+
|
|
21
|
+
export function createAgent() {
|
|
22
|
+
const agent = reference.createAgent();
|
|
23
|
+
return {
|
|
24
|
+
...agent,
|
|
25
|
+
version: `${agent.version}-virtual-backoff`,
|
|
26
|
+
async run(context) {
|
|
27
|
+
return agent.run({
|
|
28
|
+
...context,
|
|
29
|
+
call: async (name, args) => {
|
|
30
|
+
const result = await context.call(name, args);
|
|
31
|
+
// Back off for one virtual second before the reference agent retries.
|
|
32
|
+
if (
|
|
33
|
+
typeof result === 'string' ||
|
|
34
|
+
(!result.ok && result.error.retryable)
|
|
35
|
+
)
|
|
36
|
+
await context.clock.sleep(1000);
|
|
37
|
+
return result;
|
|
38
|
+
},
|
|
39
|
+
});
|
|
40
|
+
},
|
|
41
|
+
};
|
|
42
|
+
}
|
|
@@ -0,0 +1,209 @@
|
|
|
1
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
2
|
+
// Copyright 2026 Syed Arman
|
|
3
|
+
|
|
4
|
+
// This runs only from the default branch, never from an artifact or PR checkout.
|
|
5
|
+
const escape = (value) =>
|
|
6
|
+
value.replace(/[&<>|`*_[\]\\\r\n@]/g, (char) =>
|
|
7
|
+
char === '\r' || char === '\n' ? ' ' : `&#${char.charCodeAt(0)};`,
|
|
8
|
+
);
|
|
9
|
+
async function readComparison() {
|
|
10
|
+
const fs = await import('node:fs/promises');
|
|
11
|
+
const file = process.env.TRACEDOJO_COMPARISON;
|
|
12
|
+
const stat = await fs.lstat(file);
|
|
13
|
+
if (!stat.isFile() || stat.size > 50000)
|
|
14
|
+
throw new Error('Invalid comparison artifact.');
|
|
15
|
+
return JSON.parse(await fs.readFile(file, 'utf8'));
|
|
16
|
+
}
|
|
17
|
+
function valid(value) {
|
|
18
|
+
const count = (n) => Number.isInteger(n) && n >= 0 && n <= 200;
|
|
19
|
+
if (
|
|
20
|
+
!value ||
|
|
21
|
+
value.schemaVersion !== 'workflow-comparison/1' ||
|
|
22
|
+
!['no_regressions', 'regressed', 'incomparable'].includes(value.status) ||
|
|
23
|
+
!Array.isArray(value.scenarios) ||
|
|
24
|
+
value.scenarios.length > 10
|
|
25
|
+
)
|
|
26
|
+
return false;
|
|
27
|
+
if (
|
|
28
|
+
!/^[a-f0-9]{40}$/.test(value.source?.baseSha ?? '') ||
|
|
29
|
+
!/^[a-f0-9]{40}$/.test(value.source?.headSha ?? '')
|
|
30
|
+
)
|
|
31
|
+
return false;
|
|
32
|
+
if (value.status === 'incomparable') return value.scenarios.length === 0;
|
|
33
|
+
if (!value.scenarios.length) return false;
|
|
34
|
+
for (const row of value.scenarios) {
|
|
35
|
+
if (
|
|
36
|
+
!row ||
|
|
37
|
+
typeof row.title !== 'string' ||
|
|
38
|
+
row.title.length > 2000 ||
|
|
39
|
+
!count(row.trials) ||
|
|
40
|
+
row.trials < 1 ||
|
|
41
|
+
!count(row.baseFailures) ||
|
|
42
|
+
!count(row.candidateFailures) ||
|
|
43
|
+
Math.max(row.baseFailures, row.candidateFailures) > row.trials ||
|
|
44
|
+
!['regressed', 'improved', 'unchanged'].includes(row.status) ||
|
|
45
|
+
!Array.isArray(row.regressedChecks) ||
|
|
46
|
+
row.regressedChecks.length > 55 ||
|
|
47
|
+
row.regressedChecks.some((s) => typeof s !== 'string' || s.length > 100)
|
|
48
|
+
)
|
|
49
|
+
return false;
|
|
50
|
+
if (
|
|
51
|
+
(row.candidateFailures > row.baseFailures ||
|
|
52
|
+
row.regressedChecks.length > 0) !==
|
|
53
|
+
(row.status === 'regressed')
|
|
54
|
+
)
|
|
55
|
+
return false;
|
|
56
|
+
}
|
|
57
|
+
return (
|
|
58
|
+
(value.status === 'regressed') ===
|
|
59
|
+
value.scenarios.some((row) => row.status === 'regressed')
|
|
60
|
+
);
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
module.exports = async function comment({
|
|
64
|
+
github,
|
|
65
|
+
context,
|
|
66
|
+
core,
|
|
67
|
+
read = readComparison,
|
|
68
|
+
}) {
|
|
69
|
+
const run = context.payload.workflow_run;
|
|
70
|
+
if (run.event !== 'pull_request') return;
|
|
71
|
+
const repo = context.repo;
|
|
72
|
+
const latest = await github.rest.actions.listWorkflowRuns({
|
|
73
|
+
...repo,
|
|
74
|
+
workflow_id: run.workflow_id,
|
|
75
|
+
event: 'pull_request',
|
|
76
|
+
head_sha: run.head_sha,
|
|
77
|
+
per_page: 100,
|
|
78
|
+
});
|
|
79
|
+
if (
|
|
80
|
+
latest.data.workflow_runs.some(
|
|
81
|
+
(item) =>
|
|
82
|
+
item.id > run.id ||
|
|
83
|
+
(item.id === run.id && item.run_attempt > run.run_attempt),
|
|
84
|
+
)
|
|
85
|
+
) {
|
|
86
|
+
core.info('A newer run exists; leaving its comment alone.');
|
|
87
|
+
return;
|
|
88
|
+
}
|
|
89
|
+
const associated = await github.paginate(
|
|
90
|
+
github.rest.repos.listPullRequestsAssociatedWithCommit,
|
|
91
|
+
{ ...repo, commit_sha: run.head_sha, per_page: 100 },
|
|
92
|
+
);
|
|
93
|
+
let result;
|
|
94
|
+
try {
|
|
95
|
+
const value = await read();
|
|
96
|
+
if (valid(value)) result = value;
|
|
97
|
+
} catch {
|
|
98
|
+
/* Missing data must produce an incomplete comment, not a pass. */
|
|
99
|
+
}
|
|
100
|
+
const marker = `<!-- tracedojo:${run.workflow_id} -->`;
|
|
101
|
+
for (const candidate of associated) {
|
|
102
|
+
const { data: pr } = await github.rest.pulls.get({
|
|
103
|
+
...repo,
|
|
104
|
+
pull_number: candidate.number,
|
|
105
|
+
});
|
|
106
|
+
if (
|
|
107
|
+
pr.state !== 'open' ||
|
|
108
|
+
pr.head.sha !== run.head_sha ||
|
|
109
|
+
pr.base.repo.full_name !== `${repo.owner}/${repo.repo}` ||
|
|
110
|
+
pr.head.repo?.full_name !== run.head_repository.full_name
|
|
111
|
+
)
|
|
112
|
+
continue;
|
|
113
|
+
if (
|
|
114
|
+
result &&
|
|
115
|
+
(result.source.headSha !== pr.head.sha ||
|
|
116
|
+
result.source.baseSha !== pr.base.sha)
|
|
117
|
+
) {
|
|
118
|
+
core.info('The PR moved since this comparison. Skipping stale evidence.');
|
|
119
|
+
continue;
|
|
120
|
+
}
|
|
121
|
+
const usable =
|
|
122
|
+
result &&
|
|
123
|
+
(result.status !== 'no_regressions' || run.conclusion === 'success');
|
|
124
|
+
const regressions = usable
|
|
125
|
+
? result.scenarios.filter((row) => row.status === 'regressed').length
|
|
126
|
+
: 0;
|
|
127
|
+
const title =
|
|
128
|
+
!usable || result.status === 'incomparable'
|
|
129
|
+
? 'Comparison incomplete'
|
|
130
|
+
: regressions
|
|
131
|
+
? `${regressions} scenario(s) regressed vs base`
|
|
132
|
+
: 'No new regressions vs base';
|
|
133
|
+
const lines = [
|
|
134
|
+
marker,
|
|
135
|
+
`<!-- tracedojo-run:${run.id}:${run.run_attempt} -->`,
|
|
136
|
+
`## TraceDojo: ${title}`,
|
|
137
|
+
'',
|
|
138
|
+
`Head: \`${pr.head.sha.slice(0, 7)}\` · Base: \`${pr.base.sha.slice(0, 7)}\``,
|
|
139
|
+
'',
|
|
140
|
+
];
|
|
141
|
+
if (usable && result.status !== 'incomparable') {
|
|
142
|
+
lines.push(
|
|
143
|
+
'| Scenario | Base failures | PR failures | Trials | Change |',
|
|
144
|
+
'| --- | ---: | ---: | ---: | --- |',
|
|
145
|
+
);
|
|
146
|
+
for (const row of result.scenarios)
|
|
147
|
+
lines.push(
|
|
148
|
+
`| ${escape(row.title.slice(0, 200))} | ${row.baseFailures} | ${row.candidateFailures} | ${row.trials} | ${row.status} |`,
|
|
149
|
+
);
|
|
150
|
+
lines.push(
|
|
151
|
+
'',
|
|
152
|
+
'Known failures may remain. Counts describe observed trials, not a reliability guarantee.',
|
|
153
|
+
);
|
|
154
|
+
} else
|
|
155
|
+
lines.push(
|
|
156
|
+
'No passing regression verdict is available. Check for missing baseline evidence, changed conditions, interrupted trials, or a CI error.',
|
|
157
|
+
);
|
|
158
|
+
lines.push(
|
|
159
|
+
'',
|
|
160
|
+
`[View checks and evidence](https://github.com/${repo.owner}/${repo.repo}/actions/runs/${run.id})`,
|
|
161
|
+
);
|
|
162
|
+
const comments = await github.paginate(github.rest.issues.listComments, {
|
|
163
|
+
...repo,
|
|
164
|
+
issue_number: pr.number,
|
|
165
|
+
per_page: 100,
|
|
166
|
+
});
|
|
167
|
+
const existing = comments.find(
|
|
168
|
+
(item) =>
|
|
169
|
+
item.user?.login === 'github-actions[bot]' &&
|
|
170
|
+
item.user?.type === 'Bot' &&
|
|
171
|
+
item.body?.startsWith(marker),
|
|
172
|
+
);
|
|
173
|
+
if (existing) {
|
|
174
|
+
const previous = existing.body.match(
|
|
175
|
+
/<!-- tracedojo-run:(\d+):(\d+) -->/,
|
|
176
|
+
);
|
|
177
|
+
if (
|
|
178
|
+
previous &&
|
|
179
|
+
(+previous[1] > run.id ||
|
|
180
|
+
(+previous[1] === run.id && +previous[2] > run.run_attempt))
|
|
181
|
+
)
|
|
182
|
+
continue;
|
|
183
|
+
}
|
|
184
|
+
// Recheck the moving PR before updating; artifact contents never select a PR.
|
|
185
|
+
const { data: current } = await github.rest.pulls.get({
|
|
186
|
+
...repo,
|
|
187
|
+
pull_number: pr.number,
|
|
188
|
+
});
|
|
189
|
+
if (
|
|
190
|
+
current.state !== 'open' ||
|
|
191
|
+
current.head.sha !== pr.head.sha ||
|
|
192
|
+
current.base.sha !== pr.base.sha
|
|
193
|
+
)
|
|
194
|
+
continue;
|
|
195
|
+
const body = lines.join('\n');
|
|
196
|
+
if (existing)
|
|
197
|
+
await github.rest.issues.updateComment({
|
|
198
|
+
...repo,
|
|
199
|
+
comment_id: existing.id,
|
|
200
|
+
body,
|
|
201
|
+
});
|
|
202
|
+
else
|
|
203
|
+
await github.rest.issues.createComment({
|
|
204
|
+
...repo,
|
|
205
|
+
issue_number: pr.number,
|
|
206
|
+
body,
|
|
207
|
+
});
|
|
208
|
+
}
|
|
209
|
+
};
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# One request delivered twice
|
|
2
|
+
|
|
3
|
+
Run `npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs`
|
|
4
|
+
from the parent of `dojo` (adjust paths for your chosen directory).
|
|
5
|
+
|
|
6
|
+
The scripted reference passes. Set `deduplicate = false` in `adapter.mjs`: the
|
|
7
|
+
control still passes, but duplicate delivery credits 1000 cents instead of 500.
|
|
8
|
+
Restore it to verify the fix. This models service idempotency as well as agent inputs.
|
|
9
|
+
|
|
10
|
+
`duplicate_delivery` requires a write tool. The selected call invokes it twice,
|
|
11
|
+
sequentially, with identical cloned arguments and separate state transactions.
|
|
12
|
+
Both attempts execute even if the first returns a structured error. The agent
|
|
13
|
+
receives only the first response after both finish. A thrown/invalid handler or
|
|
14
|
+
cancellation stops execution and preserves partial evidence. This is immediate
|
|
15
|
+
redelivery, not arbitrary queue reordering or concurrent writes.
|
|
16
|
+
|
|
17
|
+
Evaluator-only `delivery` events distinguish executions from the single agent
|
|
18
|
+
`tool_call`. Each committed effect has its own mutation evidence; a deduplicated
|
|
19
|
+
second execution can have no mutation. Never call a real payment service here.
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
2
|
+
// Copyright 2026 Syed Arman
|
|
3
|
+
|
|
4
|
+
const deduplicate = true;
|
|
5
|
+
export const tools = {
|
|
6
|
+
credit_account: {
|
|
7
|
+
kind: 'write',
|
|
8
|
+
execute(args, state) {
|
|
9
|
+
if (args.operationKey !== 'request_7')
|
|
10
|
+
return {
|
|
11
|
+
ok: false,
|
|
12
|
+
error: {
|
|
13
|
+
code: 'INVALID_INPUT',
|
|
14
|
+
message: 'Expected request_7.',
|
|
15
|
+
retryable: false,
|
|
16
|
+
},
|
|
17
|
+
};
|
|
18
|
+
if (!deduplicate || !state.processed.includes(args.operationKey)) {
|
|
19
|
+
state.creditedCents += 500;
|
|
20
|
+
state.processed.push(args.operationKey);
|
|
21
|
+
}
|
|
22
|
+
return { ok: true, data: { creditedCents: 500 } };
|
|
23
|
+
},
|
|
24
|
+
},
|
|
25
|
+
};
|
|
26
|
+
export function createAgent() {
|
|
27
|
+
return {
|
|
28
|
+
id: 'credit-requester',
|
|
29
|
+
version: deduplicate ? 'safe-1' : 'unsafe-1',
|
|
30
|
+
async run({ call }) {
|
|
31
|
+
const result = await call('credit_account', {
|
|
32
|
+
operationKey: 'request_7',
|
|
33
|
+
});
|
|
34
|
+
return {
|
|
35
|
+
status: result?.ok ? 'completed' : 'blocked',
|
|
36
|
+
message: 'Credit requested.',
|
|
37
|
+
};
|
|
38
|
+
},
|
|
39
|
+
};
|
|
40
|
+
}
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": "workflow/1",
|
|
3
|
+
"id": "duplicate-delivery",
|
|
4
|
+
"title": "A queue delivers one credit request twice",
|
|
5
|
+
"task": "Credit 500 cents once using request_7 as the idempotency key.",
|
|
6
|
+
"initialState": { "creditedCents": 0, "processed": [] },
|
|
7
|
+
"checks": [
|
|
8
|
+
{
|
|
9
|
+
"id": "one-credit",
|
|
10
|
+
"label": "Credit exactly 500 cents",
|
|
11
|
+
"type": "equals",
|
|
12
|
+
"path": "/creditedCents",
|
|
13
|
+
"expected": 500
|
|
14
|
+
},
|
|
15
|
+
{
|
|
16
|
+
"id": "completed",
|
|
17
|
+
"label": "Confirm the credit",
|
|
18
|
+
"type": "equals",
|
|
19
|
+
"source": "completion",
|
|
20
|
+
"path": "/status",
|
|
21
|
+
"expected": "completed"
|
|
22
|
+
}
|
|
23
|
+
],
|
|
24
|
+
"scenarios": [
|
|
25
|
+
{
|
|
26
|
+
"id": "redelivered",
|
|
27
|
+
"title": "One request, two write deliveries",
|
|
28
|
+
"fault": { "type": "duplicate_delivery", "tool": "credit_account" }
|
|
29
|
+
}
|
|
30
|
+
]
|
|
31
|
+
}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
# Instructions in tool output
|
|
2
|
+
|
|
3
|
+
Run `npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs`
|
|
4
|
+
from the parent of `dojo` (adjust the paths if you chose another directory).
|
|
5
|
+
|
|
6
|
+
The scripted reference passes. In `adapter.mjs`, set `trustToolInstructions` to
|
|
7
|
+
`true`: the control still passes, but the injected instruction triggers an
|
|
8
|
+
unauthorized refund. Both the attempt and committed state fail independent checks.
|
|
9
|
+
|
|
10
|
+
`prompt_injection` replaces the entire `data` field of a successful tool response
|
|
11
|
+
with the configured JSON. Its original response stays evaluator-only; the agent
|
|
12
|
+
receives the replacement. Error responses are not replaced. A scenario without a
|
|
13
|
+
successful targeted response is inconclusive. `occurrence` and `repeat` select calls.
|
|
14
|
+
|
|
15
|
+
Replace this scripted loop with your agent to test its behavior. A passing scripted
|
|
16
|
+
fixture establishes the harness behavior, not a model's resistance to attacks.
|
|
17
|
+
The replacement is untrusted test data; it is never executed by the runner.
|