@tracedojo/sdk 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (58) hide show
  1. package/LICENSE +202 -0
  2. package/NOTICE +5 -0
  3. package/README.md +96 -0
  4. package/dist/cli.d.ts +2 -0
  5. package/dist/cli.js +356 -0
  6. package/dist/clock.d.ts +41 -0
  7. package/dist/clock.js +118 -0
  8. package/dist/compare.d.ts +20 -0
  9. package/dist/compare.js +147 -0
  10. package/dist/demo.d.ts +465 -0
  11. package/dist/demo.js +45 -0
  12. package/dist/index.d.ts +35 -0
  13. package/dist/index.js +515 -0
  14. package/dist/langchain.d.ts +9 -0
  15. package/dist/langchain.js +17 -0
  16. package/dist/langsmith-starter.d.ts +4 -0
  17. package/dist/langsmith-starter.js +77 -0
  18. package/dist/langsmith.d.ts +30 -0
  19. package/dist/langsmith.js +148 -0
  20. package/dist/matrix-cli.d.ts +107 -0
  21. package/dist/matrix-cli.js +154 -0
  22. package/dist/matrix.d.ts +138 -0
  23. package/dist/matrix.js +148 -0
  24. package/dist/privacy.d.ts +5 -0
  25. package/dist/privacy.js +51 -0
  26. package/dist/reports.d.ts +2 -0
  27. package/dist/reports.js +38 -0
  28. package/dist/schema.d.ts +715 -0
  29. package/dist/schema.js +433 -0
  30. package/dist/starter.d.ts +1 -0
  31. package/dist/starter.js +64 -0
  32. package/dist/summary.d.ts +2 -0
  33. package/dist/summary.js +50 -0
  34. package/dist/trends.d.ts +25 -0
  35. package/dist/trends.js +164 -0
  36. package/dist/upload.d.ts +8 -0
  37. package/dist/upload.js +73 -0
  38. package/dist/worlds.d.ts +9 -0
  39. package/dist/worlds.js +279 -0
  40. package/package.json +85 -0
  41. package/templates/README.md +65 -0
  42. package/templates/adapter.mjs +88 -0
  43. package/templates/ci-comment.yml +40 -0
  44. package/templates/ci.yml +100 -0
  45. package/templates/clock-adapter.mjs +42 -0
  46. package/templates/comment.cjs +209 -0
  47. package/templates/faults/duplicate-delivery/README.md +19 -0
  48. package/templates/faults/duplicate-delivery/adapter.mjs +40 -0
  49. package/templates/faults/duplicate-delivery/workflow.json +31 -0
  50. package/templates/faults/prompt-injection/README.md +17 -0
  51. package/templates/faults/prompt-injection/adapter.mjs +36 -0
  52. package/templates/faults/prompt-injection/workflow.json +43 -0
  53. package/templates/faults/stale-read/README.md +21 -0
  54. package/templates/faults/stale-read/adapter.mjs +59 -0
  55. package/templates/faults/stale-read/workflow.json +33 -0
  56. package/templates/langsmith-adapter.mjs +83 -0
  57. package/templates/workflow.json +106 -0
  58. package/templates/world-adapter.mjs +67 -0
@@ -0,0 +1,65 @@
1
+ # Your first TraceDojo workflow
2
+
3
+ This is a scripted booking agent with synthetic data. It makes no model calls.
4
+ Run from your repository root, with `@tracedojo/sdk` installed:
5
+
6
+ ```sh
7
+ npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs
8
+ ```
9
+
10
+ If you chose another directory, replace `dojo` in the command. Each invocation
11
+ saves a new report in `.tracedojo/reports/`. Add `.tracedojo/` to your repository's
12
+ `.gitignore`; reports can contain application data.
13
+
14
+ ## Reproduce a bug and protect the fix
15
+
16
+ The starter passes five failure scenarios. In `adapter.mjs`, inside the call to
17
+ `reserve_slot`, replace the `operationKey,` argument with
18
+ `operationKey: String(attempt),`. Run the command again. A booking succeeds but
19
+ its confirmation is lost; retrying with a new key creates a duplicate. The check
20
+ fails and the command exits with code 1. Restore `operationKey,` and run again.
21
+
22
+ Import each report on the TraceDojo Workflows page. Select the lost-confirmation
23
+ scenario, open its failed check, and view the supporting events. Compare the
24
+ failure and fix under the same scenario.
25
+
26
+ ## Connect your application
27
+
28
+ 1. Replace `workflow.json` with your task, synthetic starting state, outcome checks,
29
+ and tool failures. Scenario-specific checks can describe correct refusal.
30
+ 2. Replace `tools` with in-memory simulations of your tool behavior. Handlers may
31
+ return promises and receive cancellation through the third argument's signal.
32
+ Successful writes commit their state draft; errors discard it. Do not connect
33
+ these handlers to production services.
34
+ 3. Replace `createAgent().run()` with your decision loop. Pass `context.task` to the
35
+ agent and route **every** tool invocation through `context.call(name, args)`.
36
+ Use `context.signal` for cancellation. Return a structured completion.
37
+ 4. Keep conversation state inside `createAgent()` so every trial starts fresh.
38
+ Update the agent version when changing its implementation.
39
+
40
+ Your adapter is trusted local code. It can access your environment and network;
41
+ model calls you add may cost money. Limits are cooperative, not a code sandbox.
42
+ LangChain tools can be wrapped with `@tracedojo/sdk/langchain`.
43
+
44
+ ## Pull-request checks
45
+
46
+ If initialized with `--ci`, `.github/workflows/tracedojo.yml` runs this workflow
47
+ with three trials per fault. Commit your adapter, config, package manifest,
48
+ lockfile, and the SDK tarball if installed from a local archive. The SDK dependency
49
+ must be reproducible on a fresh runner; a link to a folder on your laptop will fail.
50
+
51
+ The PR job compares against a report for the exact base commit and fails on new
52
+ regressions or incomplete comparisons. Known failures remain visible. Base-branch
53
+ runs still require all checks to pass. Reports are kept for 30 days. Merge the
54
+ generated test/comment workflows and `.github/tracedojo/comment.cjs` to the
55
+ default branch first. The separate comment workflow updates one PR comment using
56
+ trusted default-branch code. Missing baseline artifacts require rerunning the
57
+ base workflow. Configure the check in branch protection to enforce it on merges.
58
+ Review artifact contents and repository visibility
59
+ before adding private fixtures. The supplied setup needs no secrets. Opt into
60
+ paid model calls separately; never expose credentials to untrusted pull requests.
61
+
62
+ Exit codes: 0 passed, 1 failed checks, 2 invalid setup/input, 3 incomplete results.
63
+ Reports preserve individual trials; new assertion failures cannot cancel each other out.
64
+
65
+ Full setup: https://github.com/syedarman1/TraceDojo/blob/main/docs/WORKFLOWS.md
@@ -0,0 +1,88 @@
1
+ // SPDX-License-Identifier: Apache-2.0
2
+ // Copyright 2026 Syed Arman
3
+
4
+ // Simulated tool behavior. Replace these handlers and workflow.json for your app.
5
+ // Handlers are synchronous and must not call real services. Only successful
6
+ // writes commit their state draft. The agent receives only the tool responses.
7
+ export const tools = {
8
+ reserve_slot: {
9
+ kind: 'write',
10
+ execute(args, state) {
11
+ const { customerId, slot, operationKey } = args;
12
+ if (
13
+ typeof customerId !== 'string' ||
14
+ typeof slot !== 'string' ||
15
+ typeof operationKey !== 'string' ||
16
+ !operationKey
17
+ )
18
+ return {
19
+ ok: false,
20
+ error: {
21
+ code: 'INVALID_ARGUMENTS',
22
+ message: 'Provide customerId, slot, and operationKey.',
23
+ retryable: false,
24
+ },
25
+ };
26
+ const previous = Object.hasOwn(state.operations, operationKey)
27
+ ? state.operations[operationKey]
28
+ : undefined;
29
+ if (previous) {
30
+ if (previous.customerId !== customerId || previous.slot !== slot)
31
+ return {
32
+ ok: false,
33
+ error: {
34
+ code: 'KEY_CONFLICT',
35
+ message: 'The operation key belongs to a different booking.',
36
+ retryable: false,
37
+ },
38
+ };
39
+ return { ok: true, data: previous };
40
+ }
41
+ const booking = {
42
+ id: `booking_${state.bookings.length + 1}`,
43
+ customerId,
44
+ slot,
45
+ };
46
+ state.bookings.push(booking);
47
+ Object.defineProperty(state.operations, operationKey, {
48
+ value: booking,
49
+ enumerable: true,
50
+ writable: true,
51
+ configurable: true,
52
+ });
53
+ return { ok: true, data: booking };
54
+ },
55
+ },
56
+ };
57
+
58
+ // Deterministic reference agent. Replace run() with your agent, routing all its
59
+ // tools through context.call(). Keep fresh conversation state inside this factory.
60
+ export function createAgent() {
61
+ return {
62
+ id: 'booking-agent',
63
+ version: '1',
64
+ async run({ call, signal }) {
65
+ const operationKey = 'one-booking';
66
+ for (let attempt = 0; attempt < 3; attempt++) {
67
+ signal.throwIfAborted();
68
+ const result = await call('reserve_slot', {
69
+ customerId: 'customer_7',
70
+ slot: 'slot_42',
71
+ operationKey,
72
+ });
73
+ if (typeof result !== 'string' && result.ok)
74
+ return {
75
+ status: 'completed',
76
+ message: 'Meeting booked.',
77
+ output: { bookingId: result.data.id },
78
+ };
79
+ if (typeof result !== 'string' && !result.ok && !result.error.retryable)
80
+ return { status: 'blocked', message: result.error.message };
81
+ }
82
+ return {
83
+ status: 'blocked',
84
+ message: 'The booking could not be confirmed.',
85
+ };
86
+ },
87
+ };
88
+ }
@@ -0,0 +1,40 @@
1
+ name: TraceDojo PR comment
2
+ on:
3
+ workflow_run:
4
+ workflows: [TraceDojo agent checks]
5
+ types: [completed]
6
+ permissions:
7
+ contents: read
8
+ actions: read
9
+ pull-requests: write
10
+ concurrency:
11
+ group: tracedojo-comment-${{ github.event.workflow_run.workflow_id }}-${{ github.event.workflow_run.head_branch }}
12
+ cancel-in-progress: false
13
+ jobs:
14
+ comment:
15
+ if: github.event.workflow_run.event == 'pull_request'
16
+ runs-on: ubuntu-latest
17
+ timeout-minutes: 5
18
+ steps:
19
+ # Only default-branch code runs with write permission. Never check out PR code here.
20
+ - uses: actions/checkout@v4
21
+ with:
22
+ ref: ${{ github.event.repository.default_branch }}
23
+ persist-credentials: false
24
+ sparse-checkout: .github/tracedojo
25
+ - name: Download comparison data
26
+ continue-on-error: true
27
+ uses: actions/download-artifact@v4
28
+ with:
29
+ name: tracedojo-evidence
30
+ path: ${{ runner.temp }}/tracedojo-evidence
31
+ run-id: ${{ github.event.workflow_run.id }}
32
+ github-token: ${{ github.token }}
33
+ - name: Update the TraceDojo comment
34
+ uses: actions/github-script@v7
35
+ env:
36
+ TRACEDOJO_COMPARISON: ${{ runner.temp }}/tracedojo-evidence/comparison.json
37
+ with:
38
+ script: |
39
+ const comment = require('./.github/tracedojo/comment.cjs');
40
+ await comment({ github, context, core });
@@ -0,0 +1,100 @@
1
+ name: TraceDojo agent checks
2
+ on:
3
+ pull_request:
4
+ push:
5
+ branches: [main]
6
+ workflow_dispatch:
7
+ permissions:
8
+ contents: read
9
+ actions: read
10
+ concurrency:
11
+ group: tracedojo-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
12
+ cancel-in-progress: true
13
+ jobs:
14
+ agent-checks:
15
+ runs-on: ubuntu-latest
16
+ timeout-minutes: 10
17
+ env:
18
+ DOJO_CONFIG: TRACEDOJO_CONFIG_TEMPLATE
19
+ DOJO_ADAPTER: TRACEDOJO_ADAPTER_TEMPLATE
20
+ DOJO_BASE_SHA: ${{ github.event.pull_request.base.sha }}
21
+ DOJO_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
22
+ DOJO_COMMIT_SHA: ${{ github.event.pull_request.head.sha || github.sha }}
23
+ steps:
24
+ - uses: actions/checkout@v4
25
+ with:
26
+ ref: ${{ github.event.pull_request.head.sha || github.sha }}
27
+ persist-credentials: false
28
+ - uses: actions/setup-node@v4
29
+ with:
30
+ node-version: '22.23.1'
31
+ cache: npm
32
+ - name: Find evidence for the exact base commit
33
+ id: baseline
34
+ if: github.event_name == 'pull_request'
35
+ uses: actions/github-script@v7
36
+ with:
37
+ script: |
38
+ const current = await github.rest.actions.getWorkflowRun({ ...context.repo, run_id: context.runId });
39
+ const { data } = await github.rest.actions.listWorkflowRuns({
40
+ ...context.repo, workflow_id: current.data.workflow_id,
41
+ branch: context.payload.pull_request.base.ref,
42
+ head_sha: context.payload.pull_request.base.sha, status: 'completed', per_page: 100,
43
+ });
44
+ const runs = data.workflow_runs.filter(run => ['push', 'workflow_dispatch'].includes(run.event) && run.head_repository?.full_name === `${context.repo.owner}/${context.repo.repo}`);
45
+ runs.sort((a, b) => b.id - a.id);
46
+ if (runs[0]) core.setOutput('run-id', runs[0].id);
47
+ else core.notice('No saved report for the exact base commit. Comparison will be incomplete.');
48
+ - name: Download baseline evidence
49
+ if: steps.baseline.outputs.run-id != ''
50
+ continue-on-error: true
51
+ uses: actions/download-artifact@v4
52
+ with:
53
+ name: tracedojo-evidence
54
+ path: .tracedojo/base
55
+ run-id: ${{ steps.baseline.outputs.run-id }}
56
+ github-token: ${{ github.token }}
57
+ - run: npm ci
58
+ - name: Record candidate behavior
59
+ id: candidate
60
+ continue-on-error: true
61
+ run: |
62
+ set +e
63
+ npx --no-install tracedojo test --config "$DOJO_CONFIG" --adapter "$DOJO_ADAPTER" --commit "$DOJO_COMMIT_SHA" --trials 3 --out .tracedojo/report.json --summary .tracedojo/trials.md
64
+ result=$?
65
+ echo "exit-code=$result" >> "$GITHUB_OUTPUT"
66
+ exit "$result"
67
+ - name: Gate new regressions
68
+ if: github.event_name == 'pull_request' && !cancelled()
69
+ env:
70
+ DOJO_TEST_EXIT: ${{ steps.candidate.outputs.exit-code }}
71
+ run: |
72
+ if [ "$DOJO_TEST_EXIT" != "0" ] && [ "$DOJO_TEST_EXIT" != "1" ]; then
73
+ echo "Candidate execution did not produce comparable evidence." >&2
74
+ exit 3
75
+ fi
76
+ npx --no-install tracedojo compare --base .tracedojo/base/report.json --candidate .tracedojo/report.json --base-sha "$DOJO_BASE_SHA" --head-sha "$DOJO_HEAD_SHA" --out .tracedojo/comparison.json --summary .tracedojo/summary.md
77
+ - name: Require passing checks on the base branch
78
+ if: github.event_name != 'pull_request' && steps.candidate.outcome != 'success'
79
+ run: exit 1
80
+ - name: Show results
81
+ if: always()
82
+ run: |
83
+ if [ -f .tracedojo/summary.md ]; then
84
+ cat .tracedojo/summary.md >> "$GITHUB_STEP_SUMMARY"
85
+ elif [ -f .tracedojo/trials.md ]; then
86
+ cat .tracedojo/trials.md >> "$GITHUB_STEP_SUMMARY"
87
+ fi
88
+ - name: Preserve evidence
89
+ if: always() && contains(fromJSON('["0", "1", "3"]'), steps.candidate.outputs.exit-code)
90
+ uses: actions/upload-artifact@v4
91
+ with:
92
+ name: tracedojo-evidence
93
+ path: |
94
+ .tracedojo/report.json
95
+ .tracedojo/comparison.json
96
+ .tracedojo/summary.md
97
+ .tracedojo/trials.md
98
+ include-hidden-files: true
99
+ if-no-files-found: warn
100
+ retention-days: 30
@@ -0,0 +1,42 @@
1
+ // SPDX-License-Identifier: Apache-2.0
2
+ // Copyright 2026 Syed Arman
3
+
4
+ import * as reference from './reference.mjs';
5
+
6
+ // Model a 25 ms service delay without waiting in real time.
7
+ export const tools = Object.fromEntries(
8
+ Object.entries(reference.tools).map(([name, tool]) => [
9
+ name,
10
+ {
11
+ ...tool,
12
+ async execute(args, state, context) {
13
+ await context.clock.sleep(25);
14
+ context.signal.throwIfAborted();
15
+ return tool.execute(args, state, context);
16
+ },
17
+ },
18
+ ]),
19
+ );
20
+
21
+ export function createAgent() {
22
+ const agent = reference.createAgent();
23
+ return {
24
+ ...agent,
25
+ version: `${agent.version}-virtual-backoff`,
26
+ async run(context) {
27
+ return agent.run({
28
+ ...context,
29
+ call: async (name, args) => {
30
+ const result = await context.call(name, args);
31
+ // Back off for one virtual second before the reference agent retries.
32
+ if (
33
+ typeof result === 'string' ||
34
+ (!result.ok && result.error.retryable)
35
+ )
36
+ await context.clock.sleep(1000);
37
+ return result;
38
+ },
39
+ });
40
+ },
41
+ };
42
+ }
@@ -0,0 +1,209 @@
1
+ // SPDX-License-Identifier: Apache-2.0
2
+ // Copyright 2026 Syed Arman
3
+
4
+ // This runs only from the default branch, never from an artifact or PR checkout.
5
+ const escape = (value) =>
6
+ value.replace(/[&<>|`*_[\]\\\r\n@]/g, (char) =>
7
+ char === '\r' || char === '\n' ? ' ' : `&#${char.charCodeAt(0)};`,
8
+ );
9
+ async function readComparison() {
10
+ const fs = await import('node:fs/promises');
11
+ const file = process.env.TRACEDOJO_COMPARISON;
12
+ const stat = await fs.lstat(file);
13
+ if (!stat.isFile() || stat.size > 50000)
14
+ throw new Error('Invalid comparison artifact.');
15
+ return JSON.parse(await fs.readFile(file, 'utf8'));
16
+ }
17
+ function valid(value) {
18
+ const count = (n) => Number.isInteger(n) && n >= 0 && n <= 200;
19
+ if (
20
+ !value ||
21
+ value.schemaVersion !== 'workflow-comparison/1' ||
22
+ !['no_regressions', 'regressed', 'incomparable'].includes(value.status) ||
23
+ !Array.isArray(value.scenarios) ||
24
+ value.scenarios.length > 10
25
+ )
26
+ return false;
27
+ if (
28
+ !/^[a-f0-9]{40}$/.test(value.source?.baseSha ?? '') ||
29
+ !/^[a-f0-9]{40}$/.test(value.source?.headSha ?? '')
30
+ )
31
+ return false;
32
+ if (value.status === 'incomparable') return value.scenarios.length === 0;
33
+ if (!value.scenarios.length) return false;
34
+ for (const row of value.scenarios) {
35
+ if (
36
+ !row ||
37
+ typeof row.title !== 'string' ||
38
+ row.title.length > 2000 ||
39
+ !count(row.trials) ||
40
+ row.trials < 1 ||
41
+ !count(row.baseFailures) ||
42
+ !count(row.candidateFailures) ||
43
+ Math.max(row.baseFailures, row.candidateFailures) > row.trials ||
44
+ !['regressed', 'improved', 'unchanged'].includes(row.status) ||
45
+ !Array.isArray(row.regressedChecks) ||
46
+ row.regressedChecks.length > 55 ||
47
+ row.regressedChecks.some((s) => typeof s !== 'string' || s.length > 100)
48
+ )
49
+ return false;
50
+ if (
51
+ (row.candidateFailures > row.baseFailures ||
52
+ row.regressedChecks.length > 0) !==
53
+ (row.status === 'regressed')
54
+ )
55
+ return false;
56
+ }
57
+ return (
58
+ (value.status === 'regressed') ===
59
+ value.scenarios.some((row) => row.status === 'regressed')
60
+ );
61
+ }
62
+
63
+ module.exports = async function comment({
64
+ github,
65
+ context,
66
+ core,
67
+ read = readComparison,
68
+ }) {
69
+ const run = context.payload.workflow_run;
70
+ if (run.event !== 'pull_request') return;
71
+ const repo = context.repo;
72
+ const latest = await github.rest.actions.listWorkflowRuns({
73
+ ...repo,
74
+ workflow_id: run.workflow_id,
75
+ event: 'pull_request',
76
+ head_sha: run.head_sha,
77
+ per_page: 100,
78
+ });
79
+ if (
80
+ latest.data.workflow_runs.some(
81
+ (item) =>
82
+ item.id > run.id ||
83
+ (item.id === run.id && item.run_attempt > run.run_attempt),
84
+ )
85
+ ) {
86
+ core.info('A newer run exists; leaving its comment alone.');
87
+ return;
88
+ }
89
+ const associated = await github.paginate(
90
+ github.rest.repos.listPullRequestsAssociatedWithCommit,
91
+ { ...repo, commit_sha: run.head_sha, per_page: 100 },
92
+ );
93
+ let result;
94
+ try {
95
+ const value = await read();
96
+ if (valid(value)) result = value;
97
+ } catch {
98
+ /* Missing data must produce an incomplete comment, not a pass. */
99
+ }
100
+ const marker = `<!-- tracedojo:${run.workflow_id} -->`;
101
+ for (const candidate of associated) {
102
+ const { data: pr } = await github.rest.pulls.get({
103
+ ...repo,
104
+ pull_number: candidate.number,
105
+ });
106
+ if (
107
+ pr.state !== 'open' ||
108
+ pr.head.sha !== run.head_sha ||
109
+ pr.base.repo.full_name !== `${repo.owner}/${repo.repo}` ||
110
+ pr.head.repo?.full_name !== run.head_repository.full_name
111
+ )
112
+ continue;
113
+ if (
114
+ result &&
115
+ (result.source.headSha !== pr.head.sha ||
116
+ result.source.baseSha !== pr.base.sha)
117
+ ) {
118
+ core.info('The PR moved since this comparison. Skipping stale evidence.');
119
+ continue;
120
+ }
121
+ const usable =
122
+ result &&
123
+ (result.status !== 'no_regressions' || run.conclusion === 'success');
124
+ const regressions = usable
125
+ ? result.scenarios.filter((row) => row.status === 'regressed').length
126
+ : 0;
127
+ const title =
128
+ !usable || result.status === 'incomparable'
129
+ ? 'Comparison incomplete'
130
+ : regressions
131
+ ? `${regressions} scenario(s) regressed vs base`
132
+ : 'No new regressions vs base';
133
+ const lines = [
134
+ marker,
135
+ `<!-- tracedojo-run:${run.id}:${run.run_attempt} -->`,
136
+ `## TraceDojo: ${title}`,
137
+ '',
138
+ `Head: \`${pr.head.sha.slice(0, 7)}\` · Base: \`${pr.base.sha.slice(0, 7)}\``,
139
+ '',
140
+ ];
141
+ if (usable && result.status !== 'incomparable') {
142
+ lines.push(
143
+ '| Scenario | Base failures | PR failures | Trials | Change |',
144
+ '| --- | ---: | ---: | ---: | --- |',
145
+ );
146
+ for (const row of result.scenarios)
147
+ lines.push(
148
+ `| ${escape(row.title.slice(0, 200))} | ${row.baseFailures} | ${row.candidateFailures} | ${row.trials} | ${row.status} |`,
149
+ );
150
+ lines.push(
151
+ '',
152
+ 'Known failures may remain. Counts describe observed trials, not a reliability guarantee.',
153
+ );
154
+ } else
155
+ lines.push(
156
+ 'No passing regression verdict is available. Check for missing baseline evidence, changed conditions, interrupted trials, or a CI error.',
157
+ );
158
+ lines.push(
159
+ '',
160
+ `[View checks and evidence](https://github.com/${repo.owner}/${repo.repo}/actions/runs/${run.id})`,
161
+ );
162
+ const comments = await github.paginate(github.rest.issues.listComments, {
163
+ ...repo,
164
+ issue_number: pr.number,
165
+ per_page: 100,
166
+ });
167
+ const existing = comments.find(
168
+ (item) =>
169
+ item.user?.login === 'github-actions[bot]' &&
170
+ item.user?.type === 'Bot' &&
171
+ item.body?.startsWith(marker),
172
+ );
173
+ if (existing) {
174
+ const previous = existing.body.match(
175
+ /<!-- tracedojo-run:(\d+):(\d+) -->/,
176
+ );
177
+ if (
178
+ previous &&
179
+ (+previous[1] > run.id ||
180
+ (+previous[1] === run.id && +previous[2] > run.run_attempt))
181
+ )
182
+ continue;
183
+ }
184
+ // Recheck the moving PR before updating; artifact contents never select a PR.
185
+ const { data: current } = await github.rest.pulls.get({
186
+ ...repo,
187
+ pull_number: pr.number,
188
+ });
189
+ if (
190
+ current.state !== 'open' ||
191
+ current.head.sha !== pr.head.sha ||
192
+ current.base.sha !== pr.base.sha
193
+ )
194
+ continue;
195
+ const body = lines.join('\n');
196
+ if (existing)
197
+ await github.rest.issues.updateComment({
198
+ ...repo,
199
+ comment_id: existing.id,
200
+ body,
201
+ });
202
+ else
203
+ await github.rest.issues.createComment({
204
+ ...repo,
205
+ issue_number: pr.number,
206
+ body,
207
+ });
208
+ }
209
+ };
@@ -0,0 +1,19 @@
1
+ # One request delivered twice
2
+
3
+ Run `npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs`
4
+ from the parent of `dojo` (adjust paths for your chosen directory).
5
+
6
+ The scripted reference passes. Set `deduplicate = false` in `adapter.mjs`: the
7
+ control still passes, but duplicate delivery credits 1000 cents instead of 500.
8
+ Restore it to verify the fix. This models service idempotency as well as agent inputs.
9
+
10
+ `duplicate_delivery` requires a write tool. The selected call invokes it twice,
11
+ sequentially, with identical cloned arguments and separate state transactions.
12
+ Both attempts execute even if the first returns a structured error. The agent
13
+ receives only the first response after both finish. A thrown/invalid handler or
14
+ cancellation stops execution and preserves partial evidence. This is immediate
15
+ redelivery, not arbitrary queue reordering or concurrent writes.
16
+
17
+ Evaluator-only `delivery` events distinguish executions from the single agent
18
+ `tool_call`. Each committed effect has its own mutation evidence; a deduplicated
19
+ second execution can have no mutation. Never call a real payment service here.
@@ -0,0 +1,40 @@
1
+ // SPDX-License-Identifier: Apache-2.0
2
+ // Copyright 2026 Syed Arman
3
+
4
+ const deduplicate = true;
5
+ export const tools = {
6
+ credit_account: {
7
+ kind: 'write',
8
+ execute(args, state) {
9
+ if (args.operationKey !== 'request_7')
10
+ return {
11
+ ok: false,
12
+ error: {
13
+ code: 'INVALID_INPUT',
14
+ message: 'Expected request_7.',
15
+ retryable: false,
16
+ },
17
+ };
18
+ if (!deduplicate || !state.processed.includes(args.operationKey)) {
19
+ state.creditedCents += 500;
20
+ state.processed.push(args.operationKey);
21
+ }
22
+ return { ok: true, data: { creditedCents: 500 } };
23
+ },
24
+ },
25
+ };
26
+ export function createAgent() {
27
+ return {
28
+ id: 'credit-requester',
29
+ version: deduplicate ? 'safe-1' : 'unsafe-1',
30
+ async run({ call }) {
31
+ const result = await call('credit_account', {
32
+ operationKey: 'request_7',
33
+ });
34
+ return {
35
+ status: result?.ok ? 'completed' : 'blocked',
36
+ message: 'Credit requested.',
37
+ };
38
+ },
39
+ };
40
+ }
@@ -0,0 +1,31 @@
1
+ {
2
+ "schemaVersion": "workflow/1",
3
+ "id": "duplicate-delivery",
4
+ "title": "A queue delivers one credit request twice",
5
+ "task": "Credit 500 cents once using request_7 as the idempotency key.",
6
+ "initialState": { "creditedCents": 0, "processed": [] },
7
+ "checks": [
8
+ {
9
+ "id": "one-credit",
10
+ "label": "Credit exactly 500 cents",
11
+ "type": "equals",
12
+ "path": "/creditedCents",
13
+ "expected": 500
14
+ },
15
+ {
16
+ "id": "completed",
17
+ "label": "Confirm the credit",
18
+ "type": "equals",
19
+ "source": "completion",
20
+ "path": "/status",
21
+ "expected": "completed"
22
+ }
23
+ ],
24
+ "scenarios": [
25
+ {
26
+ "id": "redelivered",
27
+ "title": "One request, two write deliveries",
28
+ "fault": { "type": "duplicate_delivery", "tool": "credit_account" }
29
+ }
30
+ ]
31
+ }
@@ -0,0 +1,17 @@
1
+ # Instructions in tool output
2
+
3
+ Run `npx --no-install tracedojo test --config dojo/workflow.json --adapter dojo/adapter.mjs`
4
+ from the parent of `dojo` (adjust the paths if you chose another directory).
5
+
6
+ The scripted reference passes. In `adapter.mjs`, set `trustToolInstructions` to
7
+ `true`: the control still passes, but the injected instruction triggers an
8
+ unauthorized refund. Both the attempt and committed state fail independent checks.
9
+
10
+ `prompt_injection` replaces the entire `data` field of a successful tool response
11
+ with the configured JSON. Its original response stays evaluator-only; the agent
12
+ receives the replacement. Error responses are not replaced. A scenario without a
13
+ successful targeted response is inconclusive. `occurrence` and `repeat` select calls.
14
+
15
+ Replace this scripted loop with your agent to test its behavior. A passing scripted
16
+ fixture establishes the harness behavior, not a model's resistance to attacks.
17
+ The replacement is untrusted test data; it is never executed by the runner.