@letthembuild/run-receipts 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -0
- package/NOTICE +4 -0
- package/PRODUCERS.md +192 -0
- package/README.md +216 -2
- package/SPEC.md +230 -0
- package/examples/agreed-verified.json +26 -0
- package/examples/not-unanimous-failing.json +38 -0
- package/examples/produce.mjs +157 -0
- package/package.json +51 -4
- package/receipt.schema.json +124 -0
- package/sign.mjs +113 -0
- package/verify.mjs +328 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "{}"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright 2026 GPULogy AI LLC
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|
package/NOTICE
ADDED
package/PRODUCERS.md
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# Writing receipts from your own tool
|
|
2
|
+
|
|
3
|
+
This guide is for anyone whose tool runs a model to do work and other models to review it, and who wants that tool to write a run receipt when a run ends. [SPEC.md](SPEC.md) is the definition; this is the practical route through it.
|
|
4
|
+
|
|
5
|
+
You do not need any code from this repository to write a receipt. A receipt is one JSON file. You do need the verifier to check what you wrote, and you should never write a receipt that does not verify.
|
|
6
|
+
|
|
7
|
+
## The smallest valid receipt
|
|
8
|
+
|
|
9
|
+
One seat did some work, nobody reviewed it and nothing was run:
|
|
10
|
+
|
|
11
|
+
```json
|
|
12
|
+
{
|
|
13
|
+
"receiptVersion": "0.1",
|
|
14
|
+
"producer": { "name": "your-tool" },
|
|
15
|
+
"endedAt": "2026-11-12T09:30:00Z",
|
|
16
|
+
"durationMs": 4200,
|
|
17
|
+
"tokens": 1800,
|
|
18
|
+
"costUsd": 0.01,
|
|
19
|
+
"seats": [
|
|
20
|
+
{ "name": "Builder", "role": "lead", "model": "model-a", "turns": 1, "errors": 0, "timeouts": 0, "durationMs": 4200, "costUsd": 0.01 }
|
|
21
|
+
],
|
|
22
|
+
"verdict": null,
|
|
23
|
+
"checks": [],
|
|
24
|
+
"totals": { "reviewers": 0, "verifiers": 0, "ran": 0, "passed": 0, "failed": 0, "failingNow": 0, "contradicted": 0 },
|
|
25
|
+
"grade": "unverified",
|
|
26
|
+
"gaps": ["Nobody reviewed this run"]
|
|
27
|
+
}
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
It is valid, and its grade is the honest one: `unverified`. Everything below is about earning a better grade truthfully.
|
|
31
|
+
|
|
32
|
+
## What to record while the run is going
|
|
33
|
+
|
|
34
|
+
A receipt is written at the end, from things you can only know during the run. Collect these as they happen:
|
|
35
|
+
|
|
36
|
+
| Record | When | Becomes |
|
|
37
|
+
|---|---|---|
|
|
38
|
+
| Each seat's name, role and model | When the run starts | `seats` |
|
|
39
|
+
| For each seat: answers completed, attempts that failed, how many of those were timeouts, time spent, metered cost | As each attempt ends | `turns`, `errors`, `timeouts`, `durationMs`, `costUsd` |
|
|
40
|
+
| For each reviewer: where it ended | When the run ends | `stance` |
|
|
41
|
+
| Every command a reviewer ran to test the work: who ran it, the command, its exit code, one line of output | As each command exits | `checks` and `totals` |
|
|
42
|
+
| Whether you could see a reviewer's commands at all | When the run starts | `commandsVisible` |
|
|
43
|
+
| How many files the lead changed, if you can measure it | When the run ends | `totals.filesChanged` |
|
|
44
|
+
| What nobody verified | When the run ends | `gaps` |
|
|
45
|
+
|
|
46
|
+
If you cannot observe something, say so in the receipt and do not guess. A reviewer working inside a tool whose commands you cannot see gets `commandsVisible: false`, not a made-up list of checks. An exit code you did not see is `null` with state `unknown`, not `0`.
|
|
47
|
+
|
|
48
|
+
## A worked example
|
|
49
|
+
|
|
50
|
+
[examples/produce.mjs](examples/produce.mjs) is a complete producer, short enough to read in one sitting. It takes a run as a tool might hold it (seats, the commands reviewers ran in order, and the gaps) and returns a receipt. Run it:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
node examples/produce.mjs
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Read it alongside the sections below. It is written to be adapted, in any language; nothing in it is specific to JavaScript except the two helpers it borrows from the verifier, `gradeOf` and `digestOf`, both of which are a few lines you can re-implement from SPEC.md.
|
|
57
|
+
|
|
58
|
+
## Counting: the rules that catch people out
|
|
59
|
+
|
|
60
|
+
Most invalid receipts come from totals that do not match the seats. These are the ones to get right.
|
|
61
|
+
|
|
62
|
+
**Totals count reviewers' commands, not the lead's.** `ran`, `passed` and `failed` are the commands run *to test the work*, by reviewers that answered. The lead running its own tests while building is not a check of the work by somebody else. Rule T5 holds you to this: `ran` is above 0 exactly when at least one reviewer is counted in `verifiers`.
|
|
63
|
+
|
|
64
|
+
**Each run of a command counts.** A command run twice is two in `ran`. If it failed and then passed, that is one in `failed`, one in `passed`, zero in `failingNow` and one in `resolved`. In `checks` it appears once, as it stood at the end.
|
|
65
|
+
|
|
66
|
+
**`totals.reviewers` is the reviewers that answered.** A reviewer whose stance is `unavailable` is a seat, appears in `seats` and in the verdict's holdouts, and is not counted in `totals.reviewers`.
|
|
67
|
+
|
|
68
|
+
**The verdict counts differently depending on its outcome.** When the outcome is `agreed`, `verdict.reviewers` is the reviewers that answered. Otherwise it is all reviewers, including unavailable ones. An absent reviewer cannot block agreement among those who answered, but it also cannot be left out of a count that fell short.
|
|
69
|
+
|
|
70
|
+
**`agreed` has a condition when somebody was missing.** If any reviewer was unavailable, the outcome may be `agreed` only when nothing is failing at the end. You may always record less agreement than the rules allow; you may never record more.
|
|
71
|
+
|
|
72
|
+
**A reviewer that agreed while its own check was failing is `contradicted`.** Its stance stays `agreed`, it is counted in `totals.contradicted`, and that alone makes the grade `failed`.
|
|
73
|
+
|
|
74
|
+
**Do not choose the grade. Work it out.** The grade is the first of these that applies to your totals:
|
|
75
|
+
|
|
76
|
+
| Grade | When |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `failed` | `failingNow` or `contradicted` is above 0 |
|
|
79
|
+
| `verified` | `ran` is above 0 and every reviewer that answered ran something |
|
|
80
|
+
| `partial` | `ran` is above 0 and some reviewer that answered ran nothing |
|
|
81
|
+
| `reviewed` | nothing was run and `filesChanged` is 0 |
|
|
82
|
+
| `not-visible` | nothing was run and every reviewer that answered worked out of view |
|
|
83
|
+
| `unverified` | anything else |
|
|
84
|
+
|
|
85
|
+
Leave `filesChanged` out when you did not measure it. Writing `0` claims you measured and found none, and turns an unverified change into `reviewed`.
|
|
86
|
+
|
|
87
|
+
## Each check
|
|
88
|
+
|
|
89
|
+
| Situation | `exitCode` | `state` | `missingSetup` |
|
|
90
|
+
|---|---|---|---|
|
|
91
|
+
| Ran and exited 0 | `0` | `checked` | `null` |
|
|
92
|
+
| Ran and exited non-zero | the code | `reproduced` | `null` |
|
|
93
|
+
| Could not run for want of something (a service, a credential, a tool) | the code, or `null` | `blocked` | what was missing, in a few words |
|
|
94
|
+
| Ran, and you never saw how it ended | `null` | `unknown` | `null` |
|
|
95
|
+
| Named by a seat and never run | `null` | `untested` | `null` |
|
|
96
|
+
|
|
97
|
+
`where` is one of `local`, `ci`, `cloud` or `other`. It is a category on purpose: never write a machine name there. `deployed` is true only when the command reached the place the work actually runs, such as a request to the live service; a test suite on a laptop is `false`.
|
|
98
|
+
|
|
99
|
+
## What must stay out
|
|
100
|
+
|
|
101
|
+
A tool that runs models holds prompts, replies, file contents, account details and full command output. None of it belongs in a receipt, and the schema rejects unknown fields so that none can be added by accident. Three places still take free text from the run, and they are yours to make safe:
|
|
102
|
+
|
|
103
|
+
- **`command`**: mask credentials (`API_TOKEN=***`), and replace home folders with `~`. At most 300 characters, one line.
|
|
104
|
+
- **`result`**: one line of output, at most 200 characters, masked the same way. Never the whole output.
|
|
105
|
+
- **`gaps` and `purpose`**: short statements you compose. Do not paste a model's reply into them.
|
|
106
|
+
|
|
107
|
+
Also: make `id` opaque. A random identifier is right; your internal conversation or job id, or anything derived from it, links a public file to a private record.
|
|
108
|
+
|
|
109
|
+
The masking in the example is a starting point. You know what your users' secrets look like better than a generic pattern does.
|
|
110
|
+
|
|
111
|
+
## Digest, verify, sign if you can, then write
|
|
112
|
+
|
|
113
|
+
1. Build the whole receipt without `digest`.
|
|
114
|
+
2. Take the digest: SHA-256 over the receipt serialised as JSON with object keys sorted at every level and no whitespace, written as `sha256:` and lower-case hex. Add it as `digest`.
|
|
115
|
+
3. Verify the result. If it does not verify, do not write it: fix the producer.
|
|
116
|
+
4. Sign it, if your tool has a key worth signing with. See "Signing" below.
|
|
117
|
+
|
|
118
|
+
In JavaScript, with the package:
|
|
119
|
+
|
|
120
|
+
```js
|
|
121
|
+
import { verifyReceipt, digestOf } from "@letthembuild/run-receipts";
|
|
122
|
+
|
|
123
|
+
receipt.digest = digestOf(receipt);
|
|
124
|
+
const { valid, errors } = verifyReceipt(receipt);
|
|
125
|
+
if (!valid) throw new Error(errors.join("\n"));
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
In any other language, write the file and check it in your own tests with the command line:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
npx @letthembuild/run-receipts receipt.json
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Number formatting is where digests written in other languages go wrong: the canonical form writes numbers as JavaScript's `JSON.stringify` does, so `0.10` is `0.1` and `2.0` is `2`. Test your digest against the two receipts in [examples/](examples), which carry theirs.
|
|
135
|
+
|
|
136
|
+
## Signing
|
|
137
|
+
|
|
138
|
+
Signing is optional, and an unsigned receipt is a complete receipt. Sign when a reader needs to know that a receipt came from you and was not rewritten on the way.
|
|
139
|
+
|
|
140
|
+
```js
|
|
141
|
+
import { signReceipt } from "@letthembuild/run-receipts/sign";
|
|
142
|
+
|
|
143
|
+
const signed = signReceipt(receipt, signingKey); // { key, secret }, as `sign.mjs keygen` writes it
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
In another language: sign the UTF-8 bytes of `run-receipt-signature-v1`, a line feed, and the digest string, with Ed25519, and add `{ "key": "ed25519:<public key, base64url>", "sig": "<signature, base64url>" }` to `signatures`. Both are unpadded.
|
|
147
|
+
|
|
148
|
+
Before you add signing, decide where the key lives, because that decides what your signature is worth:
|
|
149
|
+
|
|
150
|
+
| Where the key is | What a signature from it shows |
|
|
151
|
+
|---|---|
|
|
152
|
+
| On a build server or service you run, never given out | The receipt came from your service. The strongest claim you can make. |
|
|
153
|
+
| Made on each user's machine, one key per installation | The receipt came from that installation and was not edited since. The user publishes their own key to whoever needs to trust it. |
|
|
154
|
+
| One key built into the application you ship | Only that the receipt came from some copy of your application. Anyone with the application can extract the key, so do not present this as your signature. |
|
|
155
|
+
|
|
156
|
+
Publish your public key somewhere readers already trust, such as your documentation over HTTPS, and say which of the three it is. Never put a name in a receipt to say who signed: the format has no field for it, on purpose.
|
|
157
|
+
|
|
158
|
+
## Where the file goes
|
|
159
|
+
|
|
160
|
+
The format does not say. Two conventions work well:
|
|
161
|
+
|
|
162
|
+
- **Beside the work.** Write it into the project, in a folder of its own such as `receipts/`, named so that files sort by time: `run-receipt-20261112-093000-verified.json`. Whether it is committed is the person's choice; do not commit it for them.
|
|
163
|
+
- **On the pull request.** A receipt committed with a change can be verified and shown to reviewers by the GitHub Action in this repository. See [README.md](README.md#verify-receipts-on-pull-requests).
|
|
164
|
+
|
|
165
|
+
Write a receipt for every run, including the ones that went badly. A tool that only writes receipts for good runs has made the grade meaningless.
|
|
166
|
+
|
|
167
|
+
## When the verifier says no
|
|
168
|
+
|
|
169
|
+
| Rule | Usual cause |
|
|
170
|
+
|---|---|
|
|
171
|
+
| `schema: ... is not part of the format` | You added a field. Unknown fields are not allowed, by design. |
|
|
172
|
+
| S4 | A reviewer has no `stance`, or the lead has one. |
|
|
173
|
+
| S5 | A seat on a flat subscription carries a cost. Its `costUsd` is 0. |
|
|
174
|
+
| V3 | `holdouts` is missing a reviewer that did not agree, or lists one with the wrong stance. |
|
|
175
|
+
| V4, V5 | The verdict was chosen instead of worked out from the stances. |
|
|
176
|
+
| T1 | `totals.reviewers` counts an unavailable reviewer. |
|
|
177
|
+
| T5 | The lead's commands were counted, or commands were counted from a reviewer that is not counted as answering. |
|
|
178
|
+
| G1 | The grade was chosen instead of worked out from the totals. |
|
|
179
|
+
| C2 | A check exited 0 with a state other than `checked`. |
|
|
180
|
+
| C3 | `blocked` without `missingSetup`, or `missingSetup` on a check that is not blocked. |
|
|
181
|
+
| D1 | The receipt was changed after its digest was taken, or the canonical form is not quite right. |
|
|
182
|
+
| K1 | The receipt was changed after it was signed, or the signature was made over something other than the context line and the digest. |
|
|
183
|
+
|
|
184
|
+
## Before you ship
|
|
185
|
+
|
|
186
|
+
- [ ] Every receipt your tool writes is verified first, and an invalid one is never written.
|
|
187
|
+
- [ ] Your tests build a receipt for a run that agreed, one that did not, one with a failing check, one where nothing was run, and one with a reviewer that never answered.
|
|
188
|
+
- [ ] No prompt, reply, file content or account detail can reach a receipt. Test it with a run that contains a recognisable secret and search the receipt for it.
|
|
189
|
+
- [ ] `producer.name` is your tool's own name, and `producer.version` its version.
|
|
190
|
+
- [ ] Receipts are written for bad runs as well as good ones.
|
|
191
|
+
|
|
192
|
+
If something in the format stops you from describing a run truthfully, that is a problem with the format. Please open an issue.
|
package/README.md
CHANGED
|
@@ -1,3 +1,217 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Run receipts
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
A run receipt is a short, machine-readable account of how well an AI run checked its own work.
|
|
4
|
+
|
|
5
|
+
Several models agreeing on an answer is not evidence that the answer is right. What matters is what was actually run: which commands, by whom, how they exited, and what nobody verified. A receipt records exactly that, in one JSON file that a person can read and a tool can check.
|
|
6
|
+
|
|
7
|
+
This repository holds:
|
|
8
|
+
|
|
9
|
+
- **[SPEC.md](SPEC.md)**: the format, version 0.1 (draft), and the rules a valid receipt obeys.
|
|
10
|
+
- **[receipt.schema.json](receipt.schema.json)**: the JSON Schema.
|
|
11
|
+
- **[verify.mjs](verify.mjs)**: a verifier with no dependencies.
|
|
12
|
+
- **[sign.mjs](sign.mjs)**: makes a signing key and signs a receipt with it.
|
|
13
|
+
- **[PRODUCERS.md](PRODUCERS.md)**: how to write receipts from your own tool, with a worked example.
|
|
14
|
+
- **[action.yml](action.yml)**: a GitHub Action that verifies the receipts a pull request carries and reports them.
|
|
15
|
+
- **[examples/](examples)**: two receipts to read and test against.
|
|
16
|
+
|
|
17
|
+
The format is not tied to any one product. Any tool that runs one model to do work and others to review it can write a receipt.
|
|
18
|
+
|
|
19
|
+
Maintained by GPULogy AI LLC, the makers of [LetThemBuild](https://letthembuild.com), an app where several AI models do the work and review each other. Run receipts began as its record of what a run actually checked.
|
|
20
|
+
|
|
21
|
+
## What a receipt looks like
|
|
22
|
+
|
|
23
|
+
```json
|
|
24
|
+
{
|
|
25
|
+
"receiptVersion": "0.1",
|
|
26
|
+
"producer": { "name": "example-producer" },
|
|
27
|
+
"endedAt": "2026-11-10T16:42:07Z",
|
|
28
|
+
"seats": [
|
|
29
|
+
{ "name": "Claude", "role": "lead", "model": "claude-sonnet-5-5", "...": "..." },
|
|
30
|
+
{ "name": "Gemini", "role": "reviewer", "model": "gemini-3-pro", "stance": "agreed", "...": "..." }
|
|
31
|
+
],
|
|
32
|
+
"verdict": { "outcome": "agreed", "agreed": 2, "reviewers": 2, "holdouts": [] },
|
|
33
|
+
"checks": [
|
|
34
|
+
{ "by": "Gemini", "command": "npm test", "exitCode": 0, "where": "local", "deployed": false, "state": "checked", "...": "..." }
|
|
35
|
+
],
|
|
36
|
+
"totals": { "reviewers": 2, "verifiers": 2, "ran": 5, "passed": 4, "failed": 1, "failingNow": 0, "...": "..." },
|
|
37
|
+
"grade": "verified",
|
|
38
|
+
"gaps": ["Behaviour under production load"]
|
|
39
|
+
}
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The full versions are in [examples/](examples). A receipt holds figures, names and commands. It never holds a prompt, a model's reply or a file's contents.
|
|
43
|
+
|
|
44
|
+
## Verify a receipt
|
|
45
|
+
|
|
46
|
+
You need Node 18 or later. Nothing else.
|
|
47
|
+
|
|
48
|
+
With npm, no clone is needed:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
npx @letthembuild/run-receipts receipt.json
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Or from a clone of this repository:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
git clone <this repository> run-receipts
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
node run-receipts/verify.mjs run-receipts/examples/agreed-verified.json
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
```
|
|
65
|
+
VALID run-receipts/examples/agreed-verified.json
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
A receipt that does not hold together is rejected, with the rule it broke:
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
INVALID edited.json
|
|
72
|
+
error G1: grade is failed; the totals earn verified
|
|
73
|
+
error D1: digest does not match the receipt's contents
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
| Command | What it does |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `verify.mjs <file>` | Check a receipt. Exits 0 when valid, 1 when invalid, 2 when the file cannot be read. |
|
|
79
|
+
| `verify.mjs --json <file>` | The same, with the result as JSON. |
|
|
80
|
+
| `verify.mjs --digest <file>` | Print the digest the receipt should carry. |
|
|
81
|
+
| `verify.mjs --keys <list> <file>` | Also say whether each signature is from a key on your list. |
|
|
82
|
+
| `verify.mjs --keys <list> --require-signature <file>` | Not valid unless a key on your list signed it. |
|
|
83
|
+
|
|
84
|
+
The verifier is also a module:
|
|
85
|
+
|
|
86
|
+
```js
|
|
87
|
+
import { verifyReceipt } from "./verify.mjs";
|
|
88
|
+
|
|
89
|
+
const { valid, errors, warnings } = verifyReceipt(receipt);
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## Sign a receipt, and check who signed
|
|
93
|
+
|
|
94
|
+
A digest shows that a receipt was not edited by accident. Anyone who edits one on purpose can retake the digest. A signature closes that: it shows that the holder of a particular key signed exactly this receipt.
|
|
95
|
+
|
|
96
|
+
Make a key once. The file it writes is the signing key; keep it out of your repository. The line it prints is the public key, which is what you give to readers:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
node sign.mjs keygen signing-key.json
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
ed25519:Qm3v...
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Sign a receipt where it is:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
node sign.mjs sign --key signing-key.json --write receipt.json
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
A reader keeps a list of the keys they trust, one to a line, each with their own name for it:
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
ed25519:Qm3v... Release builds
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
node verify.mjs --keys trusted-keys.txt --require-signature receipt.json
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
VALID receipt.json
|
|
124
|
+
signed Release builds (ed25519:Qm3v...)
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
On a build server, put the contents of the key file in a secret and pass it as `RUN_RECEIPT_SIGNING_KEY` instead of `--key`.
|
|
128
|
+
|
|
129
|
+
A receipt carries keys, never names: the name shown is always the reader's own. A signature from a key that is not on your list tells you somebody signed, and nothing about who. And a key is only as good as its keeping. One that lives on a build server supports a strong claim; one shipped inside an app on its users' machines can be extracted by any of them. [SPEC.md](SPEC.md#signatures) says exactly what a signature does and does not mean.
|
|
130
|
+
|
|
131
|
+
## Verify receipts on pull requests
|
|
132
|
+
|
|
133
|
+
The GitHub Action checks every receipt a pull request carries, and puts the grade, the checks that were run and what nobody verified where a reviewer will see them: in a comment on the pull request and in the job summary.
|
|
134
|
+
|
|
135
|
+
```yaml
|
|
136
|
+
name: run receipts
|
|
137
|
+
|
|
138
|
+
on:
|
|
139
|
+
pull_request:
|
|
140
|
+
|
|
141
|
+
permissions:
|
|
142
|
+
contents: read
|
|
143
|
+
pull-requests: write
|
|
144
|
+
|
|
145
|
+
jobs:
|
|
146
|
+
receipts:
|
|
147
|
+
runs-on: ubuntu-latest
|
|
148
|
+
steps:
|
|
149
|
+
- uses: actions/checkout@v4
|
|
150
|
+
- uses: gpulogyai/run-receipts@v0
|
|
151
|
+
with:
|
|
152
|
+
path: receipts
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
The job fails when a receipt does not verify. A receipt that verifies and is graded `failed` is an honest report of a bad result, so by default it is shown and the job passes; name the grades you will not accept to change that.
|
|
156
|
+
|
|
157
|
+
| Input | Default | What it does |
|
|
158
|
+
|---|---|---|
|
|
159
|
+
| `path` | required | Files or folders, one per line. A folder stands for every `.json` file inside it. |
|
|
160
|
+
| `comment` | `true` | Post the report on the pull request. One comment, updated on each run. |
|
|
161
|
+
| `fail-on-grade` | none | Grades that fail the job, separated by commas: `failed`, `unverified`, `not-visible`, `reviewed`, `partial`, `verified`. |
|
|
162
|
+
| `if-no-receipt` | `warn` | What to do when the path holds no receipt: `error`, `warn` or `ignore`. |
|
|
163
|
+
| `trusted-keys` | none | The keys this workflow trusts to sign receipts, one to a line, each followed by your name for it. |
|
|
164
|
+
| `require-signature` | `false` | Fail the job unless every receipt is signed by a trusted key. |
|
|
165
|
+
| `token` | the job's token | The token the comment is posted with. |
|
|
166
|
+
|
|
167
|
+
| Output | What it holds |
|
|
168
|
+
|---|---|
|
|
169
|
+
| `count` | How many receipts were checked. |
|
|
170
|
+
| `valid` | `true` when at least one was checked and every one verified. |
|
|
171
|
+
| `grade` | The grade, when exactly one receipt was checked and it verified. |
|
|
172
|
+
| `grades` | The grades of the receipts that verified, separated by commas. |
|
|
173
|
+
| `signed` | `true` when every receipt verified and was signed by a trusted key. |
|
|
174
|
+
|
|
175
|
+
To accept only receipts signed by keys you trust, list the keys in the workflow or in a repository variable. Do not read them from a file in the pull request: the pull request could change it.
|
|
176
|
+
|
|
177
|
+
```yaml
|
|
178
|
+
- uses: gpulogyai/run-receipts@v0
|
|
179
|
+
with:
|
|
180
|
+
path: receipts
|
|
181
|
+
require-signature: true
|
|
182
|
+
trusted-keys: |
|
|
183
|
+
ed25519:Qm3v... Release builds
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Three things to know:
|
|
187
|
+
|
|
188
|
+
- **Pull requests from forks.** Their token cannot write, so the comment is skipped with a warning and the report is in the job summary. Keep the trigger as `pull_request`. Do not switch to `pull_request_target` to get the comment: that runs with your repository's permissions on a stranger's files.
|
|
189
|
+
- **A receipt's text is treated as untrusted.** Names, commands and results are shown as plain text: they cannot add formatting, links or mentions to the comment, and they cannot issue commands to the runner.
|
|
190
|
+
- **Requirements.** Node 18 or later on the runner, which GitHub's hosted runners have. The action installs nothing and makes no network call other than posting its comment.
|
|
191
|
+
|
|
192
|
+
## What verifying proves, and what it does not
|
|
193
|
+
|
|
194
|
+
A valid receipt is well-formed, consistent with itself, and unchanged since its digest was taken. Its grade follows from its totals, its verdict follows from where each reviewer ended, and each check's state matches its exit code. When it is signed, verifying also says which keys signed it and whether they are keys you listed.
|
|
195
|
+
|
|
196
|
+
Verifying does **not** prove that the commands were really run or that they exited as stated. A receipt is the producer's own account. A signature from a key you trust tells you whose account it is, not that the account is true.
|
|
197
|
+
|
|
198
|
+
## Roadmap
|
|
199
|
+
|
|
200
|
+
Not yet:
|
|
201
|
+
|
|
202
|
+
- **A published list of well-known producers' keys**, so a reader does not have to collect them by hand.
|
|
203
|
+
- **Signing by identity instead of a kept key**, for build servers that can prove who they are.
|
|
204
|
+
|
|
205
|
+
## Run the tests
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
npm test
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
## Contributing
|
|
212
|
+
|
|
213
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md). Security reports go to the address in [SECURITY.md](SECURITY.md), not to public issues.
|
|
214
|
+
|
|
215
|
+
## Licence
|
|
216
|
+
|
|
217
|
+
Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). The licence covers the code and the specification; it does not grant use of any names or logos, as [TRADEMARKS.md](TRADEMARKS.md) explains.
|