@letthembuild/run-receipts 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/PRODUCERS.md +192 -0
- package/README.md +112 -6
- package/SPEC.md +34 -3
- package/examples/produce.mjs +157 -0
- package/package.json +8 -3
- package/receipt.schema.json +11 -1
- package/sign.mjs +113 -0
- package/verify.mjs +112 -13
package/PRODUCERS.md
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# Writing receipts from your own tool
|
|
2
|
+
|
|
3
|
+
This guide is for anyone whose tool runs a model to do work and other models to review it, and who wants that tool to write a run receipt when a run ends. [SPEC.md](SPEC.md) is the definition; this is the practical route through it.
|
|
4
|
+
|
|
5
|
+
You do not need any code from this repository to write a receipt. A receipt is one JSON file. You do need the verifier to check what you wrote, and you should never write a receipt that does not verify.
|
|
6
|
+
|
|
7
|
+
## The smallest valid receipt
|
|
8
|
+
|
|
9
|
+
One seat did some work, nobody reviewed it and nothing was run:
|
|
10
|
+
|
|
11
|
+
```json
|
|
12
|
+
{
|
|
13
|
+
"receiptVersion": "0.1",
|
|
14
|
+
"producer": { "name": "your-tool" },
|
|
15
|
+
"endedAt": "2026-11-12T09:30:00Z",
|
|
16
|
+
"durationMs": 4200,
|
|
17
|
+
"tokens": 1800,
|
|
18
|
+
"costUsd": 0.01,
|
|
19
|
+
"seats": [
|
|
20
|
+
{ "name": "Builder", "role": "lead", "model": "model-a", "turns": 1, "errors": 0, "timeouts": 0, "durationMs": 4200, "costUsd": 0.01 }
|
|
21
|
+
],
|
|
22
|
+
"verdict": null,
|
|
23
|
+
"checks": [],
|
|
24
|
+
"totals": { "reviewers": 0, "verifiers": 0, "ran": 0, "passed": 0, "failed": 0, "failingNow": 0, "contradicted": 0 },
|
|
25
|
+
"grade": "unverified",
|
|
26
|
+
"gaps": ["Nobody reviewed this run"]
|
|
27
|
+
}
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
It is valid, and its grade is the honest one: `unverified`. Everything below is about earning a better grade truthfully.
|
|
31
|
+
|
|
32
|
+
## What to record while the run is going
|
|
33
|
+
|
|
34
|
+
A receipt is written at the end, from things you can only know during the run. Collect these as they happen:
|
|
35
|
+
|
|
36
|
+
| Record | When | Becomes |
|
|
37
|
+
|---|---|---|
|
|
38
|
+
| Each seat's name, role and model | When the run starts | `seats` |
|
|
39
|
+
| For each seat: answers completed, attempts that failed, how many of those were timeouts, time spent, metered cost | As each attempt ends | `turns`, `errors`, `timeouts`, `durationMs`, `costUsd` |
|
|
40
|
+
| For each reviewer: where it ended | When the run ends | `stance` |
|
|
41
|
+
| Every command a reviewer ran to test the work: who ran it, the command, its exit code, one line of output | As each command exits | `checks` and `totals` |
|
|
42
|
+
| Whether you could see a reviewer's commands at all | When the run starts | `commandsVisible` |
|
|
43
|
+
| How many files the lead changed, if you can measure it | When the run ends | `totals.filesChanged` |
|
|
44
|
+
| What nobody verified | When the run ends | `gaps` |
|
|
45
|
+
|
|
46
|
+
If you cannot observe something, say so in the receipt and do not guess. A reviewer working inside a tool whose commands you cannot see gets `commandsVisible: false`, not a made-up list of checks. An exit code you did not see is `null` with state `unknown`, not `0`.
|
|
47
|
+
|
|
48
|
+
## A worked example
|
|
49
|
+
|
|
50
|
+
[examples/produce.mjs](examples/produce.mjs) is a complete producer, short enough to read in one sitting. It takes a run as a tool might hold it (seats, the commands reviewers ran in order, and the gaps) and returns a receipt. Run it:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
node examples/produce.mjs
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Read it alongside the sections below. It is written to be adapted, in any language; nothing in it is specific to JavaScript except the two helpers it borrows from the verifier, `gradeOf` and `digestOf`, both of which are a few lines you can re-implement from SPEC.md.
|
|
57
|
+
|
|
58
|
+
## Counting: the rules that catch people out
|
|
59
|
+
|
|
60
|
+
Most invalid receipts come from totals that do not match the seats. These are the ones to get right.
|
|
61
|
+
|
|
62
|
+
**Totals count reviewers' commands, not the lead's.** `ran`, `passed` and `failed` are the commands run *to test the work*, by reviewers that answered. The lead running its own tests while building is not a check of the work by somebody else. Rule T5 holds you to this: `ran` is above 0 exactly when at least one reviewer is counted in `verifiers`.
|
|
63
|
+
|
|
64
|
+
**Each run of a command counts.** A command run twice is two in `ran`. If it failed and then passed, that is one in `failed`, one in `passed`, zero in `failingNow` and one in `resolved`. In `checks` it appears once, as it stood at the end.
|
|
65
|
+
|
|
66
|
+
**`totals.reviewers` is the reviewers that answered.** A reviewer whose stance is `unavailable` is a seat, appears in `seats` and in the verdict's holdouts, and is not counted in `totals.reviewers`.
|
|
67
|
+
|
|
68
|
+
**The verdict counts differently depending on its outcome.** When the outcome is `agreed`, `verdict.reviewers` is the reviewers that answered. Otherwise it is all reviewers, including unavailable ones. An absent reviewer cannot block agreement among those who answered, but it also cannot be left out of a count that fell short.
|
|
69
|
+
|
|
70
|
+
**`agreed` has a condition when somebody was missing.** If any reviewer was unavailable, the outcome may be `agreed` only when nothing is failing at the end. You may always record less agreement than the rules allow; you may never record more.
|
|
71
|
+
|
|
72
|
+
**A reviewer that agreed while its own check was failing is `contradicted`.** Its stance stays `agreed`, it is counted in `totals.contradicted`, and that alone makes the grade `failed`.
|
|
73
|
+
|
|
74
|
+
**Do not choose the grade. Work it out.** The grade is the first of these that applies to your totals:
|
|
75
|
+
|
|
76
|
+
| Grade | When |
|
|
77
|
+
|---|---|
|
|
78
|
+
| `failed` | `failingNow` or `contradicted` is above 0 |
|
|
79
|
+
| `verified` | `ran` is above 0 and every reviewer that answered ran something |
|
|
80
|
+
| `partial` | `ran` is above 0 and some reviewer that answered ran nothing |
|
|
81
|
+
| `reviewed` | nothing was run and `filesChanged` is 0 |
|
|
82
|
+
| `not-visible` | nothing was run and every reviewer that answered worked out of view |
|
|
83
|
+
| `unverified` | anything else |
|
|
84
|
+
|
|
85
|
+
Leave `filesChanged` out when you did not measure it. Writing `0` claims you measured and found none, and turns an unverified change into `reviewed`.
|
|
86
|
+
|
|
87
|
+
## Each check
|
|
88
|
+
|
|
89
|
+
| Situation | `exitCode` | `state` | `missingSetup` |
|
|
90
|
+
|---|---|---|---|
|
|
91
|
+
| Ran and exited 0 | `0` | `checked` | `null` |
|
|
92
|
+
| Ran and exited non-zero | the code | `reproduced` | `null` |
|
|
93
|
+
| Could not run for want of something (a service, a credential, a tool) | the code, or `null` | `blocked` | what was missing, in a few words |
|
|
94
|
+
| Ran, and you never saw how it ended | `null` | `unknown` | `null` |
|
|
95
|
+
| Named by a seat and never run | `null` | `untested` | `null` |
|
|
96
|
+
|
|
97
|
+
`where` is one of `local`, `ci`, `cloud` or `other`. It is a category on purpose: never write a machine name there. `deployed` is true only when the command reached the place the work actually runs, such as a request to the live service; a test suite on a laptop is `false`.
|
|
98
|
+
|
|
99
|
+
## What must stay out
|
|
100
|
+
|
|
101
|
+
A tool that runs models holds prompts, replies, file contents, account details and full command output. None of it belongs in a receipt, and the schema rejects unknown fields so that none can be added by accident. Three places still take free text from the run, and they are yours to make safe:
|
|
102
|
+
|
|
103
|
+
- **`command`**: mask credentials (`API_TOKEN=***`), and replace home folders with `~`. At most 300 characters, one line.
|
|
104
|
+
- **`result`**: one line of output, at most 200 characters, masked the same way. Never the whole output.
|
|
105
|
+
- **`gaps` and `purpose`**: short statements you compose. Do not paste a model's reply into them.
|
|
106
|
+
|
|
107
|
+
Also: make `id` opaque. A random identifier is right; your internal conversation or job id, or anything derived from it, links a public file to a private record.
|
|
108
|
+
|
|
109
|
+
The masking in the example is a starting point. You know what your users' secrets look like better than a generic pattern does.
|
|
110
|
+
|
|
111
|
+
## Digest, verify, sign if you can, then write
|
|
112
|
+
|
|
113
|
+
1. Build the whole receipt without `digest`.
|
|
114
|
+
2. Take the digest: SHA-256 over the receipt serialised as JSON with object keys sorted at every level and no whitespace, written as `sha256:` and lower-case hex. Add it as `digest`.
|
|
115
|
+
3. Verify the result. If it does not verify, do not write it: fix the producer.
|
|
116
|
+
4. Sign it, if your tool has a key worth signing with. See "Signing" below.
|
|
117
|
+
|
|
118
|
+
In JavaScript, with the package:
|
|
119
|
+
|
|
120
|
+
```js
|
|
121
|
+
import { verifyReceipt, digestOf } from "@letthembuild/run-receipts";
|
|
122
|
+
|
|
123
|
+
receipt.digest = digestOf(receipt);
|
|
124
|
+
const { valid, errors } = verifyReceipt(receipt);
|
|
125
|
+
if (!valid) throw new Error(errors.join("\n"));
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
In any other language, write the file and check it in your own tests with the command line:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
npx @letthembuild/run-receipts receipt.json
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Number formatting is where digests written in other languages go wrong: the canonical form writes numbers as JavaScript's `JSON.stringify` does, so `0.10` is `0.1` and `2.0` is `2`. Test your digest against the two receipts in [examples/](examples), which carry theirs.
|
|
135
|
+
|
|
136
|
+
## Signing
|
|
137
|
+
|
|
138
|
+
Signing is optional, and an unsigned receipt is a complete receipt. Sign when a reader needs to know that a receipt came from you and was not rewritten on the way.
|
|
139
|
+
|
|
140
|
+
```js
|
|
141
|
+
import { signReceipt } from "@letthembuild/run-receipts/sign";
|
|
142
|
+
|
|
143
|
+
const signed = signReceipt(receipt, signingKey); // { key, secret }, as `sign.mjs keygen` writes it
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
In another language: sign the UTF-8 bytes of `run-receipt-signature-v1`, a line feed, and the digest string, with Ed25519, and add `{ "key": "ed25519:<public key, base64url>", "sig": "<signature, base64url>" }` to `signatures`. Both are unpadded.
|
|
147
|
+
|
|
148
|
+
Before you add signing, decide where the key lives, because that decides what your signature is worth:
|
|
149
|
+
|
|
150
|
+
| Where the key is | What a signature from it shows |
|
|
151
|
+
|---|---|
|
|
152
|
+
| On a build server or service you run, never given out | The receipt came from your service. The strongest claim you can make. |
|
|
153
|
+
| Made on each user's machine, one key per installation | The receipt came from that installation and was not edited since. The user publishes their own key to whoever needs to trust it. |
|
|
154
|
+
| One key built into the application you ship | Only that the receipt came from some copy of your application. Anyone with the application can extract the key, so do not present this as your signature. |
|
|
155
|
+
|
|
156
|
+
Publish your public key somewhere readers already trust, such as your documentation over HTTPS, and say which of the three it is. Never put a name in a receipt to say who signed: the format has no field for it, on purpose.
|
|
157
|
+
|
|
158
|
+
## Where the file goes
|
|
159
|
+
|
|
160
|
+
The format does not say. Two conventions work well:
|
|
161
|
+
|
|
162
|
+
- **Beside the work.** Write it into the project, in a folder of its own such as `receipts/`, named so that files sort by time: `run-receipt-20261112-093000-verified.json`. Whether it is committed is the person's choice; do not commit it for them.
|
|
163
|
+
- **On the pull request.** A receipt committed with a change can be verified and shown to reviewers by the GitHub Action in this repository. See [README.md](README.md#verify-receipts-on-pull-requests).
|
|
164
|
+
|
|
165
|
+
Write a receipt for every run, including the ones that went badly. A tool that only writes receipts for good runs has made the grade meaningless.
|
|
166
|
+
|
|
167
|
+
## When the verifier says no
|
|
168
|
+
|
|
169
|
+
| Rule | Usual cause |
|
|
170
|
+
|---|---|
|
|
171
|
+
| `schema: ... is not part of the format` | You added a field. Unknown fields are not allowed, by design. |
|
|
172
|
+
| S4 | A reviewer has no `stance`, or the lead has one. |
|
|
173
|
+
| S5 | A seat on a flat subscription carries a cost. Its `costUsd` is 0. |
|
|
174
|
+
| V3 | `holdouts` is missing a reviewer that did not agree, or lists one with the wrong stance. |
|
|
175
|
+
| V4, V5 | The verdict was chosen instead of worked out from the stances. |
|
|
176
|
+
| T1 | `totals.reviewers` counts an unavailable reviewer. |
|
|
177
|
+
| T5 | The lead's commands were counted, or commands were counted from a reviewer that is not counted as answering. |
|
|
178
|
+
| G1 | The grade was chosen instead of worked out from the totals. |
|
|
179
|
+
| C2 | A check exited 0 with a state other than `checked`. |
|
|
180
|
+
| C3 | `blocked` without `missingSetup`, or `missingSetup` on a check that is not blocked. |
|
|
181
|
+
| D1 | The receipt was changed after its digest was taken, or the canonical form is not quite right. |
|
|
182
|
+
| K1 | The receipt was changed after it was signed, or the signature was made over something other than the context line and the digest. |
|
|
183
|
+
|
|
184
|
+
## Before you ship
|
|
185
|
+
|
|
186
|
+
- [ ] Every receipt your tool writes is verified first, and an invalid one is never written.
|
|
187
|
+
- [ ] Your tests build a receipt for a run that agreed, one that did not, one with a failing check, one where nothing was run, and one with a reviewer that never answered.
|
|
188
|
+
- [ ] No prompt, reply, file content or account detail can reach a receipt. Test it with a run that contains a recognisable secret and search the receipt for it.
|
|
189
|
+
- [ ] `producer.name` is your tool's own name, and `producer.version` its version.
|
|
190
|
+
- [ ] Receipts are written for bad runs as well as good ones.
|
|
191
|
+
|
|
192
|
+
If something in the format stops you from describing a run truthfully, that is a problem with the format. Please open an issue.
|
package/README.md
CHANGED
|
@@ -9,10 +9,15 @@ This repository holds:
|
|
|
9
9
|
- **[SPEC.md](SPEC.md)**: the format, version 0.1 (draft), and the rules a valid receipt obeys.
|
|
10
10
|
- **[receipt.schema.json](receipt.schema.json)**: the JSON Schema.
|
|
11
11
|
- **[verify.mjs](verify.mjs)**: a verifier with no dependencies.
|
|
12
|
+
- **[sign.mjs](sign.mjs)**: makes a signing key and signs a receipt with it.
|
|
13
|
+
- **[PRODUCERS.md](PRODUCERS.md)**: how to write receipts from your own tool, with a worked example.
|
|
14
|
+
- **[action.yml](action.yml)**: a GitHub Action that verifies the receipts a pull request carries and reports them.
|
|
12
15
|
- **[examples/](examples)**: two receipts to read and test against.
|
|
13
16
|
|
|
14
17
|
The format is not tied to any one product. Any tool that runs one model to do work and others to review it can write a receipt.
|
|
15
18
|
|
|
19
|
+
Maintained by GPULogy AI LLC, the makers of [LetThemBuild](https://letthembuild.com), an app where several AI models do the work and review each other. Run receipts began as its record of what a run actually checked.
|
|
20
|
+
|
|
16
21
|
## What a receipt looks like
|
|
17
22
|
|
|
18
23
|
```json
|
|
@@ -73,6 +78,8 @@ INVALID edited.json
|
|
|
73
78
|
| `verify.mjs <file>` | Check a receipt. Exits 0 when valid, 1 when invalid, 2 when the file cannot be read. |
|
|
74
79
|
| `verify.mjs --json <file>` | The same, with the result as JSON. |
|
|
75
80
|
| `verify.mjs --digest <file>` | Print the digest the receipt should carry. |
|
|
81
|
+
| `verify.mjs --keys <list> <file>` | Also say whether each signature is from a key on your list. |
|
|
82
|
+
| `verify.mjs --keys <list> --require-signature <file>` | Not valid unless a key on your list signed it. |
|
|
76
83
|
|
|
77
84
|
The verifier is also a module:
|
|
78
85
|
|
|
@@ -82,19 +89,118 @@ import { verifyReceipt } from "./verify.mjs";
|
|
|
82
89
|
const { valid, errors, warnings } = verifyReceipt(receipt);
|
|
83
90
|
```
|
|
84
91
|
|
|
92
|
+
## Sign a receipt, and check who signed
|
|
93
|
+
|
|
94
|
+
A digest shows that a receipt was not edited by accident. Anyone who edits one on purpose can retake the digest. A signature closes that: it shows that the holder of a particular key signed exactly this receipt.
|
|
95
|
+
|
|
96
|
+
Make a key once. The file it writes is the signing key; keep it out of your repository. The line it prints is the public key, which is what you give to readers:
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
node sign.mjs keygen signing-key.json
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
```
|
|
103
|
+
ed25519:Qm3v...
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Sign a receipt where it is:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
node sign.mjs sign --key signing-key.json --write receipt.json
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
A reader keeps a list of the keys they trust, one to a line, each with their own name for it:
|
|
113
|
+
|
|
114
|
+
```
|
|
115
|
+
ed25519:Qm3v... Release builds
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
node verify.mjs --keys trusted-keys.txt --require-signature receipt.json
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
```
|
|
123
|
+
VALID receipt.json
|
|
124
|
+
signed Release builds (ed25519:Qm3v...)
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
On a build server, put the contents of the key file in a secret and pass it as `RUN_RECEIPT_SIGNING_KEY` instead of `--key`.
|
|
128
|
+
|
|
129
|
+
A receipt carries keys, never names: the name shown is always the reader's own. A signature from a key that is not on your list tells you somebody signed, and nothing about who. And a key is only as good as its keeping. One that lives on a build server supports a strong claim; one shipped inside an app on its users' machines can be extracted by any of them. [SPEC.md](SPEC.md#signatures) says exactly what a signature does and does not mean.
|
|
130
|
+
|
|
131
|
+
## Verify receipts on pull requests
|
|
132
|
+
|
|
133
|
+
The GitHub Action checks every receipt a pull request carries, and puts the grade, the checks that were run and what nobody verified where a reviewer will see them: in a comment on the pull request and in the job summary.
|
|
134
|
+
|
|
135
|
+
```yaml
|
|
136
|
+
name: run receipts
|
|
137
|
+
|
|
138
|
+
on:
|
|
139
|
+
pull_request:
|
|
140
|
+
|
|
141
|
+
permissions:
|
|
142
|
+
contents: read
|
|
143
|
+
pull-requests: write
|
|
144
|
+
|
|
145
|
+
jobs:
|
|
146
|
+
receipts:
|
|
147
|
+
runs-on: ubuntu-latest
|
|
148
|
+
steps:
|
|
149
|
+
- uses: actions/checkout@v4
|
|
150
|
+
- uses: gpulogyai/run-receipts@v0
|
|
151
|
+
with:
|
|
152
|
+
path: receipts
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
The job fails when a receipt does not verify. A receipt that verifies and is graded `failed` is an honest report of a bad result, so by default it is shown and the job passes; name the grades you will not accept to change that.
|
|
156
|
+
|
|
157
|
+
| Input | Default | What it does |
|
|
158
|
+
|---|---|---|
|
|
159
|
+
| `path` | required | Files or folders, one per line. A folder stands for every `.json` file inside it. |
|
|
160
|
+
| `comment` | `true` | Post the report on the pull request. One comment, updated on each run. |
|
|
161
|
+
| `fail-on-grade` | none | Grades that fail the job, separated by commas: `failed`, `unverified`, `not-visible`, `reviewed`, `partial`, `verified`. |
|
|
162
|
+
| `if-no-receipt` | `warn` | What to do when the path holds no receipt: `error`, `warn` or `ignore`. |
|
|
163
|
+
| `trusted-keys` | none | The keys this workflow trusts to sign receipts, one to a line, each followed by your name for it. |
|
|
164
|
+
| `require-signature` | `false` | Fail the job unless every receipt is signed by a trusted key. |
|
|
165
|
+
| `token` | the job's token | The token the comment is posted with. |
|
|
166
|
+
|
|
167
|
+
| Output | What it holds |
|
|
168
|
+
|---|---|
|
|
169
|
+
| `count` | How many receipts were checked. |
|
|
170
|
+
| `valid` | `true` when at least one was checked and every one verified. |
|
|
171
|
+
| `grade` | The grade, when exactly one receipt was checked and it verified. |
|
|
172
|
+
| `grades` | The grades of the receipts that verified, separated by commas. |
|
|
173
|
+
| `signed` | `true` when every receipt verified and was signed by a trusted key. |
|
|
174
|
+
|
|
175
|
+
To accept only receipts signed by keys you trust, list the keys in the workflow or in a repository variable. Do not read them from a file in the pull request: the pull request could change it.
|
|
176
|
+
|
|
177
|
+
```yaml
|
|
178
|
+
- uses: gpulogyai/run-receipts@v0
|
|
179
|
+
with:
|
|
180
|
+
path: receipts
|
|
181
|
+
require-signature: true
|
|
182
|
+
trusted-keys: |
|
|
183
|
+
ed25519:Qm3v... Release builds
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
Three things to know:
|
|
187
|
+
|
|
188
|
+
- **Pull requests from forks.** Their token cannot write, so the comment is skipped with a warning and the report is in the job summary. Keep the trigger as `pull_request`. Do not switch to `pull_request_target` to get the comment: that runs with your repository's permissions on a stranger's files.
|
|
189
|
+
- **A receipt's text is treated as untrusted.** Names, commands and results are shown as plain text: they cannot add formatting, links or mentions to the comment, and they cannot issue commands to the runner.
|
|
190
|
+
- **Requirements.** Node 18 or later on the runner, which GitHub's hosted runners have. The action installs nothing and makes no network call other than posting its comment.
|
|
191
|
+
|
|
85
192
|
## What verifying proves, and what it does not
|
|
86
193
|
|
|
87
|
-
A valid receipt is well-formed, consistent with itself, and unchanged since its digest was taken. Its grade follows from its totals, its verdict follows from where each reviewer ended, and each check's state matches its exit code.
|
|
194
|
+
A valid receipt is well-formed, consistent with itself, and unchanged since its digest was taken. Its grade follows from its totals, its verdict follows from where each reviewer ended, and each check's state matches its exit code. When it is signed, verifying also says which keys signed it and whether they are keys you listed.
|
|
88
195
|
|
|
89
|
-
Verifying does **not** prove that the commands were really run or that they exited as stated. A receipt is the producer's own account.
|
|
196
|
+
Verifying does **not** prove that the commands were really run or that they exited as stated. A receipt is the producer's own account. A signature from a key you trust tells you whose account it is, not that the account is true.
|
|
90
197
|
|
|
91
198
|
## Roadmap
|
|
92
199
|
|
|
93
|
-
Not
|
|
200
|
+
Not yet:
|
|
94
201
|
|
|
95
|
-
- **
|
|
96
|
-
- **
|
|
97
|
-
- **A guide for producers**: how to write a receipt from your own tool.
|
|
202
|
+
- **A published list of well-known producers' keys**, so a reader does not have to collect them by hand.
|
|
203
|
+
- **Signing by identity instead of a kept key**, for build servers that can prove who they are.
|
|
98
204
|
|
|
99
205
|
## Run the tests
|
|
100
206
|
|
package/SPEC.md
CHANGED
|
@@ -8,7 +8,7 @@ Several models agreeing on an answer is not evidence that the answer is right. A
|
|
|
8
8
|
|
|
9
9
|
A receipt holds figures, names and commands. It never holds a prompt, a model's reply, a file's contents or a command's full output.
|
|
10
10
|
|
|
11
|
-
A receipt is the producer's account of the run. A verifier can prove that the account is well-formed, consistent with itself and unchanged since its digest was taken. It cannot prove the run happened as described. See "What a verifier cannot tell you".
|
|
11
|
+
A receipt is the producer's account of the run. A verifier can prove that the account is well-formed, consistent with itself and unchanged since its digest was taken, and, when it is signed, which keys put their name to it. It cannot prove the run happened as described. See "What a verifier cannot tell you".
|
|
12
12
|
|
|
13
13
|
## Vocabulary
|
|
14
14
|
|
|
@@ -43,6 +43,7 @@ A receipt is one JSON object, UTF-8. `receipt.schema.json` is the normative shap
|
|
|
43
43
|
| `grade` | yes | One word for how well the result was checked. |
|
|
44
44
|
| `gaps` | yes | What was not verified. May be empty, and is listed even when everyone agreed. |
|
|
45
45
|
| `digest` | no | `sha256:` and 64 hex characters over the canonical form. |
|
|
46
|
+
| `signatures` | no | One to eight signatures over the digest. See "Signatures". |
|
|
46
47
|
|
|
47
48
|
### Seat
|
|
48
49
|
|
|
@@ -166,21 +167,51 @@ A producer may record `majority` or `none` where V4 would allow `agreed`: a run
|
|
|
166
167
|
- D1. If `digest` is present it equals the digest of the receipt.
|
|
167
168
|
- D2 (warning). No digest is present.
|
|
168
169
|
|
|
170
|
+
**Signatures**
|
|
171
|
+
- K1. Every signature is its key's signature over this receipt.
|
|
172
|
+
- K2. No key signs twice.
|
|
173
|
+
- K3. Only when the reader requires a signature: at least one signature is from a key the reader trusts.
|
|
174
|
+
|
|
169
175
|
Warnings do not make a receipt invalid.
|
|
170
176
|
|
|
171
177
|
## Digest
|
|
172
178
|
|
|
173
|
-
The digest is SHA-256 over the receipt's canonical form, written `sha256:` followed by lower-case hex. The canonical form is the receipt without its top-level `digest`
|
|
179
|
+
The digest is SHA-256 over the receipt's canonical form, written `sha256:` followed by lower-case hex. The canonical form is the receipt without its top-level `digest` and `signatures` fields, serialised as JSON with object keys sorted by code unit at every level, no whitespace, and numbers and strings as `JSON.stringify` writes them, encoded as UTF-8.
|
|
174
180
|
|
|
175
181
|
A digest lets a copy of a receipt, such as one quoted in a pull-request comment, be matched against the file it came from. It is not a signature: anyone who edits a receipt can retake its digest.
|
|
176
182
|
|
|
183
|
+
## Signatures
|
|
184
|
+
|
|
185
|
+
A signature lets a reader tell who put their name to a receipt. Signing is optional. Signatures were added to this draft after its first release; a verifier older than 0.2.0 rejects a signed receipt as carrying an unknown field.
|
|
186
|
+
|
|
187
|
+
Each entry in `signatures` has two fields and no others:
|
|
188
|
+
|
|
189
|
+
| Field | Meaning |
|
|
190
|
+
|---|---|
|
|
191
|
+
| `key` | The signer's Ed25519 public key: `ed25519:` followed by its 32 bytes in unpadded base64url. |
|
|
192
|
+
| `sig` | The signature's 64 bytes in unpadded base64url. |
|
|
193
|
+
|
|
194
|
+
What is signed is the UTF-8 bytes of the line `run-receipt-signature-v1`, a line feed, and the receipt's digest as written (`sha256:` and the hex). The digest signed is the one worked out from the receipt, not the one it states, so a signature holds only over the receipt as it stands. The leading line keeps a signature made for a receipt from being valid for anything else the same key signs.
|
|
195
|
+
|
|
196
|
+
Because `signatures` is left out of the canonical form, signing does not change a receipt's digest, and a second party can add its own signature without disturbing the first.
|
|
197
|
+
|
|
198
|
+
### What a signature means
|
|
199
|
+
|
|
200
|
+
A signature that holds says that the holder of that key signed exactly this receipt. That is all it says. Three things follow.
|
|
201
|
+
|
|
202
|
+
- **A key is not a name.** A receipt carries keys and never says whose they are: a signature entry has no room for a name, so a receipt cannot introduce its own signer. The reader decides which keys to trust and what to call them, in a list the reader keeps. A signature from a key the reader has not listed proves that somebody signed, and nothing about who.
|
|
203
|
+
- **A signature is worth what the key's keeping is worth.** A key held by a build server and never shown to anyone supports a strong claim. A key shipped inside an application that runs on its users' machines can be taken out by any of them, and proves only that the receipt came from a copy of that application. Say which kind your key is when you publish it.
|
|
204
|
+
- **A signature does not make a receipt true.** It moves the question from "was this file edited?" to "do I believe this signer's account?". The signer can still be wrong, or lie.
|
|
205
|
+
|
|
206
|
+
A reader that requires a signature (rule K3) must name the keys it trusts. Requiring a signature from any key at all is meaningless, since anyone can make a key.
|
|
207
|
+
|
|
177
208
|
## What a verifier cannot tell you
|
|
178
209
|
|
|
179
210
|
- That the commands were really run, or exited as stated.
|
|
180
211
|
- That the seats were the models named.
|
|
181
212
|
- That the receipt belongs to the change it is attached to, beyond the `revision` it names.
|
|
182
213
|
|
|
183
|
-
|
|
214
|
+
A signature from a key you trust narrows the first two to how far you trust the signer. Nothing in a receipt closes them outright.
|
|
184
215
|
|
|
185
216
|
## What a receipt must not carry
|
|
186
217
|
|
|
@@ -0,0 +1,157 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Copyright 2026 GPULogy AI LLC
|
|
3
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
4
|
+
//
|
|
5
|
+
// An example producer: the least a tool needs to turn what it saw during a
|
|
6
|
+
// run into a valid receipt. PRODUCERS.md walks through it. It is an example
|
|
7
|
+
// to read and adapt, not a library; the rules it follows are SPEC.md's.
|
|
8
|
+
//
|
|
9
|
+
// node examples/produce.mjs print a receipt for the sample run below
|
|
10
|
+
|
|
11
|
+
import { realpathSync } from "node:fs";
|
|
12
|
+
import { fileURLToPath } from "node:url";
|
|
13
|
+
import { gradeOf, digestOf, verifyReceipt } from "../verify.mjs";
|
|
14
|
+
|
|
15
|
+
/** One line, cut to length: `command` and `result` never carry more. */
|
|
16
|
+
const line = (s, max) => Array.from(String(s ?? "").replace(/\s+/g, " ").trim()).slice(0, max).join("");
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* A starting point for masking, and no more than that. A real producer knows
|
|
20
|
+
* what its own secrets look like and must mask those too.
|
|
21
|
+
*/
|
|
22
|
+
export const mask = (s) => String(s ?? "")
|
|
23
|
+
.replace(/\b([A-Z][A-Z0-9_]*(?:KEY|TOKEN|SECRET|PASSWORD)[A-Z0-9_]*)=\S+/g, "$1=***")
|
|
24
|
+
.replace(/\b(Bearer|Basic)\s+[A-Za-z0-9._~+/=-]{8,}/g, "$1 ***")
|
|
25
|
+
.replace(/(?:\/Users|\/home)\/[^/\s]+/g, "~");
|
|
26
|
+
|
|
27
|
+
/**
|
|
28
|
+
* Build a receipt from a run.
|
|
29
|
+
*
|
|
30
|
+
* run.seats every seat, the lead and the reviewers, with its figures and,
|
|
31
|
+
* for a reviewer, where it ended (stance)
|
|
32
|
+
* run.commands every command a reviewer ran to test the work, in the order
|
|
33
|
+
* they were run: { by, command, exitCode, purpose?, result?,
|
|
34
|
+
* where?, deployed?, missingSetup? }
|
|
35
|
+
* run.gaps what nobody verified, in words
|
|
36
|
+
*/
|
|
37
|
+
export function buildReceipt(run) {
|
|
38
|
+
const reviewers = run.seats.filter((s) => s.role === "reviewer");
|
|
39
|
+
const answered = reviewers.filter((s) => s.stance !== "unavailable");
|
|
40
|
+
const agreed = reviewers.filter((s) => s.stance === "agreed");
|
|
41
|
+
const counted = new Set(answered.map((s) => s.name));
|
|
42
|
+
|
|
43
|
+
// Totals count what reviewers that answered ran. Each run of a command is
|
|
44
|
+
// counted; a command run twice is two.
|
|
45
|
+
const runs = run.commands.filter((c) => counted.has(c.by));
|
|
46
|
+
const failedRun = (c) => typeof c.exitCode === "number" && c.exitCode !== 0;
|
|
47
|
+
const last = new Map(); // command -> its last run
|
|
48
|
+
const everFailed = new Set(); // commands that failed at any point
|
|
49
|
+
const lastBySeat = new Map(); // "seat command" -> that seat's last run of it
|
|
50
|
+
for (const c of runs) {
|
|
51
|
+
last.set(c.command, c);
|
|
52
|
+
lastBySeat.set(`${c.by}\n${c.command}`, c);
|
|
53
|
+
if (failedRun(c)) everFailed.add(c.command);
|
|
54
|
+
}
|
|
55
|
+
const failingNow = [...last.values()].filter(failedRun).length;
|
|
56
|
+
const verifiers = answered.filter((s) => runs.some((c) => c.by === s.name));
|
|
57
|
+
|
|
58
|
+
const totals = {
|
|
59
|
+
reviewers: answered.length,
|
|
60
|
+
verifiers: verifiers.length,
|
|
61
|
+
notVisible: answered.filter((s) => s.commandsVisible === false && !verifiers.includes(s)).length,
|
|
62
|
+
// Agreed while a command of its own was still failing: that agreement does not count.
|
|
63
|
+
contradicted: agreed.filter((s) => [...lastBySeat.entries()].some(([k, c]) => k.startsWith(`${s.name}\n`) && failedRun(c))).length,
|
|
64
|
+
ran: runs.length,
|
|
65
|
+
passed: runs.filter((c) => c.exitCode === 0).length,
|
|
66
|
+
failed: runs.filter(failedRun).length,
|
|
67
|
+
failingNow,
|
|
68
|
+
resolved: [...everFailed].filter((cmd) => last.get(cmd).exitCode === 0).length,
|
|
69
|
+
...(Number.isInteger(run.filesChanged) ? { filesChanged: run.filesChanged } : {}),
|
|
70
|
+
};
|
|
71
|
+
|
|
72
|
+
// The verdict follows from the stances (rules V1 to V5).
|
|
73
|
+
let verdict = null;
|
|
74
|
+
if (reviewers.length) {
|
|
75
|
+
const everyAnswerAgreed = answered.length > 0 && answered.every((s) => s.stance === "agreed");
|
|
76
|
+
const isAgreed = everyAnswerAgreed && (answered.length === reviewers.length || failingNow === 0);
|
|
77
|
+
verdict = {
|
|
78
|
+
outcome: isAgreed ? "agreed" : agreed.length * 2 > reviewers.length ? "majority" : "none",
|
|
79
|
+
agreed: agreed.length,
|
|
80
|
+
reviewers: isAgreed ? answered.length : reviewers.length,
|
|
81
|
+
holdouts: reviewers.filter((s) => s.stance !== "agreed").map((s) => ({ name: s.name, stance: s.stance })),
|
|
82
|
+
};
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
// One check per command, as it stood at the end.
|
|
86
|
+
const checks = [...last.values()].map((c) => {
|
|
87
|
+
const others = [...new Set(runs.filter((o) => o.command === c.command && o.by !== c.by && o.exitCode === c.exitCode).map((o) => o.by))].slice(0, 4);
|
|
88
|
+
const blocked = Boolean(c.missingSetup);
|
|
89
|
+
return {
|
|
90
|
+
by: c.by,
|
|
91
|
+
...(others.length ? { alsoBy: others } : {}),
|
|
92
|
+
...(c.purpose ? { purpose: line(c.purpose, 100) } : {}),
|
|
93
|
+
command: line(mask(c.command), 300),
|
|
94
|
+
exitCode: typeof c.exitCode === "number" ? c.exitCode : null,
|
|
95
|
+
result: line(mask(c.result), 200),
|
|
96
|
+
where: c.where ?? "local",
|
|
97
|
+
deployed: c.deployed === true,
|
|
98
|
+
missingSetup: blocked ? line(c.missingSetup, 120) : null,
|
|
99
|
+
state: blocked ? "blocked" : typeof c.exitCode !== "number" ? "unknown" : c.exitCode === 0 ? "checked" : "reproduced",
|
|
100
|
+
};
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
const receipt = {
|
|
104
|
+
receiptVersion: "0.1",
|
|
105
|
+
producer: run.producer,
|
|
106
|
+
endedAt: new Date(run.endedAt).toISOString().replace(/\.\d{3}Z$/, "Z"),
|
|
107
|
+
...(run.revision ? { revision: run.revision } : {}),
|
|
108
|
+
durationMs: run.durationMs,
|
|
109
|
+
tokens: run.tokens,
|
|
110
|
+
costUsd: Number(run.seats.reduce((n, s) => n + s.costUsd, 0).toFixed(4)),
|
|
111
|
+
seats: run.seats,
|
|
112
|
+
verdict,
|
|
113
|
+
checks,
|
|
114
|
+
totals,
|
|
115
|
+
grade: gradeOf(totals),
|
|
116
|
+
gaps: run.gaps.map((g) => line(g, 200)),
|
|
117
|
+
};
|
|
118
|
+
// The digest is taken last, over everything else.
|
|
119
|
+
receipt.digest = digestOf(receipt);
|
|
120
|
+
return receipt;
|
|
121
|
+
}
|
|
122
|
+
|
|
123
|
+
/** A run as a tool might hold it when the run ends. */
|
|
124
|
+
export const sampleRun = {
|
|
125
|
+
producer: { name: "example-producer", version: "0.0.1" },
|
|
126
|
+
endedAt: "2026-11-12T09:30:00Z",
|
|
127
|
+
revision: "4f1c2ab90d3e",
|
|
128
|
+
durationMs: 95000,
|
|
129
|
+
tokens: 41800,
|
|
130
|
+
filesChanged: 2,
|
|
131
|
+
seats: [
|
|
132
|
+
{ name: "Builder", role: "lead", model: "model-a", turns: 2, errors: 0, timeouts: 0, durationMs: 52000, costUsd: 0.21, costBasis: "metered" },
|
|
133
|
+
{ name: "First reviewer", role: "reviewer", model: "model-b", turns: 2, errors: 0, timeouts: 0, durationMs: 30000, costUsd: 0.08, costBasis: "metered", stance: "agreed" },
|
|
134
|
+
{ name: "Second reviewer", role: "reviewer", model: "model-c", turns: 1, errors: 0, timeouts: 0, durationMs: 21000, costUsd: 0, costBasis: "subscription", stance: "agreed" },
|
|
135
|
+
],
|
|
136
|
+
commands: [
|
|
137
|
+
{ by: "First reviewer", command: "npm test", exitCode: 1, purpose: "Run the unit tests", result: "1 failing" },
|
|
138
|
+
{ by: "First reviewer", command: "npm test", exitCode: 0, purpose: "Run the unit tests", result: "18 passing" },
|
|
139
|
+
{ by: "Second reviewer", command: "npm test", exitCode: 0, result: "18 passing" },
|
|
140
|
+
{ by: "Second reviewer", command: "API_TOKEN=abc123 npm run smoke", exitCode: 0, purpose: "Smoke test", result: "ok" },
|
|
141
|
+
],
|
|
142
|
+
gaps: ["Checked locally only; deployed behaviour not verified"],
|
|
143
|
+
};
|
|
144
|
+
|
|
145
|
+
function isMain() {
|
|
146
|
+
if (!process.argv[1]) return false;
|
|
147
|
+
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
|
148
|
+
catch { return false; }
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
if (isMain()) {
|
|
152
|
+
const receipt = buildReceipt(sampleRun);
|
|
153
|
+
// Never write a receipt that does not verify.
|
|
154
|
+
const { valid, errors } = verifyReceipt(receipt);
|
|
155
|
+
if (!valid) { console.error(errors.join("\n")); process.exit(1); }
|
|
156
|
+
console.log(JSON.stringify(receipt, null, 2));
|
|
157
|
+
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@letthembuild/run-receipts",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "A format for recording how well an AI run checked its own work, and a verifier for it.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"author": "GPULogy AI LLC",
|
|
@@ -16,15 +16,20 @@
|
|
|
16
16
|
"main": "./verify.mjs",
|
|
17
17
|
"exports": {
|
|
18
18
|
".": "./verify.mjs",
|
|
19
|
+
"./sign": "./sign.mjs",
|
|
19
20
|
"./schema": "./receipt.schema.json"
|
|
20
21
|
},
|
|
21
22
|
"bin": {
|
|
22
|
-
"run-
|
|
23
|
+
"run-receipts": "verify.mjs",
|
|
24
|
+
"run-receipt-verify": "verify.mjs",
|
|
25
|
+
"run-receipt-sign": "sign.mjs"
|
|
23
26
|
},
|
|
24
27
|
"files": [
|
|
25
28
|
"verify.mjs",
|
|
29
|
+
"sign.mjs",
|
|
26
30
|
"receipt.schema.json",
|
|
27
31
|
"SPEC.md",
|
|
32
|
+
"PRODUCERS.md",
|
|
28
33
|
"examples",
|
|
29
34
|
"NOTICE"
|
|
30
35
|
],
|
|
@@ -32,7 +37,7 @@
|
|
|
32
37
|
"node": ">=18"
|
|
33
38
|
},
|
|
34
39
|
"scripts": {
|
|
35
|
-
"test": "node --test verify.test.mjs"
|
|
40
|
+
"test": "node --test verify.test.mjs sign.test.mjs action.test.mjs produce.test.mjs"
|
|
36
41
|
},
|
|
37
42
|
"keywords": [
|
|
38
43
|
"receipt",
|
package/receipt.schema.json
CHANGED
|
@@ -30,9 +30,19 @@
|
|
|
30
30
|
"totals": { "$ref": "#/$defs/totals" },
|
|
31
31
|
"grade": { "enum": ["failed", "verified", "partial", "not-visible", "reviewed", "unverified"] },
|
|
32
32
|
"gaps": { "type": "array", "maxItems": 50, "items": { "type": "string", "minLength": 1, "maxLength": 200 } },
|
|
33
|
-
"digest": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" }
|
|
33
|
+
"digest": { "type": "string", "pattern": "^sha256:[0-9a-f]{64}$" },
|
|
34
|
+
"signatures": { "type": "array", "minItems": 1, "maxItems": 8, "items": { "$ref": "#/$defs/signature" }, "description": "Signatures over the digest. Left out of the digest themselves." }
|
|
34
35
|
},
|
|
35
36
|
"$defs": {
|
|
37
|
+
"signature": {
|
|
38
|
+
"type": "object",
|
|
39
|
+
"additionalProperties": false,
|
|
40
|
+
"required": ["key", "sig"],
|
|
41
|
+
"properties": {
|
|
42
|
+
"key": { "type": "string", "pattern": "^ed25519:[A-Za-z0-9_-]{43}$", "description": "The signer's Ed25519 public key: its 32 bytes in base64url." },
|
|
43
|
+
"sig": { "type": "string", "pattern": "^[A-Za-z0-9_-]{86}$", "description": "The signature's 64 bytes in base64url." }
|
|
44
|
+
}
|
|
45
|
+
},
|
|
36
46
|
"stance": { "enum": ["agreed", "objected", "silent", "unavailable"] },
|
|
37
47
|
"seat": {
|
|
38
48
|
"type": "object",
|
package/sign.mjs
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// Copyright 2026 GPULogy AI LLC
|
|
3
|
+
// SPDX-License-Identifier: Apache-2.0
|
|
4
|
+
/**
|
|
5
|
+
* Sign a run receipt, and make the key to sign with.
|
|
6
|
+
*
|
|
7
|
+
* A signature says one thing: whoever holds this key put their name to this
|
|
8
|
+
* receipt, and it has not changed since. It does not say the commands were
|
|
9
|
+
* run. What a signature is worth is what the key is worth: who holds it, and
|
|
10
|
+
* how hard it is for anyone else to.
|
|
11
|
+
*
|
|
12
|
+
* No dependencies: Node 18 or later, and verify.mjs beside this file.
|
|
13
|
+
*/
|
|
14
|
+
import { readFileSync, writeFileSync, realpathSync, existsSync } from "node:fs";
|
|
15
|
+
import { generateKeyPairSync, createPrivateKey, sign as signBytes } from "node:crypto";
|
|
16
|
+
import { fileURLToPath } from "node:url";
|
|
17
|
+
import { digestOf, signedBytes, verifyReceipt, publicKeyOf } from "./verify.mjs";
|
|
18
|
+
|
|
19
|
+
/** A new key pair: `key` is public and goes in receipts and trust lists; `secret` signs and goes nowhere. */
|
|
20
|
+
export function generateKey() {
|
|
21
|
+
const { privateKey } = generateKeyPairSync("ed25519");
|
|
22
|
+
const jwk = privateKey.export({ format: "jwk" });
|
|
23
|
+
return { key: `ed25519:${jwk.x}`, secret: jwk.d };
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/** The signing key Node needs, from a { key, secret } pair. Throws when they are not a pair. */
|
|
27
|
+
function privateKeyOf(pair) {
|
|
28
|
+
const x = /^ed25519:([A-Za-z0-9_-]{43})$/.exec(String(pair?.key ?? ""))?.[1];
|
|
29
|
+
if (!x || !/^[A-Za-z0-9_-]{43}$/.test(String(pair?.secret ?? ""))) throw new Error("not a signing key");
|
|
30
|
+
if (!publicKeyOf(pair.key)) throw new Error("not a signing key");
|
|
31
|
+
try { return createPrivateKey({ key: { kty: "OKP", crv: "Ed25519", x, d: pair.secret }, format: "jwk" }); }
|
|
32
|
+
catch { throw new Error("not a signing key"); }
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
/**
|
|
36
|
+
* The receipt, signed: its digest retaken, and this key's signature added in
|
|
37
|
+
* place of any earlier one from the same key. Other keys' signatures are kept
|
|
38
|
+
* only while they still hold. Refuses a receipt that does not verify: a
|
|
39
|
+
* signature on a broken receipt would put a name to nothing.
|
|
40
|
+
*/
|
|
41
|
+
export function signReceipt(receipt, pair) {
|
|
42
|
+
const key = privateKeyOf(pair);
|
|
43
|
+
const { signatures: before = [], ...rest } = receipt;
|
|
44
|
+
const unsigned = { ...rest, digest: digestOf(rest) };
|
|
45
|
+
const check = verifyReceipt(unsigned);
|
|
46
|
+
if (!check.valid) throw new Error(`the receipt does not verify, so it is not signed:\n${check.errors.join("\n")}`);
|
|
47
|
+
const sig = signBytes(null, signedBytes(unsigned.digest), key).toString("base64url");
|
|
48
|
+
const kept = before.filter((s) => s.key !== pair.key && verifyReceipt({ ...unsigned, signatures: [s] }).valid);
|
|
49
|
+
const signed = { ...unsigned, signatures: [...kept, { key: pair.key, sig }] };
|
|
50
|
+
const after = verifyReceipt(signed);
|
|
51
|
+
if (!after.valid) throw new Error("the key's two halves do not belong together");
|
|
52
|
+
return signed;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/* ── Command line ────────────────────────────────────────────────────── */
|
|
56
|
+
|
|
57
|
+
const USAGE = `Usage:
|
|
58
|
+
sign.mjs keygen <key-file> make a signing key; prints the public half
|
|
59
|
+
sign.mjs sign --key <key-file> <receipt> print the receipt, signed
|
|
60
|
+
sign.mjs sign --key <key-file> --write <receipt> sign it where it is
|
|
61
|
+
|
|
62
|
+
The key can also come from the environment, for a build server:
|
|
63
|
+
RUN_RECEIPT_SIGNING_KEY the contents of a key file`;
|
|
64
|
+
|
|
65
|
+
function readPair(file, env) {
|
|
66
|
+
const text = file ? readFileSync(file, "utf8") : env.RUN_RECEIPT_SIGNING_KEY;
|
|
67
|
+
if (!text) throw new Error("no signing key: pass --key <key-file> or set RUN_RECEIPT_SIGNING_KEY");
|
|
68
|
+
return JSON.parse(text);
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
export function main(argv, env = process.env) {
|
|
72
|
+
const [command, ...rest] = argv;
|
|
73
|
+
if (command === "keygen") {
|
|
74
|
+
if (rest.length !== 1 || rest[0].startsWith("--")) { console.error(USAGE); return 2; }
|
|
75
|
+
if (existsSync(rest[0])) { console.error(`${rest[0]} exists. A key is not overwritten: receipts signed with it would have nobody to answer for them.`); return 2; }
|
|
76
|
+
const pair = generateKey();
|
|
77
|
+
writeFileSync(rest[0], JSON.stringify(pair, null, 2) + "\n", { mode: 0o600, flag: "wx" });
|
|
78
|
+
console.log(pair.key);
|
|
79
|
+
console.error(`The signing key is in ${rest[0]}. Keep it out of your repository. Give readers the line above.`);
|
|
80
|
+
return 0;
|
|
81
|
+
}
|
|
82
|
+
if (command === "sign") {
|
|
83
|
+
const args = [...rest];
|
|
84
|
+
let keyFile = null;
|
|
85
|
+
const at = args.indexOf("--key");
|
|
86
|
+
if (at !== -1) { keyFile = args[at + 1] ?? null; args.splice(at, 2); if (!keyFile) { console.error(USAGE); return 2; } }
|
|
87
|
+
const write = args.includes("--write");
|
|
88
|
+
const files = args.filter((a) => !a.startsWith("--"));
|
|
89
|
+
if (files.length !== 1 || args.some((a) => a.startsWith("--") && a !== "--write")) { console.error(USAGE); return 2; }
|
|
90
|
+
let receipt, pair;
|
|
91
|
+
try { receipt = JSON.parse(readFileSync(files[0], "utf8")); }
|
|
92
|
+
catch (e) { console.error(`Cannot read ${files[0]}: ${e.message}`); return 2; }
|
|
93
|
+
try { pair = readPair(keyFile, env); }
|
|
94
|
+
catch (e) { console.error(`Cannot read the signing key: ${e.message}`); return 2; }
|
|
95
|
+
let signed;
|
|
96
|
+
try { signed = signReceipt(receipt, pair); }
|
|
97
|
+
catch (e) { console.error(e.message); return 1; }
|
|
98
|
+
const text = JSON.stringify(signed, null, 2) + "\n";
|
|
99
|
+
if (write) { writeFileSync(files[0], text); console.error(`Signed ${files[0]} as ${pair.key}`); }
|
|
100
|
+
else process.stdout.write(text);
|
|
101
|
+
return 0;
|
|
102
|
+
}
|
|
103
|
+
console.error(USAGE);
|
|
104
|
+
return 2;
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
function isMain() {
|
|
108
|
+
if (!process.argv[1]) return false;
|
|
109
|
+
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
|
110
|
+
catch { return false; }
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
if (isMain()) process.exit(main(process.argv.slice(2)));
|
package/verify.mjs
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
/**
|
|
3
|
-
* Run-receipt verifier,
|
|
3
|
+
* Run-receipt verifier, for format 0.1.
|
|
4
4
|
*
|
|
5
5
|
* Answers one question about a receipt file: does it hold together? First
|
|
6
6
|
* against the schema (shape, bounds), then against itself (the rules in
|
|
@@ -9,12 +9,13 @@
|
|
|
9
9
|
*
|
|
10
10
|
* It does not prove the run happened. A receipt is the producer's account;
|
|
11
11
|
* this proves the account is well-formed, internally consistent and, when it
|
|
12
|
-
* carries a digest, unaltered since the digest was taken.
|
|
12
|
+
* carries a digest, unaltered since the digest was taken. When it is signed,
|
|
13
|
+
* it also says which keys signed it, and whether you listed them as trusted.
|
|
13
14
|
*
|
|
14
15
|
* No dependencies: Node 18 or later and this file, with the schema beside it.
|
|
15
16
|
*/
|
|
16
17
|
import { readFileSync, realpathSync } from "node:fs";
|
|
17
|
-
import { createHash } from "node:crypto";
|
|
18
|
+
import { createHash, createPublicKey, verify as verifyBytes } from "node:crypto";
|
|
18
19
|
import { fileURLToPath } from "node:url";
|
|
19
20
|
|
|
20
21
|
const SCHEMA_URL = new URL("./receipt.schema.json", import.meta.url);
|
|
@@ -73,7 +74,7 @@ export function validateSchema(value, schema, root = schema, path = "$") {
|
|
|
73
74
|
|
|
74
75
|
/* ── Digest ──────────────────────────────────────────────────────────── */
|
|
75
76
|
|
|
76
|
-
/** The receipt as bytes to hash: keys sorted, no whitespace, `digest` left out. */
|
|
77
|
+
/** The receipt as bytes to hash: keys sorted, no whitespace, `digest` and `signatures` left out. */
|
|
77
78
|
export function canonical(value) {
|
|
78
79
|
if (value === null || typeof value !== "object") return JSON.stringify(value);
|
|
79
80
|
if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
|
|
@@ -81,10 +82,71 @@ export function canonical(value) {
|
|
|
81
82
|
}
|
|
82
83
|
|
|
83
84
|
export function digestOf(receipt) {
|
|
84
|
-
const { digest: _left_out, ...rest } = receipt;
|
|
85
|
+
const { digest: _left_out, signatures: _nor_these, ...rest } = receipt;
|
|
85
86
|
return `sha256:${createHash("sha256").update(canonical(rest), "utf8").digest("hex")}`;
|
|
86
87
|
}
|
|
87
88
|
|
|
89
|
+
/* ── Signatures ──────────────────────────────────────────────────────── */
|
|
90
|
+
|
|
91
|
+
/** What a signature is made over: these words, then the receipt's digest. The words keep a signature made for a receipt from being one for anything else. */
|
|
92
|
+
export const SIGNATURE_CONTEXT = "run-receipt-signature-v1\n";
|
|
93
|
+
export const signedBytes = (digest) => Buffer.from(SIGNATURE_CONTEXT + digest, "utf8");
|
|
94
|
+
|
|
95
|
+
const KEY_FORM = /^ed25519:([A-Za-z0-9_-]{43})$/;
|
|
96
|
+
|
|
97
|
+
/** A public key as Node holds it, from its written form, or null when it is not one. */
|
|
98
|
+
export function publicKeyOf(key) {
|
|
99
|
+
const m = KEY_FORM.exec(String(key ?? ""));
|
|
100
|
+
if (!m) return null;
|
|
101
|
+
try { return createPublicKey({ key: { kty: "OKP", crv: "Ed25519", x: m[1] }, format: "jwk" }); }
|
|
102
|
+
catch { return null; }
|
|
103
|
+
}
|
|
104
|
+
|
|
105
|
+
/** True when `signature` is its key's signature over this receipt as it stands. */
|
|
106
|
+
export function signatureHolds(receipt, signature) {
|
|
107
|
+
const key = publicKeyOf(signature?.key);
|
|
108
|
+
if (!key || typeof signature.sig !== "string") return false;
|
|
109
|
+
try { return verifyBytes(null, signedBytes(digestOf(receipt)), key, Buffer.from(signature.sig, "base64url")); }
|
|
110
|
+
catch { return false; }
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
/**
|
|
114
|
+
* A list of keys somebody trusts, from text: either JSON,
|
|
115
|
+
* { "keys": [{ "key": "ed25519:…", "name": "…" }] }, or one key to a line with
|
|
116
|
+
* an optional name after it. Lines starting with # are comments. The names are
|
|
117
|
+
* the reader's own words for each key; a receipt never supplies them.
|
|
118
|
+
*/
|
|
119
|
+
export function parseKeys(text) {
|
|
120
|
+
const src = String(text ?? "").trim();
|
|
121
|
+
const keys = [], problems = [];
|
|
122
|
+
const add = (key, name, where) => {
|
|
123
|
+
if (!publicKeyOf(key)) problems.push(`${where}: not a key`);
|
|
124
|
+
else if (!keys.some((k) => k.key === key)) keys.push({ key, name: String(name ?? "").trim().slice(0, 80) });
|
|
125
|
+
};
|
|
126
|
+
if (src.startsWith("{")) {
|
|
127
|
+
let doc;
|
|
128
|
+
try { doc = JSON.parse(src); } catch { return { keys, problems: ["the key list is not readable as JSON"] }; }
|
|
129
|
+
if (!Array.isArray(doc?.keys)) return { keys, problems: ["the key list has no keys"] };
|
|
130
|
+
doc.keys.forEach((k, i) => add(k?.key, k?.name, `keys[${i}]`));
|
|
131
|
+
} else {
|
|
132
|
+
src.split(/\r?\n/).forEach((raw, i) => {
|
|
133
|
+
const line = raw.trim();
|
|
134
|
+
if (!line || line.startsWith("#")) return;
|
|
135
|
+
const [key, ...name] = line.split(/\s+/);
|
|
136
|
+
add(key, name.join(" "), `line ${i + 1}`);
|
|
137
|
+
});
|
|
138
|
+
}
|
|
139
|
+
return { keys, problems };
|
|
140
|
+
}
|
|
141
|
+
|
|
142
|
+
/** Who signed a receipt: each key whose signature holds, with the reader's name for it when they listed it. */
|
|
143
|
+
export function signersOf(receipt, trustedKeys = []) {
|
|
144
|
+
return (receipt.signatures ?? []).filter((s) => signatureHolds(receipt, s)).map((s) => {
|
|
145
|
+
const known = trustedKeys.find((k) => k.key === s.key);
|
|
146
|
+
return { key: s.key, trusted: Boolean(known), name: known?.name ?? "" };
|
|
147
|
+
});
|
|
148
|
+
}
|
|
149
|
+
|
|
88
150
|
/* ── Consistency: the rules of SPEC.md, by number ────────────────────── */
|
|
89
151
|
|
|
90
152
|
/** The grade the totals earn (SPEC rule G). */
|
|
@@ -179,15 +241,30 @@ export function checkConsistency(r) {
|
|
|
179
241
|
if (r.digest && r.digest !== digestOf(r)) err("D1", "digest does not match the receipt's contents");
|
|
180
242
|
if (!r.digest) warnings.push("D2: no digest, so a copy cannot be told from an edited one");
|
|
181
243
|
|
|
244
|
+
// K — signatures.
|
|
245
|
+
const signed = r.signatures ?? [];
|
|
246
|
+
if (new Set(signed.map((s) => s.key)).size !== signed.length) err("K2", "one key signed twice");
|
|
247
|
+
signed.forEach((s, i) => { if (!signatureHolds(r, s)) err("K1", `signatures[${i}]: is not that key's signature over this receipt`); });
|
|
248
|
+
|
|
182
249
|
return { errors, warnings };
|
|
183
250
|
}
|
|
184
251
|
|
|
185
252
|
/** Schema first; the consistency rules only run on a receipt with the right shape. */
|
|
186
|
-
|
|
253
|
+
/**
|
|
254
|
+
* `signers` lists the keys whose signatures hold, and is empty for a receipt
|
|
255
|
+
* that is not valid. Pass `trustedKeys` (as parseKeys returns them) to have
|
|
256
|
+
* each marked as one you listed. With `requireSignature`, a receipt no listed
|
|
257
|
+
* key has signed is not valid.
|
|
258
|
+
*/
|
|
259
|
+
export function verifyReceipt(receipt, { trustedKeys = [], requireSignature = false } = {}) {
|
|
187
260
|
const schemaErrors = validateSchema(receipt, loadSchema());
|
|
188
|
-
if (schemaErrors.length) return { valid: false, errors: schemaErrors.map((e) => `schema: ${e}`), warnings: [] };
|
|
261
|
+
if (schemaErrors.length) return { valid: false, errors: schemaErrors.map((e) => `schema: ${e}`), warnings: [], signers: [] };
|
|
189
262
|
const { errors, warnings } = checkConsistency(receipt);
|
|
190
|
-
|
|
263
|
+
const signers = errors.length ? [] : signersOf(receipt, trustedKeys);
|
|
264
|
+
if (requireSignature && !errors.length && !signers.some((s) => s.trusted)) {
|
|
265
|
+
errors.push(signers.length ? "K3: signed, but by no key on the trusted list" : "K3: not signed, and a signature from a trusted key is required");
|
|
266
|
+
}
|
|
267
|
+
return { valid: errors.length === 0, errors, warnings, signers: errors.length ? [] : signers };
|
|
191
268
|
}
|
|
192
269
|
|
|
193
270
|
/* ── Command line ────────────────────────────────────────────────────── */
|
|
@@ -195,12 +272,33 @@ export function verifyReceipt(receipt) {
|
|
|
195
272
|
const USAGE = `Usage:
|
|
196
273
|
verify.mjs <receipt.json> check a receipt; exit 0 valid, 1 invalid
|
|
197
274
|
verify.mjs --json <receipt.json> the same, as JSON
|
|
198
|
-
verify.mjs --digest <receipt.json> print the digest the receipt should carry
|
|
275
|
+
verify.mjs --digest <receipt.json> print the digest the receipt should carry
|
|
276
|
+
|
|
277
|
+
Signatures:
|
|
278
|
+
--keys <file> the keys you trust: JSON, or one "ed25519:… name" to a line
|
|
279
|
+
--require-signature not valid unless a key from --keys signed it`;
|
|
280
|
+
|
|
281
|
+
/** A key as people read it: enough of it to tell two apart. */
|
|
282
|
+
export const shortKey = (key) => `${String(key).slice(0, 16)}…`;
|
|
199
283
|
|
|
200
284
|
function main(argv) {
|
|
201
|
-
const
|
|
202
|
-
|
|
203
|
-
|
|
285
|
+
const args = [...argv];
|
|
286
|
+
let keysFile = null;
|
|
287
|
+
const at = args.indexOf("--keys");
|
|
288
|
+
if (at !== -1) { keysFile = args[at + 1] ?? null; args.splice(at, 2); if (!keysFile || keysFile.startsWith("--")) { console.error(USAGE); return 2; } }
|
|
289
|
+
const flags = args.filter((a) => a.startsWith("--"));
|
|
290
|
+
const files = args.filter((a) => !a.startsWith("--"));
|
|
291
|
+
if (files.length !== 1 || flags.some((f) => !["--json", "--digest", "--require-signature"].includes(f))) { console.error(USAGE); return 2; }
|
|
292
|
+
const requireSignature = flags.includes("--require-signature");
|
|
293
|
+
let trustedKeys = [];
|
|
294
|
+
if (keysFile) {
|
|
295
|
+
let parsed;
|
|
296
|
+
try { parsed = parseKeys(readFileSync(keysFile, "utf8")); }
|
|
297
|
+
catch (e) { console.error(`Cannot read ${keysFile}: ${e.message}`); return 2; }
|
|
298
|
+
if (parsed.problems.length) { console.error(`${keysFile}: ${parsed.problems.join("; ")}`); return 2; }
|
|
299
|
+
trustedKeys = parsed.keys;
|
|
300
|
+
}
|
|
301
|
+
if (requireSignature && !trustedKeys.length) { console.error("--require-signature needs --keys: a signature only means something from a key you trust."); return 2; }
|
|
204
302
|
let receipt;
|
|
205
303
|
try { receipt = JSON.parse(readFileSync(files[0], "utf8")); }
|
|
206
304
|
catch (e) { console.error(`Cannot read ${files[0]}: ${e.message}`); return 2; }
|
|
@@ -209,12 +307,13 @@ function main(argv) {
|
|
|
209
307
|
console.log(digestOf(receipt));
|
|
210
308
|
return 0;
|
|
211
309
|
}
|
|
212
|
-
const result = verifyReceipt(receipt);
|
|
310
|
+
const result = verifyReceipt(receipt, { trustedKeys, requireSignature });
|
|
213
311
|
if (flags.includes("--json")) console.log(JSON.stringify(result, null, 2));
|
|
214
312
|
else {
|
|
215
313
|
console.log(result.valid ? `VALID ${files[0]}` : `INVALID ${files[0]}`);
|
|
216
314
|
for (const e of result.errors) console.log(` error ${e}`);
|
|
217
315
|
for (const w of result.warnings) console.log(` warning ${w}`);
|
|
316
|
+
for (const s of result.signers) console.log(s.trusted ? ` signed ${s.name || "a trusted key"} (${shortKey(s.key)})` : ` signed ${shortKey(s.key)}, which ${keysFile ? "is not on your list" : "nothing here vouches for (pass --keys)"}`);
|
|
218
317
|
}
|
|
219
318
|
return result.valid ? 0 : 1;
|
|
220
319
|
}
|