@ssheleg/sheleg-dev 0.7.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +114 -0
- package/README.md +24 -9
- package/SECURITY.md +28 -18
- package/package.json +4 -2
- package/plugins/sheleg-dev/.claude-plugin/plugin.json +7 -3
- package/plugins/sheleg-dev/skills/ad-tracking/fixtures/README.md +5 -4
- package/plugins/sheleg-dev/skills/ad-tracking/fixtures/assert-dedup-contract.mjs +168 -20
- package/plugins/sheleg-dev/skills/ad-tracking/references/meta-linkedin.md +6 -2
- package/plugins/sheleg-dev/skills/crypto-payments/SKILL.md +17 -80
- package/plugins/sheleg-dev/skills/crypto-payments/references/callback-route-hardening.md +55 -0
- package/plugins/sheleg-dev/skills/crypto-payments/references/testing-and-local-dev.md +60 -0
- package/plugins/sheleg-dev/skills/error-tracking/SKILL.md +259 -0
- package/plugins/sheleg-dev/skills/error-tracking/references/releases.md +159 -0
- package/plugins/sheleg-dev/skills/error-tracking/references/scrubbing.md +170 -0
- package/plugins/sheleg-dev/skills/error-tracking/references/setup.md +176 -0
- package/plugins/sheleg-dev/skills/stripe-billing/fixtures/README.md +5 -4
- package/plugins/sheleg-dev/skills/stripe-billing/fixtures/assert-money-invariants.mjs +195 -35
- package/plugins/sheleg-dev/skills/stripe-billing/references/testing-and-local-dev.md +4 -3
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# Sentry setup — install, auth, projects, DSN, MCP
|
|
2
|
+
|
|
3
|
+
Read when adding Sentry to a project for the first time, or when a CLI call is
|
|
4
|
+
refused and you need to tell a policy problem from an auth problem.
|
|
5
|
+
|
|
6
|
+
## Contents
|
|
7
|
+
|
|
8
|
+
- [Install](#install)
|
|
9
|
+
- [Authenticate](#authenticate)
|
|
10
|
+
- [Token taxonomy — which kind can do what](#token-taxonomy--which-kind-can-do-what)
|
|
11
|
+
- [Find what already exists](#find-what-already-exists)
|
|
12
|
+
- [Create a project](#create-a-project)
|
|
13
|
+
- [Get the DSN and wire it in](#get-the-dsn-and-wire-it-in)
|
|
14
|
+
- [Declare the MCP server](#declare-the-mcp-server)
|
|
15
|
+
- [SDK options for a backend service](#sdk-options-for-a-backend-service)
|
|
16
|
+
|
|
17
|
+
All commands below were executed on 2026-08-24 against `sentry` 0.43.0 and
|
|
18
|
+
`sentry-sdk` 2.19.2. Where a version matters, it is named.
|
|
19
|
+
|
|
20
|
+
## Install
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
brew install getsentry/tools/sentry # macOS, preferred: no piping a remote script to a shell
|
|
24
|
+
curl https://cli.sentry.dev/install -fsS | bash # the documented alternative
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
Then check for a shadow — two prefixes, and the older copy usually wins:
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
which -a sentry | while read p; do printf '%-34s %s\n' "$p" "$("$p" --version 2>&1 | head -1)"; done
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Remove the stale copy rather than reordering `PATH`. Credentials live in
|
|
34
|
+
`~/.sentry/`, not beside the binary, so this does not log you out.
|
|
35
|
+
|
|
36
|
+
## Authenticate
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
sentry auth # OAuth device flow: prints a URL and a code
|
|
40
|
+
sentry auth --token <TOKEN> # non-interactive, for CI
|
|
41
|
+
sentry auth status # who, which token, when it expires
|
|
42
|
+
sentry auth logout
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
`sentry auth` writes to `~/.sentry/cli.db`. Override with `SENTRY_CONFIG_DIR`.
|
|
46
|
+
`SENTRY_URL` points at self-hosted instances.
|
|
47
|
+
|
|
48
|
+
**The device-flow token expires.** `auth status` prints the expiry — one
|
|
49
|
+
observed token had three days left. A CI job depending on it fails on a
|
|
50
|
+
Wednesday for no visible reason; CI wants a token, not the device flow.
|
|
51
|
+
|
|
52
|
+
## Token taxonomy — which kind can do what
|
|
53
|
+
|
|
54
|
+
| Kind | Created in | Scopes | Use it for |
|
|
55
|
+
|---|---|---|---|
|
|
56
|
+
| **OAuth device flow** (`sentry auth`) | the CLI | fixed by the flow; measured: `event:read`, `event:write`, `member:read`, `org:read`, `project:admin`, `project:read`, `project:write`, `team:read`, `team:write` — **no `org:write`/`org:admin`** | interactive work at a desk |
|
|
57
|
+
| **Organization Token** | Settings → Developer Settings → Organization Tokens | limited, not customizable | CI. Sentry's own recommendation for it |
|
|
58
|
+
| **Internal Integration** | the integration platform | customizable, **full API access** | org-level automation, including creating projects |
|
|
59
|
+
| **Personal Token** | Account → Personal Tokens | chosen at creation, never editable after | a one-off at a desk. Explicitly discouraged for CI |
|
|
60
|
+
|
|
61
|
+
Print the scopes a token actually carries, rather than inferring them from how
|
|
62
|
+
it was made:
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
sentry api "/" --json # {"auth": {"scopes": [...]}, "user": {...}}
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Create a project
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
sentry project create <org>/<name>:<platform> --team <team> --dry-run
|
|
72
|
+
sentry project create <org>/<name>:<platform> --team <team> --json
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
`--dry-run` prints the resolved org, team, slug and platform without creating
|
|
76
|
+
anything. Use it: the org must be in the argument or the command refuses.
|
|
77
|
+
|
|
78
|
+
### When it returns 403
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
Your organization has disabled this feature for members.
|
|
82
|
+
This is an org-level policy setting, not an auth issue.
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
The message is accurate and easy to disbelieve, because it appears **even for an
|
|
86
|
+
org owner**. The policy is evaluated against the token's scopes, not the human's
|
|
87
|
+
role, so an owner acting through a device-flow token is treated as a member.
|
|
88
|
+
|
|
89
|
+
```bash
|
|
90
|
+
sentry org view <org> --json | grep -E 'allowMemberProjectCreation|allowMemberInvite'
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
`allowMemberProjectCreation: false` is the cause. `POST /teams/{org}/{team}/projects/`
|
|
94
|
+
fails identically — the error names it as an alternative, but it is the same
|
|
95
|
+
policy underneath.
|
|
96
|
+
|
|
97
|
+
Resolutions, cheapest first:
|
|
98
|
+
|
|
99
|
+
1. Create the project once in the web UI. One browser action, then everything
|
|
100
|
+
else is scriptable forever.
|
|
101
|
+
2. Flip the org setting. Also a browser action, and it loosens a policy that
|
|
102
|
+
exists for a reason — decide deliberately.
|
|
103
|
+
3. Issue an Internal Integration token and `sentry auth --token` with it. The
|
|
104
|
+
only route that makes project creation itself scriptable.
|
|
105
|
+
|
|
106
|
+
## Get the DSN and wire it in
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
sentry project view <org>/<project> --json \
|
|
110
|
+
| python3 -c "import json,sys; print(json.load(sys.stdin)[0]['dsn'])"
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
The response is a **list**, hence the `[0]`. The DSN is the app's ingest
|
|
114
|
+
address; put it wherever that host keeps configuration:
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
heroku config:set SENTRY_DSN='https://…@o….ingest.sentry.io/…' -a <app>
|
|
118
|
+
fly secrets set SENTRY_DSN='…'
|
|
119
|
+
gh secret set SENTRY_DSN --body '…' # only if CI itself reports errors
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
A DSN is write-only ingest, not an API credential — see the table in `SKILL.md`.
|
|
123
|
+
Treat it as configuration, not as a secret to be guarded like a token, and still
|
|
124
|
+
do not commit it.
|
|
125
|
+
|
|
126
|
+
## Declare the MCP server
|
|
127
|
+
|
|
128
|
+
```bash
|
|
129
|
+
curl -si https://mcp.sentry.dev/mcp | grep -i www-authenticate
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
A `401` whose `www-authenticate` carries `resource_metadata=` means OAuth, which
|
|
133
|
+
means **the agent's own config, never a shared MCP gateway** — an OAuth upstream
|
|
134
|
+
names itself as the resource at its own address, so a client pointed at a
|
|
135
|
+
gateway must reject the mismatch and the flow cannot complete.
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
claude mcp add --transport http sentry https://mcp.sentry.dev/mcp/<org>/<project>
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Scoping is optional and narrows what the server sees:
|
|
142
|
+
`…/mcp`, `…/mcp/<org>`, `…/mcp/<org>/<project>`. Narrow it.
|
|
143
|
+
|
|
144
|
+
Everything the MCP does, the CLI also does. A machine without it loses no
|
|
145
|
+
capability, so never let a procedure depend on it.
|
|
146
|
+
|
|
147
|
+
## SDK options for a backend service
|
|
148
|
+
|
|
149
|
+
```python
|
|
150
|
+
sentry_sdk.init(
|
|
151
|
+
dsn=SENTRY_DSN,
|
|
152
|
+
environment=ENVIRONMENT, # "production" | "staging" | "local"
|
|
153
|
+
release=RELEASE, # MUST equal the version you create in Sentry
|
|
154
|
+
event_scrubber=EventScrubber( # see references/scrubbing.md
|
|
155
|
+
denylist=DEFAULT_DENYLIST + ["dsn", "database_url", "auth_key"],
|
|
156
|
+
recursive=True,
|
|
157
|
+
),
|
|
158
|
+
before_send=scrub_values,
|
|
159
|
+
send_default_pii=False, # excludes headers, IPs, and similar
|
|
160
|
+
traces_sample_rate=0.0, # tracing off unless someone reads the spans
|
|
161
|
+
)
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Notes that decide behaviour:
|
|
165
|
+
|
|
166
|
+
- `environment` defaults to `"production"`. A local run that forgets to set it
|
|
167
|
+
pollutes the production feed with a developer's laptop.
|
|
168
|
+
- `release` defaults to `None` and the SDK's auto-detection is not reliable
|
|
169
|
+
enough to depend on. Set it.
|
|
170
|
+
- `traces_sample_rate` defaults to `None`; tracing is off unless it or
|
|
171
|
+
`traces_sampler` is set. Turning it on costs quota — do it when someone is
|
|
172
|
+
going to read the result, not by default.
|
|
173
|
+
- `profiles_sample_rate` is the current profiling option and is not deprecated.
|
|
174
|
+
- `send_default_pii=False` (the default) keeps integrations from attaching
|
|
175
|
+
request headers and IP addresses. It does **not** scrub your own payloads;
|
|
176
|
+
that is the scrubber's job.
|
|
@@ -2,14 +2,15 @@
|
|
|
2
2
|
|
|
3
3
|
**Copy this whole directory into your repository.** It is the executable half of what
|
|
4
4
|
`SKILL.md` and `references/webhook-events.md` state in prose: nine real Stripe event
|
|
5
|
-
bodies, twelve
|
|
6
|
-
breaks the handler one rule at a time so you can watch
|
|
5
|
+
bodies, twelve invariants a correct webhook handler must hold — 45 assertions between
|
|
6
|
+
them — and a self-test that breaks the handler one rule at a time so you can watch each
|
|
7
|
+
assertion fail on its own.
|
|
7
8
|
|
|
8
9
|
Two commands, no dependencies, no network, no key:
|
|
9
10
|
|
|
10
11
|
```bash
|
|
11
|
-
node assert-money-invariants.mjs # 12
|
|
12
|
-
node assert-money-invariants.mjs --self-test # break one rule at a time;
|
|
12
|
+
node assert-money-invariants.mjs # 12 invariants / 45 assertions
|
|
13
|
+
node assert-money-invariants.mjs --self-test # break one rule at a time; EACH ASSERTION must go red
|
|
13
14
|
```
|
|
14
15
|
|
|
15
16
|
Point it at **your** handler by replacing the two imports at the top of
|
|
@@ -72,14 +72,20 @@ export const INVARIANTS = [
|
|
|
72
72
|
fixtures: [CYCLE_JAN],
|
|
73
73
|
states: 'a paid renewal becomes exactly one grant, and the mirror carries its period',
|
|
74
74
|
breaks: ['grant-on-renewal'],
|
|
75
|
-
async run({ store, handler }) {
|
|
75
|
+
async run({ store, handler, assert }) {
|
|
76
76
|
const event = fixture(CYCLE_JAN);
|
|
77
77
|
const response = await handler.deliver(event);
|
|
78
|
-
|
|
79
|
-
|
|
78
|
+
// `unmutated`: no rule in RULES makes this route answer anything but 200 with that
|
|
79
|
+
// body, so neither assertion has ever been watched failing. They pin the envelope a
|
|
80
|
+
// reader copies; they are not evidence, and the verdict line counts them apart.
|
|
81
|
+
assert.unmutated.equal(response.status, 200);
|
|
82
|
+
assert.unmutated.deepEqual(response.body, { received: true });
|
|
80
83
|
assert.equal(store.grants.length, 1, 'a paid renewal granted nothing');
|
|
81
84
|
assert.equal(store.credits.get('usr_PLACEHOLDER_alice'), CREDITS_PER_PERIOD);
|
|
82
|
-
|
|
85
|
+
// `|| {}` on purpose: with no mirror row at all these two assertions must FAIL
|
|
86
|
+
// rather than throw a TypeError on the line above them. A throw aborts the run, and
|
|
87
|
+
// an assertion the runner never reached is one no mutant has been watched breaking.
|
|
88
|
+
const mirror = store.subscriptions.get('sub_PLACEHOLDER_alice') || {};
|
|
83
89
|
assert.equal(mirror.periodStart, 1767225600);
|
|
84
90
|
assert.equal(mirror.periodEnd, 1769904000, 'the period is not the line period');
|
|
85
91
|
},
|
|
@@ -89,10 +95,11 @@ export const INVARIANTS = [
|
|
|
89
95
|
fixtures: [CYCLE_JAN, PRORATION],
|
|
90
96
|
states: 'billing_reason subscription_update is a $40 proration, not a month of product',
|
|
91
97
|
breaks: ['billing-reason', 'grant-on-renewal'],
|
|
92
|
-
async run({ store, handler }) {
|
|
98
|
+
async run({ store, handler, assert }) {
|
|
93
99
|
await handler.deliver(fixture(CYCLE_JAN));
|
|
94
100
|
const response = await handler.deliver(fixture(PRORATION));
|
|
95
|
-
|
|
101
|
+
// unmutated: nothing in RULES makes the endpoint answer 500 for a proration.
|
|
102
|
+
assert.unmutated.equal(response.status, 200, 'a proration invoice must not fail the endpoint');
|
|
96
103
|
assert.equal(store.grants.length, 1,
|
|
97
104
|
'the proration granted a second period — a user who adds and removes a seat '
|
|
98
105
|
+ 'four times has been given four months of product');
|
|
@@ -107,10 +114,10 @@ export const INVARIANTS = [
|
|
|
107
114
|
fixtures: [CYCLE_JAN, CYCLE_JAN_REDELIVERY],
|
|
108
115
|
states: 'the same evt_ delivered twice grants once AND runs its side effects once',
|
|
109
116
|
breaks: ['claim', 'grant-on-renewal'],
|
|
110
|
-
async run({ store, handler }) {
|
|
117
|
+
async run({ store, handler, assert }) {
|
|
111
118
|
const first = await handler.deliver(fixture(CYCLE_JAN));
|
|
112
119
|
const second = await handler.deliver(fixture(CYCLE_JAN_REDELIVERY));
|
|
113
|
-
assert.deepEqual(first.body, { received: true });
|
|
120
|
+
assert.unmutated.deepEqual(first.body, { received: true }); // the envelope, again
|
|
114
121
|
assert.deepEqual(second.body, { received: true, duplicate: true },
|
|
115
122
|
'the second delivery was not recognised as a duplicate');
|
|
116
123
|
assert.equal(store.grants.length, 1, 'the redelivery granted a second time');
|
|
@@ -124,7 +131,7 @@ export const INVARIANTS = [
|
|
|
124
131
|
fixtures: [CYCLE_JAN, CYCLE_JAN_REDELIVERY],
|
|
125
132
|
states: 'two deliveries in flight at once grant once — the claim is an INSERT, not a SELECT',
|
|
126
133
|
breaks: ['claim', 'grant-on-renewal'],
|
|
127
|
-
async run({ store, handler }) {
|
|
134
|
+
async run({ store, handler, assert }) {
|
|
128
135
|
// 40 ms apart in production; here, both before either has finished its first read.
|
|
129
136
|
// This is the fixture that separates the event claim from the grant marker: the
|
|
130
137
|
// marker reads before it writes, and a read is a round trip.
|
|
@@ -146,7 +153,7 @@ export const INVARIANTS = [
|
|
|
146
153
|
fixtures: [CYCLE_JAN],
|
|
147
154
|
states: 'the nightly repair reuses the webhook marker — a nightly job is not a nightly gift',
|
|
148
155
|
breaks: ['grant-marker', 'grant-on-renewal'],
|
|
149
|
-
async run({ store, handler }) {
|
|
156
|
+
async run({ store, handler, assert }) {
|
|
150
157
|
const event = fixture(CYCLE_JAN);
|
|
151
158
|
await handler.deliver(event);
|
|
152
159
|
const subId = subscriptionIdOf(event.data.object);
|
|
@@ -163,11 +170,11 @@ export const INVARIANTS = [
|
|
|
163
170
|
fixtures: [CYCLE_FEB, CYCLE_JAN],
|
|
164
171
|
states: 'February delivered before January leaves February in the mirror, and grants both',
|
|
165
172
|
breaks: ['ordering', 'grant-on-renewal'],
|
|
166
|
-
async run({ store, handler }) {
|
|
173
|
+
async run({ store, handler, assert }) {
|
|
167
174
|
await handler.deliver(fixture(CYCLE_FEB)); // arrives first
|
|
168
175
|
await handler.deliver(fixture(CYCLE_JAN)); // the earlier period, delivered late
|
|
169
176
|
assert.equal(store.grants.length, 2, 'both periods were paid, so both are owed');
|
|
170
|
-
const mirror = store.subscriptions.get('sub_PLACEHOLDER_alice');
|
|
177
|
+
const mirror = store.subscriptions.get('sub_PLACEHOLDER_alice') || {};
|
|
171
178
|
assert.equal(mirror.periodStart, 1769904000,
|
|
172
179
|
'the mirror was rewound to the older period — state derived from arrival order');
|
|
173
180
|
assert.equal(mirror.periodEnd, 1772323200,
|
|
@@ -179,7 +186,7 @@ export const INVARIANTS = [
|
|
|
179
186
|
fixtures: [REFUND_PARTIAL, REFUND_REMAINDER],
|
|
180
187
|
states: 'amount_refunded is the total so far: two partials claw back $90, not $130',
|
|
181
188
|
breaks: ['cumulative-refund'],
|
|
182
|
-
async run({ store, handler }) {
|
|
189
|
+
async run({ store, handler, assert }) {
|
|
183
190
|
seedOneOffPurchase(store);
|
|
184
191
|
await handler.deliver(fixture(REFUND_PARTIAL));
|
|
185
192
|
await handler.deliver(fixture(REFUND_REMAINDER));
|
|
@@ -201,7 +208,7 @@ export const INVARIANTS = [
|
|
|
201
208
|
// and the claim still refuses the second delivery. A suite that judged a redelivery by
|
|
202
209
|
// its grant count alone would ship with no idempotency and stay green.
|
|
203
210
|
breaks: [['claim', 'grant-marker'], 'grant-on-renewal'],
|
|
204
|
-
async run({ store, handler }) {
|
|
211
|
+
async run({ store, handler, assert }) {
|
|
205
212
|
await handler.deliver(fixture(CYCLE_JAN));
|
|
206
213
|
await handler.deliver(fixture(CYCLE_JAN_REDELIVERY));
|
|
207
214
|
assert.equal(store.grants.length, 1, 'the redelivery granted a second time');
|
|
@@ -216,7 +223,7 @@ export const INVARIANTS = [
|
|
|
216
223
|
// `increment <= 0` catches a sequential replay by itself, and the claim catches it
|
|
217
224
|
// without the swap. Neither is tested by this fixture alone.
|
|
218
225
|
breaks: [['claim', 'cumulative-refund']],
|
|
219
|
-
async run({ store, handler }) {
|
|
226
|
+
async run({ store, handler, assert }) {
|
|
220
227
|
seedOneOffPurchase(store);
|
|
221
228
|
await handler.deliver(fixture(REFUND_PARTIAL));
|
|
222
229
|
await handler.deliver(fixture(REFUND_PARTIAL));
|
|
@@ -229,16 +236,20 @@ export const INVARIANTS = [
|
|
|
229
236
|
fixtures: [SESSION_UNPAID, SESSION_FAILED],
|
|
230
237
|
states: 'the redirect proves a browser: an unpaid async session grants and converts nothing',
|
|
231
238
|
breaks: ['async-gate'],
|
|
232
|
-
async run({ store, handler }) {
|
|
239
|
+
async run({ store, handler, assert }) {
|
|
233
240
|
const completed = await handler.deliver(fixture(SESSION_UNPAID));
|
|
234
|
-
|
|
241
|
+
// unmutated: the envelope. Removing `async-gate` grants the session and still answers 200.
|
|
242
|
+
assert.unmutated.equal(completed.status, 200, 'an unpaid session must still answer 200');
|
|
235
243
|
assert.equal(store.grants.length, 0,
|
|
236
244
|
'a session whose payment_status is unpaid was granted — the money has not moved');
|
|
237
|
-
|
|
245
|
+
// unmutated, and the reason is a gap worth seeing: this handler emits conversions on
|
|
246
|
+
// `invoice.paid` only, so no rule here can make a checkout session report one. The
|
|
247
|
+
// assertion states the rule for a reader whose handler DOES emit from a session.
|
|
248
|
+
assert.unmutated.equal(store.conversions.length, 0,
|
|
238
249
|
'a purchase conversion was reported for a charge that has not cleared');
|
|
239
250
|
await handler.deliver(fixture(SESSION_FAILED));
|
|
240
251
|
assert.equal(store.grants.length, 0, 'the failed async payment still granted');
|
|
241
|
-
assert.equal(store.conversions.length, 0);
|
|
252
|
+
assert.unmutated.equal(store.conversions.length, 0); // same gap as above
|
|
242
253
|
assert.equal(store.credits.get('usr_PLACEHOLDER_bob'), undefined);
|
|
243
254
|
},
|
|
244
255
|
},
|
|
@@ -249,9 +260,9 @@ export const INVARIANTS = [
|
|
|
249
260
|
// Without it, a handler that refuses every checkout session satisfies the assertion
|
|
250
261
|
// above and this pack would have taught the opposite defect.
|
|
251
262
|
breaks: ['grant-on-checkout'],
|
|
252
|
-
async run({ store, handler }) {
|
|
263
|
+
async run({ store, handler, assert }) {
|
|
253
264
|
const response = await handler.deliver(fixture(SESSION_PAID));
|
|
254
|
-
assert.deepEqual(response.body, { received: true });
|
|
265
|
+
assert.unmutated.deepEqual(response.body, { received: true }); // the envelope
|
|
255
266
|
assert.equal(store.grants.length, 1, 'a cleared card session granted nothing');
|
|
256
267
|
assert.equal(store.credits.get('usr_PLACEHOLDER_carol'), CREDITS_PER_PERIOD);
|
|
257
268
|
assert.ok(store.subscriptions.has('sub_PLACEHOLDER_carol'));
|
|
@@ -262,10 +273,10 @@ export const INVARIANTS = [
|
|
|
262
273
|
fixtures: [CYCLE_JAN],
|
|
263
274
|
states: 'the conversion id comes from subscription metadata, so browser and server share it',
|
|
264
275
|
breaks: ['conversion-id-from-metadata', 'grant-on-renewal'],
|
|
265
|
-
async run({ store, handler }) {
|
|
276
|
+
async run({ store, handler, assert }) {
|
|
266
277
|
await handler.deliver(fixture(CYCLE_JAN));
|
|
267
278
|
assert.equal(store.conversions.length, 1);
|
|
268
|
-
const [conversion] = store.conversions;
|
|
279
|
+
const [conversion = {}] = store.conversions;
|
|
269
280
|
assert.equal(conversion.eventId, 'evtid_PLACEHOLDER_purchase_jan',
|
|
270
281
|
'the server generated its own id — nothing will ever deduplicate against the pixel');
|
|
271
282
|
assert.equal(conversion.eventName, 'Purchase', 'the event name is not byte-equal');
|
|
@@ -276,34 +287,131 @@ export const INVARIANTS = [
|
|
|
276
287
|
},
|
|
277
288
|
];
|
|
278
289
|
|
|
290
|
+
// ------------------------------------------------- one assertion, one measurement
|
|
291
|
+
//
|
|
292
|
+
// Why the assertions are not called straight through `node:assert`. Until 2026-08-20 the
|
|
293
|
+
// runner returned on the FIRST throw and the matrix compared one row per invariant, so an
|
|
294
|
+
// invariant was measured as a single unit: any assertion that was not the sole
|
|
295
|
+
// discriminator could be replaced by `assert.ok(true)` and nothing went red. Measured that
|
|
296
|
+
// day — three neutered assertions (the `clawbacks.map` deepEqual, the `mirror.periodEnd`
|
|
297
|
+
// equality, and the `response.body` deepEqual) left `--self-test` at exit 0 still printing
|
|
298
|
+
// *"12 assertions, each watched failing"*, and `npm test` at 0 with them.
|
|
299
|
+
//
|
|
300
|
+
// So the `assert` an invariant receives is a RECORDER: same API, but a failure is
|
|
301
|
+
// remembered instead of thrown and the assertions after it still run. Each call site is one
|
|
302
|
+
// assertion with its own identity — its line in this file — and `--self-test` asks the
|
|
303
|
+
// question per assertion rather than per invariant.
|
|
304
|
+
//
|
|
305
|
+
// Two kinds, and the difference is declared at the call site rather than in a table:
|
|
306
|
+
//
|
|
307
|
+
// assert.equal(...) a DISCRIMINATING assertion. At least one mutant must turn it
|
|
308
|
+
// red, or it is not evidence of anything.
|
|
309
|
+
// assert.unmutated.equal() an assertion NO mutant in this pack varies. It pins the
|
|
310
|
+
// envelope or a field the rules do not touch, so it has never
|
|
311
|
+
// been watched failing and the verdict line counts it apart
|
|
312
|
+
// rather than as evidence. Measured, not asserted: the self-test
|
|
313
|
+
// fails if a mutant does break one, because that is a
|
|
314
|
+
// discriminator hiding in the wrong list.
|
|
315
|
+
//
|
|
316
|
+
// A reader pointing this at their own handler keeps writing `assert.equal`; the recorder is
|
|
317
|
+
// API-compatible with `node:assert/strict` for the methods below.
|
|
318
|
+
|
|
319
|
+
const SELF = import.meta.url;
|
|
320
|
+
const METHODS = ['equal', 'notEqual', 'deepEqual', 'notDeepEqual', 'ok', 'match', 'doesNotMatch'];
|
|
321
|
+
|
|
322
|
+
/**
|
|
323
|
+
* The line in THIS file that called the recorder.
|
|
324
|
+
*
|
|
325
|
+
* `captureStackTrace(holder, boundary)` drops `boundary` and everything above it, so the
|
|
326
|
+
* top frame is the assertion's own line — no frame counting, which is what makes the
|
|
327
|
+
* identity survive a refactor of the recorder itself.
|
|
328
|
+
*/
|
|
329
|
+
function siteOf(boundary) {
|
|
330
|
+
const holder = {};
|
|
331
|
+
Error.captureStackTrace(holder, boundary);
|
|
332
|
+
const frame = (holder.stack || '').split('\n').slice(1).find((f) => f.includes(SELF));
|
|
333
|
+
const m = frame && frame.match(/:(\d+):\d+\)?\s*$/);
|
|
334
|
+
return m ? Number(m[1]) : 0;
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
function recorder() {
|
|
338
|
+
const seen = new Map(); // line -> { site, unmutated, ok, why }
|
|
339
|
+
|
|
340
|
+
function record(site, unmutated, name, args) {
|
|
341
|
+
let ok = true;
|
|
342
|
+
let why = null;
|
|
343
|
+
try {
|
|
344
|
+
assert[name](...args);
|
|
345
|
+
} catch (error) {
|
|
346
|
+
ok = false;
|
|
347
|
+
why = error.message.split('\n')[0];
|
|
348
|
+
}
|
|
349
|
+
const prev = seen.get(site);
|
|
350
|
+
if (prev) {
|
|
351
|
+
prev.ok = prev.ok && ok;
|
|
352
|
+
prev.why = prev.why || why;
|
|
353
|
+
} else {
|
|
354
|
+
seen.set(site, { site, unmutated, ok, why });
|
|
355
|
+
}
|
|
356
|
+
}
|
|
357
|
+
|
|
358
|
+
const bind = (unmutated) => {
|
|
359
|
+
const api = {};
|
|
360
|
+
for (const name of METHODS) {
|
|
361
|
+
const call = (...args) => record(siteOf(call), unmutated, name, args);
|
|
362
|
+
api[name] = call;
|
|
363
|
+
}
|
|
364
|
+
return api;
|
|
365
|
+
};
|
|
366
|
+
|
|
367
|
+
const api = bind(false);
|
|
368
|
+
api.unmutated = bind(true);
|
|
369
|
+
return { api, seen };
|
|
370
|
+
}
|
|
371
|
+
|
|
279
372
|
// ------------------------------------------------------------------------- runners
|
|
280
373
|
|
|
281
374
|
async function runOne(invariant, without) {
|
|
282
|
-
const
|
|
375
|
+
const { api, seen } = recorder();
|
|
376
|
+
const { store, handler } = harness(without);
|
|
377
|
+
let threw = null;
|
|
283
378
|
try {
|
|
284
|
-
await invariant.run(
|
|
285
|
-
return null;
|
|
379
|
+
await invariant.run({ store, handler, assert: api });
|
|
286
380
|
} catch (error) {
|
|
287
|
-
|
|
381
|
+
// A real error — a `TypeError` on a row the mutant never wrote, say. It aborts the
|
|
382
|
+
// remaining assertions, and it is still the invariant going red.
|
|
383
|
+
threw = error.message.split('\n')[0];
|
|
288
384
|
}
|
|
385
|
+
const sites = [...seen.values()];
|
|
386
|
+
return {
|
|
387
|
+
threw,
|
|
388
|
+
sites,
|
|
389
|
+
failed: sites.filter((s) => !s.ok),
|
|
390
|
+
broke: Boolean(threw) || sites.some((s) => !s.ok),
|
|
391
|
+
};
|
|
289
392
|
}
|
|
290
393
|
|
|
291
394
|
async function runAll() {
|
|
292
395
|
let failed = 0;
|
|
396
|
+
let asserted = 0;
|
|
293
397
|
for (const invariant of INVARIANTS) {
|
|
294
|
-
const
|
|
295
|
-
|
|
398
|
+
const result = await runOne(invariant, []);
|
|
399
|
+
asserted += result.sites.length;
|
|
400
|
+
if (result.broke) {
|
|
296
401
|
failed += 1;
|
|
297
|
-
console.error(`FAIL ${invariant.id}
|
|
402
|
+
console.error(`FAIL ${invariant.id}`);
|
|
403
|
+
for (const s of result.failed) console.error(` :${s.site} ${s.why}`);
|
|
404
|
+
if (result.threw) console.error(` threw: ${result.threw}`);
|
|
298
405
|
} else {
|
|
299
|
-
console.log(`pass ${invariant.id} — ${invariant.states}`);
|
|
406
|
+
console.log(`pass ${invariant.id} (${result.sites.length} assertions) — ${invariant.states}`);
|
|
300
407
|
}
|
|
301
408
|
}
|
|
302
409
|
if (failed) {
|
|
303
410
|
console.error(`\n${failed} of ${INVARIANTS.length} money invariants failed`);
|
|
304
411
|
return 1;
|
|
305
412
|
}
|
|
306
|
-
console.log(`\nOK: ${INVARIANTS.length} money invariants hold against the handler`
|
|
413
|
+
console.log(`\nOK: ${INVARIANTS.length} money invariants hold against the handler `
|
|
414
|
+
+ `(${asserted} assertions)`);
|
|
307
415
|
return 0;
|
|
308
416
|
}
|
|
309
417
|
|
|
@@ -333,21 +441,49 @@ async function selfTest() {
|
|
|
333
441
|
const mutants = [...RULES.map((rule) => [rule]), ...combos];
|
|
334
442
|
|
|
335
443
|
const matrix = [];
|
|
444
|
+
const coverage = []; // one row per ASSERTION: which mutants turn that call site red
|
|
336
445
|
for (const invariant of INVARIANTS) {
|
|
446
|
+
// The clean run is the census: it names every assertion this invariant executes, and
|
|
447
|
+
// whether each is declared discriminating or shape. An invariant that records nothing
|
|
448
|
+
// is one whose `run` never took the recorder — the failure mode of the injection.
|
|
449
|
+
const clean = await runOne(invariant, []);
|
|
450
|
+
if (clean.broke) {
|
|
451
|
+
problems.push(`${invariant.id}: fails against the unmutated handler `
|
|
452
|
+
+ `(${(clean.failed[0] && `:${clean.failed[0].site} ${clean.failed[0].why}`) || clean.threw})`);
|
|
453
|
+
}
|
|
454
|
+
if (!clean.sites.length) {
|
|
455
|
+
problems.push(`${invariant.id}: recorded no assertion — its run() does not take the `
|
|
456
|
+
+ '`assert` the runner passes it, so nothing about it is measured');
|
|
457
|
+
}
|
|
458
|
+
const broken = new Map(clean.sites.map((s) => [s.site, []]));
|
|
459
|
+
|
|
337
460
|
const singles = [];
|
|
338
461
|
for (const rule of RULES) {
|
|
339
|
-
|
|
462
|
+
const r = await runOne(invariant, [rule]);
|
|
463
|
+
if (r.broke) singles.push(rule);
|
|
464
|
+
for (const s of r.failed) if (broken.has(s.site)) broken.get(s.site).push(rule);
|
|
340
465
|
}
|
|
341
466
|
// Minimal mutants only: a combination containing a rule that already breaks the
|
|
342
467
|
// assertion on its own says nothing new.
|
|
343
468
|
const minimalCombos = [];
|
|
344
469
|
for (const combo of combos) {
|
|
345
470
|
if (combo.some((rule) => singles.includes(rule))) continue;
|
|
346
|
-
|
|
471
|
+
const r = await runOne(invariant, combo);
|
|
472
|
+
if (r.broke) minimalCombos.push(combo);
|
|
473
|
+
for (const s of r.failed) if (broken.has(s.site)) broken.get(s.site).push(key(combo));
|
|
347
474
|
}
|
|
348
475
|
const broke = [...singles.map((r) => [r]), ...minimalCombos];
|
|
349
476
|
matrix.push({ id: invariant.id, broke, singles });
|
|
350
477
|
|
|
478
|
+
for (const site of clean.sites) {
|
|
479
|
+
coverage.push({
|
|
480
|
+
invariant: invariant.id,
|
|
481
|
+
site: site.site,
|
|
482
|
+
unmutated: site.unmutated,
|
|
483
|
+
by: broken.get(site.site) || [],
|
|
484
|
+
});
|
|
485
|
+
}
|
|
486
|
+
|
|
351
487
|
const expected = invariant.breaks
|
|
352
488
|
.map((b) => key(Array.isArray(b) ? b : [b])).sort().join(', ');
|
|
353
489
|
const measured = broke.map(key).sort().join(', ');
|
|
@@ -368,6 +504,27 @@ async function selfTest() {
|
|
|
368
504
|
console.log(` ${row.id.padEnd(width)} ${row.broke.map(key).join(', ') || '(nothing)'}`);
|
|
369
505
|
}
|
|
370
506
|
|
|
507
|
+
// The per-ASSERTION half, and the one the per-invariant table above cannot give: an
|
|
508
|
+
// invariant goes red as a unit, so a dead assertion inside a live one is invisible there.
|
|
509
|
+
console.log('\nassertion x mutant — every call site, and what turns IT red\n');
|
|
510
|
+
for (const row of coverage) {
|
|
511
|
+
const label = `${row.invariant}:${row.site}`.padEnd(width + 6);
|
|
512
|
+
if (row.unmutated) {
|
|
513
|
+
console.log(` ${label} unmutated — no rule varies what it checks`);
|
|
514
|
+
if (row.by.length) {
|
|
515
|
+
problems.push(`${row.invariant} :${row.site} is declared assert.unmutated and `
|
|
516
|
+
+ `[${row.by.join(', ')}] breaks it — it discriminates, so drop the .unmutated`);
|
|
517
|
+
}
|
|
518
|
+
} else {
|
|
519
|
+
console.log(` ${label} ${row.by.join(', ') || '(nothing)'}`);
|
|
520
|
+
if (!row.by.length) {
|
|
521
|
+
problems.push(`${row.invariant} :${row.site} is broken by no mutant — the assertion `
|
|
522
|
+
+ 'cannot fail. Either it is not testing anything, or no rule varies what it '
|
|
523
|
+
+ 'checks and it belongs on assert.unmutated, which the verdict counts apart');
|
|
524
|
+
}
|
|
525
|
+
}
|
|
526
|
+
}
|
|
527
|
+
|
|
371
528
|
console.log('\nisolation — the fixture that separates each rule from every other\n');
|
|
372
529
|
for (const rule of RULES) {
|
|
373
530
|
const isolating = matrix.filter((row) => {
|
|
@@ -392,7 +549,10 @@ async function selfTest() {
|
|
|
392
549
|
for (const p of problems) console.error(` - ${p}`);
|
|
393
550
|
return 1;
|
|
394
551
|
}
|
|
395
|
-
|
|
552
|
+
const pinned = coverage.filter((r) => r.unmutated).length;
|
|
553
|
+
console.log(`\nOK: ${INVARIANTS.length} invariants over ${coverage.length} assertions — `
|
|
554
|
+
+ `${coverage.length - pinned} watched failing ONE CALL SITE AT A TIME against `
|
|
555
|
+
+ `${mutants.length} mutants; ${pinned} declared unmutated and measured unbreakable; `
|
|
396
556
|
+ `${RULES.length} rules, each isolated by a fixture`);
|
|
397
557
|
return 0;
|
|
398
558
|
}
|
|
@@ -158,12 +158,13 @@ the helper works and says nothing about the route Stripe posts to.
|
|
|
158
158
|
## The shipped fixtures and the assertion pack
|
|
159
159
|
|
|
160
160
|
Everything above tells you what to write. `fixtures/` is that suite already written:
|
|
161
|
-
nine Stripe event bodies, twelve assertions, and the mutants that prove
|
|
162
|
-
fail
|
|
161
|
+
nine Stripe event bodies, twelve invariants over 45 assertions, and the mutants that prove
|
|
162
|
+
each assertion can fail — per call site, not per invariant, which is a distinction this
|
|
163
|
+
pack got wrong until 2026-08-20. Copy the directory into your repository and run it.
|
|
163
164
|
|
|
164
165
|
```bash
|
|
165
166
|
node fixtures/assert-money-invariants.mjs # the assertions
|
|
166
|
-
node fixtures/assert-money-invariants.mjs --self-test #
|
|
167
|
+
node fixtures/assert-money-invariants.mjs --self-test # each ASSERTION, watched failing
|
|
167
168
|
```
|
|
168
169
|
|
|
169
170
|
Point it at your handler by replacing the two imports at the top of
|