@autotelic/joggle 0.4.5 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/JOGGLE.md +39 -25
- package/dist/answer-store.js +116 -29
- package/dist/check.js +15 -5
- package/dist/decision.js +1 -0
- package/dist/main.js +74 -6
- package/dist/state.js +23 -0
- package/package.json +1 -1
package/JOGGLE.md
CHANGED
|
@@ -229,10 +229,11 @@ on every machine — which means **run from the repository root**. Evidence
|
|
|
229
229
|
carries root-relative paths; running from a subdirectory changes the key and
|
|
230
230
|
misses every cached verdict. Successful judgements are written to
|
|
231
231
|
`.joggle/answers.jsonl`, one sorted, line-oriented answer per line; commit that
|
|
232
|
-
file and CI replays them with `--offline` and **no API key at all**. A
|
|
233
|
-
answer
|
|
234
|
-
git's built-in `merge=union
|
|
235
|
-
|
|
232
|
+
file and CI replays them with `--offline` and **no API key at all**. A line is the
|
|
233
|
+
key and the answer and nothing else, so two branches that judge different
|
|
234
|
+
candidates merge with git's built-in `merge=union` and the same answer merges to
|
|
235
|
+
one line rather than a copy; `joggle cache compact` and `cache prune --max-bytes`
|
|
236
|
+
keep the store from growing without a ceiling. A store large enough to make a whole-file rewrite or a
|
|
236
237
|
merge expensive can be sharded instead (`cache.layout: "sharded"`, or
|
|
237
238
|
`joggle cache migrate --layout sharded`); both layouts are read, and
|
|
238
239
|
`cache migrate` moves between them. The older `answers.json` object is read and
|
|
@@ -427,15 +428,19 @@ Committing the answer cache buys keyless replay, and it costs bytes in every clo
|
|
|
427
428
|
and every CI checkout. GitHub warns above **50 MB per file** and refuses above
|
|
428
429
|
100 MB; a cache is worth nowhere near that.
|
|
429
430
|
|
|
430
|
-
- `joggle cache prune --max-
|
|
431
|
-
`
|
|
432
|
-
are safe: an answer is regenerable by judging again.
|
|
431
|
+
- `joggle cache prune --max-bytes 4000000` caps the file, oldest first, on any store.
|
|
432
|
+
`--max-age 90` needs ages, which only the machine cache records.
|
|
433
433
|
- A run says so when the store passes 8 MB, because a cache nobody measured is how
|
|
434
434
|
a 0-byte ledger on the base branch becomes 55 MB at the top of a stack.
|
|
435
|
-
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
435
|
+
- **A scoped run does not write the committed ledger.** `--since`, `--pr` and
|
|
436
|
+
`--changed` answer into the machine cache unless `--cache-dir` names a store: a
|
|
437
|
+
branch's verdicts are a cache, and committing them is what put 131,440 lines in
|
|
438
|
+
every pull request of a stack instead of the 5,890 answers they held.
|
|
439
|
+
- **`joggle cache compact` repairs a duplicated ledger in place.** A line used to
|
|
440
|
+
carry `writtenAt`, so the same answer serialised differently on every branch and
|
|
441
|
+
`merge=union` kept both sides on every rebase. The line is now the key and the
|
|
442
|
+
answer only, so merges collapse to one line; for a history that already grew,
|
|
443
|
+
compact reads each key once and rewrites the set. No re-judging, no key needed.
|
|
439
444
|
|
|
440
445
|
## Two committed artifacts, two machine artifacts
|
|
441
446
|
|
|
@@ -488,23 +493,32 @@ full-repository budget: at ~2.5k candidates a keyed run, a repository with 12k
|
|
|
488
493
|
unjudged candidates is a 25-minute run that still refuses to write.
|
|
489
494
|
|
|
490
495
|
```sh
|
|
491
|
-
|
|
496
|
+
# the gate: what this branch adds, judged
|
|
497
|
+
joggle check --types --no-replay --since origin/<parent> --baseline .joggle/baseline
|
|
498
|
+
```
|
|
499
|
+
|
|
500
|
+
**A scoped run is for the gate. Acceptance is a full-run act**, and the command
|
|
501
|
+
enforces it: `--update-baseline` refuses a scoped run unless
|
|
502
|
+
`--allow-scoped-baseline` says otherwise.
|
|
503
|
+
|
|
504
|
+
The reason is the fold. Two findings at the same site with the same move are folded
|
|
505
|
+
into the richer one, and which survivor wins depends on what ran: a scoped run sees
|
|
506
|
+
fewer candidates at that site than a full run does, so it can name the problem with
|
|
507
|
+
one rule where a full run names it with another. Identities are therefore not
|
|
508
|
+
scope-invariant, a scoped accept can miss exactly the identity CI fails on, and no
|
|
509
|
+
amount of `--merge-baseline` repairs a ledger that never saw it. (The fix is aliases
|
|
510
|
+
on a folded survivor, so the ledger can match any of them; until that lands, the
|
|
511
|
+
refusal is the honest state.) Accept with one full run at the head of a stack:
|
|
512
|
+
|
|
513
|
+
```sh
|
|
514
|
+
joggle check --types --no-replay \
|
|
492
515
|
--baseline .joggle/baseline --update-baseline .joggle/baseline --merge-baseline
|
|
493
516
|
```
|
|
494
517
|
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
ledger. `--allow-unjudged-baseline` is not the shortcut here: it ratifies the
|
|
500
|
-
thousands nobody looked at, which is a commit nobody can defend.
|
|
501
|
-
|
|
502
|
-
A scoped accept is necessary, not sufficient. Cross-file rules only fire over the
|
|
503
|
-
whole repository, and a scoped run folds some findings into another rule at the same
|
|
504
|
-
site (the note counts them), so an identity CI fails on can be one this run never had.
|
|
505
|
-
Use the scoped accept to keep a stack moving and treat the full run as the authority:
|
|
506
|
-
when CI names an identity the scoped ledger lacks, accept that one and move on. Budget
|
|
507
|
-
one full judged run for the stack's head, not one per branch.
|
|
518
|
+
`--merge-baseline` keeps the identities already accepted and adds this run's, so
|
|
519
|
+
accepting at the head does not discard the stack's accepted set. `--allow-unjudged-baseline`
|
|
520
|
+
is not the shortcut here: it ratifies thousands nobody looked at, which is a commit
|
|
521
|
+
nobody can defend.
|
|
508
522
|
|
|
509
523
|
### The cascade, not the conflicts
|
|
510
524
|
|
package/dist/answer-store.js
CHANGED
|
@@ -4,9 +4,17 @@ import { StoredAnswer } from "./plans.js";
|
|
|
4
4
|
/**
|
|
5
5
|
* The answer cache: one logical ledger, two on-disk layouts.
|
|
6
6
|
*
|
|
7
|
-
* An entry is a line of `<key>\t<
|
|
8
|
-
*
|
|
9
|
-
*
|
|
7
|
+
* An entry is a line of `<key>\t<json>`, sorted by key. The key
|
|
8
|
+
* (`answerKeyFor`) already content-addresses the decision, the atoms it read,
|
|
9
|
+
* the model and the decision version, so the line IS the identity — and a line
|
|
10
|
+
* that is a function of the key is a line two branches can merge.
|
|
11
|
+
*
|
|
12
|
+
* The timestamp is deliberately NOT in that line. A committed ledger is merged
|
|
13
|
+
* with git's built-in `merge=union`, which keeps both sides of a differing line;
|
|
14
|
+
* stamping every copy with the wall clock made the same answer a different line in
|
|
15
|
+
* every clone, so each rebase of a 21-branch stack duplicated it again — 5,890
|
|
16
|
+
* answers became 131,440 lines, 20.5 MB for 0.9 MB of content. Age is a machine
|
|
17
|
+
* fact: the machine cache records it, the committed ledger does not.
|
|
10
18
|
*
|
|
11
19
|
* **Single** (the default) is one file, `answers.jsonl`. It is the shape a
|
|
12
20
|
* reviewer can read: a verdict change is one changed line in one diff. Sharding
|
|
@@ -69,10 +77,14 @@ const AnswersObject = Schema.Struct({
|
|
|
69
77
|
});
|
|
70
78
|
const decodeAnswer = Schema.fromJsonString(StoredAnswer);
|
|
71
79
|
/**
|
|
72
|
-
* One line: `<key>\t<unix-seconds>\t<json
|
|
73
|
-
*
|
|
80
|
+
* One line: `<key>\t<json>`, or `<key>\t<unix-seconds>\t<json>` when the store
|
|
81
|
+
* records ages. A tab cannot appear unescaped in any field, and a payload always
|
|
82
|
+
* starts with `{`, so the split is unambiguous.
|
|
74
83
|
*/
|
|
75
|
-
export const renderEntry = (entry) => entry.key +
|
|
84
|
+
export const renderEntry = (entry) => entry.key +
|
|
85
|
+
"\t" +
|
|
86
|
+
(entry.writtenAt === undefined ? "" : entry.writtenAt + "\t") +
|
|
87
|
+
canonical(entry.answer);
|
|
76
88
|
/** Parse a ledger file. A line that does not decode is skipped and COUNTED, never fatal. */
|
|
77
89
|
export const parseLines = (text) => {
|
|
78
90
|
const entries = [];
|
|
@@ -81,18 +93,24 @@ export const parseLines = (text) => {
|
|
|
81
93
|
if (line === "")
|
|
82
94
|
continue;
|
|
83
95
|
const first = line.indexOf("\t");
|
|
84
|
-
|
|
85
|
-
if (first < 0 || second < 0) {
|
|
96
|
+
if (first < 0) {
|
|
86
97
|
skippedLines += 1;
|
|
87
98
|
continue;
|
|
88
99
|
}
|
|
89
|
-
const
|
|
90
|
-
|
|
91
|
-
|
|
100
|
+
const rest = line.slice(first + 1);
|
|
101
|
+
// A payload is JSON and starts with `{`; anything else is a legacy age field.
|
|
102
|
+
const stamped = /^(\d+)\t/.exec(rest);
|
|
103
|
+
const writtenAt = stamped === null ? undefined : Number(stamped[1]);
|
|
104
|
+
const payload = stamped === null ? rest : rest.slice(stamped[0].length);
|
|
105
|
+
const answer = Result.getOrUndefined(SchemaParser.decodeResult(decodeAnswer)(payload));
|
|
106
|
+
if (answer === undefined || (writtenAt !== undefined && !Number.isFinite(writtenAt))) {
|
|
92
107
|
skippedLines += 1;
|
|
93
108
|
continue;
|
|
94
109
|
}
|
|
95
|
-
|
|
110
|
+
// Two shapes rather than a spread or a widened binding: an entry either has an age
|
|
111
|
+
// (machine store) or it does not (committed ledger).
|
|
112
|
+
const key = line.slice(0, first);
|
|
113
|
+
entries.push(writtenAt === undefined ? { key, answer } : { key, writtenAt, answer });
|
|
96
114
|
}
|
|
97
115
|
return { entries, skippedLines };
|
|
98
116
|
};
|
|
@@ -103,12 +121,32 @@ export const renderShard = (entries) => {
|
|
|
103
121
|
return sorted.length === 0 ? "" : sorted.map(renderEntry).join("\n") + "\n";
|
|
104
122
|
};
|
|
105
123
|
/**
|
|
106
|
-
*
|
|
124
|
+
* Which of two entries for one key survives.
|
|
125
|
+
*
|
|
126
|
+
* An age is evidence of what the model produced last, so a stamped entry wins over an
|
|
127
|
+
* unstamped one. Two unstamped entries with one key are the same answer by construction.
|
|
128
|
+
*
|
|
129
|
+
* @param pair.left - The existing entry.
|
|
130
|
+
* @param pair.right - The challenger.
|
|
131
|
+
* @returns The entry that stands.
|
|
132
|
+
*/
|
|
133
|
+
const newer = ({ left, right }) => {
|
|
134
|
+
if (left.writtenAt === undefined)
|
|
135
|
+
return right;
|
|
136
|
+
if (right.writtenAt === undefined)
|
|
137
|
+
return left;
|
|
138
|
+
return right.writtenAt > left.writtenAt ? right : left;
|
|
139
|
+
};
|
|
140
|
+
/**
|
|
141
|
+
* Union two sides of one ledger by key.
|
|
107
142
|
*
|
|
108
|
-
* A key
|
|
109
|
-
*
|
|
110
|
-
*
|
|
111
|
-
*
|
|
143
|
+
* A key only one side carries is kept. A key both sides carry keeps the newer answer when
|
|
144
|
+
* ages exist, and otherwise ours — for a content-addressed key the payloads are identical,
|
|
145
|
+
* which is what makes a merge collapse to one line instead of copying it.
|
|
146
|
+
*
|
|
147
|
+
* @param input.ours - Our side.
|
|
148
|
+
* @param input.theirs - Their side.
|
|
149
|
+
* @returns One entry per key.
|
|
112
150
|
*/
|
|
113
151
|
export const mergeEntries = ({ ours, theirs, }) => {
|
|
114
152
|
const merged = new Map();
|
|
@@ -116,8 +154,10 @@ export const mergeEntries = ({ ours, theirs, }) => {
|
|
|
116
154
|
merged.set(entry.key, entry);
|
|
117
155
|
for (const entry of theirs) {
|
|
118
156
|
const existing = merged.get(entry.key);
|
|
119
|
-
if (existing === undefined
|
|
157
|
+
if (existing === undefined)
|
|
120
158
|
merged.set(entry.key, entry);
|
|
159
|
+
else
|
|
160
|
+
merged.set(entry.key, newer({ left: existing, right: entry }));
|
|
121
161
|
}
|
|
122
162
|
return [...merged.values()];
|
|
123
163
|
};
|
|
@@ -168,10 +208,15 @@ const loadAll = (fs, path, dir) => Effect.gen(function* () {
|
|
|
168
208
|
const merged = new Map();
|
|
169
209
|
const absorb = (entry) => {
|
|
170
210
|
const existing = merged.get(entry.key);
|
|
171
|
-
if (existing === undefined
|
|
211
|
+
if (existing === undefined)
|
|
172
212
|
merged.set(entry.key, entry);
|
|
213
|
+
else
|
|
214
|
+
merged.set(entry.key, newer({ left: existing, right: entry }));
|
|
173
215
|
};
|
|
174
216
|
let files = 0;
|
|
217
|
+
// Lines and skipped lines together are what the file held; the difference from
|
|
218
|
+
// `merged.size` is duplication a union merge left behind.
|
|
219
|
+
let lines = 0;
|
|
175
220
|
let bytes = 0;
|
|
176
221
|
let skippedLines = 0;
|
|
177
222
|
const older = answersObject(path, dir);
|
|
@@ -193,6 +238,7 @@ const loadAll = (fs, path, dir) => Effect.gen(function* () {
|
|
|
193
238
|
files += 1;
|
|
194
239
|
bytes += bytesOf(text);
|
|
195
240
|
skippedLines += parsed.skippedLines;
|
|
241
|
+
lines += parsed.entries.length + parsed.skippedLines;
|
|
196
242
|
for (const entry of parsed.entries)
|
|
197
243
|
absorb(entry);
|
|
198
244
|
}
|
|
@@ -210,6 +256,7 @@ const loadAll = (fs, path, dir) => Effect.gen(function* () {
|
|
|
210
256
|
files += 1;
|
|
211
257
|
bytes += bytesOf(text);
|
|
212
258
|
skippedLines += parsed.skippedLines;
|
|
259
|
+
lines += parsed.entries.length + parsed.skippedLines;
|
|
213
260
|
for (const entry of parsed.entries)
|
|
214
261
|
absorb(entry);
|
|
215
262
|
}
|
|
@@ -219,6 +266,7 @@ const loadAll = (fs, path, dir) => Effect.gen(function* () {
|
|
|
219
266
|
entries: merged,
|
|
220
267
|
layout: shards > 0 ? "sharded" : "single",
|
|
221
268
|
files,
|
|
269
|
+
lines,
|
|
222
270
|
bytes,
|
|
223
271
|
skippedLines,
|
|
224
272
|
};
|
|
@@ -299,11 +347,18 @@ export const make = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
|
299
347
|
const entries = yield* Ref.make(loaded.entries);
|
|
300
348
|
const dirty = yield* Ref.make(new Set());
|
|
301
349
|
const layoutRef = yield* Ref.make(chosen);
|
|
302
|
-
const
|
|
350
|
+
const aged = [...loaded.entries.values()].some((entry) => entry.writtenAt !== undefined);
|
|
351
|
+
const fullWrite = yield* Ref.make(loaded.layout !== chosen || (options.timestamps !== true && aged));
|
|
303
352
|
const writer = yield* Semaphore.make(1);
|
|
353
|
+
// A ledger written before the line was stable carries ages; a store that does not record
|
|
354
|
+
// ages rewrites them away on its first write, so a merge stops seeing two shapes of one key.
|
|
355
|
+
const strip = options.timestamps !== true;
|
|
356
|
+
const sanitise = (current) => strip
|
|
357
|
+
? new Map([...current].map(([key, entry]) => [key, { key, answer: entry.answer }]))
|
|
358
|
+
: current;
|
|
304
359
|
const flush = writer.withPermits(1)(Effect.gen(function* () {
|
|
305
360
|
const changed = yield* Ref.getAndSet(dirty, new Set());
|
|
306
|
-
const current = yield* Ref.get(entries);
|
|
361
|
+
const current = sanitise(yield* Ref.get(entries));
|
|
307
362
|
const layout = yield* Ref.get(layoutRef);
|
|
308
363
|
const rewrite = yield* Ref.getAndSet(fullWrite, false);
|
|
309
364
|
if (rewrite || layout === "single") {
|
|
@@ -326,7 +381,9 @@ export const make = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
|
326
381
|
return {
|
|
327
382
|
get: (key) => Effect.map(Ref.get(entries), (current) => Option.fromUndefinedOr(current.get(key)?.answer)),
|
|
328
383
|
put: (key, answer) => Effect.gen(function* () {
|
|
329
|
-
const entry =
|
|
384
|
+
const entry = options.timestamps === true
|
|
385
|
+
? { key, writtenAt: seconds(yield* Clock.currentTimeMillis), answer }
|
|
386
|
+
: { key, answer };
|
|
330
387
|
yield* Ref.update(entries, (current) => new Map(current).set(key, entry));
|
|
331
388
|
yield* Ref.update(dirty, (current) => new Set(current).add(key));
|
|
332
389
|
yield* flush;
|
|
@@ -384,8 +441,10 @@ export const census = (fs, path, dir) => Effect.gen(function* () {
|
|
|
384
441
|
let oldest;
|
|
385
442
|
let newest;
|
|
386
443
|
for (const entry of loaded.entries.values()) {
|
|
387
|
-
|
|
388
|
-
|
|
444
|
+
if (entry.writtenAt !== undefined) {
|
|
445
|
+
oldest = oldest === undefined ? entry.writtenAt : Math.min(oldest, entry.writtenAt);
|
|
446
|
+
newest = newest === undefined ? entry.writtenAt : Math.max(newest, entry.writtenAt);
|
|
447
|
+
}
|
|
389
448
|
kinds.set(entry.answer.kind, (kinds.get(entry.answer.kind) ?? 0) + 1);
|
|
390
449
|
if (entry.answer.kind === "Probability") {
|
|
391
450
|
if (entry.answer.probability >= 0.5)
|
|
@@ -399,6 +458,7 @@ export const census = (fs, path, dir) => Effect.gen(function* () {
|
|
|
399
458
|
}
|
|
400
459
|
return {
|
|
401
460
|
entries: loaded.entries.size,
|
|
461
|
+
lines: loaded.lines,
|
|
402
462
|
files: loaded.files,
|
|
403
463
|
layout: loaded.layout,
|
|
404
464
|
bytes: loaded.bytes,
|
|
@@ -417,9 +477,11 @@ export const census = (fs, path, dir) => Effect.gen(function* () {
|
|
|
417
477
|
/**
|
|
418
478
|
* Drop cache entries and rewrite only the file or shards that changed.
|
|
419
479
|
*
|
|
420
|
-
* The age cut is per entry, from the timestamp each line carries
|
|
421
|
-
*
|
|
422
|
-
*
|
|
480
|
+
* The age cut is per entry, from the timestamp each line carries — a committed
|
|
481
|
+
* ledger carries none, so age pruning leaves it alone and only `--max-bytes`
|
|
482
|
+
* retires what it holds. The size cut is oldest-first across the whole ledger,
|
|
483
|
+
* with ageless entries last. An emptied file or shard is deleted rather than
|
|
484
|
+
* written empty.
|
|
423
485
|
*/
|
|
424
486
|
export const prune = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
425
487
|
const now = options.now ?? seconds(yield* Clock.currentTimeMillis);
|
|
@@ -428,7 +490,8 @@ export const prune = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
|
428
490
|
const kept = new Map();
|
|
429
491
|
let removed = 0;
|
|
430
492
|
for (const entry of loaded.entries.values()) {
|
|
431
|
-
|
|
493
|
+
// A store that does not record ages cannot be pruned by age: absent is not "old".
|
|
494
|
+
if (cutoff !== undefined && entry.writtenAt !== undefined && entry.writtenAt < cutoff) {
|
|
432
495
|
removed += 1;
|
|
433
496
|
continue;
|
|
434
497
|
}
|
|
@@ -441,7 +504,8 @@ export const prune = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
|
441
504
|
if (total > options.maxBytes) {
|
|
442
505
|
// A total order: oldest first, ties by key, so the result does not
|
|
443
506
|
// depend on directory iteration.
|
|
444
|
-
const
|
|
507
|
+
const age = (entry) => entry.writtenAt ?? Number.POSITIVE_INFINITY;
|
|
508
|
+
const ordered = [...kept.values()].sort((left, right) => age(left) - age(right) || byKey(left, right));
|
|
445
509
|
for (const entry of ordered) {
|
|
446
510
|
if (total <= options.maxBytes)
|
|
447
511
|
break;
|
|
@@ -460,3 +524,26 @@ export const prune = (fs, path, dir, options = {}) => Effect.gen(function* () {
|
|
|
460
524
|
bytes += sizeOf(entry);
|
|
461
525
|
return { removed, kept: kept.size, bytes };
|
|
462
526
|
});
|
|
527
|
+
/**
|
|
528
|
+
* Rewrite the ledger as the set it is: one sorted line per key.
|
|
529
|
+
*
|
|
530
|
+
* `merge=union` keeps both sides of a *differing* line, so before the line was a function of
|
|
531
|
+
* the key alone, every rebase of a stack copied the same answers again with a new stamp. The
|
|
532
|
+
* repair is not a prune (the copies are not stale) and not a migration (the layout is right):
|
|
533
|
+
* read every key once, newest answer winning, and write the file a fresh run would write.
|
|
534
|
+
*/
|
|
535
|
+
export const compact = (fs, path, dir) => Effect.gen(function* () {
|
|
536
|
+
const bytesBefore = yield* answerStoreBytes(fs, path, dir);
|
|
537
|
+
const loaded = yield* loadAll(fs, path, dir);
|
|
538
|
+
// Ages are a machine fact: the repaired file is the set, in the one committed shape.
|
|
539
|
+
const set = new Map([...loaded.entries].map(([key, entry]) => [key, { key, answer: entry.answer }]));
|
|
540
|
+
yield* writeAll(fs, path, dir, loaded.layout, set);
|
|
541
|
+
const bytesAfter = yield* answerStoreBytes(fs, path, dir);
|
|
542
|
+
return {
|
|
543
|
+
lines: loaded.lines,
|
|
544
|
+
entries: loaded.entries.size,
|
|
545
|
+
duplicates: Math.max(0, loaded.lines - loaded.entries.size),
|
|
546
|
+
bytesBefore,
|
|
547
|
+
bytesAfter,
|
|
548
|
+
};
|
|
549
|
+
});
|
package/dist/check.js
CHANGED
|
@@ -227,6 +227,18 @@ export const checkRepository = Effect.fn("joggle.check")(function* (options) {
|
|
|
227
227
|
});
|
|
228
228
|
}
|
|
229
229
|
if (options.updateBaselinePath !== undefined) {
|
|
230
|
+
const target = baselinePath(path, options.updateBaselinePath);
|
|
231
|
+
// A scoped run is for the gate. Acceptance is a full-run act until folded findings
|
|
232
|
+
// carry the identities they absorbed, because which survivor wins a fold depends on
|
|
233
|
+
// what ran -- so a scoped accept can miss the identity a full run reports, silently.
|
|
234
|
+
const scoped = options.changed === true || options.changedPaths !== undefined;
|
|
235
|
+
if (scoped && options.allowScopedBaseline !== true) {
|
|
236
|
+
return yield* WorkspaceError.make({
|
|
237
|
+
path: options.updateBaselinePath,
|
|
238
|
+
operation: "baseline",
|
|
239
|
+
cause: new Error(`refusing to write ${target} for a scoped run: a scoped run's identities are not a subset of a full run's — the same site can fold to a different survivor when candidates outside the scope are missing — so this ledger could be missing the identity a full run reports. Accept with one full run (at the head of a stack), or pass --allow-scoped-baseline to accept the risk. Unscoped runs are unaffected.`),
|
|
240
|
+
});
|
|
241
|
+
}
|
|
230
242
|
if (unjudged > 0 && options.allowUnjudgedBaseline !== true) {
|
|
231
243
|
return yield* WorkspaceError.make({
|
|
232
244
|
path: options.updateBaselinePath,
|
|
@@ -235,13 +247,11 @@ export const checkRepository = Effect.fn("joggle.check")(function* (options) {
|
|
|
235
247
|
"Judging needs a model key in this run's environment (TYPESAFE_API_KEY); without one, " +
|
|
236
248
|
"the same findings are named from what the rule could see alone, so a ledger written " +
|
|
237
249
|
"now would not match a judged run. Judge first (raise --max-tokens if the budget was " +
|
|
238
|
-
"the limit), or
|
|
239
|
-
"
|
|
240
|
-
"
|
|
241
|
-
"Pass --allow-unjudged-baseline only to accept the unjudged debt itself."),
|
|
250
|
+
"the limit), or run scoped (--since <base>) to see the change judged with a pull " +
|
|
251
|
+
"request's budget — a scoped run reports the gate's answer but cannot write the " +
|
|
252
|
+
"ledger. Pass --allow-unjudged-baseline only to accept the unjudged debt itself."),
|
|
242
253
|
});
|
|
243
254
|
}
|
|
244
|
-
const target = baselinePath(path, options.updateBaselinePath);
|
|
245
255
|
const fresh = report.diagnostics
|
|
246
256
|
.map((entry) => entry.identity)
|
|
247
257
|
.filter((identity) => identity !== undefined);
|
package/dist/decision.js
CHANGED
|
@@ -185,6 +185,7 @@ export const layer = (options) => Layer.effectContext(Effect.gen(function* () {
|
|
|
185
185
|
// `docs/artifacts.md`.
|
|
186
186
|
const answers = yield* makeAnswerStore(fs, path, options.cacheDir, {
|
|
187
187
|
layout: options.layout,
|
|
188
|
+
timestamps: options.timestamps,
|
|
188
189
|
});
|
|
189
190
|
const planAnswers = {
|
|
190
191
|
get: (key) => Effect.map(answers.get(key), (stored) => Option.map(stored, answerOf)),
|
package/dist/main.js
CHANGED
|
@@ -5,9 +5,9 @@ import * as FetchHttpClient from "effect/http/FetchHttpClient";
|
|
|
5
5
|
import { NodeRuntime, NodeServices } from "@effect/platform-node";
|
|
6
6
|
import { checkRepository } from "./check.js";
|
|
7
7
|
import { layer as decisionLayer } from "./decision.js";
|
|
8
|
-
import { census as censusAnswers, make as makeAnswerStore, mergeShardText, prune as pruneAnswers, } from "./answer-store.js";
|
|
8
|
+
import { census as censusAnswers, compact as compactAnswers, make as makeAnswerStore, mergeShardText, prune as pruneAnswers, } from "./answer-store.js";
|
|
9
9
|
import { ask } from "./ask.js";
|
|
10
|
-
import { cacheDirFor, runCacheDirFor, shortHash } from "./state.js";
|
|
10
|
+
import { cacheDirFor, runCacheDirFor, scopedAnswerCacheDir, shortHash } from "./state.js";
|
|
11
11
|
import { canonical } from "./canonical.js";
|
|
12
12
|
import { loadConfig, isEnabled, severityFor } from "./config.js";
|
|
13
13
|
import { layer as gitLayer, Service as Git } from "./git.js";
|
|
@@ -88,6 +88,7 @@ const checkCommand = Command.make("check", {
|
|
|
88
88
|
baseline: Flag.String("baseline").pipe(Flag.withDescription("Report only findings not already accepted in this baseline file."), Flag.optional),
|
|
89
89
|
updateBaseline: Flag.String("update-baseline").pipe(Flag.withDescription("Write the current findings to this file as the accepted baseline."), Flag.optional),
|
|
90
90
|
mergeBaseline: Flag.Boolean("merge-baseline").pipe(Flag.withDescription("With --update-baseline, keep the identities already in that file and add this run's. A scoped run (--since) judges all of its own files, so this grows a ledger by one change instead of replacing it."), Flag.withDefault(false)),
|
|
91
|
+
allowScopedBaseline: Flag.Boolean("allow-scoped-baseline").pipe(Flag.withDescription("Write a ledger from a scoped run. Refused by default: a scoped run's identities are not a subset of a full run's, so the ledger can miss the identity CI reports. Accept from a full run instead."), Flag.withDefault(false)),
|
|
91
92
|
allowUnjudgedBaseline: Flag.Boolean("allow-unjudged-baseline").pipe(Flag.withDescription("Write the baseline even though some candidates were not judged. A baseline is what was reviewed; this is the explicit way to accept that some of it was not."), Flag.withDefault(false)),
|
|
92
93
|
cacheDir: Flag.String("cache-dir").pipe(Flag.withDescription("Where the answer cache lives: <cwd>/.joggle when it exists (an onboarded repository), otherwise the machine cache."), Flag.optional),
|
|
93
94
|
cwd: Flag.String("cwd").pipe(Flag.withDescription("Project root to analyse (default: the current directory)."), Flag.optional),
|
|
@@ -98,7 +99,8 @@ const checkCommand = Command.make("check", {
|
|
|
98
99
|
// file list against the root, git's changed files against the workspace --
|
|
99
100
|
// and a relative root makes those comparisons fail quietly.
|
|
100
101
|
const cwd = path.resolve(Option.getOrUndefined(config.cwd) ?? process.cwd());
|
|
101
|
-
const
|
|
102
|
+
const declaredCacheDir = Option.getOrUndefined(config.cacheDir);
|
|
103
|
+
const repositoryCache = yield* cacheDirFor(cwd, declaredCacheDir);
|
|
102
104
|
const apiKey = yield* Config.option(Config.String("TYPESAFE_API_KEY"));
|
|
103
105
|
const ruleFlag = Option.getOrUndefined(config.rule);
|
|
104
106
|
const rules = ruleFlag === undefined
|
|
@@ -137,6 +139,16 @@ const checkCommand = Command.make("check", {
|
|
|
137
139
|
// The machine cache holds the run cache and the wire cache. The
|
|
138
140
|
// per-decision answer cache in `cacheDir` is the one CI replays.
|
|
139
141
|
const machineCache = yield* runCacheDirFor(cwd);
|
|
142
|
+
// A branch's answers are a cache, not a commit: a scoped run warms this machine and
|
|
143
|
+
// leaves the committed ledger alone. Otherwise every pull request in a stack carries
|
|
144
|
+
// thousands of lines of verdicts nobody can review (see docs/artifacts.md).
|
|
145
|
+
const scopedRun = config.changed || changedPaths !== undefined;
|
|
146
|
+
const cacheDir = scopedAnswerCacheDir({
|
|
147
|
+
declared: declaredCacheDir,
|
|
148
|
+
repository: repositoryCache,
|
|
149
|
+
machine: machineCache,
|
|
150
|
+
scoped: scopedRun,
|
|
151
|
+
});
|
|
140
152
|
const report = yield* checkRepository({
|
|
141
153
|
config: effective,
|
|
142
154
|
plugins: loaded,
|
|
@@ -165,18 +177,35 @@ const checkCommand = Command.make("check", {
|
|
|
165
177
|
updateBaselinePath: Option.getOrUndefined(config.updateBaseline),
|
|
166
178
|
allowUnjudgedBaseline: config.allowUnjudgedBaseline,
|
|
167
179
|
mergeBaseline: config.mergeBaseline,
|
|
180
|
+
allowScopedBaseline: config.allowScopedBaseline,
|
|
168
181
|
}).pipe(Effect.provide(Layer.mergeAll(rulesLayer, decisionLayer({
|
|
169
182
|
cacheDir,
|
|
170
183
|
wireCacheDir: machineCache,
|
|
171
184
|
offline: config.offline,
|
|
172
185
|
apiKey,
|
|
173
186
|
layout: effective.cache?.layout,
|
|
187
|
+
timestamps: cacheDir === machineCache,
|
|
174
188
|
}), tsgoLayer(cwd, {
|
|
175
189
|
runExternalCode: config.runExternalCode || effective.runExternalCode === true,
|
|
176
190
|
}))));
|
|
177
|
-
|
|
191
|
+
// Say where the answers went when the run left the committed ledger alone: a reader
|
|
192
|
+
// who expected the cache to grow would otherwise think judging did nothing.
|
|
193
|
+
const answered = scopedRun && declaredCacheDir === undefined && repositoryCache !== machineCache;
|
|
194
|
+
const reported = answered
|
|
195
|
+
? {
|
|
196
|
+
...report,
|
|
197
|
+
notes: [
|
|
198
|
+
{
|
|
199
|
+
ruleId: "joggle",
|
|
200
|
+
reason: "answers for this scoped run went to the machine cache, not the committed ledger: a branch's verdicts are a cache, and committing them puts thousands of lines in every pull request",
|
|
201
|
+
},
|
|
202
|
+
...report.notes,
|
|
203
|
+
],
|
|
204
|
+
}
|
|
205
|
+
: report;
|
|
206
|
+
yield* write(render(reported, config.format, { color: process.stdout.isTTY === true }));
|
|
178
207
|
yield* Effect.sync(() => {
|
|
179
|
-
process.exitCode = exitCodeFor(
|
|
208
|
+
process.exitCode = exitCodeFor(reported, config.maxWarnings);
|
|
180
209
|
});
|
|
181
210
|
}).pipe(Effect.provide(gitLayer))).pipe(Command.withDescription("Report cross-file duplication and naming drift. With no paths, analyse the project tsgo sees. --since, --pr and --pr-number answer what a change introduced; --changed does the same against the last stored run."), Command.withShortDescription("Report cross-file duplication and naming drift."), Command.withExamples([
|
|
182
211
|
{ command: "joggle check", description: "Analyse the project tsgo sees." },
|
|
@@ -590,11 +619,50 @@ const cacheReport = Command.make("report", {
|
|
|
590
619
|
" verdicts Probability " + found.verdicts.yes + " yes / " + found.verdicts.no + " no",
|
|
591
620
|
" labels " + (labels === "" ? "none" : labels),
|
|
592
621
|
" written " + date(found.oldest) + " .. " + date(found.newest),
|
|
622
|
+
...(found.lines > found.entries
|
|
623
|
+
? [
|
|
624
|
+
" duplicates " +
|
|
625
|
+
(found.lines - found.entries) +
|
|
626
|
+
" line(s) over " +
|
|
627
|
+
found.entries +
|
|
628
|
+
" answer(s); `joggle cache compact` rewrites the set",
|
|
629
|
+
]
|
|
630
|
+
: []),
|
|
593
631
|
...(found.skippedLines === 0 ? [] : [" skipped " + found.skippedLines + " unreadable line(s)"]),
|
|
594
632
|
].join("\n"));
|
|
595
633
|
})).pipe(Command.withDescription("Render the committed answer ledger for a reader: entry and file counts, layout, verdict distribution, labels, and the date range. A summary cannot be replayed; this is what to read when reviewing the diff."), Command.withShortDescription("Report on the answer cache."), Command.withExamples([
|
|
596
634
|
{ command: "joggle cache report", description: "Count what the committed ledger holds." },
|
|
597
635
|
]));
|
|
636
|
+
const cacheCompact = Command.make("compact", {
|
|
637
|
+
cwd: Flag.String("cwd").pipe(Flag.withDescription("Project root whose cache to compact (default: the current directory)."), Flag.optional),
|
|
638
|
+
cacheDir: Flag.String("cache-dir").pipe(Flag.withDescription("Where the answer cache lives: <cwd>/.joggle when it exists, otherwise the machine cache."), Flag.optional),
|
|
639
|
+
}, (config) => Effect.gen(function* () {
|
|
640
|
+
const fs = yield* FileSystem.FileSystem;
|
|
641
|
+
const path = yield* Path.Path;
|
|
642
|
+
const cwd = path.resolve(Option.getOrUndefined(config.cwd) ?? process.cwd());
|
|
643
|
+
const dir = yield* cacheDirFor(cwd, Option.getOrUndefined(config.cacheDir));
|
|
644
|
+
const result = yield* compactAnswers(fs, path, dir);
|
|
645
|
+
if (result.duplicates === 0) {
|
|
646
|
+
yield* write("joggle: the ledger is already a set: " + result.entries + " answer(s), " + result.bytesAfter + " bytes");
|
|
647
|
+
return;
|
|
648
|
+
}
|
|
649
|
+
yield* write("joggle: " +
|
|
650
|
+
result.lines +
|
|
651
|
+
" line(s) held " +
|
|
652
|
+
result.entries +
|
|
653
|
+
" answer(s); compacted " +
|
|
654
|
+
result.duplicates +
|
|
655
|
+
" duplicate(s) away, " +
|
|
656
|
+
result.bytesBefore +
|
|
657
|
+
" -> " +
|
|
658
|
+
result.bytesAfter +
|
|
659
|
+
" bytes");
|
|
660
|
+
})).pipe(Command.withDescription("Rewrite the answer ledger as the set it is: one sorted line per key, newest answer winning. Repairs the duplication a union merge leaves behind (a differing line is kept from both sides), without re-judging anything."), Command.withShortDescription("Deduplicate the answer cache."), Command.withExamples([
|
|
661
|
+
{
|
|
662
|
+
command: "joggle cache compact",
|
|
663
|
+
description: "Collapse duplicate lines left by a stack's merges.",
|
|
664
|
+
},
|
|
665
|
+
]));
|
|
598
666
|
const cacheMigrate = Command.make("migrate", {
|
|
599
667
|
cwd: Flag.String("cwd").pipe(Flag.withDescription("Project root whose cache to migrate (default: the current directory)."), Flag.optional),
|
|
600
668
|
cacheDir: Flag.String("cache-dir").pipe(Flag.withDescription("Where the answer cache lives: <cwd>/.joggle when it exists, otherwise the machine cache."), Flag.optional),
|
|
@@ -609,7 +677,7 @@ const cacheMigrate = Command.make("migrate", {
|
|
|
609
677
|
const target = config.layout === "single" ? path.join(dir, "answers.jsonl") : path.join(dir, "answers");
|
|
610
678
|
yield* write("joggle: wrote " + written + " answer(s) to " + target);
|
|
611
679
|
})).pipe(Command.withDescription("Rewrite the cache in one layout and remove the other: a single sorted answers.jsonl by default, or shards with --layout sharded. Idempotent."), Command.withShortDescription("Consolidate or shard the answer cache."));
|
|
612
|
-
const cacheCommand = Command.make("cache", {}, () => write("joggle cache: run `joggle cache report`, `prune`, or `migrate --layout single|sharded`; docs/artifacts.md has the reasoning.")).pipe(Command.withDescription("Work on the answer cache itself: prune it, migrate it, and see how it is laid out."), Command.withShortDescription("
|
|
680
|
+
const cacheCommand = Command.make("cache", {}, () => write("joggle cache: run `joggle cache compact`, `report`, `prune`, or `migrate --layout single|sharded`; docs/artifacts.md has the reasoning.")).pipe(Command.withDescription("Work on the answer cache itself: compact it, prune it, migrate it, and see how it is laid out."), Command.withShortDescription("Work on the answer cache."), Command.withSubcommands([cacheCompact, cachePrune, cacheMigrate, cacheReport]));
|
|
613
681
|
const cli = Command.make("joggle").pipe(Command.withDescription("Cross-file patterns and idioms for TypeScript, enforced like a linter: deterministic where it can prove, System One where it has to judge."), Command.withExamples([
|
|
614
682
|
{
|
|
615
683
|
command: "joggle check --pr",
|
package/dist/state.js
CHANGED
|
@@ -40,6 +40,29 @@ export const machineCacheRoot = Effect.gen(function* () {
|
|
|
40
40
|
}).pipe(Effect.orElseSucceed(() => homedir()));
|
|
41
41
|
/** A machine-local directory for one analysed root. */
|
|
42
42
|
export const runCacheDirFor = (root) => Effect.map(machineCacheRoot, (base) => `${base}/joggle/${shortHash(root)}`);
|
|
43
|
+
/**
|
|
44
|
+
* Where a SCOPED run's answers go.
|
|
45
|
+
*
|
|
46
|
+
* A scoped run answers "what did this branch introduce", and a branch's answers are a cache,
|
|
47
|
+
* not a commit. Writing them into the repository's `.joggle/` puts thousands of lines into
|
|
48
|
+
* every pull request of a stack — unreviewable, and a blob in every clone and every CI
|
|
49
|
+
* checkout — which is what a 22-branch migration produced: a 55 MB file, 777 commits touching
|
|
50
|
+
* it, and a diff nobody could read.
|
|
51
|
+
*
|
|
52
|
+
* So a scoped run writes the machine cache unless the caller names a store with
|
|
53
|
+
* `--cache-dir`. The committed ledger stays the trunk's: one full run on the base branch
|
|
54
|
+
* decides what is accepted, and a branch's answers warm that machine and nobody else's diff.
|
|
55
|
+
*
|
|
56
|
+
* @param input.declared - `--cache-dir`, when the caller gave one. It always wins.
|
|
57
|
+
* @param input.repository - What {@link cacheDirFor} resolved: `.joggle/` or the machine.
|
|
58
|
+
* @param input.machine - The machine-local cache for this root.
|
|
59
|
+
* @param input.scoped - Whether the run was scoped to a change (`--since`, `--pr`, `--changed`).
|
|
60
|
+
*/
|
|
61
|
+
export const scopedAnswerCacheDir = (input) => {
|
|
62
|
+
if (input.declared !== undefined)
|
|
63
|
+
return input.declared;
|
|
64
|
+
return input.scoped ? input.machine : input.repository;
|
|
65
|
+
};
|
|
43
66
|
/**
|
|
44
67
|
* Where a run's answer cache goes.
|
|
45
68
|
*
|