@niroai/niro 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +99 -0
- package/package.json +1 -1
- package/src/commands/release.js +16 -0
- package/src/commands/watch.js +247 -64
- package/src/lib/watchState.js +14 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,105 @@
|
|
|
2
2
|
|
|
3
3
|
Notable changes to `@niroai/niro`. Dates are release dates.
|
|
4
4
|
|
|
5
|
+
## 0.7.0 — 2026-08-17
|
|
6
|
+
|
|
7
|
+
A data-loss release. If you use `niro watch`, upgrade.
|
|
8
|
+
|
|
9
|
+
### Upgrading
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
npm install -g @niroai/niro@latest
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Nothing extra is needed from 0.6.0. **From 0.4.x or earlier, upgrade urgently** — every fix
|
|
16
|
+
below is client-side, so a server update alone does not protect you.
|
|
17
|
+
|
|
18
|
+
### Why this release exists
|
|
19
|
+
|
|
20
|
+
On 31 July a repository lost 494 source files. The client reported them deleted, the server
|
|
21
|
+
removed the working copy and the durable backup, and the client's own record then said
|
|
22
|
+
"deleted", so nothing ever re-sent them. Eleven days passed before anyone noticed.
|
|
23
|
+
|
|
24
|
+
The cause was a `git checkout`. Git removed a directory of test files and wrote it straight
|
|
25
|
+
back a moment later. In that gap the watcher looked, saw nothing, and reported 249 files
|
|
26
|
+
deleted. Everything below follows from that one sentence: **the watcher believed a file was
|
|
27
|
+
gone because it could not see it at one instant.**
|
|
28
|
+
|
|
29
|
+
### Fixed — the client no longer infers a deletion from a failure to look
|
|
30
|
+
|
|
31
|
+
- **Deletes now wait for the folder to go quiet.** A file that disappears is queued, not
|
|
32
|
+
reported. One timer per folder resets on every new disappearance, so a three-file checkout
|
|
33
|
+
holds for a few seconds and a ninety-second rebase holds for ninety seconds. When the folder
|
|
34
|
+
settles, whatever is still missing is deleted and whatever came back is dropped.
|
|
35
|
+
|
|
36
|
+
This covers `git checkout`, `git stash`, rebases, `npm ci`, `make clean`, a Docker remount and
|
|
37
|
+
editors that save by deleting and rewriting — without needing to know which of them is
|
|
38
|
+
happening. A real deletion now reaches the server about five seconds later than before.
|
|
39
|
+
Set `NIRO_DELETE_SETTLE_MS` to change that.
|
|
40
|
+
|
|
41
|
+
- **A file that cannot be READ is no longer treated as deleted.** A permission error, a
|
|
42
|
+
transient lock, a file being rewritten or simply too many open files all made the hash return
|
|
43
|
+
nothing, and the file was then reported gone. It now keeps its previous state.
|
|
44
|
+
|
|
45
|
+
- **A DIRECTORY that cannot be read is no longer treated as "everything under it was deleted".**
|
|
46
|
+
The same mistake one level up, and worse: an unreadable folder lists as empty, which is
|
|
47
|
+
indistinguishable from one whose files were all removed. Renaming a watched folder and
|
|
48
|
+
restarting reported the entire project as deleted, in a pass that reported itself successful.
|
|
49
|
+
An unreadable root now aborts the pass; an unreadable subdirectory keeps its files while the
|
|
50
|
+
rest of the tree syncs.
|
|
51
|
+
|
|
52
|
+
- **A delete the server refuses is retried instead of forgotten.** It was marked done regardless
|
|
53
|
+
of the outcome, and once out of the local record it could never be re-attempted — so a
|
|
54
|
+
transient error became permanent.
|
|
55
|
+
|
|
56
|
+
- **`niro discard-temp-project` no longer loses the work done in the temporary project.** The
|
|
57
|
+
file record followed the folder back to the source repository, so the source appeared to
|
|
58
|
+
already hold edits it had never received, and the next sync uploaded nothing. Discarding now
|
|
59
|
+
clears that record and re-uploads once.
|
|
60
|
+
|
|
61
|
+
- **Excluding a path removes it again.** A guard added in the same work stopped an excluded file
|
|
62
|
+
ever being removed from the index, which mattered most for a path excluded *because* it holds
|
|
63
|
+
secrets.
|
|
64
|
+
|
|
65
|
+
- **No delete calls for files that were never uploaded.** Editor scratch files are rejected on
|
|
66
|
+
the way in, but a delete was still sent when the editor renamed them away — 21% of all file
|
|
67
|
+
events on one real repository, every one of them pointless traffic.
|
|
68
|
+
|
|
69
|
+
### Fixed — the retry flood
|
|
70
|
+
|
|
71
|
+
- **The 60-second re-scan now backs off.** It retried at full rate forever, so a two-day server
|
|
72
|
+
problem became roughly 1,400 failed requests a day per folder and buried the real error. The
|
|
73
|
+
delay now doubles to a 30-minute ceiling and resets on success. It never gives up entirely, so
|
|
74
|
+
a recovered server is picked up within half an hour on its own.
|
|
75
|
+
|
|
76
|
+
- **A pass whose file operations failed is no longer counted as a recovery**, which previously
|
|
77
|
+
reset the backoff mid-incident and printed a green "Sync recovered" while the outage continued.
|
|
78
|
+
|
|
79
|
+
### Changed — worktrees and temporary projects
|
|
80
|
+
|
|
81
|
+
You get one temporary project per project, and creating another moves it rather than adding one.
|
|
82
|
+
With several worktrees open that meant they silently took it from each other.
|
|
83
|
+
|
|
84
|
+
- A second worktree is no longer told to create one. It is told which branch currently holds it
|
|
85
|
+
and to use its own tools for that folder.
|
|
86
|
+
- `niro discard-temp-project` asks before moving the slot away from another folder.
|
|
87
|
+
- A folder that has lost the slot stops uploading, instead of blending two branches into one copy.
|
|
88
|
+
- A branch switch is now told apart from another folder taking the slot.
|
|
89
|
+
- `niro info` shows which local folder and branch syncs each project, and marks the one you are
|
|
90
|
+
standing in.
|
|
91
|
+
|
|
92
|
+
### Also fixed
|
|
93
|
+
|
|
94
|
+
- **A sync finishing after the folder was re-pointed no longer writes its stale record.** A pass takes
|
|
95
|
+
minutes on a large tree; if `niro new-temp-project` or `niro discard-temp-project` re-pointed the
|
|
96
|
+
folder while it ran, its results were written anyway, against a repository they did not describe.
|
|
97
|
+
For discard that undid the clean slate above and the re-upload was skipped. The record now belongs
|
|
98
|
+
to one repository and a pass that no longer matches is dropped, the same way a folder release has
|
|
99
|
+
always been handled.
|
|
100
|
+
|
|
101
|
+
- **A folder whose root cannot be read now says so once**, and says when it can be read again. It used
|
|
102
|
+
to repeat the same error every sixty seconds for as long as the folder stayed away.
|
|
103
|
+
|
|
5
104
|
## 0.6.0 — 2026-08-11
|
|
6
105
|
|
|
7
106
|
### Upgrading
|
package/package.json
CHANGED
package/src/commands/release.js
CHANGED
|
@@ -151,6 +151,22 @@ async function release(pathArg, opts = {}) {
|
|
|
151
151
|
...e,
|
|
152
152
|
gitId: backup.previousGitId || null,
|
|
153
153
|
projectId: sourceProjectId,
|
|
154
|
+
// The baseline describes what the TEMPORARY project holds, so it MUST NOT survive the move
|
|
155
|
+
// back to the source repo. Carrying it across told the daemon the source already had every
|
|
156
|
+
// file at its sandbox-era fingerprint — which it did not, because those edits only ever went
|
|
157
|
+
// to the temporary copy. The next pass then compared disk against that baseline, found no
|
|
158
|
+
// difference, and uploaded nothing. Every edit made during the temporary project was silently
|
|
159
|
+
// lost to the source project, with no error and nothing to prompt a re-sync.
|
|
160
|
+
//
|
|
161
|
+
// Clearing it makes the next pass treat every file as new and upload it. That is a full
|
|
162
|
+
// re-upload once, which is the cost of being sure: the client cannot know which files the
|
|
163
|
+
// source repo is missing, so the only safe assumption is all of them.
|
|
164
|
+
//
|
|
165
|
+
// The opposite direction already gets this right — takeover.js seeds fileHashes with what it
|
|
166
|
+
// actually uploaded to the new gitId. A baseline belongs to ONE gitId; when that changes, so
|
|
167
|
+
// must it.
|
|
168
|
+
fileHashes: {},
|
|
169
|
+
lastSyncedAt: null,
|
|
154
170
|
takeover: undefined,
|
|
155
171
|
}, e.apiUrl || baseUrl);
|
|
156
172
|
repinned.push(e.folderPath);
|
package/src/commands/watch.js
CHANGED
|
@@ -38,6 +38,47 @@ const syncInFlight = new Set();
|
|
|
38
38
|
// honoured by the live file handler, so one check per sync pass covers both paths rather than asking
|
|
39
39
|
// the server on every save.
|
|
40
40
|
const notHolding = new Set();
|
|
41
|
+
|
|
42
|
+
/**
|
|
43
|
+
* How long the deletes for a folder must stay quiet before any of them is believed.
|
|
44
|
+
*
|
|
45
|
+
* Not a guess at how long a git checkout takes — the timer RESETS on every new delete, so a 90-second
|
|
46
|
+
* rebase holds the queue for 90 seconds and a three-file checkout holds it for five. This only has to
|
|
47
|
+
* outlast the gap between one file vanishing and the next.
|
|
48
|
+
*/
|
|
49
|
+
const DELETE_SETTLE_MS = Number(process.env.NIRO_DELETE_SETTLE_MS) || 5_000;
|
|
50
|
+
|
|
51
|
+
/**
|
|
52
|
+
* Deletes waiting to be believed: resolved folder -> { paths:Set, timer }.
|
|
53
|
+
*
|
|
54
|
+
* A file disappearing is not evidence it was deleted. On 2026-07-31 a customer lost 494 files because
|
|
55
|
+
* a `git checkout` removed a tree, this handler saw the gap and reported every file gone, and git then
|
|
56
|
+
* wrote them all back — by which time the server had destroyed its copies. The client's own log shows
|
|
57
|
+
* the burst (249 consecutive deletes) followed immediately by re-uploads of the very same paths.
|
|
58
|
+
*
|
|
59
|
+
* So deletes are queued rather than sent, and ONE timer per folder is reset by every new delete. When
|
|
60
|
+
* the folder finally goes quiet, whatever is still missing is really gone and is sent; whatever came
|
|
61
|
+
* back was never deleted and is dropped.
|
|
62
|
+
*
|
|
63
|
+
* Deliberately NOT volume-based, and deliberately not "detect a git operation":
|
|
64
|
+
* - a threshold misses a small checkout that touches three files, which is just as wrong;
|
|
65
|
+
* - lock-file detection is git-only, and the same damage comes from `npm ci`, `make clean`, a Docker
|
|
66
|
+
* remount, Perforce (no local lock at all) or any other VCS.
|
|
67
|
+
* Waiting for quiet needs to know none of that. It is the same rule the server already applies before
|
|
68
|
+
* it rebuilds a graph.
|
|
69
|
+
*
|
|
70
|
+
* Holding a delete costs nothing: the graph only updates after its own quiet period, so a few seconds
|
|
71
|
+
* is invisible. A wrong delete costs the file AND its durable backup, at the moment the call lands.
|
|
72
|
+
*/
|
|
73
|
+
const pendingDeletes = new Map();
|
|
74
|
+
|
|
75
|
+
/**
|
|
76
|
+
* Folders whose root we could not read, so the "cannot read" error is said ONCE rather than every
|
|
77
|
+
* 60 seconds for as long as the folder stays away. A watched folder that is renamed or unmounted is
|
|
78
|
+
* often left that way for days; repeating a red error every minute trains people to ignore the log,
|
|
79
|
+
* which is how the errors that DO matter get missed.
|
|
80
|
+
*/
|
|
81
|
+
const unreadableRoots = new Set();
|
|
41
82
|
// The state file is rewritten several times during a `niro takeover` (per-file hashes), so debounce
|
|
42
83
|
// before reconciling the watch set. Also the interval for the mtime backstop re-scan below.
|
|
43
84
|
const STATE_RELOAD_DEBOUNCE_MS = 1_500;
|
|
@@ -233,46 +274,7 @@ async function startWatch(targetDir, opts = {}) {
|
|
|
233
274
|
// that drifted — the reliable, content-checked form of "also pick up changes we didn't get a live
|
|
234
275
|
// event for". Re-reads fresh state, so a folder re-pointed by `niro takeover` or removed by `niro
|
|
235
276
|
// release` mid-session is handled (or skipped) rather than synced to a stale gitId.
|
|
236
|
-
const rescanTimer = setInterval(() => {
|
|
237
|
-
const fresh = watchState.get(targetDir, api.currentBaseUrl());
|
|
238
|
-
if (!fresh || !fresh.gitId) return; // folder released — nothing to sync
|
|
239
|
-
// Back off after failures instead of retrying every 60s forever. Retrying at full rate against a
|
|
240
|
-
// server that is failing does not fix anything and buries the real error in its own noise.
|
|
241
|
-
const backoffKey = path.resolve(targetDir);
|
|
242
|
-
if (!shouldAttemptBackstop(backoffKey)) return;
|
|
243
|
-
deltaSyncOnResume(fresh.gitId, targetDir, fresh, { quiet: true })
|
|
244
|
-
.then(result => {
|
|
245
|
-
// A run that never reached the server proves nothing about the server. The in-flight guard
|
|
246
|
-
// returns at once when another sync for this folder is already going, and during an outage a
|
|
247
|
-
// large delta holds that flag for minutes — so a skipped tick was clearing the failure count
|
|
248
|
-
// and printing a green "Sync recovered" in the middle of the incident, while the real run was
|
|
249
|
-
// still heading for its own failure. The count then never grew past one, so the backoff never
|
|
250
|
-
// climbed towards its ceiling and kept hammering every two minutes: both things this backoff
|
|
251
|
-
// was added to stop.
|
|
252
|
-
// Only a pass that actually reached the server and had nothing fail counts as a recovery.
|
|
253
|
-
// A skipped run (another sync in flight) and a no-change run both contact nothing, and a pass
|
|
254
|
-
// whose per-file deletes were all refused resolves cleanly — so all three used to clear the
|
|
255
|
-
// failure count and print a green "Sync recovered" during an outage, which is how the retry rate
|
|
256
|
-
// never decayed and the real error stayed buried.
|
|
257
|
-
if (!result || result.skipped || result.noContact || result.notHolder) return;
|
|
258
|
-
if (result.failures > 0) {
|
|
259
|
-
const delayMs = recordBackstopFailure(backoffKey, Date.now());
|
|
260
|
-
console.error(color.red(
|
|
261
|
-
` ${result.failures} file operation(s) failed for ${targetDir} — retrying in ` +
|
|
262
|
-
`${Math.round(delayMs / 60_000)} min.`));
|
|
263
|
-
return;
|
|
264
|
-
}
|
|
265
|
-
if (recordBackstopSuccess(backoffKey)) {
|
|
266
|
-
console.log(color.green(` Sync recovered for ${targetDir}.`));
|
|
267
|
-
}
|
|
268
|
-
})
|
|
269
|
-
.catch(err => {
|
|
270
|
-
const delayMs = recordBackstopFailure(backoffKey, Date.now());
|
|
271
|
-
console.error(color.red(
|
|
272
|
-
` Backstop re-scan failed for ${targetDir}: ${err.message} — retrying in ` +
|
|
273
|
-
`${Math.round(delayMs / 60_000)} min.`));
|
|
274
|
-
});
|
|
275
|
-
}, RESCAN_INTERVAL_MS);
|
|
277
|
+
const rescanTimer = setInterval(() => { backstopTick(targetDir); }, RESCAN_INTERVAL_MS);
|
|
276
278
|
|
|
277
279
|
// Start file watcher
|
|
278
280
|
const debounceTimers = new Map();
|
|
@@ -373,7 +375,27 @@ async function deltaSyncOnResume(gitId, targetDir, state, opts = {}) {
|
|
|
373
375
|
const deleted = [];
|
|
374
376
|
|
|
375
377
|
// Walk all currently included files
|
|
376
|
-
const manifest = buildMinimalManifest(targetDir, state.includePatterns,
|
|
378
|
+
const { files: manifest, failedDirs } = buildMinimalManifest(targetDir, state.includePatterns,
|
|
379
|
+
state.excludePatterns);
|
|
380
|
+
|
|
381
|
+
// The repo root itself could not be read — the folder was moved, unmounted, or its permissions
|
|
382
|
+
// changed. EVERY file looks missing, so a delete pass here would wipe the whole project. Abort the
|
|
383
|
+
// pass entirely; a later one recovers by itself once the folder is back.
|
|
384
|
+
if (failedDirs.includes("")) {
|
|
385
|
+
// Said once per folder, then silence until it reads again. Repeating this every 60s for a folder
|
|
386
|
+
// that has simply been moved is noise, and noise is how a real error gets missed.
|
|
387
|
+
if (!unreadableRoots.has(path.resolve(targetDir))) {
|
|
388
|
+
unreadableRoots.add(path.resolve(targetDir));
|
|
389
|
+
console.error(color.red(
|
|
390
|
+
` Cannot read ${targetDir} — skipping this sync entirely rather than reporting every file ` +
|
|
391
|
+
`deleted. Check the folder still exists and is readable. ` +
|
|
392
|
+
`This will not be repeated until it can be read again.`));
|
|
393
|
+
}
|
|
394
|
+
return;
|
|
395
|
+
}
|
|
396
|
+
if (unreadableRoots.delete(path.resolve(targetDir))) {
|
|
397
|
+
console.log(color.green(` ${targetDir} is readable again — syncing resumed.`));
|
|
398
|
+
}
|
|
377
399
|
|
|
378
400
|
const unreadable = [];
|
|
379
401
|
for (const relativePath of manifest) {
|
|
@@ -406,15 +428,32 @@ async function deltaSyncOnResume(gitId, targetDir, state, opts = {}) {
|
|
|
406
428
|
}
|
|
407
429
|
|
|
408
430
|
// Files that were in stored hashes but are no longer present in the walk at all.
|
|
431
|
+
//
|
|
432
|
+
// "Not in the walk" only means "deleted" for parts of the tree the walk could actually READ. A
|
|
433
|
+
// directory that failed to list (permissions, a build holding it, too many open files, an NFS or
|
|
434
|
+
// Docker mount blip) yields no entries, which is indistinguishable from one whose files were all
|
|
435
|
+
// removed. Judging those would delete a subtree because we could not look at it.
|
|
436
|
+
const unreadableDirs = [];
|
|
409
437
|
for (const rp of Object.keys(storedHashes)) {
|
|
410
|
-
if (
|
|
411
|
-
|
|
438
|
+
if (Object.prototype.hasOwnProperty.call(currentHashes, rp)) continue;
|
|
439
|
+
if (failedDirs.some(d => d && (rp === d || rp.startsWith(d + "/")))) {
|
|
440
|
+
unreadableDirs.push(rp);
|
|
441
|
+
currentHashes[rp] = storedHashes[rp]; // keep it tracked; we learned nothing about it
|
|
442
|
+
continue;
|
|
412
443
|
}
|
|
444
|
+
deleted.push(rp);
|
|
445
|
+
}
|
|
446
|
+
|
|
447
|
+
if (unreadableDirs.length > 0) {
|
|
448
|
+
console.warn(color.yellow(
|
|
449
|
+
` ${failedDirs.length} directory(ies) could not be read this pass, so ${unreadableDirs.length} ` +
|
|
450
|
+
`file(s) under them are being left alone rather than reported deleted. ` +
|
|
451
|
+
`Sample: ${failedDirs.slice(0, 3).join(", ")}`));
|
|
413
452
|
}
|
|
414
453
|
|
|
415
454
|
if (changed.length === 0 && deleted.length === 0) {
|
|
416
455
|
if (!quiet) console.log(` No changes since last sync.`);
|
|
417
|
-
watchState.mergeFileHashes(targetDir, { set: currentHashes });
|
|
456
|
+
watchState.mergeFileHashes(targetDir, { set: currentHashes, expectedGitId: gitId });
|
|
418
457
|
// Nothing to send, so the server was never contacted and this pass is not evidence it is healthy.
|
|
419
458
|
// This is the MAJORITY of backstop ticks: treated as a success it cleared the failure count on the
|
|
420
459
|
// very next tick after one, so the backoff could never grow past its first step.
|
|
@@ -503,11 +542,16 @@ async function deltaSyncOnResume(gitId, targetDir, state, opts = {}) {
|
|
|
503
542
|
|
|
504
543
|
// Merge rather than replace. Removals are explicit: the files we actually deleted, plus any the
|
|
505
544
|
// gitignore policy dropped (those were never uploaded, so they must not stay recorded as synced).
|
|
506
|
-
|
|
545
|
+
// expectedGitId: this whole pass computed hashes for `gitId`. If a takeover or discard re-pointed the
|
|
546
|
+
// folder while it ran, these describe a repo it is no longer registered against and must be dropped.
|
|
547
|
+
if (!watchState.mergeFileHashes(targetDir, {
|
|
507
548
|
set: currentHashes,
|
|
508
549
|
remove: [...actuallyDeleted, ...gitignoreDropped],
|
|
509
550
|
patch: { lastSyncedAt: new Date().toISOString() },
|
|
510
|
-
|
|
551
|
+
expectedGitId: gitId,
|
|
552
|
+
})) {
|
|
553
|
+
console.log(color.dim(" Folder was re-pointed during this sync — its results were not recorded."));
|
|
554
|
+
}
|
|
511
555
|
console.log(` ${color.green("✓")} Delta sync complete.`);
|
|
512
556
|
return { failures };
|
|
513
557
|
} finally {
|
|
@@ -515,6 +559,124 @@ async function deltaSyncOnResume(gitId, targetDir, state, opts = {}) {
|
|
|
515
559
|
}
|
|
516
560
|
}
|
|
517
561
|
|
|
562
|
+
/** Queue one vanished path and (re)start this folder's settle timer. */
|
|
563
|
+
function queueDelete(targetDir, relativePath, settleMs = DELETE_SETTLE_MS) {
|
|
564
|
+
const key = path.resolve(targetDir);
|
|
565
|
+
let entry = pendingDeletes.get(key);
|
|
566
|
+
if (!entry) {
|
|
567
|
+
entry = { paths: new Set(), timer: null };
|
|
568
|
+
pendingDeletes.set(key, entry);
|
|
569
|
+
}
|
|
570
|
+
entry.paths.add(relativePath);
|
|
571
|
+
// Reset, never extend-if-absent: while files keep vanishing the operation is still running, and
|
|
572
|
+
// nothing in the queue should be believed until it stops.
|
|
573
|
+
if (entry.timer) clearTimeout(entry.timer);
|
|
574
|
+
entry.timer = setTimeout(() => {
|
|
575
|
+
flushPendingDeletes(targetDir).catch(err =>
|
|
576
|
+
console.error(color.red(` Delete flush failed for ${targetDir}: ${err.message}`)));
|
|
577
|
+
}, settleMs);
|
|
578
|
+
if (entry.timer.unref) entry.timer.unref(); // must never hold the daemon open
|
|
579
|
+
}
|
|
580
|
+
|
|
581
|
+
/**
|
|
582
|
+
* The folder has gone quiet. Look again at everything queued: send the deletes that are still real,
|
|
583
|
+
* and forget the ones that came back.
|
|
584
|
+
*
|
|
585
|
+
* A file that returned is left ALONE rather than re-uploaded here. Its baseline entry was never
|
|
586
|
+
* removed (we never sent the delete), so if the bytes changed, git's own write fires a change event
|
|
587
|
+
* that takes the normal upload path, and the 60-second backstop re-scan covers a coalesced event.
|
|
588
|
+
* Duplicating the upload path here would mean duplicating its gitignore and secret checks too.
|
|
589
|
+
*/
|
|
590
|
+
async function flushPendingDeletes(targetDir) {
|
|
591
|
+
const key = path.resolve(targetDir);
|
|
592
|
+
const entry = pendingDeletes.get(key);
|
|
593
|
+
if (!entry) return;
|
|
594
|
+
pendingDeletes.delete(key);
|
|
595
|
+
if (entry.timer) clearTimeout(entry.timer);
|
|
596
|
+
|
|
597
|
+
// Re-read state at flush time: a `niro takeover` or `niro release` during the settle window changes
|
|
598
|
+
// (or removes) the gitId these deletes were queued against.
|
|
599
|
+
const fresh = watchState.get(targetDir, api.currentBaseUrl());
|
|
600
|
+
if (!fresh || !fresh.gitId) return;
|
|
601
|
+
if (notHolding.has(key)) return;
|
|
602
|
+
|
|
603
|
+
const stillGone = [];
|
|
604
|
+
const returned = [];
|
|
605
|
+
for (const rp of entry.paths) {
|
|
606
|
+
if (fs.existsSync(path.join(targetDir, rp))) returned.push(rp);
|
|
607
|
+
else stillGone.push(rp);
|
|
608
|
+
}
|
|
609
|
+
|
|
610
|
+
if (returned.length > 0) {
|
|
611
|
+
console.log(color.dim(
|
|
612
|
+
` ${returned.length} file(s) reappeared while settling — not deleted` +
|
|
613
|
+
(returned.length <= 3 ? `: ${returned.join(", ")}` : "")));
|
|
614
|
+
}
|
|
615
|
+
|
|
616
|
+
for (const rp of stillGone) {
|
|
617
|
+
try {
|
|
618
|
+
await agentApi.deleteFile(fresh.gitId, rp);
|
|
619
|
+
watchState.mergeFileHashes(targetDir, {
|
|
620
|
+
remove: [rp],
|
|
621
|
+
patch: { lastSyncedAt: new Date().toISOString() },
|
|
622
|
+
expectedGitId: fresh.gitId,
|
|
623
|
+
});
|
|
624
|
+
console.log(` ${color.yellow("−")} ${rp}`);
|
|
625
|
+
} catch (err) {
|
|
626
|
+
// Keep the baseline entry so a later pass retries. Dropping it here is how a refused delete
|
|
627
|
+
// became permanent (see fb3ff0a).
|
|
628
|
+
console.error(color.red(` Failed to delete ${rp}: ${err.message}`));
|
|
629
|
+
}
|
|
630
|
+
}
|
|
631
|
+
}
|
|
632
|
+
|
|
633
|
+
|
|
634
|
+
/**
|
|
635
|
+
* One backstop pass for a folder. Extracted from the interval so the backoff gate at the top can
|
|
636
|
+
* actually be exercised — adversarial review found the gate could be deleted with the whole suite
|
|
637
|
+
* still green, because the only thing that called it was a timer no test drives.
|
|
638
|
+
*/
|
|
639
|
+
function backstopTick(targetDir) {
|
|
640
|
+
const fresh = watchState.get(targetDir, api.currentBaseUrl());
|
|
641
|
+
if (!fresh || !fresh.gitId) return; // folder released — nothing to sync
|
|
642
|
+
// Back off after failures instead of retrying every 60s forever. Retrying at full rate against a
|
|
643
|
+
// server that is failing does not fix anything and buries the real error in its own noise.
|
|
644
|
+
const backoffKey = path.resolve(targetDir);
|
|
645
|
+
if (!shouldAttemptBackstop(backoffKey)) return;
|
|
646
|
+
deltaSyncOnResume(fresh.gitId, targetDir, fresh, { quiet: true })
|
|
647
|
+
.then(result => {
|
|
648
|
+
// A run that never reached the server proves nothing about the server. The in-flight guard
|
|
649
|
+
// returns at once when another sync for this folder is already going, and during an outage a
|
|
650
|
+
// large delta holds that flag for minutes — so a skipped tick was clearing the failure count
|
|
651
|
+
// and printing a green "Sync recovered" in the middle of the incident, while the real run was
|
|
652
|
+
// still heading for its own failure. The count then never grew past one, so the backoff never
|
|
653
|
+
// climbed towards its ceiling and kept hammering every two minutes: both things this backoff
|
|
654
|
+
// was added to stop.
|
|
655
|
+
// Only a pass that actually reached the server and had nothing fail counts as a recovery.
|
|
656
|
+
// A skipped run (another sync in flight) and a no-change run both contact nothing, and a pass
|
|
657
|
+
// whose per-file deletes were all refused resolves cleanly — so all three used to clear the
|
|
658
|
+
// failure count and print a green "Sync recovered" during an outage, which is how the retry rate
|
|
659
|
+
// never decayed and the real error stayed buried.
|
|
660
|
+
if (!result || result.skipped || result.noContact || result.notHolder) return;
|
|
661
|
+
if (result.failures > 0) {
|
|
662
|
+
const delayMs = recordBackstopFailure(backoffKey, Date.now());
|
|
663
|
+
console.error(color.red(
|
|
664
|
+
` ${result.failures} file operation(s) failed for ${targetDir} — retrying in ` +
|
|
665
|
+
`${Math.round(delayMs / 60_000)} min.`));
|
|
666
|
+
return;
|
|
667
|
+
}
|
|
668
|
+
if (recordBackstopSuccess(backoffKey)) {
|
|
669
|
+
console.log(color.green(` Sync recovered for ${targetDir}.`));
|
|
670
|
+
}
|
|
671
|
+
})
|
|
672
|
+
.catch(err => {
|
|
673
|
+
const delayMs = recordBackstopFailure(backoffKey, Date.now());
|
|
674
|
+
console.error(color.red(
|
|
675
|
+
` Backstop re-scan failed for ${targetDir}: ${err.message} — retrying in ` +
|
|
676
|
+
`${Math.round(delayMs / 60_000)} min.`));
|
|
677
|
+
});
|
|
678
|
+
}
|
|
679
|
+
|
|
518
680
|
async function handleFileEvent(gitId, targetDir, relativePath, state) {
|
|
519
681
|
// Re-read this folder's current state on every event instead of trusting the value
|
|
520
682
|
// captured when the daemon started. A mid-session `niro takeover` re-points the folder
|
|
@@ -577,26 +739,29 @@ async function handleFileEvent(gitId, targetDir, relativePath, state) {
|
|
|
577
739
|
|
|
578
740
|
const absPath = path.join(targetDir, relativePath);
|
|
579
741
|
|
|
742
|
+
// Symmetry: if we would never UPLOAD this path, we must never ask the server to DELETE it. The
|
|
743
|
+
// accept-set check used to sit below the delete branch, so an editor's scratch file — never sent
|
|
744
|
+
// to us in the first place — still produced a delete call for a path the server had never heard
|
|
745
|
+
// of. That is 21% of all file events in the customer's log (3,226 of 15,223), every one of them a
|
|
746
|
+
// wasted round trip, a wasted journal record, and a count against the server's delete-burst budget
|
|
747
|
+
// that could trip the guard on churn alone.
|
|
748
|
+
//
|
|
749
|
+
// Safe because uploadability is decided by the filename, which does not change for a given path:
|
|
750
|
+
// if it is not uploadable now it was never uploaded. A file the accept set USED to include is
|
|
751
|
+
// still reachable by the backstop re-scan, which works from the stored baseline rather than from
|
|
752
|
+
// live events.
|
|
753
|
+
if (!folderSync.isUploadableName(relativePath)) return;
|
|
754
|
+
|
|
580
755
|
if (!fs.existsSync(absPath)) {
|
|
581
|
-
//
|
|
582
|
-
|
|
583
|
-
await agentApi.deleteFile(gitId, relativePath);
|
|
584
|
-
watchState.mergeFileHashes(targetDir, {
|
|
585
|
-
remove: [relativePath],
|
|
586
|
-
patch: { lastSyncedAt: new Date().toISOString() },
|
|
587
|
-
});
|
|
588
|
-
console.log(` ${color.yellow("−")} ${relativePath}`);
|
|
589
|
-
} catch (err) {
|
|
590
|
-
console.error(color.red(` Failed to delete ${relativePath}: ${err.message}`));
|
|
591
|
-
}
|
|
756
|
+
// Gone RIGHT NOW is not the same as deleted. Queue it and look again once the folder is quiet.
|
|
757
|
+
queueDelete(targetDir, relativePath);
|
|
592
758
|
return;
|
|
593
759
|
}
|
|
594
760
|
|
|
595
|
-
//
|
|
596
|
-
//
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
if (!folderSync.isUploadableName(relativePath)) return;
|
|
761
|
+
// It exists, so any queued delete for it was a transient gap (a rewrite, not a removal). Drop it
|
|
762
|
+
// immediately rather than waiting for the settle timer to work it out.
|
|
763
|
+
const pending = pendingDeletes.get(path.resolve(targetDir));
|
|
764
|
+
if (pending) pending.paths.delete(relativePath);
|
|
600
765
|
|
|
601
766
|
// File changed or created
|
|
602
767
|
const newHash = watchState.hashFile(absPath);
|
|
@@ -617,6 +782,7 @@ async function handleFileEvent(gitId, targetDir, relativePath, state) {
|
|
|
617
782
|
watchState.mergeFileHashes(targetDir, {
|
|
618
783
|
set: { [relativePath]: newHash },
|
|
619
784
|
patch: { lastSyncedAt: new Date().toISOString() },
|
|
785
|
+
expectedGitId: gitId,
|
|
620
786
|
});
|
|
621
787
|
console.log(` ${color.green("+")} ${relativePath}`);
|
|
622
788
|
} catch (err) {
|
|
@@ -655,12 +821,27 @@ function minimatch(filePath, pattern) {
|
|
|
655
821
|
}
|
|
656
822
|
|
|
657
823
|
/** Walk the directory respecting stored patterns to find all currently included files. */
|
|
824
|
+
/**
|
|
825
|
+
* @returns {{files: string[], failedDirs: string[]}} — `failedDirs` holds the repo-relative directories
|
|
826
|
+
* the walk could NOT read ("" means the repo root itself). A caller must treat those as UNKNOWN, never
|
|
827
|
+
* as empty: a directory that cannot be listed looks exactly like a directory whose files were all
|
|
828
|
+
* deleted, and acting on that difference is how a whole project gets wiped.
|
|
829
|
+
*/
|
|
658
830
|
function buildMinimalManifest(rootDir, includePatterns, excludePatterns) {
|
|
659
831
|
const results = [];
|
|
832
|
+
const failedDirs = [];
|
|
660
833
|
|
|
661
834
|
function walk(dir) {
|
|
662
835
|
let entries;
|
|
663
|
-
try {
|
|
836
|
+
try {
|
|
837
|
+
entries = fs.readdirSync(dir, { withFileTypes: true });
|
|
838
|
+
} catch {
|
|
839
|
+
// Swallowing this was the defect. The walk returned a SHORT list with no signal, the caller read
|
|
840
|
+
// "absent from the walk" as "deleted", and every file under here was reported gone. Renaming a
|
|
841
|
+
// watched folder emptied the whole manifest and deleted the entire project server-side.
|
|
842
|
+
failedDirs.push(path.relative(rootDir, dir).replace(/\\/g, "/"));
|
|
843
|
+
return;
|
|
844
|
+
}
|
|
664
845
|
for (const entry of entries) {
|
|
665
846
|
const fullPath = path.join(dir, entry.name);
|
|
666
847
|
const relativePath = path.relative(rootDir, fullPath).replace(/\\/g, "/");
|
|
@@ -691,10 +872,12 @@ function buildMinimalManifest(rootDir, includePatterns, excludePatterns) {
|
|
|
691
872
|
}
|
|
692
873
|
}
|
|
693
874
|
walk(rootDir);
|
|
694
|
-
return results;
|
|
875
|
+
return { files: results, failedDirs };
|
|
695
876
|
}
|
|
696
877
|
|
|
697
878
|
// handleFileEvent + deltaSyncOnResume are exported for unit testing the
|
|
698
879
|
// gitignore/fail-closed guards without a live server or fs.watch loop.
|
|
699
880
|
module.exports = { watch, startWatch, handleFileEvent, deltaSyncOnResume, handleEntryFailure,
|
|
881
|
+
queueDelete, flushPendingDeletes, pendingDeletes, DELETE_SETTLE_MS, backstopTick, notHolding,
|
|
882
|
+
unreadableRoots,
|
|
700
883
|
nextBackoffMs, shouldAttemptBackstop, recordBackstopFailure, recordBackstopSuccess };
|
package/src/lib/watchState.js
CHANGED
|
@@ -143,16 +143,28 @@ function update(folderPath, patch) {
|
|
|
143
143
|
* THIS process. The watch daemon is a separate process, so a cross-process interleave is still
|
|
144
144
|
* possible; this shrinks the window from minutes to microseconds rather than closing it entirely.
|
|
145
145
|
*/
|
|
146
|
-
function mergeFileHashes(folderPath, { set = {}, remove = [], patch = {} } = {}) {
|
|
146
|
+
function mergeFileHashes(folderPath, { set = {}, remove = [], patch = {}, expectedGitId } = {}) {
|
|
147
147
|
const state = load();
|
|
148
148
|
const key = path.resolve(folderPath);
|
|
149
|
-
if (!state[key]) return;
|
|
149
|
+
if (!state[key]) return false;
|
|
150
|
+
// Compare-and-merge, exactly as `remove` above does its compare-and-delete, and for the same reason.
|
|
151
|
+
// A sync pass can take minutes on a large tree. If a `niro new-temp-project` or
|
|
152
|
+
// `niro discard-temp-project` re-points the folder DURING that pass, the hashes it computed describe
|
|
153
|
+
// a repo the folder is no longer registered against — and writing them lands a stale baseline on the
|
|
154
|
+
// new gitId. Discard was the case that bit: it clears the baseline so the source repo gets a full
|
|
155
|
+
// re-upload, and an in-flight pass finishing afterwards restored the old map, so the next pass said
|
|
156
|
+
// "No changes since last sync" and the sandbox edits were never uploaded. Exactly the silent loss
|
|
157
|
+
// clearing the baseline was meant to prevent.
|
|
158
|
+
if (expectedGitId !== undefined && state[key].gitId !== expectedGitId) {
|
|
159
|
+
return false;
|
|
160
|
+
}
|
|
150
161
|
const merged = { ...(state[key].fileHashes || {}) };
|
|
151
162
|
for (const rp of remove) delete merged[rp];
|
|
152
163
|
// Applied after the removals, so a file deleted and re-created within one run keeps its new hash.
|
|
153
164
|
Object.assign(merged, set);
|
|
154
165
|
state[key] = { ...state[key], ...patch, fileHashes: merged };
|
|
155
166
|
save(state);
|
|
167
|
+
return true;
|
|
156
168
|
}
|
|
157
169
|
|
|
158
170
|
/**
|