@smthrs/engine-store 1.0.0-rc.7 → 1.0.0-rc.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. package/README.md +3 -1
  2. package/dist/cjs/ArtifactSync.d.ts.map +1 -1
  3. package/dist/cjs/ArtifactSync.js +5 -0
  4. package/dist/cjs/ArtifactSync.js.map +2 -2
  5. package/dist/cjs/DisasterRecovery.d.ts +9 -26
  6. package/dist/cjs/DisasterRecovery.d.ts.map +1 -1
  7. package/dist/cjs/DisasterRecovery.js +27 -18
  8. package/dist/cjs/DisasterRecovery.js.map +2 -2
  9. package/dist/cjs/index.d.ts +12 -0
  10. package/dist/cjs/index.d.ts.map +1 -1
  11. package/dist/cjs/index.js.map +2 -2
  12. package/dist/cjs/internal/ActionPersistence.d.ts.map +1 -1
  13. package/dist/cjs/internal/ActionPersistence.js +27 -2
  14. package/dist/cjs/internal/ActionPersistence.js.map +3 -3
  15. package/dist/esm/ArtifactSync.d.ts.map +1 -1
  16. package/dist/esm/ArtifactSync.js +6 -1
  17. package/dist/esm/ArtifactSync.js.map +1 -1
  18. package/dist/esm/DisasterRecovery.d.ts +9 -26
  19. package/dist/esm/DisasterRecovery.d.ts.map +1 -1
  20. package/dist/esm/DisasterRecovery.js +46 -27
  21. package/dist/esm/DisasterRecovery.js.map +1 -1
  22. package/dist/esm/index.d.ts +12 -0
  23. package/dist/esm/index.d.ts.map +1 -1
  24. package/dist/esm/index.js +12 -0
  25. package/dist/esm/index.js.map +1 -1
  26. package/dist/esm/internal/ActionPersistence.d.ts.map +1 -1
  27. package/dist/esm/internal/ActionPersistence.js +59 -3
  28. package/dist/esm/internal/ActionPersistence.js.map +1 -1
  29. package/docs/README.md +3 -1
  30. package/docs/api.md +4 -1
  31. package/docs/concepts/attempts-and-replay.md +15 -2
  32. package/docs/guides/back-up-and-restore.md +9 -3
  33. package/docs/guides/reclaim-runs-from-a-dead-host.md +16 -0
  34. package/docs/installation.md +5 -1
  35. package/docs/quickstart.md +9 -1
  36. package/docs/troubleshooting.md +3 -0
  37. package/package.json +18 -18
  38. package/src/ArtifactSync.ts +7 -0
  39. package/src/DisasterRecovery.ts +90 -58
  40. package/src/index.ts +14 -0
  41. package/src/internal/ActionPersistence.ts +64 -3
package/docs/README.md CHANGED
@@ -11,7 +11,9 @@ over.
11
11
  It is an [Effect](https://effect.website) library: every surface below is a
12
12
  `Layer` you compose or an `Effect` you run.
13
13
 
14
- Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
14
+ ```bash
15
+ npm install @smthrs/engine-store@next effect
16
+ ```
15
17
 
16
18
  ## What it solves
17
19
 
package/docs/api.md CHANGED
@@ -895,7 +895,10 @@ durable fails and the shared entry is withheld.
895
895
 
896
896
  `hydrate(digests)` establishes that this host can resolve every referenced
897
897
  artifact and reports whether the replay is now worth retrying. It never fails a
898
- run: a shared tier that is down must not stop work that can simply be done.
898
+ run: a shared tier that is down must not stop work that can simply be done. A
899
+ downloaded blob counts only when this host's store files it under the requested
900
+ digest, so a shared tier that serves the wrong bytes costs the replay, never
901
+ its correctness.
899
902
 
900
903
  The two tiers cannot both be layers, because they inhabit the same tag and
901
904
  composing them would shadow one with the other.
@@ -30,15 +30,21 @@ question, answered in [Cache admission](./cache-admission.md).
30
30
  | ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
31
31
  | `succeeded` | Returns the recorded result, and calls `StepBoundary.replayOutputs` first so the workspace holds the outputs the step produced. |
32
32
  | `failed` | Rethrows the persisted domain failure. It never re-admits the attempt. |
33
- | `running` | Recovers under the run fence and shared admission permit, honoring any recorded effect crossing. |
33
+ | `running` | Asks the step cache first when the step is cache-eligible, then recovers under the run fence and admission permit. |
34
34
  | `suspended` | Continues the same attempt rather than burning a new one against the retry budget. |
35
35
 
36
- An admitted attempt remains authoritative when a matching shared cache row
36
+ A settled attempt remains authoritative when a matching shared cache row
37
37
  exists; another run's cached success cannot replace its result or failure.
38
38
  Failure values were schema-encoded before persistence, so a `_tag` survives
39
39
  the JSON round trip and a `RetryPolicy`'s non-retryable matching still applies
40
40
  on replay.
41
41
 
42
+ A `running` attempt has no result of its own. When its step is cache-eligible
43
+ (sealed, with a hard boundary), a verified row another run recorded under the
44
+ same key answers it, and the row is closed as `succeeded` with that result. A
45
+ recorded successful effect crossing still wins over the cache, because it
46
+ carries this run's own result.
47
+
42
48
  ## Which columns are redacted
43
49
 
44
50
  Attempt and run rows share a database with the journal, so failure text meets
@@ -62,6 +68,13 @@ A permit covers one run and step key through admission, execution, and
62
68
  settlement. Concurrent same-key dispatches wait, then replay the settled
63
69
  attempt instead of executing another body. Interruption releases the permit.
64
70
 
71
+ A cache-eligible dispatch also holds a permit for its cache address, which runs
72
+ share. A second run that asks one engine for a step that engine is already
73
+ computing waits, then takes the first run's cache row instead of running the
74
+ body beside it. Engines in other processes are not excluded: two processes can
75
+ still execute one key at once, and the second recording resolves as described
76
+ in [Cache admission](./cache-admission.md).
77
+
65
78
  Holding the permit and the run fence permits recovery of a stranded `running`
66
79
  attempt. A recorded successful effect crossing supplies the result directly;
67
80
  an unresolved irreversible crossing requires an idempotency key before retry.
@@ -110,10 +110,16 @@ file (`store.sqlite3`), an `objects` directory, and a `restored.json` marker, an
110
110
  is a fresh store directory to compose `NodeDatabase.layer` and
111
111
  `ArtifactStore.layerFileSystem` over.
112
112
 
113
+ A failed restore removes what it wrote, so nothing store-shaped is left in the
114
+ target and you can retry into the same directory. A target that was not empty
115
+ is refused with `not_empty` and left untouched.
116
+
113
117
  `restore` alone leaves the pre-backup owner fences in place. Run
114
118
  `DisasterRecovery.fence(manifest)` against the restored file before any engine
115
119
  adopts it, or use `restoreAndFence`, which does both and returns a
116
- `FencedRestoredStore`.
120
+ `FencedRestoredStore`. When fencing refuses, `restoreAndFence` also removes the
121
+ restored files, so no engine can adopt a store that still carries pre-backup
122
+ owners.
117
123
 
118
124
  ## What fencing does
119
125
 
@@ -149,8 +155,8 @@ its own fresh owner.
149
155
  | `not_empty` | A target or backup directory exists and is not empty. |
150
156
  | `invalid_manifest` | The manifest is missing or does not decode. |
151
157
  | `missing_file` | A file the manifest lists is absent. |
152
- | `digest_mismatch` | The database snapshot does not hash to its recorded digest. |
153
- | `artifact_corruption` | A blob does not hash to its address. |
158
+ | `digest_mismatch` | A backup file does not hash to its manifest digest. |
159
+ | `artifact_corruption` | A live blob does not hash to its address during a backup. |
154
160
  | `snapshot_incomplete` | The snapshot references artifacts the capture could not take. |
155
161
  | `schema_mismatch` | A recorded migration is missing or has a different name. |
156
162
  | `io` | A host read or write refused, including an oversized file. |
@@ -93,6 +93,22 @@ at the staleness cutoff. A deferral is forgotten as soon as the row leaves the
93
93
  stale window, so a fresh stall under a new lease is probed on the first tick
94
94
  that sees it.
95
95
 
96
+ ## What a reclaimed run does with its unfinished step
97
+
98
+ The step that was running when the owner died left an attempt row in state
99
+ `running`. The re-drive reaches that attempt again, and what it does depends on
100
+ the step:
101
+
102
+ - A cache-eligible step (sealed, with a hard boundary) asks the step cache
103
+ first. When another run already recorded the same key, for example the run
104
+ you started to replace the killed one, the reclaimed attempt takes that
105
+ result and the body does not run again. When that run is still computing it
106
+ in the same process, the reclaimed attempt waits for it.
107
+ - Any other step re-executes under its original attempt number. An
108
+ irreversible step whose effect was dispatched but never recorded as settled
109
+ needs an idempotency key to re-execute; without one the run fails with
110
+ `IrreversibleRetryRequiresIdempotencyKey`.
111
+
96
112
  ## Check a run's own view
97
113
 
98
114
  `DurableEngineState` answers directly, which is useful in a diagnostic tool:
@@ -7,7 +7,11 @@ sidebar:
7
7
 
8
8
  ## Install the package
9
9
 
10
- Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
10
+ ```bash
11
+ npm install @smthrs/engine-store@next effect
12
+ ```
13
+
14
+ The `next` tag carries the 1.0 release candidates.
11
15
 
12
16
  The package requires Node.js 26.4.0 or later and ships as both ESM and
13
17
  CommonJS with TypeScript declarations. Its runtime dependencies, including
@@ -13,7 +13,11 @@ the whole point of the package.
13
13
  ## Prerequisites
14
14
 
15
15
  - Node.js 26.4.0 or later.
16
- - A package with the dependencies installed. Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
16
+ - A project with the packages installed:
17
+
18
+ ```bash
19
+ npm install @smthrs/engine-store@next @smthrs/artifacts@next @smthrs/database@next @smthrs/flow@next @smthrs/journal@next @smthrs/kernel@next @smthrs/platform-node@next @smthrs/run-store@next @smthrs/step-cache@next @effect/platform-node effect
20
+ ```
17
21
 
18
22
  ## Declare a sealed action and a flow
19
23
 
@@ -193,6 +197,10 @@ console.log(await Effect.runPromise(main("./.quickstart/.flows/engine.db", "./.q
193
197
  { first: '42', second: '42', executions: 1 }
194
198
  ```
195
199
 
200
+ The assembled program is
201
+ [`examples/quickstart.ts`](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/engine-store/examples/quickstart.ts);
202
+ the package's test suite runs it on every change.
203
+
196
204
  ## What just happened
197
205
 
198
206
  The first run claimed `analyse-1`, opened an attempt row for the sealed step,
@@ -304,6 +304,9 @@ migrations omit or rename a historical entry recorded in the manifest. Forward
304
304
  additions within installed lower blocks are compatible; check historical IDs
305
305
  and names before fencing again.
306
306
 
307
+ A failed backup, `restore`, or `restoreAndFence` removes the files it wrote, so
308
+ fix the cause and retry into the same directory.
309
+
307
310
  ## The composition does not compile
308
311
 
309
312
  **What happened.** A required service is missing. A composition failure is an
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@smthrs/engine-store",
3
3
  "type": "module",
4
- "version": "1.0.0-rc.7",
4
+ "version": "1.0.0-rc.8",
5
5
  "license": "MIT",
6
6
  "smthrs": {
7
7
  "group": "engine"
@@ -386,29 +386,29 @@
386
386
  "tag": "next"
387
387
  },
388
388
  "dependencies": {
389
- "@smthrs/artifacts": "1.0.0-rc.7",
390
- "@smthrs/canonical": "1.0.0-rc.7",
391
- "@smthrs/crypto": "1.0.0-rc.7",
392
- "@smthrs/database": "1.0.0-rc.7",
393
- "@smthrs/engine": "1.0.0-rc.7",
394
- "@smthrs/flow": "1.0.0-rc.7",
395
- "@smthrs/journal": "1.0.0-rc.7",
396
- "@smthrs/kernel": "1.0.0-rc.7",
397
- "@smthrs/keys": "1.0.0-rc.7",
398
- "@smthrs/plan": "1.0.0-rc.7",
399
- "@smthrs/plan-store": "1.0.0-rc.7",
400
- "@smthrs/run-store": "1.0.0-rc.7",
401
- "@smthrs/step-cache": "1.0.0-rc.7",
402
- "@smthrs/capability": "1.0.0-rc.7"
389
+ "@smthrs/artifacts": "1.0.0-rc.8",
390
+ "@smthrs/canonical": "1.0.0-rc.8",
391
+ "@smthrs/crypto": "1.0.0-rc.8",
392
+ "@smthrs/database": "1.0.0-rc.8",
393
+ "@smthrs/engine": "1.0.0-rc.8",
394
+ "@smthrs/flow": "1.0.0-rc.8",
395
+ "@smthrs/journal": "1.0.0-rc.8",
396
+ "@smthrs/kernel": "1.0.0-rc.8",
397
+ "@smthrs/keys": "1.0.0-rc.8",
398
+ "@smthrs/plan": "1.0.0-rc.8",
399
+ "@smthrs/plan-store": "1.0.0-rc.8",
400
+ "@smthrs/run-store": "1.0.0-rc.8",
401
+ "@smthrs/step-cache": "1.0.0-rc.8",
402
+ "@smthrs/capability": "1.0.0-rc.8"
403
403
  },
404
404
  "devDependencies": {
405
405
  "@effect/language-service": "0.87.2",
406
406
  "@effect/platform-node": "4.0.4",
407
407
  "@effect/vitest": "4.0.4",
408
408
  "@eslint/js": "9.39.5",
409
- "@smthrs/core": "1.0.0-rc.7",
410
- "@smthrs/patterns": "1.0.0-rc.7",
411
- "@smthrs/platform-node": "1.0.0-rc.7",
409
+ "@smthrs/core": "1.0.0-rc.8",
410
+ "@smthrs/patterns": "1.0.0-rc.8",
411
+ "@smthrs/platform-node": "1.0.0-rc.8",
412
412
  "@types/node": "26.4.1",
413
413
  "@vitest/coverage-v8": "5.0.0",
414
414
  "dprint": "0.57.1",
@@ -241,6 +241,13 @@ export const make = (options: {
241
241
  // them written into this workspace.
242
242
  const written = yield* remote.get(digest).pipe(
243
243
  Effect.flatMap((bytes) => local.put(bytes)),
244
+ // A custom tier may skip that verification; the address `put`
245
+ // returned is the proof the requested blob is now local.
246
+ Effect.flatMap((address) =>
247
+ address === digest ?
248
+ Effect.void :
249
+ Effect.fail({ message: `the shared tier served the bytes of ${address}` })
250
+ ),
244
251
  Effect.tapError(fallBack("the artifact could not be fetched", [digest])),
245
252
  Effect.option
246
253
  )
@@ -407,6 +407,23 @@ const ensureEmptyDirectory = (
407
407
  }
408
408
  })
409
409
 
410
+ /**
411
+ * Removes every entry of a directory this module filled after admitting it
412
+ * empty, leaving the directory reusable for a retry. Cleanup is best effort;
413
+ * the caller reports the original failure.
414
+ */
415
+ const discardEntries = (fs: FileSystem.FileSystem, directory: string): Effect.Effect<void> =>
416
+ fs.readDirectory(directory).pipe(
417
+ Effect.flatMap((entries) =>
418
+ Effect.forEach(
419
+ entries,
420
+ (entry) => fs.remove(`${directory}/${entry}`, { recursive: true, force: true }),
421
+ { discard: true }
422
+ )
423
+ ),
424
+ Effect.ignore
425
+ )
426
+
410
427
  /**
411
428
  * A relative blob path in the store's fanout layout: a two-hex directory
412
429
  * holding a 64-hex digest that begins with it. Anything else under the
@@ -660,18 +677,8 @@ export const backup = <R = never, E = never>(
660
677
  // The directory was empty when the capture began, so everything in it is
661
678
  // this capture's: the snapshot, any SQLite sidecar the inspection
662
679
  // connection left, and the copied blobs. A failed capture removes them and
663
- // leaves the directory reusable for a retry. Cleanup is best effort; the
664
- // original failure is the one reported.
665
- const discardPartial = fs.readDirectory(options.directory).pipe(
666
- Effect.flatMap((entries) =>
667
- Effect.forEach(
668
- entries,
669
- (entry) => fs.remove(`${options.directory}/${entry}`, { recursive: true, force: true }),
670
- { discard: true }
671
- )
672
- ),
673
- Effect.ignore
674
- )
680
+ // leaves the directory reusable for a retry.
681
+ const discardPartial = discardEntries(fs, options.directory)
675
682
 
676
683
  return yield* (options.objectsDirectory === undefined
677
684
  ? capture
@@ -719,6 +726,57 @@ export const verify = Effect.fn("DisasterRecovery.verify")(function*(
719
726
  return manifest
720
727
  })
721
728
 
729
+ /** Copies a verified backup into an admitted empty target directory. */
730
+ const writeRestoredStore = (
731
+ fs: FileSystem.FileSystem,
732
+ options: RestoreOptions,
733
+ manifest: BackupManifest,
734
+ maxFileSizeBytes: number
735
+ ): Effect.Effect<RestoredStore, DisasterRecoveryError, Crypto.Crypto> =>
736
+ Effect.gen(function*() {
737
+ const databaseBytes = yield* checkedFile(
738
+ fs,
739
+ "restore",
740
+ `${options.backupDirectory}/${manifest.database.file}`,
741
+ manifest.database.sha256,
742
+ maxFileSizeBytes
743
+ )
744
+ const databaseFile = `${options.targetDirectory}/${databaseFileName}`
745
+ yield* fs.writeFile(databaseFile, databaseBytes).pipe(
746
+ Effect.mapError(ioFailure("restore", `writing ${databaseFile}`))
747
+ )
748
+
749
+ const objectsDirectory = `${options.targetDirectory}/${objectsDirectoryName}`
750
+ for (const entry of manifest.artifacts) {
751
+ const bytes = yield* checkedFile(
752
+ fs,
753
+ "restore",
754
+ blobPath(`${options.backupDirectory}/${objectsDirectoryName}`, entry.digest),
755
+ entry.digest,
756
+ maxFileSizeBytes
757
+ )
758
+ yield* fs.makeDirectory(`${objectsDirectory}/${entry.digest.slice(0, 2)}`, { recursive: true }).pipe(
759
+ Effect.mapError(ioFailure("restore", `creating the objects directory for ${entry.digest}`))
760
+ )
761
+ yield* fs.writeFile(blobPath(objectsDirectory, entry.digest), bytes).pipe(
762
+ Effect.mapError(ioFailure("restore", `writing the blob for ${entry.digest}`))
763
+ )
764
+ }
765
+
766
+ const restoredAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
767
+ const marker = {
768
+ formatVersion: 1,
769
+ restoredAtMs,
770
+ backupCreatedAtMs: manifest.createdAtMs,
771
+ databaseSha256: manifest.database.sha256
772
+ }
773
+ yield* fs.writeFileString(
774
+ `${options.targetDirectory}/${restoredMarkerFileName}`,
775
+ `${JSON.stringify(marker, null, 2)}\n`
776
+ ).pipe(Effect.mapError(ioFailure("restore", `writing ${restoredMarkerFileName}`)))
777
+ return { databaseFile, objectsDirectory, manifest } satisfies RestoredStore
778
+ })
779
+
722
780
  /**
723
781
  * Restores a verified backup into an empty target directory.
724
782
  *
@@ -728,7 +786,8 @@ export const verify = Effect.fn("DisasterRecovery.verify")(function*(
728
786
  * is a fresh store directory to compose `NodeDatabase.layer` and
729
787
  * `ArtifactStore.layerFileSystem` over. The database is restored with the
730
788
  * pre-backup owner fences still in place — run {@link fence} against it
731
- * before any engine adopts it.
789
+ * before any engine adopts it. A failed copy removes everything it wrote, so
790
+ * the target is left empty and reusable for a retry.
732
791
  *
733
792
  * @since 0.1.0
734
793
  * @category operations
@@ -741,47 +800,13 @@ export const restore = Effect.fn("DisasterRecovery.restore")(function*(options:
741
800
  const manifest = yield* readManifest(fs, "restore", options.backupDirectory, maxFileSizeBytes)
742
801
  yield* ensureEmptyDirectory(fs, "restore", options.targetDirectory)
743
802
 
744
- const databaseBytes = yield* checkedFile(
745
- fs,
746
- "restore",
747
- `${options.backupDirectory}/${manifest.database.file}`,
748
- manifest.database.sha256,
749
- maxFileSizeBytes
750
- )
751
- const databaseFile = `${options.targetDirectory}/${databaseFileName}`
752
- yield* fs.writeFile(databaseFile, databaseBytes).pipe(
753
- Effect.mapError(ioFailure("restore", `writing ${databaseFile}`))
803
+ // The target was empty when the copy began, so everything in it is this
804
+ // restore's. A refused blob or a failed write removes the partial store, so
805
+ // nothing store-shaped is left to compose over and a retry can reuse it.
806
+ return yield* Effect.onError(
807
+ writeRestoredStore(fs, options, manifest, maxFileSizeBytes),
808
+ () => discardEntries(fs, options.targetDirectory)
754
809
  )
755
-
756
- const objectsDirectory = `${options.targetDirectory}/${objectsDirectoryName}`
757
- for (const entry of manifest.artifacts) {
758
- const bytes = yield* checkedFile(
759
- fs,
760
- "restore",
761
- blobPath(`${options.backupDirectory}/${objectsDirectoryName}`, entry.digest),
762
- entry.digest,
763
- maxFileSizeBytes
764
- )
765
- yield* fs.makeDirectory(`${objectsDirectory}/${entry.digest.slice(0, 2)}`, { recursive: true }).pipe(
766
- Effect.mapError(ioFailure("restore", `creating the objects directory for ${entry.digest}`))
767
- )
768
- yield* fs.writeFile(blobPath(objectsDirectory, entry.digest), bytes).pipe(
769
- Effect.mapError(ioFailure("restore", `writing the blob for ${entry.digest}`))
770
- )
771
- }
772
-
773
- const restoredAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
774
- const marker = {
775
- formatVersion: 1,
776
- restoredAtMs,
777
- backupCreatedAtMs: manifest.createdAtMs,
778
- databaseSha256: manifest.database.sha256
779
- }
780
- yield* fs.writeFileString(
781
- `${options.targetDirectory}/${restoredMarkerFileName}`,
782
- `${JSON.stringify(marker, null, 2)}\n`
783
- ).pipe(Effect.mapError(ioFailure("restore", `writing ${restoredMarkerFileName}`)))
784
- return { databaseFile, objectsDirectory, manifest } satisfies RestoredStore
785
810
  })
786
811
 
787
812
  /**
@@ -868,10 +893,12 @@ export const fence = Effect.fn("DisasterRecovery.fence")(function*(manifest: Bac
868
893
  /**
869
894
  * Restores a backup and immediately fences the restored database.
870
895
  *
871
- * This is the one-shot operator API for disaster recovery. The database
872
- * driver itself still arrives through a caller-supplied Layer so this module
873
- * remains host-neutral and browser-bundleable; the bundled Node script passes
874
- * `NodeDatabase.layer` for the restored file.
896
+ * This is the one-shot operator API for disaster recovery. When fencing
897
+ * refuses, the restored files are removed, so no engine can adopt a store that
898
+ * still carries pre-backup owners. The database driver itself still arrives
899
+ * through a caller-supplied Layer so this module remains host-neutral and
900
+ * browser-bundleable; the bundled Node script passes `NodeDatabase.layer` for
901
+ * the restored file.
875
902
  *
876
903
  * @since 0.1.0
877
904
  * @category operations
@@ -880,9 +907,14 @@ export const restoreAndFence = <R = never, E = never>(
880
907
  options: RestoreAndFenceOptions<R, E>
881
908
  ): Effect.Effect<FencedRestoredStore, DisasterRecoveryError | E, FileSystem.FileSystem | Crypto.Crypto | R> =>
882
909
  Effect.gen(function*() {
910
+ const fs = yield* FileSystem.FileSystem
883
911
  const restored = yield* restore(options)
912
+ // A refused fence leaves a database that still carries pre-backup owner
913
+ // fences beside a `restored.json` marker. Remove the whole restore so no
914
+ // engine can adopt it and the operator can retry into the same target.
884
915
  const summary = yield* fence(restored.manifest).pipe(
885
- Effect.provide(options.databaseLayer(restored.databaseFile))
916
+ Effect.provide(options.databaseLayer(restored.databaseFile)),
917
+ Effect.onError(() => discardEntries(fs, options.targetDirectory))
886
918
  )
887
919
  return { ...restored, fence: summary } satisfies FencedRestoredStore
888
920
  })
package/src/index.ts CHANGED
@@ -67,7 +67,21 @@ export * as RunChangeFeed from "./RunChangeFeed.ts"
67
67
  * @category services
68
68
  */
69
69
  export * as PlanInputStore from "./PlanInputStore.ts"
70
+
71
+ /**
72
+ * Owner-fenced plan merge decisions, recorded before a merge is appended.
73
+ *
74
+ * @since 1.0.0
75
+ * @category services
76
+ */
70
77
  export * as PlanMergeStore from "./PlanMergeStore.ts"
78
+
79
+ /**
80
+ * Drives a persisted plan: the node scheduler.
81
+ *
82
+ * @since 0.1.0
83
+ * @category services
84
+ */
71
85
  export * as PlanScheduler from "./PlanScheduler.ts"
72
86
 
73
87
  /**
@@ -1229,7 +1229,21 @@ export const make = (deps: Dependencies) => {
1229
1229
  // permits and interleave the very span the permit serializes. The step
1230
1230
  // key, not the narrowed cache address, is what serializes here, because
1231
1231
  // the attempt rows two scopes of one key share are what they race over.
1232
- return yield* admission.withPermit(`${deps.runId}|${stepKeyDigest}`)(
1232
+ //
1233
+ // A cache-eligible dispatch also holds the permit for its cache address,
1234
+ // taken second so every dispatch acquires the two in one order (issue
1235
+ // #3898). Runs share that address, so a second run asking for a result
1236
+ // this process is already computing waits for it and is answered by its
1237
+ // cache row instead of running the body beside it. That includes a run
1238
+ // the stale-running sweep reclaimed from a dead owner, whose in-flight
1239
+ // attempt would otherwise re-execute next to the run that superseded
1240
+ // it. Other processes are not excluded: two processes may still execute
1241
+ // one key at once, and the second recording resolves as before.
1242
+ const exclusive = <A, E, R>(effect: Effect.Effect<A, E, R>) =>
1243
+ admission.withPermit(`run:${deps.runId}|${stepKeyDigest}`)(
1244
+ cacheDeclaration._tag === "Eligible" ? admission.withPermit(`cache:${keyDigest}`)(effect) : effect
1245
+ )
1246
+ return yield* exclusive(
1233
1247
  Effect.gen(function*() {
1234
1248
  // Lifecycle announcements take a per-attempt producer identity
1235
1249
  // (issue #91): adoption — or a replay after a crash in the
@@ -1476,6 +1490,26 @@ export const make = (deps: Dependencies) => {
1476
1490
  // A run's admitted attempt is replay authority. Shared cache rows
1477
1491
  // are replaceable and cannot overwrite its result or failure.
1478
1492
  const existing = yield* attempts.get(attemptId)
1493
+ /**
1494
+ * AN ORPHANED ATTEMPT ASKS THE CACHE FIRST (issue #3898). A `running`
1495
+ * row read under the permit is crash evidence (see adoption below):
1496
+ * the attempt never settled, so it holds no result for the cache to
1497
+ * overwrite. A reclaimed run reaching it is in the same position as
1498
+ * a fresh dispatch, and must be answered by a verified row another
1499
+ * run recorded under this key rather than by re-executing the body.
1500
+ * A crossing that already succeeded is the one exception: the row
1501
+ * carries this run's own result, which adoption settles from.
1502
+ *
1503
+ * Only cache-eligible declarations get here, which are sealed with a
1504
+ * hard boundary: an irreversible or compensable attempt, or a sealed
1505
+ * one without a boundary, adopts and re-executes exactly as before.
1506
+ * An unkeyed sealed step's key folds in its run, so no other run can
1507
+ * have recorded it, and its adoption re-executes as before too.
1508
+ */
1509
+ const orphaned = Option.isSome(existing) && existing.value.state === "running" &&
1510
+ decodeMeta(existing.value.meta)?.effectCrossing !== "succeeded"
1511
+ ? existing.value
1512
+ : undefined
1479
1513
  if (cacheDeclaration._tag === "Eligible") {
1480
1514
  const observed = yield* cache.get(keyDigest).pipe(
1481
1515
  Effect.catch((error) =>
@@ -1513,7 +1547,7 @@ export const make = (deps: Dependencies) => {
1513
1547
  }
1514
1548
  ))
1515
1549
  }
1516
- if (candidate._tag === "CandidateEvidence" && Option.isNone(existing)) {
1550
+ if (candidate._tag === "CandidateEvidence" && (Option.isNone(existing) || orphaned !== undefined)) {
1517
1551
  const boundary = yield* StepBoundary.StepBoundary
1518
1552
  // Skyframe's dirty check, not "the declaration changed" (issue
1519
1553
  // #90): the read-set digests folded into the step key are caller
@@ -1560,6 +1594,32 @@ export const make = (deps: Dependencies) => {
1560
1594
  recordedRunId: cached.value.recordedRunId,
1561
1595
  recordedEventSeq: cached.value.recordedEventSeq
1562
1596
  }))
1597
+ if (orphaned !== undefined) {
1598
+ // Close the orphaned attempt with the result that answered
1599
+ // it, so the run carries no attempt that started and never
1600
+ // finished, and a later re-drive replays this row. The
1601
+ // meta records no boundary evidence: this attempt measured
1602
+ // none of its own, so the row is never published back.
1603
+ const finishedAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
1604
+ const settled = yield* settleAttempt({
1605
+ ...attemptId,
1606
+ state: "succeeded",
1607
+ finishedAtMs,
1608
+ outcome: attemptPayload(cached.value.result),
1609
+ meta: attemptPayload(
1610
+ {
1611
+ ...decodeMeta(orphaned.meta),
1612
+ ...declarationMeta,
1613
+ admittedBy: deps.owner
1614
+ } satisfies AttemptMeta
1615
+ )
1616
+ }, [
1617
+ JournalRecords.attemptFinished(attemptSource("finished"), { ...attemptId, state: "succeeded" })
1618
+ ])
1619
+ if (!settled) {
1620
+ return yield* Effect.interrupt
1621
+ }
1622
+ }
1563
1623
  // Counted after the provenance emit: `verified_hit` means
1564
1624
  // the cached result was served, and a journal failure on
1565
1625
  // the emit fails the dispatch before any result is
@@ -1953,7 +2013,8 @@ export const make = (deps: Dependencies) => {
1953
2013
  * is crash evidence, not a live admission (issue #71): the incarnation
1954
2014
  * that admitted the attempt died before finishing (SIGKILL, OOM), and
1955
2015
  * the #53 stale-running sweep re-drove the run to a new owner — or the
1956
- * same owner after restart. The attempt never completed, so it must
2016
+ * same owner after restart. The attempt never completed, so unless a
2017
+ * verified cache row answered it above (issue #3898), it must
1957
2018
  * re-execute under its original number rather than fall through to
1958
2019
  * `attempts.put` (whose `Conflict` on the differing `startedAtMs`
1959
2020
  * surfaced as `AttemptAdmissionRejected`, permanently failing a