@smthrs/engine-store 1.0.0-rc.7 → 1.0.0-rc.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -1
- package/dist/cjs/ArtifactSync.d.ts.map +1 -1
- package/dist/cjs/ArtifactSync.js +5 -0
- package/dist/cjs/ArtifactSync.js.map +2 -2
- package/dist/cjs/DisasterRecovery.d.ts +9 -26
- package/dist/cjs/DisasterRecovery.d.ts.map +1 -1
- package/dist/cjs/DisasterRecovery.js +27 -18
- package/dist/cjs/DisasterRecovery.js.map +2 -2
- package/dist/cjs/index.d.ts +12 -0
- package/dist/cjs/index.d.ts.map +1 -1
- package/dist/cjs/index.js.map +2 -2
- package/dist/cjs/internal/ActionPersistence.d.ts.map +1 -1
- package/dist/cjs/internal/ActionPersistence.js +27 -2
- package/dist/cjs/internal/ActionPersistence.js.map +3 -3
- package/dist/esm/ArtifactSync.d.ts.map +1 -1
- package/dist/esm/ArtifactSync.js +6 -1
- package/dist/esm/ArtifactSync.js.map +1 -1
- package/dist/esm/DisasterRecovery.d.ts +9 -26
- package/dist/esm/DisasterRecovery.d.ts.map +1 -1
- package/dist/esm/DisasterRecovery.js +46 -27
- package/dist/esm/DisasterRecovery.js.map +1 -1
- package/dist/esm/index.d.ts +12 -0
- package/dist/esm/index.d.ts.map +1 -1
- package/dist/esm/index.js +12 -0
- package/dist/esm/index.js.map +1 -1
- package/dist/esm/internal/ActionPersistence.d.ts.map +1 -1
- package/dist/esm/internal/ActionPersistence.js +59 -3
- package/dist/esm/internal/ActionPersistence.js.map +1 -1
- package/docs/README.md +3 -1
- package/docs/api.md +4 -1
- package/docs/concepts/attempts-and-replay.md +15 -2
- package/docs/guides/back-up-and-restore.md +9 -3
- package/docs/guides/reclaim-runs-from-a-dead-host.md +16 -0
- package/docs/installation.md +5 -1
- package/docs/quickstart.md +9 -1
- package/docs/troubleshooting.md +3 -0
- package/package.json +18 -18
- package/src/ArtifactSync.ts +7 -0
- package/src/DisasterRecovery.ts +90 -58
- package/src/index.ts +14 -0
- package/src/internal/ActionPersistence.ts +64 -3
package/docs/README.md
CHANGED
|
@@ -11,7 +11,9 @@ over.
|
|
|
11
11
|
It is an [Effect](https://effect.website) library: every surface below is a
|
|
12
12
|
`Layer` you compose or an `Effect` you run.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
```bash
|
|
15
|
+
npm install @smthrs/engine-store@next effect
|
|
16
|
+
```
|
|
15
17
|
|
|
16
18
|
## What it solves
|
|
17
19
|
|
package/docs/api.md
CHANGED
|
@@ -895,7 +895,10 @@ durable fails and the shared entry is withheld.
|
|
|
895
895
|
|
|
896
896
|
`hydrate(digests)` establishes that this host can resolve every referenced
|
|
897
897
|
artifact and reports whether the replay is now worth retrying. It never fails a
|
|
898
|
-
run: a shared tier that is down must not stop work that can simply be done.
|
|
898
|
+
run: a shared tier that is down must not stop work that can simply be done. A
|
|
899
|
+
downloaded blob counts only when this host's store files it under the requested
|
|
900
|
+
digest, so a shared tier that serves the wrong bytes costs the replay, never
|
|
901
|
+
its correctness.
|
|
899
902
|
|
|
900
903
|
The two tiers cannot both be layers, because they inhabit the same tag and
|
|
901
904
|
composing them would shadow one with the other.
|
|
@@ -30,15 +30,21 @@ question, answered in [Cache admission](./cache-admission.md).
|
|
|
30
30
|
| ------------- | ------------------------------------------------------------------------------------------------------------------------------- |
|
|
31
31
|
| `succeeded` | Returns the recorded result, and calls `StepBoundary.replayOutputs` first so the workspace holds the outputs the step produced. |
|
|
32
32
|
| `failed` | Rethrows the persisted domain failure. It never re-admits the attempt. |
|
|
33
|
-
| `running` |
|
|
33
|
+
| `running` | Asks the step cache first when the step is cache-eligible, then recovers under the run fence and admission permit. |
|
|
34
34
|
| `suspended` | Continues the same attempt rather than burning a new one against the retry budget. |
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
A settled attempt remains authoritative when a matching shared cache row
|
|
37
37
|
exists; another run's cached success cannot replace its result or failure.
|
|
38
38
|
Failure values were schema-encoded before persistence, so a `_tag` survives
|
|
39
39
|
the JSON round trip and a `RetryPolicy`'s non-retryable matching still applies
|
|
40
40
|
on replay.
|
|
41
41
|
|
|
42
|
+
A `running` attempt has no result of its own. When its step is cache-eligible
|
|
43
|
+
(sealed, with a hard boundary), a verified row another run recorded under the
|
|
44
|
+
same key answers it, and the row is closed as `succeeded` with that result. A
|
|
45
|
+
recorded successful effect crossing still wins over the cache, because it
|
|
46
|
+
carries this run's own result.
|
|
47
|
+
|
|
42
48
|
## Which columns are redacted
|
|
43
49
|
|
|
44
50
|
Attempt and run rows share a database with the journal, so failure text meets
|
|
@@ -62,6 +68,13 @@ A permit covers one run and step key through admission, execution, and
|
|
|
62
68
|
settlement. Concurrent same-key dispatches wait, then replay the settled
|
|
63
69
|
attempt instead of executing another body. Interruption releases the permit.
|
|
64
70
|
|
|
71
|
+
A cache-eligible dispatch also holds a permit for its cache address, which runs
|
|
72
|
+
share. A second run that asks one engine for a step that engine is already
|
|
73
|
+
computing waits, then takes the first run's cache row instead of running the
|
|
74
|
+
body beside it. Engines in other processes are not excluded: two processes can
|
|
75
|
+
still execute one key at once, and the second recording resolves as described
|
|
76
|
+
in [Cache admission](./cache-admission.md).
|
|
77
|
+
|
|
65
78
|
Holding the permit and the run fence permits recovery of a stranded `running`
|
|
66
79
|
attempt. A recorded successful effect crossing supplies the result directly;
|
|
67
80
|
an unresolved irreversible crossing requires an idempotency key before retry.
|
|
@@ -110,10 +110,16 @@ file (`store.sqlite3`), an `objects` directory, and a `restored.json` marker, an
|
|
|
110
110
|
is a fresh store directory to compose `NodeDatabase.layer` and
|
|
111
111
|
`ArtifactStore.layerFileSystem` over.
|
|
112
112
|
|
|
113
|
+
A failed restore removes what it wrote, so nothing store-shaped is left in the
|
|
114
|
+
target and you can retry into the same directory. A target that was not empty
|
|
115
|
+
is refused with `not_empty` and left untouched.
|
|
116
|
+
|
|
113
117
|
`restore` alone leaves the pre-backup owner fences in place. Run
|
|
114
118
|
`DisasterRecovery.fence(manifest)` against the restored file before any engine
|
|
115
119
|
adopts it, or use `restoreAndFence`, which does both and returns a
|
|
116
|
-
`FencedRestoredStore`.
|
|
120
|
+
`FencedRestoredStore`. When fencing refuses, `restoreAndFence` also removes the
|
|
121
|
+
restored files, so no engine can adopt a store that still carries pre-backup
|
|
122
|
+
owners.
|
|
117
123
|
|
|
118
124
|
## What fencing does
|
|
119
125
|
|
|
@@ -149,8 +155,8 @@ its own fresh owner.
|
|
|
149
155
|
| `not_empty` | A target or backup directory exists and is not empty. |
|
|
150
156
|
| `invalid_manifest` | The manifest is missing or does not decode. |
|
|
151
157
|
| `missing_file` | A file the manifest lists is absent. |
|
|
152
|
-
| `digest_mismatch` |
|
|
153
|
-
| `artifact_corruption` | A blob does not hash to its address.
|
|
158
|
+
| `digest_mismatch` | A backup file does not hash to its manifest digest. |
|
|
159
|
+
| `artifact_corruption` | A live blob does not hash to its address during a backup. |
|
|
154
160
|
| `snapshot_incomplete` | The snapshot references artifacts the capture could not take. |
|
|
155
161
|
| `schema_mismatch` | A recorded migration is missing or has a different name. |
|
|
156
162
|
| `io` | A host read or write refused, including an oversized file. |
|
|
@@ -93,6 +93,22 @@ at the staleness cutoff. A deferral is forgotten as soon as the row leaves the
|
|
|
93
93
|
stale window, so a fresh stall under a new lease is probed on the first tick
|
|
94
94
|
that sees it.
|
|
95
95
|
|
|
96
|
+
## What a reclaimed run does with its unfinished step
|
|
97
|
+
|
|
98
|
+
The step that was running when the owner died left an attempt row in state
|
|
99
|
+
`running`. The re-drive reaches that attempt again, and what it does depends on
|
|
100
|
+
the step:
|
|
101
|
+
|
|
102
|
+
- A cache-eligible step (sealed, with a hard boundary) asks the step cache
|
|
103
|
+
first. When another run already recorded the same key, for example the run
|
|
104
|
+
you started to replace the killed one, the reclaimed attempt takes that
|
|
105
|
+
result and the body does not run again. When that run is still computing it
|
|
106
|
+
in the same process, the reclaimed attempt waits for it.
|
|
107
|
+
- Any other step re-executes under its original attempt number. An
|
|
108
|
+
irreversible step whose effect was dispatched but never recorded as settled
|
|
109
|
+
needs an idempotency key to re-execute; without one the run fails with
|
|
110
|
+
`IrreversibleRetryRequiresIdempotencyKey`.
|
|
111
|
+
|
|
96
112
|
## Check a run's own view
|
|
97
113
|
|
|
98
114
|
`DurableEngineState` answers directly, which is useful in a diagnostic tool:
|
package/docs/installation.md
CHANGED
|
@@ -7,7 +7,11 @@ sidebar:
|
|
|
7
7
|
|
|
8
8
|
## Install the package
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
```bash
|
|
11
|
+
npm install @smthrs/engine-store@next effect
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
The `next` tag carries the 1.0 release candidates.
|
|
11
15
|
|
|
12
16
|
The package requires Node.js 26.4.0 or later and ships as both ESM and
|
|
13
17
|
CommonJS with TypeScript declarations. Its runtime dependencies, including
|
package/docs/quickstart.md
CHANGED
|
@@ -13,7 +13,11 @@ the whole point of the package.
|
|
|
13
13
|
## Prerequisites
|
|
14
14
|
|
|
15
15
|
- Node.js 26.4.0 or later.
|
|
16
|
-
- A
|
|
16
|
+
- A project with the packages installed:
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npm install @smthrs/engine-store@next @smthrs/artifacts@next @smthrs/database@next @smthrs/flow@next @smthrs/journal@next @smthrs/kernel@next @smthrs/platform-node@next @smthrs/run-store@next @smthrs/step-cache@next @effect/platform-node effect
|
|
20
|
+
```
|
|
17
21
|
|
|
18
22
|
## Declare a sealed action and a flow
|
|
19
23
|
|
|
@@ -193,6 +197,10 @@ console.log(await Effect.runPromise(main("./.quickstart/.flows/engine.db", "./.q
|
|
|
193
197
|
{ first: '42', second: '42', executions: 1 }
|
|
194
198
|
```
|
|
195
199
|
|
|
200
|
+
The assembled program is
|
|
201
|
+
[`examples/quickstart.ts`](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/engine-store/examples/quickstart.ts);
|
|
202
|
+
the package's test suite runs it on every change.
|
|
203
|
+
|
|
196
204
|
## What just happened
|
|
197
205
|
|
|
198
206
|
The first run claimed `analyse-1`, opened an attempt row for the sealed step,
|
package/docs/troubleshooting.md
CHANGED
|
@@ -304,6 +304,9 @@ migrations omit or rename a historical entry recorded in the manifest. Forward
|
|
|
304
304
|
additions within installed lower blocks are compatible; check historical IDs
|
|
305
305
|
and names before fencing again.
|
|
306
306
|
|
|
307
|
+
A failed backup, `restore`, or `restoreAndFence` removes the files it wrote, so
|
|
308
|
+
fix the cause and retry into the same directory.
|
|
309
|
+
|
|
307
310
|
## The composition does not compile
|
|
308
311
|
|
|
309
312
|
**What happened.** A required service is missing. A composition failure is an
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@smthrs/engine-store",
|
|
3
3
|
"type": "module",
|
|
4
|
-
"version": "1.0.0-rc.
|
|
4
|
+
"version": "1.0.0-rc.8",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"smthrs": {
|
|
7
7
|
"group": "engine"
|
|
@@ -386,29 +386,29 @@
|
|
|
386
386
|
"tag": "next"
|
|
387
387
|
},
|
|
388
388
|
"dependencies": {
|
|
389
|
-
"@smthrs/artifacts": "1.0.0-rc.
|
|
390
|
-
"@smthrs/canonical": "1.0.0-rc.
|
|
391
|
-
"@smthrs/crypto": "1.0.0-rc.
|
|
392
|
-
"@smthrs/database": "1.0.0-rc.
|
|
393
|
-
"@smthrs/engine": "1.0.0-rc.
|
|
394
|
-
"@smthrs/flow": "1.0.0-rc.
|
|
395
|
-
"@smthrs/journal": "1.0.0-rc.
|
|
396
|
-
"@smthrs/kernel": "1.0.0-rc.
|
|
397
|
-
"@smthrs/keys": "1.0.0-rc.
|
|
398
|
-
"@smthrs/plan": "1.0.0-rc.
|
|
399
|
-
"@smthrs/plan-store": "1.0.0-rc.
|
|
400
|
-
"@smthrs/run-store": "1.0.0-rc.
|
|
401
|
-
"@smthrs/step-cache": "1.0.0-rc.
|
|
402
|
-
"@smthrs/capability": "1.0.0-rc.
|
|
389
|
+
"@smthrs/artifacts": "1.0.0-rc.8",
|
|
390
|
+
"@smthrs/canonical": "1.0.0-rc.8",
|
|
391
|
+
"@smthrs/crypto": "1.0.0-rc.8",
|
|
392
|
+
"@smthrs/database": "1.0.0-rc.8",
|
|
393
|
+
"@smthrs/engine": "1.0.0-rc.8",
|
|
394
|
+
"@smthrs/flow": "1.0.0-rc.8",
|
|
395
|
+
"@smthrs/journal": "1.0.0-rc.8",
|
|
396
|
+
"@smthrs/kernel": "1.0.0-rc.8",
|
|
397
|
+
"@smthrs/keys": "1.0.0-rc.8",
|
|
398
|
+
"@smthrs/plan": "1.0.0-rc.8",
|
|
399
|
+
"@smthrs/plan-store": "1.0.0-rc.8",
|
|
400
|
+
"@smthrs/run-store": "1.0.0-rc.8",
|
|
401
|
+
"@smthrs/step-cache": "1.0.0-rc.8",
|
|
402
|
+
"@smthrs/capability": "1.0.0-rc.8"
|
|
403
403
|
},
|
|
404
404
|
"devDependencies": {
|
|
405
405
|
"@effect/language-service": "0.87.2",
|
|
406
406
|
"@effect/platform-node": "4.0.4",
|
|
407
407
|
"@effect/vitest": "4.0.4",
|
|
408
408
|
"@eslint/js": "9.39.5",
|
|
409
|
-
"@smthrs/core": "1.0.0-rc.
|
|
410
|
-
"@smthrs/patterns": "1.0.0-rc.
|
|
411
|
-
"@smthrs/platform-node": "1.0.0-rc.
|
|
409
|
+
"@smthrs/core": "1.0.0-rc.8",
|
|
410
|
+
"@smthrs/patterns": "1.0.0-rc.8",
|
|
411
|
+
"@smthrs/platform-node": "1.0.0-rc.8",
|
|
412
412
|
"@types/node": "26.4.1",
|
|
413
413
|
"@vitest/coverage-v8": "5.0.0",
|
|
414
414
|
"dprint": "0.57.1",
|
package/src/ArtifactSync.ts
CHANGED
|
@@ -241,6 +241,13 @@ export const make = (options: {
|
|
|
241
241
|
// them written into this workspace.
|
|
242
242
|
const written = yield* remote.get(digest).pipe(
|
|
243
243
|
Effect.flatMap((bytes) => local.put(bytes)),
|
|
244
|
+
// A custom tier may skip that verification; the address `put`
|
|
245
|
+
// returned is the proof the requested blob is now local.
|
|
246
|
+
Effect.flatMap((address) =>
|
|
247
|
+
address === digest ?
|
|
248
|
+
Effect.void :
|
|
249
|
+
Effect.fail({ message: `the shared tier served the bytes of ${address}` })
|
|
250
|
+
),
|
|
244
251
|
Effect.tapError(fallBack("the artifact could not be fetched", [digest])),
|
|
245
252
|
Effect.option
|
|
246
253
|
)
|
package/src/DisasterRecovery.ts
CHANGED
|
@@ -407,6 +407,23 @@ const ensureEmptyDirectory = (
|
|
|
407
407
|
}
|
|
408
408
|
})
|
|
409
409
|
|
|
410
|
+
/**
|
|
411
|
+
* Removes every entry of a directory this module filled after admitting it
|
|
412
|
+
* empty, leaving the directory reusable for a retry. Cleanup is best effort;
|
|
413
|
+
* the caller reports the original failure.
|
|
414
|
+
*/
|
|
415
|
+
const discardEntries = (fs: FileSystem.FileSystem, directory: string): Effect.Effect<void> =>
|
|
416
|
+
fs.readDirectory(directory).pipe(
|
|
417
|
+
Effect.flatMap((entries) =>
|
|
418
|
+
Effect.forEach(
|
|
419
|
+
entries,
|
|
420
|
+
(entry) => fs.remove(`${directory}/${entry}`, { recursive: true, force: true }),
|
|
421
|
+
{ discard: true }
|
|
422
|
+
)
|
|
423
|
+
),
|
|
424
|
+
Effect.ignore
|
|
425
|
+
)
|
|
426
|
+
|
|
410
427
|
/**
|
|
411
428
|
* A relative blob path in the store's fanout layout: a two-hex directory
|
|
412
429
|
* holding a 64-hex digest that begins with it. Anything else under the
|
|
@@ -660,18 +677,8 @@ export const backup = <R = never, E = never>(
|
|
|
660
677
|
// The directory was empty when the capture began, so everything in it is
|
|
661
678
|
// this capture's: the snapshot, any SQLite sidecar the inspection
|
|
662
679
|
// connection left, and the copied blobs. A failed capture removes them and
|
|
663
|
-
// leaves the directory reusable for a retry.
|
|
664
|
-
|
|
665
|
-
const discardPartial = fs.readDirectory(options.directory).pipe(
|
|
666
|
-
Effect.flatMap((entries) =>
|
|
667
|
-
Effect.forEach(
|
|
668
|
-
entries,
|
|
669
|
-
(entry) => fs.remove(`${options.directory}/${entry}`, { recursive: true, force: true }),
|
|
670
|
-
{ discard: true }
|
|
671
|
-
)
|
|
672
|
-
),
|
|
673
|
-
Effect.ignore
|
|
674
|
-
)
|
|
680
|
+
// leaves the directory reusable for a retry.
|
|
681
|
+
const discardPartial = discardEntries(fs, options.directory)
|
|
675
682
|
|
|
676
683
|
return yield* (options.objectsDirectory === undefined
|
|
677
684
|
? capture
|
|
@@ -719,6 +726,57 @@ export const verify = Effect.fn("DisasterRecovery.verify")(function*(
|
|
|
719
726
|
return manifest
|
|
720
727
|
})
|
|
721
728
|
|
|
729
|
+
/** Copies a verified backup into an admitted empty target directory. */
|
|
730
|
+
const writeRestoredStore = (
|
|
731
|
+
fs: FileSystem.FileSystem,
|
|
732
|
+
options: RestoreOptions,
|
|
733
|
+
manifest: BackupManifest,
|
|
734
|
+
maxFileSizeBytes: number
|
|
735
|
+
): Effect.Effect<RestoredStore, DisasterRecoveryError, Crypto.Crypto> =>
|
|
736
|
+
Effect.gen(function*() {
|
|
737
|
+
const databaseBytes = yield* checkedFile(
|
|
738
|
+
fs,
|
|
739
|
+
"restore",
|
|
740
|
+
`${options.backupDirectory}/${manifest.database.file}`,
|
|
741
|
+
manifest.database.sha256,
|
|
742
|
+
maxFileSizeBytes
|
|
743
|
+
)
|
|
744
|
+
const databaseFile = `${options.targetDirectory}/${databaseFileName}`
|
|
745
|
+
yield* fs.writeFile(databaseFile, databaseBytes).pipe(
|
|
746
|
+
Effect.mapError(ioFailure("restore", `writing ${databaseFile}`))
|
|
747
|
+
)
|
|
748
|
+
|
|
749
|
+
const objectsDirectory = `${options.targetDirectory}/${objectsDirectoryName}`
|
|
750
|
+
for (const entry of manifest.artifacts) {
|
|
751
|
+
const bytes = yield* checkedFile(
|
|
752
|
+
fs,
|
|
753
|
+
"restore",
|
|
754
|
+
blobPath(`${options.backupDirectory}/${objectsDirectoryName}`, entry.digest),
|
|
755
|
+
entry.digest,
|
|
756
|
+
maxFileSizeBytes
|
|
757
|
+
)
|
|
758
|
+
yield* fs.makeDirectory(`${objectsDirectory}/${entry.digest.slice(0, 2)}`, { recursive: true }).pipe(
|
|
759
|
+
Effect.mapError(ioFailure("restore", `creating the objects directory for ${entry.digest}`))
|
|
760
|
+
)
|
|
761
|
+
yield* fs.writeFile(blobPath(objectsDirectory, entry.digest), bytes).pipe(
|
|
762
|
+
Effect.mapError(ioFailure("restore", `writing the blob for ${entry.digest}`))
|
|
763
|
+
)
|
|
764
|
+
}
|
|
765
|
+
|
|
766
|
+
const restoredAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
|
|
767
|
+
const marker = {
|
|
768
|
+
formatVersion: 1,
|
|
769
|
+
restoredAtMs,
|
|
770
|
+
backupCreatedAtMs: manifest.createdAtMs,
|
|
771
|
+
databaseSha256: manifest.database.sha256
|
|
772
|
+
}
|
|
773
|
+
yield* fs.writeFileString(
|
|
774
|
+
`${options.targetDirectory}/${restoredMarkerFileName}`,
|
|
775
|
+
`${JSON.stringify(marker, null, 2)}\n`
|
|
776
|
+
).pipe(Effect.mapError(ioFailure("restore", `writing ${restoredMarkerFileName}`)))
|
|
777
|
+
return { databaseFile, objectsDirectory, manifest } satisfies RestoredStore
|
|
778
|
+
})
|
|
779
|
+
|
|
722
780
|
/**
|
|
723
781
|
* Restores a verified backup into an empty target directory.
|
|
724
782
|
*
|
|
@@ -728,7 +786,8 @@ export const verify = Effect.fn("DisasterRecovery.verify")(function*(
|
|
|
728
786
|
* is a fresh store directory to compose `NodeDatabase.layer` and
|
|
729
787
|
* `ArtifactStore.layerFileSystem` over. The database is restored with the
|
|
730
788
|
* pre-backup owner fences still in place — run {@link fence} against it
|
|
731
|
-
* before any engine adopts it.
|
|
789
|
+
* before any engine adopts it. A failed copy removes everything it wrote, so
|
|
790
|
+
* the target is left empty and reusable for a retry.
|
|
732
791
|
*
|
|
733
792
|
* @since 0.1.0
|
|
734
793
|
* @category operations
|
|
@@ -741,47 +800,13 @@ export const restore = Effect.fn("DisasterRecovery.restore")(function*(options:
|
|
|
741
800
|
const manifest = yield* readManifest(fs, "restore", options.backupDirectory, maxFileSizeBytes)
|
|
742
801
|
yield* ensureEmptyDirectory(fs, "restore", options.targetDirectory)
|
|
743
802
|
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
manifest
|
|
749
|
-
|
|
750
|
-
)
|
|
751
|
-
const databaseFile = `${options.targetDirectory}/${databaseFileName}`
|
|
752
|
-
yield* fs.writeFile(databaseFile, databaseBytes).pipe(
|
|
753
|
-
Effect.mapError(ioFailure("restore", `writing ${databaseFile}`))
|
|
803
|
+
// The target was empty when the copy began, so everything in it is this
|
|
804
|
+
// restore's. A refused blob or a failed write removes the partial store, so
|
|
805
|
+
// nothing store-shaped is left to compose over and a retry can reuse it.
|
|
806
|
+
return yield* Effect.onError(
|
|
807
|
+
writeRestoredStore(fs, options, manifest, maxFileSizeBytes),
|
|
808
|
+
() => discardEntries(fs, options.targetDirectory)
|
|
754
809
|
)
|
|
755
|
-
|
|
756
|
-
const objectsDirectory = `${options.targetDirectory}/${objectsDirectoryName}`
|
|
757
|
-
for (const entry of manifest.artifacts) {
|
|
758
|
-
const bytes = yield* checkedFile(
|
|
759
|
-
fs,
|
|
760
|
-
"restore",
|
|
761
|
-
blobPath(`${options.backupDirectory}/${objectsDirectoryName}`, entry.digest),
|
|
762
|
-
entry.digest,
|
|
763
|
-
maxFileSizeBytes
|
|
764
|
-
)
|
|
765
|
-
yield* fs.makeDirectory(`${objectsDirectory}/${entry.digest.slice(0, 2)}`, { recursive: true }).pipe(
|
|
766
|
-
Effect.mapError(ioFailure("restore", `creating the objects directory for ${entry.digest}`))
|
|
767
|
-
)
|
|
768
|
-
yield* fs.writeFile(blobPath(objectsDirectory, entry.digest), bytes).pipe(
|
|
769
|
-
Effect.mapError(ioFailure("restore", `writing the blob for ${entry.digest}`))
|
|
770
|
-
)
|
|
771
|
-
}
|
|
772
|
-
|
|
773
|
-
const restoredAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
|
|
774
|
-
const marker = {
|
|
775
|
-
formatVersion: 1,
|
|
776
|
-
restoredAtMs,
|
|
777
|
-
backupCreatedAtMs: manifest.createdAtMs,
|
|
778
|
-
databaseSha256: manifest.database.sha256
|
|
779
|
-
}
|
|
780
|
-
yield* fs.writeFileString(
|
|
781
|
-
`${options.targetDirectory}/${restoredMarkerFileName}`,
|
|
782
|
-
`${JSON.stringify(marker, null, 2)}\n`
|
|
783
|
-
).pipe(Effect.mapError(ioFailure("restore", `writing ${restoredMarkerFileName}`)))
|
|
784
|
-
return { databaseFile, objectsDirectory, manifest } satisfies RestoredStore
|
|
785
810
|
})
|
|
786
811
|
|
|
787
812
|
/**
|
|
@@ -868,10 +893,12 @@ export const fence = Effect.fn("DisasterRecovery.fence")(function*(manifest: Bac
|
|
|
868
893
|
/**
|
|
869
894
|
* Restores a backup and immediately fences the restored database.
|
|
870
895
|
*
|
|
871
|
-
* This is the one-shot operator API for disaster recovery.
|
|
872
|
-
*
|
|
873
|
-
*
|
|
874
|
-
*
|
|
896
|
+
* This is the one-shot operator API for disaster recovery. When fencing
|
|
897
|
+
* refuses, the restored files are removed, so no engine can adopt a store that
|
|
898
|
+
* still carries pre-backup owners. The database driver itself still arrives
|
|
899
|
+
* through a caller-supplied Layer so this module remains host-neutral and
|
|
900
|
+
* browser-bundleable; the bundled Node script passes `NodeDatabase.layer` for
|
|
901
|
+
* the restored file.
|
|
875
902
|
*
|
|
876
903
|
* @since 0.1.0
|
|
877
904
|
* @category operations
|
|
@@ -880,9 +907,14 @@ export const restoreAndFence = <R = never, E = never>(
|
|
|
880
907
|
options: RestoreAndFenceOptions<R, E>
|
|
881
908
|
): Effect.Effect<FencedRestoredStore, DisasterRecoveryError | E, FileSystem.FileSystem | Crypto.Crypto | R> =>
|
|
882
909
|
Effect.gen(function*() {
|
|
910
|
+
const fs = yield* FileSystem.FileSystem
|
|
883
911
|
const restored = yield* restore(options)
|
|
912
|
+
// A refused fence leaves a database that still carries pre-backup owner
|
|
913
|
+
// fences beside a `restored.json` marker. Remove the whole restore so no
|
|
914
|
+
// engine can adopt it and the operator can retry into the same target.
|
|
884
915
|
const summary = yield* fence(restored.manifest).pipe(
|
|
885
|
-
Effect.provide(options.databaseLayer(restored.databaseFile))
|
|
916
|
+
Effect.provide(options.databaseLayer(restored.databaseFile)),
|
|
917
|
+
Effect.onError(() => discardEntries(fs, options.targetDirectory))
|
|
886
918
|
)
|
|
887
919
|
return { ...restored, fence: summary } satisfies FencedRestoredStore
|
|
888
920
|
})
|
package/src/index.ts
CHANGED
|
@@ -67,7 +67,21 @@ export * as RunChangeFeed from "./RunChangeFeed.ts"
|
|
|
67
67
|
* @category services
|
|
68
68
|
*/
|
|
69
69
|
export * as PlanInputStore from "./PlanInputStore.ts"
|
|
70
|
+
|
|
71
|
+
/**
|
|
72
|
+
* Owner-fenced plan merge decisions, recorded before a merge is appended.
|
|
73
|
+
*
|
|
74
|
+
* @since 1.0.0
|
|
75
|
+
* @category services
|
|
76
|
+
*/
|
|
70
77
|
export * as PlanMergeStore from "./PlanMergeStore.ts"
|
|
78
|
+
|
|
79
|
+
/**
|
|
80
|
+
* Drives a persisted plan: the node scheduler.
|
|
81
|
+
*
|
|
82
|
+
* @since 0.1.0
|
|
83
|
+
* @category services
|
|
84
|
+
*/
|
|
71
85
|
export * as PlanScheduler from "./PlanScheduler.ts"
|
|
72
86
|
|
|
73
87
|
/**
|
|
@@ -1229,7 +1229,21 @@ export const make = (deps: Dependencies) => {
|
|
|
1229
1229
|
// permits and interleave the very span the permit serializes. The step
|
|
1230
1230
|
// key, not the narrowed cache address, is what serializes here, because
|
|
1231
1231
|
// the attempt rows two scopes of one key share are what they race over.
|
|
1232
|
-
|
|
1232
|
+
//
|
|
1233
|
+
// A cache-eligible dispatch also holds the permit for its cache address,
|
|
1234
|
+
// taken second so every dispatch acquires the two in one order (issue
|
|
1235
|
+
// #3898). Runs share that address, so a second run asking for a result
|
|
1236
|
+
// this process is already computing waits for it and is answered by its
|
|
1237
|
+
// cache row instead of running the body beside it. That includes a run
|
|
1238
|
+
// the stale-running sweep reclaimed from a dead owner, whose in-flight
|
|
1239
|
+
// attempt would otherwise re-execute next to the run that superseded
|
|
1240
|
+
// it. Other processes are not excluded: two processes may still execute
|
|
1241
|
+
// one key at once, and the second recording resolves as before.
|
|
1242
|
+
const exclusive = <A, E, R>(effect: Effect.Effect<A, E, R>) =>
|
|
1243
|
+
admission.withPermit(`run:${deps.runId}|${stepKeyDigest}`)(
|
|
1244
|
+
cacheDeclaration._tag === "Eligible" ? admission.withPermit(`cache:${keyDigest}`)(effect) : effect
|
|
1245
|
+
)
|
|
1246
|
+
return yield* exclusive(
|
|
1233
1247
|
Effect.gen(function*() {
|
|
1234
1248
|
// Lifecycle announcements take a per-attempt producer identity
|
|
1235
1249
|
// (issue #91): adoption — or a replay after a crash in the
|
|
@@ -1476,6 +1490,26 @@ export const make = (deps: Dependencies) => {
|
|
|
1476
1490
|
// A run's admitted attempt is replay authority. Shared cache rows
|
|
1477
1491
|
// are replaceable and cannot overwrite its result or failure.
|
|
1478
1492
|
const existing = yield* attempts.get(attemptId)
|
|
1493
|
+
/**
|
|
1494
|
+
* AN ORPHANED ATTEMPT ASKS THE CACHE FIRST (issue #3898). A `running`
|
|
1495
|
+
* row read under the permit is crash evidence (see adoption below):
|
|
1496
|
+
* the attempt never settled, so it holds no result for the cache to
|
|
1497
|
+
* overwrite. A reclaimed run reaching it is in the same position as
|
|
1498
|
+
* a fresh dispatch, and must be answered by a verified row another
|
|
1499
|
+
* run recorded under this key rather than by re-executing the body.
|
|
1500
|
+
* A crossing that already succeeded is the one exception: the row
|
|
1501
|
+
* carries this run's own result, which adoption settles from.
|
|
1502
|
+
*
|
|
1503
|
+
* Only cache-eligible declarations get here, which are sealed with a
|
|
1504
|
+
* hard boundary: an irreversible or compensable attempt, or a sealed
|
|
1505
|
+
* one without a boundary, adopts and re-executes exactly as before.
|
|
1506
|
+
* An unkeyed sealed step's key folds in its run, so no other run can
|
|
1507
|
+
* have recorded it, and its adoption re-executes as before too.
|
|
1508
|
+
*/
|
|
1509
|
+
const orphaned = Option.isSome(existing) && existing.value.state === "running" &&
|
|
1510
|
+
decodeMeta(existing.value.meta)?.effectCrossing !== "succeeded"
|
|
1511
|
+
? existing.value
|
|
1512
|
+
: undefined
|
|
1479
1513
|
if (cacheDeclaration._tag === "Eligible") {
|
|
1480
1514
|
const observed = yield* cache.get(keyDigest).pipe(
|
|
1481
1515
|
Effect.catch((error) =>
|
|
@@ -1513,7 +1547,7 @@ export const make = (deps: Dependencies) => {
|
|
|
1513
1547
|
}
|
|
1514
1548
|
))
|
|
1515
1549
|
}
|
|
1516
|
-
if (candidate._tag === "CandidateEvidence" && Option.isNone(existing)) {
|
|
1550
|
+
if (candidate._tag === "CandidateEvidence" && (Option.isNone(existing) || orphaned !== undefined)) {
|
|
1517
1551
|
const boundary = yield* StepBoundary.StepBoundary
|
|
1518
1552
|
// Skyframe's dirty check, not "the declaration changed" (issue
|
|
1519
1553
|
// #90): the read-set digests folded into the step key are caller
|
|
@@ -1560,6 +1594,32 @@ export const make = (deps: Dependencies) => {
|
|
|
1560
1594
|
recordedRunId: cached.value.recordedRunId,
|
|
1561
1595
|
recordedEventSeq: cached.value.recordedEventSeq
|
|
1562
1596
|
}))
|
|
1597
|
+
if (orphaned !== undefined) {
|
|
1598
|
+
// Close the orphaned attempt with the result that answered
|
|
1599
|
+
// it, so the run carries no attempt that started and never
|
|
1600
|
+
// finished, and a later re-drive replays this row. The
|
|
1601
|
+
// meta records no boundary evidence: this attempt measured
|
|
1602
|
+
// none of its own, so the row is never published back.
|
|
1603
|
+
const finishedAtMs = yield* Clock.currentTimeMillis.pipe(Effect.map(Math.floor))
|
|
1604
|
+
const settled = yield* settleAttempt({
|
|
1605
|
+
...attemptId,
|
|
1606
|
+
state: "succeeded",
|
|
1607
|
+
finishedAtMs,
|
|
1608
|
+
outcome: attemptPayload(cached.value.result),
|
|
1609
|
+
meta: attemptPayload(
|
|
1610
|
+
{
|
|
1611
|
+
...decodeMeta(orphaned.meta),
|
|
1612
|
+
...declarationMeta,
|
|
1613
|
+
admittedBy: deps.owner
|
|
1614
|
+
} satisfies AttemptMeta
|
|
1615
|
+
)
|
|
1616
|
+
}, [
|
|
1617
|
+
JournalRecords.attemptFinished(attemptSource("finished"), { ...attemptId, state: "succeeded" })
|
|
1618
|
+
])
|
|
1619
|
+
if (!settled) {
|
|
1620
|
+
return yield* Effect.interrupt
|
|
1621
|
+
}
|
|
1622
|
+
}
|
|
1563
1623
|
// Counted after the provenance emit: `verified_hit` means
|
|
1564
1624
|
// the cached result was served, and a journal failure on
|
|
1565
1625
|
// the emit fails the dispatch before any result is
|
|
@@ -1953,7 +2013,8 @@ export const make = (deps: Dependencies) => {
|
|
|
1953
2013
|
* is crash evidence, not a live admission (issue #71): the incarnation
|
|
1954
2014
|
* that admitted the attempt died before finishing (SIGKILL, OOM), and
|
|
1955
2015
|
* the #53 stale-running sweep re-drove the run to a new owner — or the
|
|
1956
|
-
* same owner after restart. The attempt never completed, so
|
|
2016
|
+
* same owner after restart. The attempt never completed, so unless a
|
|
2017
|
+
* verified cache row answered it above (issue #3898), it must
|
|
1957
2018
|
* re-execute under its original number rather than fall through to
|
|
1958
2019
|
* `attempts.put` (whose `Conflict` on the differing `startedAtMs`
|
|
1959
2020
|
* surfaced as `AttemptAdmissionRejected`, permanently failing a
|