@sixb/sync-worker 0.2.1 → 0.2.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +28 -46
- package/dist/index.js +317 -240
- package/dist/index.js.map +5 -5
- package/dist/run-sync-job.d.ts +6 -9
- package/dist/run-sync-job.d.ts.map +1 -1
- package/dist/types.d.ts +3 -2
- package/dist/types.d.ts.map +1 -1
- package/dist/worker.d.ts +7 -11
- package/dist/worker.d.ts.map +1 -1
- package/package.json +3 -3
- package/src/run-sync-job.ts +366 -172
- package/src/source-read.ts +30 -2
- package/src/types.ts +3 -2
- package/src/worker.ts +84 -131
package/README.md
CHANGED
|
@@ -1,60 +1,42 @@
|
|
|
1
1
|
# @sixb/sync-worker
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Worker that runs Sixb syncs.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
5
|
+
`SyncWorker` claims queued sync runs, reads each sync's source into one lake dataset version, and
|
|
6
|
+
records the run in `storage.syncRuns`. You normally do not import it: `sixb worker sync` and
|
|
7
|
+
`sixb worker-group` construct it for you.
|
|
8
8
|
|
|
9
|
-
##
|
|
9
|
+
## A run
|
|
10
10
|
|
|
11
|
-
-
|
|
12
|
-
|
|
13
|
-
-
|
|
14
|
-
|
|
15
|
-
-
|
|
16
|
-
|
|
11
|
+
- `RunWorker` claims the run: a job only says which run to look at. See
|
|
12
|
+
[`packages/core/src/workers/README.md`](../core/src/workers/README.md).
|
|
13
|
+
- `runSyncJob` reads the source from the checkpoint of the last succeeded run, staging rows in a lake
|
|
14
|
+
session, and validates them as they pass.
|
|
15
|
+
- Just before the commit, it stores what the commit will have read (`prepareCommit`). That write is
|
|
16
|
+
fenced, so an execution that lost its run stops there.
|
|
17
|
+
- It commits one version, with the run as `producer`, against the latest version it just read, then
|
|
18
|
+
finishes the run with the new checkpoint and announces the version and the finished run.
|
|
17
19
|
|
|
18
|
-
|
|
20
|
+
A failure before the commit finishes the run as `failed`. A failure at the commit asks the lake
|
|
21
|
+
first, because a commit can land and still answer with an error: if the version is there the run
|
|
22
|
+
succeeds, and if the lake cannot say the run stays open for the next delivery. Losing that guard to
|
|
23
|
+
another writer does not fail the run either: nothing was written, so it reads its source again, up to
|
|
24
|
+
its attempts.
|
|
19
25
|
|
|
20
|
-
|
|
21
|
-
|
|
26
|
+
One run of a sync runs at a time: a queued run waits for the one running, whose end wakes it, and
|
|
27
|
+
triggers join it.
|
|
22
28
|
|
|
23
|
-
|
|
24
|
-
id: sixb.id,
|
|
25
|
-
syncRunsStorage,
|
|
26
|
-
lakeStorage,
|
|
27
|
-
blobs: sixb.blobs,
|
|
28
|
-
datasets: sixb.definitions.datasets,
|
|
29
|
-
syncs: sixb.definitions.syncs,
|
|
30
|
-
connector: sixb.connector,
|
|
31
|
-
}
|
|
29
|
+
## Recovery
|
|
32
30
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
id: "run_123",
|
|
37
|
-
syncId: "sync-orders",
|
|
38
|
-
expectedLatestVersionId: "version_001",
|
|
39
|
-
commitMessage: "sync orders from queue",
|
|
40
|
-
},
|
|
41
|
-
signal,
|
|
42
|
-
})
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
`runSyncJob` returns the finalized run id, sync id, dataset id, write mode, timing,
|
|
46
|
-
row count, and committed `DatasetVersion`.
|
|
31
|
+
Nothing of a partial run exists until its commit: rows stay in a session no reader sees, and a crash
|
|
32
|
+
during the commit leaves at most data files no version refers to, which retention removes. So a run
|
|
33
|
+
that resumes has two cases, and one that never prepared a commit has only the second:
|
|
47
34
|
|
|
48
|
-
|
|
35
|
+
- Its commit landed, which `findProducedVersion` finds by `producer.runId`. It finishes with the
|
|
36
|
+
checkpoint and row count it prepared, without reading the source, and announces the version.
|
|
37
|
+
- It did not. It reads again from the same checkpoint, and nothing is written twice.
|
|
49
38
|
|
|
50
|
-
|
|
51
|
-
- `job.expectedLatestVersionId` is forwarded to the lake commit for optimistic
|
|
52
|
-
concurrency checks.
|
|
53
|
-
- a failed sync-run finalization after a successful lake commit throws a
|
|
54
|
-
`[SixbSyncWorker]` bookkeeping error so the durable dataset version can be
|
|
55
|
-
repaired or reconciled.
|
|
56
|
-
- supported sync read results are a single row object, an iterable, or an async
|
|
57
|
-
iterable.
|
|
39
|
+
A stopped worker writes nothing: the run stays `running` and the next delivery takes it up.
|
|
58
40
|
|
|
59
41
|
## Development
|
|
60
42
|
|