@sixb/sync-worker 0.2.2 → 0.2.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,60 +1,42 @@
1
1
  # @sixb/sync-worker
2
2
 
3
- Queue worker helpers for running Sixb sync jobs.
3
+ Worker that runs Sixb syncs.
4
4
 
5
- The sync worker turns one queued sync job into one sync run record and one lake dataset
6
- version. It is intentionally small: queue polling, retries, scheduling, and run-id
7
- generation belong to the caller.
5
+ `SyncWorker` claims queued sync runs, reads each sync's source into one lake dataset version, and
6
+ records the run in `storage.syncRuns`. You normally do not import it: `sixb worker sync` and
7
+ `sixb worker-group` construct it for you.
8
8
 
9
- ## Responsibilities
9
+ ## A run
10
10
 
11
- - look up the sync definition for a queued job
12
- - create and finalize the sync-run record
13
- - resolve the sync connector client
14
- - normalize rows returned by the sync read handler
15
- - write rows into lake storage and commit the dataset version
16
- - mark pre-commit failures as `failed` or `cancelled`
11
+ - `RunWorker` claims the run: a job only says which run to look at. See
12
+ [`packages/core/src/workers/README.md`](../core/src/workers/README.md).
13
+ - `runSyncJob` reads the source from the checkpoint of the last succeeded run, staging rows in a lake
14
+ session, and validates them as they pass.
15
+ - Just before the commit, it stores what the commit will have read (`prepareCommit`). That write is
16
+ fenced, so an execution that lost its run stops there.
17
+ - It commits one version, with the run as `producer`, against the latest version it just read, then
18
+ finishes the run with the new checkpoint and announces the version and the finished run.
17
19
 
18
- ## Usage
20
+ A failure before the commit finishes the run as `failed`. A failure at the commit asks the lake
21
+ first, because a commit can land and still answer with an error: if the version is there the run
22
+ succeeds, and if the lake cannot say the run stays open for the next delivery. Losing that guard to
23
+ another writer does not fail the run either: nothing was written, so it reads its source again, up to
24
+ its attempts.
19
25
 
20
- ```ts
21
- import { runSyncJob, type SyncWorkerContext } from "@sixb/sync-worker"
26
+ One run of a sync runs at a time: a queued run waits for the one running, whose end wakes it, and
27
+ triggers join it.
22
28
 
23
- const runtime: SyncWorkerContext = {
24
- id: sixb.id,
25
- syncRunsStorage,
26
- lakeStorage,
27
- blobs: sixb.blobs,
28
- datasets: sixb.definitions.datasets,
29
- syncs: sixb.definitions.syncs,
30
- connector: sixb.connector,
31
- }
29
+ ## Recovery
32
30
 
33
- const result = await runSyncJob({
34
- runtime,
35
- job: {
36
- id: "run_123",
37
- syncId: "sync-orders",
38
- expectedLatestVersionId: "version_001",
39
- commitMessage: "sync orders from queue",
40
- },
41
- signal,
42
- })
43
- ```
44
-
45
- `runSyncJob` returns the finalized run id, sync id, dataset id, write mode, timing,
46
- row count, and committed `DatasetVersion`.
31
+ Nothing of a partial run exists until its commit: rows stay in a session no reader sees, and a crash
32
+ during the commit leaves at most data files no version refers to, which retention removes. So a run
33
+ that resumes has two cases, and one that never prepared a commit has only the second:
47
34
 
48
- ## Notes
35
+ - Its commit landed, which `findProducedVersion` finds by `producer.runId`. It finishes with the
36
+ checkpoint and row count it prepared, without reading the source, and announces the version.
37
+ - It did not. It reads again from the same checkpoint, and nothing is written twice.
49
38
 
50
- - `job.id` is the sync-run id supplied by the queue layer.
51
- - `job.expectedLatestVersionId` is forwarded to the lake commit for optimistic
52
- concurrency checks.
53
- - a failed sync-run finalization after a successful lake commit throws a
54
- `[SixbSyncWorker]` bookkeeping error so the durable dataset version can be
55
- repaired or reconciled.
56
- - supported sync read results are a single row object, an iterable, or an async
57
- iterable.
39
+ A stopped worker writes nothing: the run stays `running` and the next delivery takes it up.
58
40
 
59
41
  ## Development
60
42