@homeostate/tool-simulator 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +124 -0
- package/README.md +9 -3
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/lost-keys.d.ts +35 -0
- package/dist/lost-keys.d.ts.map +1 -0
- package/dist/lost-keys.js +139 -0
- package/dist/lost-keys.js.map +1 -0
- package/dist/network.d.ts +30 -3
- package/dist/network.d.ts.map +1 -1
- package/dist/network.js +44 -8
- package/dist/network.js.map +1 -1
- package/dist/randomized.d.ts +10 -4
- package/dist/randomized.d.ts.map +1 -1
- package/dist/randomized.js +27 -6
- package/dist/randomized.js.map +1 -1
- package/package.json +7 -7
- package/src/index.ts +1 -0
- package/src/lost-keys.ts +164 -0
- package/src/network.ts +76 -14
- package/src/randomized.ts +29 -7
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,129 @@
|
|
|
1
1
|
# @homeostate/tool-simulator
|
|
2
2
|
|
|
3
|
+
## 0.3.0
|
|
4
|
+
|
|
5
|
+
### Minor Changes
|
|
6
|
+
|
|
7
|
+
- f3f22e8: `advance` and `settle` now check, after every delivery, that no peer's store and document both lose a synced top-level key that no peer deleted, and reject with a `LostKeyError` naming the peer, the key, the virtual time and the seed. `runRandomized` runs the check too, so its error names the step in which a key was lost. Before, a store that dropped a key for a while and got it back once a late update arrived still converged, so the run passed. Runs that passed before can now fail. A key deleted during a delivery, such as by a store subscriber or a timer, is reported as lost: pass `checkLostKeys: false` to `createNetwork`, or in `runRandomized`'s `network`, to turn the check off.
|
|
8
|
+
|
|
9
|
+
### Patch Changes
|
|
10
|
+
|
|
11
|
+
- e98195d: **Breaking:** a smaller sync engine API, with one way to do each thing. Migrate as follows:
|
|
12
|
+
|
|
13
|
+
- `SyncEngine` drops `isConnected()`, which returned what `getStatus()` holds. Replace
|
|
14
|
+
`engine.isConnected()` with `engine.getStatus().connected`. This also applies to the engine that
|
|
15
|
+
Zustand's and Redux's `store.homeostate`, `syncMobxStore` and `syncStateTree` return. An object
|
|
16
|
+
that implements `SyncEngine` by hand can drop the method.
|
|
17
|
+
- `SyncEvent` gains `"error"`, sent when the engine records a failure, which
|
|
18
|
+
`getStatus().failure` then holds. Work that throws, which used to send no event, now sends
|
|
19
|
+
`"error"`, before `onError` runs or the error propagates. A write loop sends it after the
|
|
20
|
+
`"remote"` of the apply that stopped it. A `switch` over `SyncEvent` must handle it.
|
|
21
|
+
- `SyncEngine` drops `subscribeStatus()`: every change of the status comes with a sync event, so
|
|
22
|
+
`subscribe` tells when to read it. Replace `engine.subscribeStatus(listener)` with
|
|
23
|
+
`engine.subscribe(() => listener())`, and read `engine.getStatus()` in it as before; it returns
|
|
24
|
+
the same object while the status is unchanged, so compare by identity to skip the events that
|
|
25
|
+
change nothing. An object that implements `SyncEngine` by hand can drop the method.
|
|
26
|
+
- `SyncEngineConfig.seed` is a boolean, `true` by default, and the `SeedStrategy` type is gone.
|
|
27
|
+
`"if-empty"` seeded each synced key the backend lacks, not only an empty backend. Replace
|
|
28
|
+
`seed: "if-empty"` with `seed: true`, or leave it out, and `seed: "never"` with `seed: false`.
|
|
29
|
+
This also applies to the options of Zustand's and Redux's `homeostate`, `syncMobxStore`,
|
|
30
|
+
`syncStateTree` and `atomWithSync`. Remove imports of `SeedStrategy`; type a variable that holds
|
|
31
|
+
the option as `boolean`. In plain JavaScript, a leftover `seed: "never"` is truthy and now
|
|
32
|
+
seeds, so replace it.
|
|
33
|
+
- The `ChangeType` object and type are removed; compare a step's `type` with its string literal.
|
|
34
|
+
Replace `ChangeType.INSERT`, `ChangeType.UPDATE` and `ChangeType.DELETE` with `"insert"`,
|
|
35
|
+
`"update"` and `"delete"`, and type a variable that holds a step type as `Change["type"]`.
|
|
36
|
+
- The `"pending"` step type is renamed `"edit"`: a step that edits the container at its key in
|
|
37
|
+
place by its nested script `changes`, as before. `getChanges` returns `"edit"` steps where it
|
|
38
|
+
returned `"pending"` ones. Replace `ChangeType.PENDING` and `"pending"` with `"edit"` wherever
|
|
39
|
+
you build a script or switch on a step's `type`; a `switch` left on `"pending"` no longer
|
|
40
|
+
matches, and TypeScript reports the comparison.
|
|
41
|
+
- `applyStringChanges`, `EditScript` and `Diffable` are no longer exported. `applyStringChanges`
|
|
42
|
+
returned the string a text script leads to, which is the second string given to `getChanges`,
|
|
43
|
+
so use that string; to edit a text in place, use `applyChanges` with `ApplyOps.editText`.
|
|
44
|
+
Replace `EditScript` with the script type of the kind diffed, `RecordChange[]`, `ListChange[]`
|
|
45
|
+
or `TextChange[]`, as `getChanges` returns it, and `Diffable` with
|
|
46
|
+
`JsonObject | unknown[] | string`; `getChanges` still needs both values narrowed to one kind.
|
|
47
|
+
- `count` is required on an array or string delete step: `ListChange` and `TextChange` declare
|
|
48
|
+
`count: number`, no longer `count?: number`. `getChanges` already set it on every such delete,
|
|
49
|
+
so only a script built by hand changes: add `count: 1` to an array or string delete that left
|
|
50
|
+
it out before passing the script to `applyChanges`. In plain JavaScript, such a delete no
|
|
51
|
+
longer removes one item, so add the `count` there too. A record delete still has no `count`.
|
|
52
|
+
- `getChanges` replaces a changed nested string whole by default, as `createCrdtBackend` and
|
|
53
|
+
`mergeConcurrent` already did: without `options.text`, a string in a record or an array that
|
|
54
|
+
changed is one `"update"` step instead of an `"edit"` step with a character-level script, and
|
|
55
|
+
so is what `applyChanges` writes. Two strings passed directly are still diffed character by
|
|
56
|
+
character. Pass `text: () => true` to keep character-level scripts for every nested string, or
|
|
57
|
+
a `TextPolicy` for some. The CRDT backends pass their `text` option, so they are unchanged.
|
|
58
|
+
- `createCrdtBackend` takes the parts as one object, `text` among them, and
|
|
59
|
+
`CrdtBackendOptions` is removed. Replace
|
|
60
|
+
`createCrdtBackend((text) => ({ read, transact, watch, revision }), { text })` with
|
|
61
|
+
`createCrdtBackend({ read, transact, watch, revision, text })`, and build the `ApplyOps` that
|
|
62
|
+
`transact` hands over from the same `text`, which is `undefined` when no string is text: core
|
|
63
|
+
no longer passes a policy with its default applied. `CrdtBackendParts` gains the optional
|
|
64
|
+
`text`. The Yjs, Loro and Automerge backends take their `text` option as before.
|
|
65
|
+
- `CrdtBackend.revision` and `CrdtBackendParts.revision` are required. A backend written by hand,
|
|
66
|
+
or parts passed to `createCrdtBackend`, must implement `revision`; return `undefined` when it
|
|
67
|
+
cannot tell, which keeps the read before every local change that a backend without `revision`
|
|
68
|
+
had. Such a backend now also gets no `previous` when the engine writes back what the store
|
|
69
|
+
changed during a remote apply, or a local change merged into one, since the engine cannot tell
|
|
70
|
+
whether the document changed during `setState`; the backend diffs against its document then.
|
|
71
|
+
The Yjs, Loro, Automerge and memory backends already implement it.
|
|
72
|
+
- `createSyncEngine` takes one options object, as the setup helpers do: the `backend`, the
|
|
73
|
+
`adapter` and the `SyncEngineConfig` options, flat. Replace
|
|
74
|
+
`createSyncEngine(backend, adapter, config)` with
|
|
75
|
+
`createSyncEngine({ backend, adapter, ...config })`, or list the options inline, such as
|
|
76
|
+
`createSyncEngine({ backend, adapter, omit: ["draft"] })`, and
|
|
77
|
+
`createSyncEngine(backend, adapter)` with `createSyncEngine({ backend, adapter })`. The state
|
|
78
|
+
type is still taken from `adapter`. The new `SyncEngineOptions<S>` type names the object. The
|
|
79
|
+
setup helpers' options are unchanged.
|
|
80
|
+
- `filter` takes the key alone, `(key: keyof S & string) => boolean`, and a key the store holds as
|
|
81
|
+
a function is never synced, whatever `filter` returns: the engine neither writes it nor reads the
|
|
82
|
+
backend's value into it. A store that makes a synced key a function deletes it from the backend
|
|
83
|
+
with its next write, as before, and syncs it again once it holds a JSON value. A backend value is
|
|
84
|
+
no longer judged: a synced key always takes it. `defaultSyncFilter` is removed. Drop
|
|
85
|
+
`filter: defaultSyncFilter`; replace `(key, value) => defaultSyncFilter(key, value) && key !==
|
|
86
|
+
"draft"` with `(key) => key !== "draft"`, or with `omit: ["draft"]`; elsewhere, replace
|
|
87
|
+
`defaultSyncFilter(key, value)` with `typeof value !== "function"`. A custom filter no longer
|
|
88
|
+
syncs functions it keeps, so it need not exclude them. A rule on any other value has no
|
|
89
|
+
replacement: choose the keys by name, or sync a slice without them with `scopeAdapter`. A filter
|
|
90
|
+
that takes a second parameter fails to compile; in plain JavaScript, it gets `undefined` there.
|
|
91
|
+
|
|
92
|
+
- 1bfdf95: `runRandomized` no longer reports peers that converge as diverged when updates written after
|
|
93
|
+
its final heal are dropped. With a `dropRate`, a store that writes in answer to remote changes,
|
|
94
|
+
such as a subscriber that keeps a derived key in synced state, writes new updates as the
|
|
95
|
+
documents `heal()` exchanges arrive, and the network can drop those too. The run healed once,
|
|
96
|
+
so a dropped update stayed lost and the run failed with a `ConvergenceError`. It now heals and
|
|
97
|
+
settles again until a round drops nothing, at most 10 rounds, as a provider that resyncs would.
|
|
98
|
+
The steps make the same random choices, so a seed replays the same steps. When the tenth round
|
|
99
|
+
still dropped updates and the peers did not converge, the error says so. `expectConverged()`
|
|
100
|
+
now also says how many updates were dropped since the last `heal()`, so tests that heal by hand
|
|
101
|
+
get the same hint: repeat `heal()` and `settle()` until `stats().dropped` stops growing.
|
|
102
|
+
- Updated dependencies [e98195d]
|
|
103
|
+
- Updated dependencies [e4e66f6]
|
|
104
|
+
- Updated dependencies [d17efad]
|
|
105
|
+
- Updated dependencies [7172cf4]
|
|
106
|
+
- Updated dependencies [3edede4]
|
|
107
|
+
- Updated dependencies [ea86a91]
|
|
108
|
+
- Updated dependencies [b2c1521]
|
|
109
|
+
- Updated dependencies [038c0e1]
|
|
110
|
+
- Updated dependencies [988dec0]
|
|
111
|
+
- Updated dependencies [48b0532]
|
|
112
|
+
- Updated dependencies [bf8324b]
|
|
113
|
+
- Updated dependencies [e97b920]
|
|
114
|
+
- Updated dependencies [2756e79]
|
|
115
|
+
- Updated dependencies [44a47f2]
|
|
116
|
+
- Updated dependencies [fab156f]
|
|
117
|
+
- Updated dependencies [e51dad7]
|
|
118
|
+
- Updated dependencies [1a6cfb8]
|
|
119
|
+
- Updated dependencies [4008d1e]
|
|
120
|
+
- Updated dependencies [76ec475]
|
|
121
|
+
- Updated dependencies [106f2a4]
|
|
122
|
+
- Updated dependencies [b296a2f]
|
|
123
|
+
- Updated dependencies [987d676]
|
|
124
|
+
- Updated dependencies [3e8a239]
|
|
125
|
+
- @homeostate/core@0.5.0
|
|
126
|
+
|
|
3
127
|
## 0.2.0
|
|
4
128
|
|
|
5
129
|
### Minor Changes
|
package/README.md
CHANGED
|
@@ -52,10 +52,16 @@ network.expectConverged();
|
|
|
52
52
|
|
|
53
53
|
Each peer's `doc` is its backend's document as binary updates, from the CRDT package's
|
|
54
54
|
`create*Persistable`: that is what the network carries. `settle()` delivers every message on
|
|
55
|
-
its way, and
|
|
55
|
+
its way, and rejects if a peer's store and document both lose a synced key that no peer
|
|
56
|
+
deleted. `expectConverged()` throws unless every peer's store and document agree.
|
|
56
57
|
|
|
57
|
-
`
|
|
58
|
-
|
|
58
|
+
With a `dropRate`, the updates stores write in answer to the exchange `heal()` starts, such as a
|
|
59
|
+
key derived from synced state, can be dropped too. Repeat `heal()` and `settle()` until
|
|
60
|
+
`stats().dropped` stops growing before you check convergence.
|
|
61
|
+
|
|
62
|
+
`runRandomized` drives random edits, link cuts and reconnections across a fresh network, checks
|
|
63
|
+
after every delivery that no peer's store loses a synced key, then heals and settles until
|
|
64
|
+
nothing more is dropped and checks convergence, and names the seed of a run that fails:
|
|
59
65
|
|
|
60
66
|
```ts
|
|
61
67
|
await runRandomized({
|
package/dist/index.d.ts
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
export { ConvergenceError, createNetwork } from "./network.js";
|
|
2
|
+
export { LostKeyError } from "./lost-keys.js";
|
|
2
3
|
export type { Network, NetworkOptions, NetworkStats, Peer, PeerSetup, } from "./network.js";
|
|
3
4
|
export { randomEdit, runRandomized } from "./randomized.js";
|
|
4
5
|
export type { RandomAction, RandomizedOptions } from "./randomized.js";
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAC/D,YAAY,EACV,OAAO,EACP,cAAc,EACd,YAAY,EACZ,IAAI,EACJ,SAAS,GACV,MAAM,cAAc,CAAC;AACtB,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAC5D,YAAY,EAAE,YAAY,EAAE,iBAAiB,EAAE,MAAM,iBAAiB,CAAC;AACvE,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,YAAY,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAC/D,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC;AAC9C,YAAY,EACV,OAAO,EACP,cAAc,EACd,YAAY,EACZ,IAAI,EACJ,SAAS,GACV,MAAM,cAAc,CAAC;AACtB,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAC5D,YAAY,EAAE,YAAY,EAAE,iBAAiB,EAAE,MAAM,iBAAiB,CAAC;AACvE,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,YAAY,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC"}
|
package/dist/index.js
CHANGED
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAC/D,OAAO,EAAE,YAAY,EAAE,MAAM,gBAAgB,CAAC;AAQ9C,OAAO,EAAE,UAAU,EAAE,aAAa,EAAE,MAAM,iBAAiB,CAAC;AAE5D,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC"}
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import type { Peer } from "./network.js";
|
|
2
|
+
/** Thrown by `advance` and `settle` when a peer lost a synced key; see `checkLostKeys`. */
|
|
3
|
+
export declare class LostKeyError extends Error {
|
|
4
|
+
readonly seed: number;
|
|
5
|
+
/** The peer's name. */
|
|
6
|
+
readonly peer: string;
|
|
7
|
+
readonly key: string;
|
|
8
|
+
/** Virtual time of the checkpoint that found the loss, in milliseconds. */
|
|
9
|
+
readonly at: number;
|
|
10
|
+
constructor(message: string, details: {
|
|
11
|
+
seed: number;
|
|
12
|
+
peer: string;
|
|
13
|
+
key: string;
|
|
14
|
+
at: number;
|
|
15
|
+
});
|
|
16
|
+
}
|
|
17
|
+
/**
|
|
18
|
+
* Checks that no peer's store and document both lose a synced top-level key that no peer
|
|
19
|
+
* deleted. The network calls it as its documents change and around its deliveries.
|
|
20
|
+
*/
|
|
21
|
+
export interface LostKeyCheck {
|
|
22
|
+
/** Start tracking a peer, before it links and connects. */
|
|
23
|
+
addPeer: (peer: Peer) => void;
|
|
24
|
+
/** The peer's document changed; `outside` when no delivery is running. */
|
|
25
|
+
changed: (peer: Peer, outside: boolean) => void;
|
|
26
|
+
/**
|
|
27
|
+
* Before deliveries: what changed since the last checkpoint was done by the test, so a key
|
|
28
|
+
* that left a store or document is deleted.
|
|
29
|
+
*/
|
|
30
|
+
beforeDeliveries: (peers: readonly Peer[]) => void;
|
|
31
|
+
/** After a delivery to `peer` from `from`, or after the last one; throws a `LostKeyError`. */
|
|
32
|
+
afterDelivery: (peer: Peer, at: number, from?: Peer) => void;
|
|
33
|
+
}
|
|
34
|
+
export declare function createLostKeyCheck(seed: number): LostKeyCheck;
|
|
35
|
+
//# sourceMappingURL=lost-keys.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"lost-keys.d.ts","sourceRoot":"","sources":["../src/lost-keys.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAEzC,2FAA2F;AAC3F,qBAAa,YAAa,SAAQ,KAAK;IACrC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,uBAAuB;IACvB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;IACrB,2EAA2E;IAC3E,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAGlB,OAAO,EAAE,MAAM,EACf,OAAO,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAC;QAAC,EAAE,EAAE,MAAM,CAAA;KAAE;CASnE;AAgBD;;;GAGG;AACH,MAAM,WAAW,YAAY;IAC3B,2DAA2D;IAC3D,OAAO,EAAE,CAAC,IAAI,EAAE,IAAI,KAAK,IAAI,CAAC;IAC9B,0EAA0E;IAC1E,OAAO,EAAE,CAAC,IAAI,EAAE,IAAI,EAAE,OAAO,EAAE,OAAO,KAAK,IAAI,CAAC;IAChD;;;OAGG;IACH,gBAAgB,EAAE,CAAC,KAAK,EAAE,SAAS,IAAI,EAAE,KAAK,IAAI,CAAC;IACnD,8FAA8F;IAC9F,aAAa,EAAE,CAAC,IAAI,EAAE,IAAI,EAAE,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,EAAE,IAAI,KAAK,IAAI,CAAC;CAC9D;AAiBD,wBAAgB,kBAAkB,CAAC,IAAI,EAAE,MAAM,GAAG,YAAY,CA2F7D"}
|
|
@@ -0,0 +1,139 @@
|
|
|
1
|
+
import { formatPath } from "./compare.js";
|
|
2
|
+
/** Thrown by `advance` and `settle` when a peer lost a synced key; see `checkLostKeys`. */
|
|
3
|
+
export class LostKeyError extends Error {
|
|
4
|
+
constructor(message, details) {
|
|
5
|
+
super(message);
|
|
6
|
+
Object.defineProperty(this, "seed", {
|
|
7
|
+
enumerable: true,
|
|
8
|
+
configurable: true,
|
|
9
|
+
writable: true,
|
|
10
|
+
value: void 0
|
|
11
|
+
});
|
|
12
|
+
/** The peer's name. */
|
|
13
|
+
Object.defineProperty(this, "peer", {
|
|
14
|
+
enumerable: true,
|
|
15
|
+
configurable: true,
|
|
16
|
+
writable: true,
|
|
17
|
+
value: void 0
|
|
18
|
+
});
|
|
19
|
+
Object.defineProperty(this, "key", {
|
|
20
|
+
enumerable: true,
|
|
21
|
+
configurable: true,
|
|
22
|
+
writable: true,
|
|
23
|
+
value: void 0
|
|
24
|
+
});
|
|
25
|
+
/** Virtual time of the checkpoint that found the loss, in milliseconds. */
|
|
26
|
+
Object.defineProperty(this, "at", {
|
|
27
|
+
enumerable: true,
|
|
28
|
+
configurable: true,
|
|
29
|
+
writable: true,
|
|
30
|
+
value: void 0
|
|
31
|
+
});
|
|
32
|
+
this.name = "LostKeyError";
|
|
33
|
+
this.seed = details.seed;
|
|
34
|
+
this.peer = details.peer;
|
|
35
|
+
this.key = details.key;
|
|
36
|
+
this.at = details.at;
|
|
37
|
+
}
|
|
38
|
+
}
|
|
39
|
+
/** The keys of `state` that JSON keeps, as `peer.syncedState()` has them. */
|
|
40
|
+
const jsonKeys = (state) => {
|
|
41
|
+
const keys = new Set();
|
|
42
|
+
for (const [key, value] of Object.entries(state))
|
|
43
|
+
if (value !== undefined &&
|
|
44
|
+
typeof value !== "function" &&
|
|
45
|
+
typeof value !== "symbol")
|
|
46
|
+
keys.add(key);
|
|
47
|
+
return keys;
|
|
48
|
+
};
|
|
49
|
+
const formatTime = (ms) => String(Number(ms.toFixed(2)));
|
|
50
|
+
export function createLostKeyCheck(seed) {
|
|
51
|
+
const tracked = new Map();
|
|
52
|
+
/** Keys a checkpoint outside a delivery saw leave a store or document; never cleared. */
|
|
53
|
+
const deleted = new Set();
|
|
54
|
+
const storeKeys = (peer) => jsonKeys(peer.engine.getSyncedState());
|
|
55
|
+
const docKeys = (peer, state) => {
|
|
56
|
+
if (!state.doc) {
|
|
57
|
+
state.doc = jsonKeys(peer.backend.read());
|
|
58
|
+
// Only the test's own changes delete; a delivery that removes a key may be a loss.
|
|
59
|
+
if (state.changedOutside && state.lastDoc)
|
|
60
|
+
for (const key of state.lastDoc)
|
|
61
|
+
if (!state.doc.has(key))
|
|
62
|
+
deleted.add(key);
|
|
63
|
+
state.lastDoc = state.doc;
|
|
64
|
+
state.changedOutside = false;
|
|
65
|
+
}
|
|
66
|
+
return state.doc;
|
|
67
|
+
};
|
|
68
|
+
/** Mark the keys that left the store since its last checkpoint as deleted. */
|
|
69
|
+
const noteRemovals = (state, store) => {
|
|
70
|
+
for (const key of state.store)
|
|
71
|
+
if (!store.has(key))
|
|
72
|
+
deleted.add(key);
|
|
73
|
+
state.store = store;
|
|
74
|
+
};
|
|
75
|
+
const hold = (peer, state, keys) => {
|
|
76
|
+
for (const key of keys) {
|
|
77
|
+
if (state.held.has(key) || deleted.has(key))
|
|
78
|
+
continue;
|
|
79
|
+
if (docKeys(peer, state).has(key))
|
|
80
|
+
state.held.add(key);
|
|
81
|
+
}
|
|
82
|
+
};
|
|
83
|
+
const track = (peer) => {
|
|
84
|
+
const state = tracked.get(peer);
|
|
85
|
+
if (!state)
|
|
86
|
+
throw new Error(`${peer.name} is not on the network.`);
|
|
87
|
+
return state;
|
|
88
|
+
};
|
|
89
|
+
return {
|
|
90
|
+
addPeer: (peer) => {
|
|
91
|
+
tracked.set(peer, {
|
|
92
|
+
store: storeKeys(peer),
|
|
93
|
+
doc: undefined,
|
|
94
|
+
lastDoc: undefined,
|
|
95
|
+
changedOutside: false,
|
|
96
|
+
held: new Set(),
|
|
97
|
+
});
|
|
98
|
+
},
|
|
99
|
+
changed: (peer, outside) => {
|
|
100
|
+
const state = track(peer);
|
|
101
|
+
state.doc = undefined;
|
|
102
|
+
if (!outside)
|
|
103
|
+
return;
|
|
104
|
+
state.changedOutside = true;
|
|
105
|
+
// A key added and removed again before the next delivery still counts as deleted.
|
|
106
|
+
noteRemovals(state, storeKeys(peer));
|
|
107
|
+
},
|
|
108
|
+
beforeDeliveries: (peers) => {
|
|
109
|
+
for (const peer of peers) {
|
|
110
|
+
const state = track(peer);
|
|
111
|
+
const store = storeKeys(peer);
|
|
112
|
+
noteRemovals(state, store);
|
|
113
|
+
if (state.changedOutside)
|
|
114
|
+
docKeys(peer, state);
|
|
115
|
+
hold(peer, state, store);
|
|
116
|
+
}
|
|
117
|
+
},
|
|
118
|
+
afterDelivery: (peer, at, from) => {
|
|
119
|
+
const state = track(peer);
|
|
120
|
+
const store = storeKeys(peer);
|
|
121
|
+
for (const key of state.held) {
|
|
122
|
+
if (store.has(key))
|
|
123
|
+
continue;
|
|
124
|
+
if (deleted.has(key)) {
|
|
125
|
+
state.held.delete(key);
|
|
126
|
+
continue;
|
|
127
|
+
}
|
|
128
|
+
if (docKeys(peer, state).has(key))
|
|
129
|
+
continue;
|
|
130
|
+
const source = from ? `, on an update from ${from.name}` : "";
|
|
131
|
+
throw new LostKeyError(`${peer.name} lost the synced key ${formatPath([key])} at ${formatTime(at)} ms (seed ${seed})${source}: its store and document both lack it, and no peer deleted it.`, { seed, peer: peer.name, key, at });
|
|
132
|
+
}
|
|
133
|
+
const appeared = [...store].filter((key) => !state.store.has(key));
|
|
134
|
+
state.store = store;
|
|
135
|
+
hold(peer, state, appeared);
|
|
136
|
+
},
|
|
137
|
+
};
|
|
138
|
+
}
|
|
139
|
+
//# sourceMappingURL=lost-keys.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"lost-keys.js","sourceRoot":"","sources":["../src/lost-keys.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAG1C,2FAA2F;AAC3F,MAAM,OAAO,YAAa,SAAQ,KAAK;IAQrC,YACE,OAAe,EACf,OAAgE;QAEhE,KAAK,CAAC,OAAO,CAAC,CAAC;QAXR;;;;;WAAa;QACtB,uBAAuB;QACd;;;;;WAAa;QACb;;;;;WAAY;QACrB,2EAA2E;QAClE;;;;;WAAW;QAOlB,IAAI,CAAC,IAAI,GAAG,cAAc,CAAC;QAC3B,IAAI,CAAC,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;QACzB,IAAI,CAAC,IAAI,GAAG,OAAO,CAAC,IAAI,CAAC;QACzB,IAAI,CAAC,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC;QACvB,IAAI,CAAC,EAAE,GAAG,OAAO,CAAC,EAAE,CAAC;IACvB,CAAC;CACF;AAkCD,6EAA6E;AAC7E,MAAM,QAAQ,GAAG,CAAC,KAAa,EAAe,EAAE;IAC9C,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;IAC/B,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,KAAK,CAAC;QAC9C,IACE,KAAK,KAAK,SAAS;YACnB,OAAO,KAAK,KAAK,UAAU;YAC3B,OAAO,KAAK,KAAK,QAAQ;YAEzB,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IAClB,OAAO,IAAI,CAAC;AACd,CAAC,CAAC;AAEF,MAAM,UAAU,GAAG,CAAC,EAAU,EAAU,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AAEzE,MAAM,UAAU,kBAAkB,CAAC,IAAY;IAC7C,MAAM,OAAO,GAAG,IAAI,GAAG,EAAiB,CAAC;IACzC,yFAAyF;IACzF,MAAM,OAAO,GAAG,IAAI,GAAG,EAAU,CAAC;IAElC,MAAM,SAAS,GAAG,CAAC,IAAU,EAAe,EAAE,CAC5C,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,cAAc,EAAE,CAAC,CAAC;IAEzC,MAAM,OAAO,GAAG,CAAC,IAAU,EAAE,KAAc,EAAe,EAAE;QAC1D,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;YACf,KAAK,CAAC,GAAG,GAAG,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC,CAAC;YAC1C,mFAAmF;YACnF,IAAI,KAAK,CAAC,cAAc,IAAI,KAAK,CAAC,OAAO;gBACvC,KAAK,MAAM,GAAG,IAAI,KAAK,CAAC,OAAO;oBAC7B,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC;wBAAE,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;YAC9C,KAAK,CAAC,OAAO,GAAG,KAAK,CAAC,GAAG,CAAC;YAC1B,KAAK,CAAC,cAAc,GAAG,KAAK,CAAC;QAC/B,CAAC;QACD,OAAO,KAAK,CAAC,GAAG,CAAC;IACnB,CAAC,CAAC;IAEF,8EAA8E;IAC9E,MAAM,YAAY,GAAG,CAAC,KAAc,EAAE,KAAkB,EAAQ,EAAE;QAChE,KAAK,MAAM,GAAG,IAAI,KAAK,CAAC,KAAK;YAAE,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACrE,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC;IACtB,CAAC,CAAC;IAEF,MAAM,IAAI,GAAG,CAAC,IAAU,EAAE,KAAc,EAAE,IAAsB,EAAQ,EAAE;QACxE,KAAK,MAAM,GAAG,IAAI,IAAI,EAAE,CAAC;YACvB,IAAI,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,SAAS;YACtD,IAAI,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC;gBAAE,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;QACzD,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,KAAK,GAAG,CAAC,IAAU,EAAW,EAAE;QACpC,MAAM,KAAK,GAAG,OAAO,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QAChC,IAAI,CAAC,KAAK;YAAE,MAAM,IAAI,KAAK,CAAC,GAAG,IAAI,CAAC,IAAI,yBAAyB,CAAC,CAAC;QACnE,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;IAEF,OAAO;QACL,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE;YAChB,OAAO,CAAC,GAAG,CAAC,IAAI,EAAE;gBAChB,KAAK,EAAE,SAAS,CAAC,IAAI,CAAC;gBACtB,GAAG,EAAE,SAAS;gBACd,OAAO,EAAE,SAAS;gBAClB,cAAc,EAAE,KAAK;gBACrB,IAAI,EAAE,IAAI,GAAG,EAAE;aAChB,CAAC,CAAC;QACL,CAAC;QAED,OAAO,EAAE,CAAC,IAAI,EAAE,OAAO,EAAE,EAAE;YACzB,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC;YAC1B,KAAK,CAAC,GAAG,GAAG,SAAS,CAAC;YACtB,IAAI,CAAC,OAAO;gBAAE,OAAO;YACrB,KAAK,CAAC,cAAc,GAAG,IAAI,CAAC;YAC5B,kFAAkF;YAClF,YAAY,CAAC,KAAK,EAAE,SAAS,CAAC,IAAI,CAAC,CAAC,CAAC;QACvC,CAAC;QAED,gBAAgB,EAAE,CAAC,KAAK,EAAE,EAAE;YAC1B,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;gBACzB,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC;gBAC1B,MAAM,KAAK,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;gBAC9B,YAAY,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;gBAC3B,IAAI,KAAK,CAAC,cAAc;oBAAE,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;gBAC/C,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;YAC3B,CAAC;QACH,CAAC;QAED,aAAa,EAAE,CAAC,IAAI,EAAE,EAAE,EAAE,IAAI,EAAE,EAAE;YAChC,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,CAAC,CAAC;YAC1B,MAAM,KAAK,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;YAC9B,KAAK,MAAM,GAAG,IAAI,KAAK,CAAC,IAAI,EAAE,CAAC;gBAC7B,IAAI,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC;oBAAE,SAAS;gBAC7B,IAAI,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,EAAE,CAAC;oBACrB,KAAK,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;oBACvB,SAAS;gBACX,CAAC;gBACD,IAAI,OAAO,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,GAAG,CAAC,GAAG,CAAC;oBAAE,SAAS;gBAC5C,MAAM,MAAM,GAAG,IAAI,CAAC,CAAC,CAAC,uBAAuB,IAAI,CAAC,IAAI,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC;gBAC9D,MAAM,IAAI,YAAY,CACpB,GAAG,IAAI,CAAC,IAAI,wBAAwB,UAAU,CAAC,CAAC,GAAG,CAAC,CAAC,OAAO,UAAU,CAAC,EAAE,CAAC,aAAa,IAAI,IAAI,MAAM,gEAAgE,EACrK,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,GAAG,EAAE,EAAE,EAAE,CACnC,CAAC;YACJ,CAAC;YACD,MAAM,QAAQ,GAAG,CAAC,GAAG,KAAK,CAAC,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC;YACnE,KAAK,CAAC,KAAK,GAAG,KAAK,CAAC;YACpB,IAAI,CAAC,IAAI,EAAE,KAAK,EAAE,QAAQ,CAAC,CAAC;QAC9B,CAAC;KACF,CAAC;AACJ,CAAC"}
|
package/dist/network.d.ts
CHANGED
|
@@ -54,6 +54,17 @@ export interface NetworkOptions {
|
|
|
54
54
|
* as two peers writing in a loop. Defaults to 100 000.
|
|
55
55
|
*/
|
|
56
56
|
maxDeliveries?: number;
|
|
57
|
+
/**
|
|
58
|
+
* Whether `advance` and `settle` reject with a `LostKeyError` when a peer's store and
|
|
59
|
+
* document both lose a synced top-level key that no peer deleted. Defaults to `true`.
|
|
60
|
+
*
|
|
61
|
+
* A peer holds a key once a checkpoint sees it in both its store's synced state and its
|
|
62
|
+
* document. A key is deleted for good once it leaves a store or document outside a delivery:
|
|
63
|
+
* in a step, or in a test's own change, between `advance` and `settle` calls. Checkpoints
|
|
64
|
+
* come after each delivery and when `advance` or `settle` returns. A key deleted during a
|
|
65
|
+
* delivery, such as by a store subscriber or a timer, is reported as lost: pass `false`.
|
|
66
|
+
*/
|
|
67
|
+
checkLostKeys?: boolean;
|
|
57
68
|
}
|
|
58
69
|
export interface NetworkStats {
|
|
59
70
|
/** Messages handed to the network, updates and the documents exchanged on connecting. */
|
|
@@ -71,6 +82,9 @@ export interface NetworkStats {
|
|
|
71
82
|
* document goes to every peer it is linked to, after the latency, unless it is dropped or its
|
|
72
83
|
* link is cut first. Linking two peers, as `connect` and `heal` do, makes them exchange their
|
|
73
84
|
* whole documents, as providers do on reconnecting, so what was lost on the way arrives.
|
|
85
|
+
*
|
|
86
|
+
* Unless `checkLostKeys` is `false`, `advance` and `settle` also check after each delivery
|
|
87
|
+
* that no peer's store and document both lost a synced key that no peer deleted.
|
|
74
88
|
*/
|
|
75
89
|
export interface Network {
|
|
76
90
|
/** The seed of this network's random choices. */
|
|
@@ -100,11 +114,21 @@ export interface Network {
|
|
|
100
114
|
/**
|
|
101
115
|
* Link every pair of peers and have each exchange documents, as if every peer reconnected,
|
|
102
116
|
* so that updates lost to drops and cut links arrive too.
|
|
117
|
+
*
|
|
118
|
+
* With a `dropRate`, the updates stores write in answer to that exchange, such as a key one
|
|
119
|
+
* derives from synced state, can be dropped too. Repeat `heal()` and `settle()` until
|
|
120
|
+
* `stats().dropped` stops growing before checking convergence.
|
|
103
121
|
*/
|
|
104
122
|
heal: () => void;
|
|
105
|
-
/**
|
|
123
|
+
/**
|
|
124
|
+
* Deliver the messages due within the next `ms` of virtual time, and move time on. Rejects
|
|
125
|
+
* with a `LostKeyError` when a peer lost a synced key; see `checkLostKeys`.
|
|
126
|
+
*/
|
|
106
127
|
advance: (ms: number) => Promise<void>;
|
|
107
|
-
/**
|
|
128
|
+
/**
|
|
129
|
+
* Deliver every message on its way, and those they lead to, until none are left. Rejects
|
|
130
|
+
* with a `LostKeyError` when a peer lost a synced key; see `checkLostKeys`.
|
|
131
|
+
*/
|
|
108
132
|
settle: () => Promise<void>;
|
|
109
133
|
stats: () => NetworkStats;
|
|
110
134
|
/**
|
|
@@ -113,7 +137,10 @@ export interface Network {
|
|
|
113
137
|
* document. Empty once the peers have converged.
|
|
114
138
|
*/
|
|
115
139
|
differences: () => string[];
|
|
116
|
-
/**
|
|
140
|
+
/**
|
|
141
|
+
* Throws a `ConvergenceError` unless `differences()` is empty. Its message also says how many
|
|
142
|
+
* messages are still on their way, and how many updates were dropped since the last `heal()`.
|
|
143
|
+
*/
|
|
117
144
|
expectConverged: () => void;
|
|
118
145
|
/** Disconnect every engine, stop listening to every document and drop what is on its way. */
|
|
119
146
|
destroy: () => void;
|
package/dist/network.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"network.d.ts","sourceRoot":"","sources":["../src/network.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,WAAW,EAChB,KAAK,cAAc,EACnB,KAAK,YAAY,EACjB,KAAK,UAAU,EACf,KAAK,gBAAgB,EAEtB,MAAM,kBAAkB,CAAC;
|
|
1
|
+
{"version":3,"file":"network.d.ts","sourceRoot":"","sources":["../src/network.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,WAAW,EAChB,KAAK,cAAc,EACnB,KAAK,YAAY,EACjB,KAAK,UAAU,EACf,KAAK,gBAAgB,EAEtB,MAAM,kBAAkB,CAAC;AAK1B;;;;GAIG;AAGH,MAAM,WAAW,SAAS,CAAC,CAAC,SAAS,MAAM,GAAG,GAAG;IAC/C,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC;IACzB,OAAO,EAAE,WAAW,CAAC;IACrB,GAAG,EAAE,cAAc,CAAC;IACpB;;;OAGG;IACH,MAAM,CAAC,EAAE,OAAO,CAAC,gBAAgB,CAAC,CAAC,CAAC,CAAC,CAAC;IACtC,wEAAwE;IACxE,IAAI,CAAC,EAAE,MAAM,CAAC;IACd;;;;OAIG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,kGAAkG;AAElG,MAAM,WAAW,IAAI,CAAC,CAAC,SAAS,MAAM,GAAG,GAAG;IAC1C,mCAAmC;IACnC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,OAAO,EAAE,YAAY,CAAC,CAAC,CAAC,CAAC;IAClC,QAAQ,CAAC,OAAO,EAAE,WAAW,CAAC;IAC9B,QAAQ,CAAC,GAAG,EAAE,cAAc,CAAC;IAC7B,QAAQ,CAAC,MAAM,EAAE,UAAU,CAAC;IAC5B,wCAAwC;IACxC,WAAW,EAAE,MAAM,OAAO,CAAC;CAC5B;AAED,MAAM,WAAW,cAAc;IAC7B,yFAAyF;IACzF,IAAI,CAAC,EAAE,MAAM,CAAC;IACd;;;OAGG;IACH,OAAO,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC7C,oEAAoE;IACpE,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB;;;OAGG;IACH,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB;;;OAGG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;;OASG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;CACzB;AAED,MAAM,WAAW,YAAY;IAC3B,yFAAyF;IACzF,IAAI,EAAE,MAAM,CAAC;IACb,SAAS,EAAE,MAAM,CAAC;IAClB,kCAAkC;IAClC,OAAO,EAAE,MAAM,CAAC;IAChB,6EAA6E;IAC7E,GAAG,EAAE,MAAM,CAAC;IACZ,6BAA6B;IAC7B,QAAQ,EAAE,MAAM,CAAC;CAClB;AAED;;;;;;;;GAQG;AACH,MAAM,WAAW,OAAO;IACtB,iDAAiD;IACjD,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,4CAA4C;IAC5C,QAAQ,CAAC,KAAK,EAAE,SAAS,IAAI,EAAE,CAAC;IAChC,qCAAqC;IACrC,GAAG,EAAE,MAAM,MAAM,CAAC;IAClB;;;OAGG;IACH,OAAO,EAAE,CAAC,CAAC,SAAS,MAAM,EAAE,KAAK,EAAE,SAAS,CAAC,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,CAAC,CAAC;IAC5D,oCAAoC;IACpC,MAAM,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,IAAI,KAAK,OAAO,CAAC;IACtC,8DAA8D;IAC9D,OAAO,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,IAAI,KAAK,IAAI,CAAC;IACpC,mFAAmF;IACnF,UAAU,EAAE,CAAC,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,IAAI,KAAK,IAAI,CAAC;IACvC,8DAA8D;IAC9D,OAAO,EAAE,CAAC,IAAI,EAAE,IAAI,KAAK,IAAI,CAAC;IAC9B;;;OAGG;IACH,SAAS,EAAE,CAAC,GAAG,MAAM,EAAE,IAAI,EAAE,EAAE,KAAK,IAAI,CAAC;IACzC;;;;;;;OAOG;IACH,IAAI,EAAE,MAAM,IAAI,CAAC;IACjB;;;OAGG;IACH,OAAO,EAAE,CAAC,EAAE,EAAE,MAAM,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;IACvC;;;OAGG;IACH,MAAM,EAAE,MAAM,OAAO,CAAC,IAAI,CAAC,CAAC;IAC5B,KAAK,EAAE,MAAM,YAAY,CAAC;IAC1B;;;;OAIG;IACH,WAAW,EAAE,MAAM,MAAM,EAAE,CAAC;IAC5B;;;OAGG;IACH,eAAe,EAAE,MAAM,IAAI,CAAC;IAC5B,6FAA6F;IAC7F,OAAO,EAAE,MAAM,IAAI,CAAC;CACrB;AAED,sEAAsE;AACtE,qBAAa,gBAAiB,SAAQ,KAAK;IACzC,QAAQ,CAAC,WAAW,EAAE,MAAM,EAAE,CAAC;IAC/B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;gBAEV,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,EAAE,EAAE,IAAI,EAAE,MAAM;CAMjE;AAqBD;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,wBAAgB,aAAa,CAAC,OAAO,GAAE,cAAmB,GAAG,OAAO,CAkRnE"}
|
package/dist/network.js
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { createSyncEngine, } from "@homeostate/core";
|
|
2
2
|
import { asJson, firstDifference, formatPath, formatValue } from "./compare.js";
|
|
3
|
+
import { createLostKeyCheck } from "./lost-keys.js";
|
|
3
4
|
import { createRandom, randomSeed } from "./random.js";
|
|
4
5
|
/** Thrown by `expectConverged`; the message lists the differences. */
|
|
5
6
|
export class ConvergenceError extends Error {
|
|
@@ -51,8 +52,11 @@ const linkKey = (a, b) => a.index < b.index ? `${a.index}:${b.index}` : `${b.ind
|
|
|
51
52
|
* ```
|
|
52
53
|
*/
|
|
53
54
|
export function createNetwork(options = {}) {
|
|
54
|
-
const { seed = randomSeed(), latency = 0, dropRate = 0, reorder = false, maxDeliveries = 100000, } = options;
|
|
55
|
+
const { seed = randomSeed(), latency = 0, dropRate = 0, reorder = false, maxDeliveries = 100000, checkLostKeys = true, } = options;
|
|
55
56
|
const random = createRandom(seed);
|
|
57
|
+
const lostKeys = checkLostKeys ? createLostKeyCheck(seed) : undefined;
|
|
58
|
+
/** Whether `advance` or `settle` is delivering, so a document's change is not the test's. */
|
|
59
|
+
let delivering = false;
|
|
56
60
|
const peers = [];
|
|
57
61
|
const unsubscribers = [];
|
|
58
62
|
/** Pairs of peers whose link is cut; every other pair is linked. */
|
|
@@ -63,6 +67,8 @@ export function createNetwork(options = {}) {
|
|
|
63
67
|
let now = 0;
|
|
64
68
|
let order = 0;
|
|
65
69
|
const stats = { sent: 0, delivered: 0, dropped: 0, cut: 0 };
|
|
70
|
+
/** `stats.dropped` at the last `heal()`, or `undefined` before the first. */
|
|
71
|
+
let droppedAtHeal;
|
|
66
72
|
const delay = () => {
|
|
67
73
|
if (typeof latency === "number")
|
|
68
74
|
return latency;
|
|
@@ -119,10 +125,11 @@ export function createNetwork(options = {}) {
|
|
|
119
125
|
const deliver = (message) => {
|
|
120
126
|
now = Math.max(now, message.at);
|
|
121
127
|
stats.delivered++;
|
|
128
|
+
lostKeys?.changed(message.to, false);
|
|
122
129
|
message.to.doc.apply(message.update);
|
|
123
130
|
};
|
|
124
131
|
/** Deliver messages due by `until`, letting each delivery's work finish before the next. */
|
|
125
|
-
const
|
|
132
|
+
const deliverUntil = async (until) => {
|
|
126
133
|
let deliveries = 0;
|
|
127
134
|
for (;;) {
|
|
128
135
|
const next = queue[0];
|
|
@@ -132,6 +139,7 @@ export function createNetwork(options = {}) {
|
|
|
132
139
|
queue.shift();
|
|
133
140
|
deliver(next);
|
|
134
141
|
await Promise.resolve();
|
|
142
|
+
lostKeys?.afterDelivery(next.to, now, next.from);
|
|
135
143
|
continue;
|
|
136
144
|
}
|
|
137
145
|
// Asynchronous stores may still send; only stop once a full turn sends nothing.
|
|
@@ -141,6 +149,19 @@ export function createNetwork(options = {}) {
|
|
|
141
149
|
return;
|
|
142
150
|
}
|
|
143
151
|
};
|
|
152
|
+
/** `deliverUntil`, with the lost key check's checkpoints before and after. */
|
|
153
|
+
const run = async (until) => {
|
|
154
|
+
lostKeys?.beforeDeliveries(peers);
|
|
155
|
+
delivering = true;
|
|
156
|
+
try {
|
|
157
|
+
await deliverUntil(until);
|
|
158
|
+
}
|
|
159
|
+
finally {
|
|
160
|
+
delivering = false;
|
|
161
|
+
}
|
|
162
|
+
for (const peer of peers)
|
|
163
|
+
lostKeys?.afterDelivery(peer, now);
|
|
164
|
+
};
|
|
144
165
|
const view = (peer) => ({
|
|
145
166
|
store: peer.syncedState(),
|
|
146
167
|
document: asJson(peer.backend.read()),
|
|
@@ -154,7 +175,7 @@ export function createNetwork(options = {}) {
|
|
|
154
175
|
};
|
|
155
176
|
const views = peers.map(view);
|
|
156
177
|
peers.forEach((peer, i) => {
|
|
157
|
-
if (peer.engine.
|
|
178
|
+
if (peer.engine.getStatus().connected)
|
|
158
179
|
describe(`${peer.name}'s store differs from its document`, views[i].store, views[i].document);
|
|
159
180
|
});
|
|
160
181
|
const [first] = peers;
|
|
@@ -170,7 +191,11 @@ export function createNetwork(options = {}) {
|
|
|
170
191
|
now: () => now,
|
|
171
192
|
addPeer: (setup) => {
|
|
172
193
|
const index = peers.length;
|
|
173
|
-
const engine = createSyncEngine(
|
|
194
|
+
const engine = createSyncEngine({
|
|
195
|
+
...setup.config,
|
|
196
|
+
backend: setup.backend,
|
|
197
|
+
adapter: setup.adapter,
|
|
198
|
+
});
|
|
174
199
|
const peer = {
|
|
175
200
|
index,
|
|
176
201
|
name: setup.name ?? `peer-${index}`,
|
|
@@ -181,7 +206,11 @@ export function createNetwork(options = {}) {
|
|
|
181
206
|
syncedState: () => asJson(engine.getSyncedState()),
|
|
182
207
|
};
|
|
183
208
|
peers.push(peer);
|
|
184
|
-
|
|
209
|
+
lostKeys?.addPeer(peer);
|
|
210
|
+
unsubscribers.push(setup.doc.subscribe((update) => {
|
|
211
|
+
lostKeys?.changed(peer, !delivering);
|
|
212
|
+
broadcast(peer, update);
|
|
213
|
+
}));
|
|
185
214
|
for (const other of peers)
|
|
186
215
|
link(peer, other);
|
|
187
216
|
if (setup.connect !== false)
|
|
@@ -209,6 +238,7 @@ export function createNetwork(options = {}) {
|
|
|
209
238
|
}
|
|
210
239
|
},
|
|
211
240
|
heal: () => {
|
|
241
|
+
droppedAtHeal = stats.dropped;
|
|
212
242
|
for (const a of peers)
|
|
213
243
|
for (const b of peers)
|
|
214
244
|
if (a.index < b.index)
|
|
@@ -226,12 +256,18 @@ export function createNetwork(options = {}) {
|
|
|
226
256
|
const lines = differences();
|
|
227
257
|
if (lines.length === 0)
|
|
228
258
|
return;
|
|
229
|
-
const inFlight = queue.length
|
|
230
|
-
?
|
|
259
|
+
const inFlight = queue.length === 1
|
|
260
|
+
? "\n1 message is still on its way: settle() delivers it."
|
|
261
|
+
: queue.length > 1
|
|
262
|
+
? `\n${queue.length} messages are still on their way: settle() delivers them.`
|
|
263
|
+
: "";
|
|
264
|
+
const dropped = stats.dropped - (droppedAtHeal ?? 0);
|
|
265
|
+
const droppedHint = dropped > 0
|
|
266
|
+
? `\n${dropped === 1 ? "1 update was" : `${dropped} updates were`} dropped${droppedAtHeal === undefined ? "" : " since the last heal()"}: heal() and settle() until stats().dropped stops growing.`
|
|
231
267
|
: "";
|
|
232
268
|
throw new ConvergenceError(`${peers.length} peers did not converge (seed ${seed}):\n${lines
|
|
233
269
|
.map((line) => `- ${line}`)
|
|
234
|
-
.join("\n")}${inFlight}`, lines, seed);
|
|
270
|
+
.join("\n")}${inFlight}${droppedHint}`, lines, seed);
|
|
235
271
|
},
|
|
236
272
|
destroy: () => {
|
|
237
273
|
for (const peer of peers)
|
package/dist/network.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"network.js","sourceRoot":"","sources":["../src/network.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,gBAAgB,GAOjB,MAAM,kBAAkB,CAAC;AAC1B,OAAO,EAAE,MAAM,EAAE,eAAe,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAChF,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAiIvD,sEAAsE;AACtE,MAAM,OAAO,gBAAiB,SAAQ,KAAK;IAIzC,YAAY,OAAe,EAAE,WAAqB,EAAE,IAAY;QAC9D,KAAK,CAAC,OAAO,CAAC,CAAC;QAJR;;;;;WAAsB;QACtB;;;;;WAAa;QAIpB,IAAI,CAAC,IAAI,GAAG,kBAAkB,CAAC;QAC/B,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;CACF;AAWD,2FAA2F;AAC3F,MAAM,cAAc,GAAG,UAAU,CAAC,UAAU,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;AAE9D,0FAA0F;AAC1F,MAAM,gBAAgB,GAAG,GAAkB,EAAE,CAC3C,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,cAAc,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,CAAC;AAEvD,MAAM,OAAO,GAAG,CAAC,CAAO,EAAE,CAAO,EAAU,EAAE,CAC3C,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC;AAExE;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,MAAM,UAAU,aAAa,CAAC,UAA0B,EAAE;IACxD,MAAM,EACJ,IAAI,GAAG,UAAU,EAAE,EACnB,OAAO,GAAG,CAAC,EACX,QAAQ,GAAG,CAAC,EACZ,OAAO,GAAG,KAAK,EACf,aAAa,GAAG,MAAO,GACxB,GAAG,OAAO,CAAC;IACZ,MAAM,MAAM,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAClC,MAAM,KAAK,GAAW,EAAE,CAAC;IACzB,MAAM,aAAa,GAAkB,EAAE,CAAC;IACxC,oEAAoE;IACpE,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;IAC9B,qFAAqF;IACrF,MAAM,OAAO,GAAG,IAAI,GAAG,EAAkB,CAAC;IAC1C,IAAI,KAAK,GAAc,EAAE,CAAC;IAC1B,IAAI,GAAG,GAAG,CAAC,CAAC;IACZ,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,MAAM,KAAK,GAAG,EAAE,IAAI,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,CAAC;IAE5D,MAAM,KAAK,GAAG,GAAW,EAAE;QACzB,IAAI,OAAO,OAAO,KAAK,QAAQ;YAAE,OAAO,OAAO,CAAC;QAChD,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,GAAG,OAAO,CAAC;QAC3B,OAAO,GAAG,GAAG,MAAM,CAAC,IAAI,EAAE,GAAG,CAAC,GAAG,GAAG,GAAG,CAAC,CAAC;IAC3C,CAAC,CAAC;IAEF,MAAM,QAAQ,GAAG,CAAC,IAAU,EAAE,EAAQ,EAAE,MAAkB,EAAQ,EAAE;QAClE,KAAK,CAAC,IAAI,EAAE,CAAC;QACb,IAAI,EAAE,GAAG,GAAG,GAAG,KAAK,EAAE,CAAC;QACvB,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,GAAG,GAAG,GAAG,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,KAAK,EAAE,CAAC;YACxC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC;YACzC,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC;QACvB,CAAC;QACD,MAAM,OAAO,GAAY,EAAE,IAAI,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,EAAE,CAAC;QAClE,4DAA4D;QAC5D,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;QACzB,OACE,KAAK,GAAG,CAAC;YACT,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE;gBACvB,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;YAEzE,KAAK,EAAE,CAAC;QACV,KAAK,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC;IAClC,CAAC,CAAC;IAEF,MAAM,MAAM,GAAG,CAAC,CAAO,EAAE,CAAO,EAAW,EAAE,CAC3C,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAErC,MAAM,SAAS,GAAG,CAAC,IAAU,EAAE,MAAkB,EAAQ,EAAE;QACzD,KAAK,MAAM,EAAE,IAAI,KAAK,EAAE,CAAC;YACvB,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC;gBAAE,SAAS;YAChC,IAAI,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;gBAC5B,KAAK,CAAC,IAAI,EAAE,CAAC;gBACb,KAAK,CAAC,OAAO,EAAE,CAAC;gBAChB,SAAS;YACX,CAAC;YACD,yDAAyD;YACzD,QAAQ,CAAC,IAAI,EAAE,EAAE,EAAE,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC;QACrC,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,IAAI,GAAG,CAAC,CAAO,EAAE,CAAO,EAAQ,EAAE;QACtC,IAAI,CAAC,KAAK,CAAC;YAAE,OAAO;QACpB,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QAC1B,4DAA4D;QAC5D,QAAQ,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;QAC/B,QAAQ,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;IACjC,CAAC,CAAC;IAEF,MAAM,MAAM,GAAG,CAAC,CAAO,EAAE,CAAO,EAAQ,EAAE;QACxC,IAAI,CAAC,KAAK,CAAC;YAAE,OAAO;QACpB,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QACvB,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC;QAC5B,KAAK,GAAG,KAAK,CAAC,MAAM,CAClB,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAC,CAC1E,CAAC;QACF,KAAK,CAAC,GAAG,IAAI,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC;IACrC,CAAC,CAAC;IAEF,MAAM,OAAO,GAAG,CAAC,OAAgB,EAAQ,EAAE;QACzC,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,OAAO,CAAC,EAAE,CAAC,CAAC;QAChC,KAAK,CAAC,SAAS,EAAE,CAAC;QAClB,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;IACvC,CAAC,CAAC;IAEF,4FAA4F;IAC5F,MAAM,GAAG,GAAG,KAAK,EAAE,KAAa,EAAiB,EAAE;QACjD,IAAI,UAAU,GAAG,CAAC,CAAC;QACnB,SAAS,CAAC;YACR,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;YACtB,IAAI,IAAI,IAAI,IAAI,CAAC,EAAE,IAAI,KAAK,EAAE,CAAC;gBAC7B,IAAI,EAAE,UAAU,GAAG,aAAa;oBAC9B,MAAM,IAAI,KAAK,CACb,oCAAoC,aAAa,2CAA2C,CAC7F,CAAC;gBACJ,KAAK,CAAC,KAAK,EAAE,CAAC;gBACd,OAAO,CAAC,IAAI,CAAC,CAAC;gBACd,MAAM,OAAO,CAAC,OAAO,EAAE,CAAC;gBACxB,SAAS;YACX,CAAC;YACD,gFAAgF;YAChF,MAAM,gBAAgB,EAAE,CAAC;YACzB,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;YACrB,IAAI,CAAC,GAAG,IAAI,GAAG,CAAC,EAAE,GAAG,KAAK;gBAAE,OAAO;QACrC,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,IAAI,GAAG,CAAC,IAAU,EAAE,EAAE,CAAC,CAAC;QAC5B,KAAK,EAAE,IAAI,CAAC,WAAW,EAAE;QACzB,QAAQ,EAAE,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC;KACtC,CAAC,CAAC;IAEH,MAAM,WAAW,GAAG,GAAa,EAAE;QACjC,MAAM,KAAK,GAAa,EAAE,CAAC;QAC3B,MAAM,QAAQ,GAAG,CAAC,OAAe,EAAE,IAAa,EAAE,KAAc,EAAQ,EAAE;YACxE,MAAM,UAAU,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;YAChD,IAAI,UAAU;gBACZ,KAAK,CAAC,IAAI,CACR,GAAG,OAAO,OAAO,UAAU,CAAC,UAAU,CAAC,IAAI,CAAC,KAAK,WAAW,CAAC,UAAU,CAAC,IAAI,CAAC,OAAO,WAAW,CAAC,UAAU,CAAC,KAAK,CAAC,EAAE,CACpH,CAAC;QACN,CAAC,CAAC;QAEF,MAAM,KAAK,GAAG,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QAC9B,KAAK,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,EAAE;YACxB,IAAI,IAAI,CAAC,MAAM,CAAC,WAAW,EAAE;gBAC3B,QAAQ,CACN,GAAG,IAAI,CAAC,IAAI,oCAAoC,EAChD,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,EACd,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,CAClB,CAAC;QACN,CAAC,CAAC,CAAC;QACH,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC;QACtB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,QAAQ,CACN,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,4BAA4B,KAAK,CAAC,IAAI,IAAI,EAC1D,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,EACjB,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,CAClB,CAAC;YACF,QAAQ,CACN,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,yBAAyB,KAAK,CAAC,IAAI,IAAI,EACvD,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,EACd,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CACf,CAAC;QACJ,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;IAEF,OAAO;QACL,IAAI;QACJ,KAAK;QACL,GAAG,EAAE,GAAG,EAAE,CAAC,GAAG;QAEd,OAAO,EAAE,CAAmB,KAAmB,EAAW,EAAE;YAC1D,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;YAC3B,MAAM,MAAM,GAAG,gBAAgB,CAC7B,KAAK,CAAC,OAAO,EACb,KAAK,CAAC,OAAO,EACb,KAAK,CAAC,MAAM,CACb,CAAC;YACF,MAAM,IAAI,GAAY;gBACpB,KAAK;gBACL,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,QAAQ,KAAK,EAAE;gBACnC,OAAO,EAAE,KAAK,CAAC,OAAO;gBACtB,OAAO,EAAE,KAAK,CAAC,OAAO;gBACtB,GAAG,EAAE,KAAK,CAAC,GAAG;gBACd,MAAM;gBACN,WAAW,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,CAAC;aACnD,CAAC;YACF,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACjB,aAAa,CAAC,IAAI,CAChB,KAAK,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,SAAS,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CACzD,CAAC;YACF,KAAK,MAAM,KAAK,IAAI,KAAK;gBAAE,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;YAC7C,IAAI,KAAK,CAAC,OAAO,KAAK,KAAK;gBAAE,MAAM,CAAC,OAAO,EAAE,CAAC;YAC9C,OAAO,IAAI,CAAC;QACd,CAAC;QAED,MAAM;QAEN,OAAO,EAAE,IAAI;QAEb,UAAU,EAAE,MAAM;QAElB,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE;YAChB,KAAK,MAAM,KAAK,IAAI,KAAK;gBAAE,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;QACjD,CAAC;QAED,SAAS,EAAE,CAAC,GAAG,MAAM,EAAE,EAAE;YACvB,MAAM,OAAO,GAAG,CAAC,IAAU,EAAU,EAAE,CACrC,MAAM,CAAC,SAAS,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;YACpD,KAAK,MAAM,CAAC,IAAI,KAAK;gBACnB,KAAK,MAAM,CAAC,IAAI,KAAK,EAAE,CAAC;oBACtB,IAAI,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK;wBAAE,SAAS;oBACjC,MAAM,QAAQ,GAAG,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC;oBAChE,IAAI,CAAC,QAAQ;wBAAE,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;yBACvB,IAAI,CAAC,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC;wBAAE,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;gBACrC,CAAC;QACL,CAAC;QAED,IAAI,EAAE,GAAG,EAAE;YACT,KAAK,MAAM,CAAC,IAAI,KAAK;gBACnB,KAAK,MAAM,CAAC,IAAI,KAAK;oBAAE,IAAI,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK;wBAAE,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC7D,CAAC;QAED,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,EAAE;YACpB,MAAM,KAAK,GAAG,GAAG,GAAG,EAAE,CAAC;YACvB,MAAM,GAAG,CAAC,KAAK,CAAC,CAAC;YACjB,GAAG,GAAG,KAAK,CAAC;QACd,CAAC;QAED,MAAM,EAAE,GAAG,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC;QAE3B,KAAK,EAAE,GAAG,EAAE,CAAC,CAAC,EAAE,GAAG,KAAK,EAAE,QAAQ,EAAE,KAAK,CAAC,MAAM,EAAE,CAAC;QAEnD,WAAW;QAEX,eAAe,EAAE,GAAG,EAAE;YACpB,MAAM,KAAK,GAAG,WAAW,EAAE,CAAC;YAC5B,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;gBAAE,OAAO;YAC/B,MAAM,QAAQ,GACZ,KAAK,CAAC,MAAM,GAAG,CAAC;gBACd,CAAC,CAAC,KAAK,KAAK,CAAC,MAAM,2DAA2D;gBAC9E,CAAC,CAAC,EAAE,CAAC;YACT,MAAM,IAAI,gBAAgB,CACxB,GAAG,KAAK,CAAC,MAAM,iCAAiC,IAAI,OAAO,KAAK;iBAC7D,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,KAAK,IAAI,EAAE,CAAC;iBAC1B,IAAI,CAAC,IAAI,CAAC,GAAG,QAAQ,EAAE,EAC1B,KAAK,EACL,IAAI,CACL,CAAC;QACJ,CAAC;QAED,OAAO,EAAE,GAAG,EAAE;YACZ,KAAK,MAAM,IAAI,IAAI,KAAK;gBAAE,IAAI,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC;YACnD,aAAa,CAAC,OAAO,CAAC,CAAC,WAAW,EAAE,EAAE,CAAC,WAAW,EAAE,CAAC,CAAC;YACtD,aAAa,CAAC,MAAM,GAAG,CAAC,CAAC;YACzB,KAAK,GAAG,EAAE,CAAC;QACb,CAAC;KACF,CAAC;AACJ,CAAC"}
|
|
1
|
+
{"version":3,"file":"network.js","sourceRoot":"","sources":["../src/network.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,gBAAgB,GAOjB,MAAM,kBAAkB,CAAC;AAC1B,OAAO,EAAE,MAAM,EAAE,eAAe,EAAE,UAAU,EAAE,WAAW,EAAE,MAAM,cAAc,CAAC;AAChF,OAAO,EAAE,kBAAkB,EAAE,MAAM,gBAAgB,CAAC;AACpD,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AA4JvD,sEAAsE;AACtE,MAAM,OAAO,gBAAiB,SAAQ,KAAK;IAIzC,YAAY,OAAe,EAAE,WAAqB,EAAE,IAAY;QAC9D,KAAK,CAAC,OAAO,CAAC,CAAC;QAJR;;;;;WAAsB;QACtB;;;;;WAAa;QAIpB,IAAI,CAAC,IAAI,GAAG,kBAAkB,CAAC;QAC/B,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;QAC/B,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;IACnB,CAAC;CACF;AAWD,2FAA2F;AAC3F,MAAM,cAAc,GAAG,UAAU,CAAC,UAAU,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC;AAE9D,0FAA0F;AAC1F,MAAM,gBAAgB,GAAG,GAAkB,EAAE,CAC3C,IAAI,OAAO,CAAC,CAAC,OAAO,EAAE,EAAE,CAAC,cAAc,CAAC,OAAO,EAAE,CAAC,CAAC,CAAC,CAAC;AAEvD,MAAM,OAAO,GAAG,CAAC,CAAO,EAAE,CAAO,EAAU,EAAE,CAC3C,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC;AAExE;;;;;;;;;;;;;;;;;;;;;;GAsBG;AACH,MAAM,UAAU,aAAa,CAAC,UAA0B,EAAE;IACxD,MAAM,EACJ,IAAI,GAAG,UAAU,EAAE,EACnB,OAAO,GAAG,CAAC,EACX,QAAQ,GAAG,CAAC,EACZ,OAAO,GAAG,KAAK,EACf,aAAa,GAAG,MAAO,EACvB,aAAa,GAAG,IAAI,GACrB,GAAG,OAAO,CAAC;IACZ,MAAM,MAAM,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAClC,MAAM,QAAQ,GAAG,aAAa,CAAC,CAAC,CAAC,kBAAkB,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,SAAS,CAAC;IACtE,6FAA6F;IAC7F,IAAI,UAAU,GAAG,KAAK,CAAC;IACvB,MAAM,KAAK,GAAW,EAAE,CAAC;IACzB,MAAM,aAAa,GAAkB,EAAE,CAAC;IACxC,oEAAoE;IACpE,MAAM,GAAG,GAAG,IAAI,GAAG,EAAU,CAAC;IAC9B,qFAAqF;IACrF,MAAM,OAAO,GAAG,IAAI,GAAG,EAAkB,CAAC;IAC1C,IAAI,KAAK,GAAc,EAAE,CAAC;IAC1B,IAAI,GAAG,GAAG,CAAC,CAAC;IACZ,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,MAAM,KAAK,GAAG,EAAE,IAAI,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,OAAO,EAAE,CAAC,EAAE,GAAG,EAAE,CAAC,EAAE,CAAC;IAC5D,6EAA6E;IAC7E,IAAI,aAAiC,CAAC;IAEtC,MAAM,KAAK,GAAG,GAAW,EAAE;QACzB,IAAI,OAAO,OAAO,KAAK,QAAQ;YAAE,OAAO,OAAO,CAAC;QAChD,MAAM,CAAC,GAAG,EAAE,GAAG,CAAC,GAAG,OAAO,CAAC;QAC3B,OAAO,GAAG,GAAG,MAAM,CAAC,IAAI,EAAE,GAAG,CAAC,GAAG,GAAG,GAAG,CAAC,CAAC;IAC3C,CAAC,CAAC;IAEF,MAAM,QAAQ,GAAG,CAAC,IAAU,EAAE,EAAQ,EAAE,MAAkB,EAAQ,EAAE;QAClE,KAAK,CAAC,IAAI,EAAE,CAAC;QACb,IAAI,EAAE,GAAG,GAAG,GAAG,KAAK,EAAE,CAAC;QACvB,IAAI,CAAC,OAAO,EAAE,CAAC;YACb,MAAM,GAAG,GAAG,GAAG,IAAI,CAAC,KAAK,IAAI,EAAE,CAAC,KAAK,EAAE,CAAC;YACxC,EAAE,GAAG,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,OAAO,CAAC,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC;YACzC,OAAO,CAAC,GAAG,CAAC,GAAG,EAAE,EAAE,CAAC,CAAC;QACvB,CAAC;QACD,MAAM,OAAO,GAAY,EAAE,IAAI,EAAE,EAAE,EAAE,MAAM,EAAE,EAAE,EAAE,KAAK,EAAE,KAAK,EAAE,EAAE,CAAC;QAClE,4DAA4D;QAC5D,IAAI,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;QACzB,OACE,KAAK,GAAG,CAAC;YACT,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,GAAG,EAAE;gBACvB,CAAC,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,IAAI,KAAK,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC;YAEzE,KAAK,EAAE,CAAC;QACV,KAAK,CAAC,MAAM,CAAC,KAAK,EAAE,CAAC,EAAE,OAAO,CAAC,CAAC;IAClC,CAAC,CAAC;IAEF,MAAM,MAAM,GAAG,CAAC,CAAO,EAAE,CAAO,EAAW,EAAE,CAC3C,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;IAErC,MAAM,SAAS,GAAG,CAAC,IAAU,EAAE,MAAkB,EAAQ,EAAE;QACzD,KAAK,MAAM,EAAE,IAAI,KAAK,EAAE,CAAC;YACvB,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC;gBAAE,SAAS;YAChC,IAAI,MAAM,CAAC,MAAM,CAAC,QAAQ,CAAC,EAAE,CAAC;gBAC5B,KAAK,CAAC,IAAI,EAAE,CAAC;gBACb,KAAK,CAAC,OAAO,EAAE,CAAC;gBAChB,SAAS;YACX,CAAC;YACD,yDAAyD;YACzD,QAAQ,CAAC,IAAI,EAAE,EAAE,EAAE,MAAM,CAAC,KAAK,EAAE,CAAC,CAAC;QACrC,CAAC;IACH,CAAC,CAAC;IAEF,MAAM,IAAI,GAAG,CAAC,CAAO,EAAE,CAAO,EAAQ,EAAE;QACtC,IAAI,CAAC,KAAK,CAAC;YAAE,OAAO;QACpB,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QAC1B,4DAA4D;QAC5D,QAAQ,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;QAC/B,QAAQ,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC,CAAC;IACjC,CAAC,CAAC;IAEF,MAAM,MAAM,GAAG,CAAC,CAAO,EAAE,CAAO,EAAQ,EAAE;QACxC,IAAI,CAAC,KAAK,CAAC;YAAE,OAAO;QACpB,GAAG,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QACvB,MAAM,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC;QAC5B,KAAK,GAAG,KAAK,CAAC,MAAM,CAClB,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,IAAI,CAAC,IAAI,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC,CAAC,CAC1E,CAAC;QACF,KAAK,CAAC,GAAG,IAAI,MAAM,GAAG,KAAK,CAAC,MAAM,CAAC;IACrC,CAAC,CAAC;IAEF,MAAM,OAAO,GAAG,CAAC,OAAgB,EAAQ,EAAE;QACzC,GAAG,GAAG,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,OAAO,CAAC,EAAE,CAAC,CAAC;QAChC,KAAK,CAAC,SAAS,EAAE,CAAC;QAClB,QAAQ,EAAE,OAAO,CAAC,OAAO,CAAC,EAAE,EAAE,KAAK,CAAC,CAAC;QACrC,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,KAAK,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC;IACvC,CAAC,CAAC;IAEF,4FAA4F;IAC5F,MAAM,YAAY,GAAG,KAAK,EAAE,KAAa,EAAiB,EAAE;QAC1D,IAAI,UAAU,GAAG,CAAC,CAAC;QACnB,SAAS,CAAC;YACR,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;YACtB,IAAI,IAAI,IAAI,IAAI,CAAC,EAAE,IAAI,KAAK,EAAE,CAAC;gBAC7B,IAAI,EAAE,UAAU,GAAG,aAAa;oBAC9B,MAAM,IAAI,KAAK,CACb,oCAAoC,aAAa,2CAA2C,CAC7F,CAAC;gBACJ,KAAK,CAAC,KAAK,EAAE,CAAC;gBACd,OAAO,CAAC,IAAI,CAAC,CAAC;gBACd,MAAM,OAAO,CAAC,OAAO,EAAE,CAAC;gBACxB,QAAQ,EAAE,aAAa,CAAC,IAAI,CAAC,EAAE,EAAE,GAAG,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC;gBACjD,SAAS;YACX,CAAC;YACD,gFAAgF;YAChF,MAAM,gBAAgB,EAAE,CAAC;YACzB,MAAM,GAAG,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;YACrB,IAAI,CAAC,GAAG,IAAI,GAAG,CAAC,EAAE,GAAG,KAAK;gBAAE,OAAO;QACrC,CAAC;IACH,CAAC,CAAC;IAEF,8EAA8E;IAC9E,MAAM,GAAG,GAAG,KAAK,EAAE,KAAa,EAAiB,EAAE;QACjD,QAAQ,EAAE,gBAAgB,CAAC,KAAK,CAAC,CAAC;QAClC,UAAU,GAAG,IAAI,CAAC;QAClB,IAAI,CAAC;YACH,MAAM,YAAY,CAAC,KAAK,CAAC,CAAC;QAC5B,CAAC;gBAAS,CAAC;YACT,UAAU,GAAG,KAAK,CAAC;QACrB,CAAC;QACD,KAAK,MAAM,IAAI,IAAI,KAAK;YAAE,QAAQ,EAAE,aAAa,CAAC,IAAI,EAAE,GAAG,CAAC,CAAC;IAC/D,CAAC,CAAC;IAEF,MAAM,IAAI,GAAG,CAAC,IAAU,EAAE,EAAE,CAAC,CAAC;QAC5B,KAAK,EAAE,IAAI,CAAC,WAAW,EAAE;QACzB,QAAQ,EAAE,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,IAAI,EAAE,CAAC;KACtC,CAAC,CAAC;IAEH,MAAM,WAAW,GAAG,GAAa,EAAE;QACjC,MAAM,KAAK,GAAa,EAAE,CAAC;QAC3B,MAAM,QAAQ,GAAG,CAAC,OAAe,EAAE,IAAa,EAAE,KAAc,EAAQ,EAAE;YACxE,MAAM,UAAU,GAAG,eAAe,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;YAChD,IAAI,UAAU;gBACZ,KAAK,CAAC,IAAI,CACR,GAAG,OAAO,OAAO,UAAU,CAAC,UAAU,CAAC,IAAI,CAAC,KAAK,WAAW,CAAC,UAAU,CAAC,IAAI,CAAC,OAAO,WAAW,CAAC,UAAU,CAAC,KAAK,CAAC,EAAE,CACpH,CAAC;QACN,CAAC,CAAC;QAEF,MAAM,KAAK,GAAG,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QAC9B,KAAK,CAAC,OAAO,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,EAAE;YACxB,IAAI,IAAI,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC,SAAS;gBACnC,QAAQ,CACN,GAAG,IAAI,CAAC,IAAI,oCAAoC,EAChD,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,EACd,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,CAClB,CAAC;QACN,CAAC,CAAC,CAAC;QACH,MAAM,CAAC,KAAK,CAAC,GAAG,KAAK,CAAC;QACtB,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;YACtC,QAAQ,CACN,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,4BAA4B,KAAK,CAAC,IAAI,IAAI,EAC1D,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,EACjB,KAAK,CAAC,CAAC,CAAC,CAAC,QAAQ,CAClB,CAAC;YACF,QAAQ,CACN,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,yBAAyB,KAAK,CAAC,IAAI,IAAI,EACvD,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,EACd,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CACf,CAAC;QACJ,CAAC;QACD,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;IAEF,OAAO;QACL,IAAI;QACJ,KAAK;QACL,GAAG,EAAE,GAAG,EAAE,CAAC,GAAG;QAEd,OAAO,EAAE,CAAmB,KAAmB,EAAW,EAAE;YAC1D,MAAM,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC;YAC3B,MAAM,MAAM,GAAG,gBAAgB,CAAC;gBAC9B,GAAG,KAAK,CAAC,MAAM;gBACf,OAAO,EAAE,KAAK,CAAC,OAAO;gBACtB,OAAO,EAAE,KAAK,CAAC,OAAO;aACvB,CAAC,CAAC;YACH,MAAM,IAAI,GAAY;gBACpB,KAAK;gBACL,IAAI,EAAE,KAAK,CAAC,IAAI,IAAI,QAAQ,KAAK,EAAE;gBACnC,OAAO,EAAE,KAAK,CAAC,OAAO;gBACtB,OAAO,EAAE,KAAK,CAAC,OAAO;gBACtB,GAAG,EAAE,KAAK,CAAC,GAAG;gBACd,MAAM;gBACN,WAAW,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,MAAM,CAAC,cAAc,EAAE,CAAC;aACnD,CAAC;YACF,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YACjB,QAAQ,EAAE,OAAO,CAAC,IAAI,CAAC,CAAC;YACxB,aAAa,CAAC,IAAI,CAChB,KAAK,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,MAAM,EAAE,EAAE;gBAC7B,QAAQ,EAAE,OAAO,CAAC,IAAI,EAAE,CAAC,UAAU,CAAC,CAAC;gBACrC,SAAS,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;YAC1B,CAAC,CAAC,CACH,CAAC;YACF,KAAK,MAAM,KAAK,IAAI,KAAK;gBAAE,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;YAC7C,IAAI,KAAK,CAAC,OAAO,KAAK,KAAK;gBAAE,MAAM,CAAC,OAAO,EAAE,CAAC;YAC9C,OAAO,IAAI,CAAC;QACd,CAAC;QAED,MAAM;QAEN,OAAO,EAAE,IAAI;QAEb,UAAU,EAAE,MAAM;QAElB,OAAO,EAAE,CAAC,IAAI,EAAE,EAAE;YAChB,KAAK,MAAM,KAAK,IAAI,KAAK;gBAAE,MAAM,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;QACjD,CAAC;QAED,SAAS,EAAE,CAAC,GAAG,MAAM,EAAE,EAAE;YACvB,MAAM,OAAO,GAAG,CAAC,IAAU,EAAU,EAAE,CACrC,MAAM,CAAC,SAAS,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,QAAQ,CAAC,IAAI,CAAC,CAAC,CAAC;YACpD,KAAK,MAAM,CAAC,IAAI,KAAK;gBACnB,KAAK,MAAM,CAAC,IAAI,KAAK,EAAE,CAAC;oBACtB,IAAI,CAAC,CAAC,KAAK,IAAI,CAAC,CAAC,KAAK;wBAAE,SAAS;oBACjC,MAAM,QAAQ,GAAG,OAAO,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC,KAAK,OAAO,CAAC,CAAC,CAAC,CAAC;oBAChE,IAAI,CAAC,QAAQ;wBAAE,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;yBACvB,IAAI,CAAC,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC;wBAAE,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;gBACrC,CAAC;QACL,CAAC;QAED,IAAI,EAAE,GAAG,EAAE;YACT,aAAa,GAAG,KAAK,CAAC,OAAO,CAAC;YAC9B,KAAK,MAAM,CAAC,IAAI,KAAK;gBACnB,KAAK,MAAM,CAAC,IAAI,KAAK;oBAAE,IAAI,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK;wBAAE,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC7D,CAAC;QAED,OAAO,EAAE,KAAK,EAAE,EAAE,EAAE,EAAE;YACpB,MAAM,KAAK,GAAG,GAAG,GAAG,EAAE,CAAC;YACvB,MAAM,GAAG,CAAC,KAAK,CAAC,CAAC;YACjB,GAAG,GAAG,KAAK,CAAC;QACd,CAAC;QAED,MAAM,EAAE,GAAG,EAAE,CAAC,GAAG,CAAC,QAAQ,CAAC;QAE3B,KAAK,EAAE,GAAG,EAAE,CAAC,CAAC,EAAE,GAAG,KAAK,EAAE,QAAQ,EAAE,KAAK,CAAC,MAAM,EAAE,CAAC;QAEnD,WAAW;QAEX,eAAe,EAAE,GAAG,EAAE;YACpB,MAAM,KAAK,GAAG,WAAW,EAAE,CAAC;YAC5B,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;gBAAE,OAAO;YAC/B,MAAM,QAAQ,GACZ,KAAK,CAAC,MAAM,KAAK,CAAC;gBAChB,CAAC,CAAC,wDAAwD;gBAC1D,CAAC,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC;oBAChB,CAAC,CAAC,KAAK,KAAK,CAAC,MAAM,2DAA2D;oBAC9E,CAAC,CAAC,EAAE,CAAC;YACX,MAAM,OAAO,GAAG,KAAK,CAAC,OAAO,GAAG,CAAC,aAAa,IAAI,CAAC,CAAC,CAAC;YACrD,MAAM,WAAW,GACf,OAAO,GAAG,CAAC;gBACT,CAAC,CAAC,KAAK,OAAO,KAAK,CAAC,CAAC,CAAC,CAAC,cAAc,CAAC,CAAC,CAAC,GAAG,OAAO,eAAe,WAC7D,aAAa,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,wBACrC,4DAA4D;gBAC9D,CAAC,CAAC,EAAE,CAAC;YACT,MAAM,IAAI,gBAAgB,CACxB,GAAG,KAAK,CAAC,MAAM,iCAAiC,IAAI,OAAO,KAAK;iBAC7D,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,KAAK,IAAI,EAAE,CAAC;iBAC1B,IAAI,CAAC,IAAI,CAAC,GAAG,QAAQ,GAAG,WAAW,EAAE,EACxC,KAAK,EACL,IAAI,CACL,CAAC;QACJ,CAAC;QAED,OAAO,EAAE,GAAG,EAAE;YACZ,KAAK,MAAM,IAAI,IAAI,KAAK;gBAAE,IAAI,CAAC,MAAM,CAAC,UAAU,EAAE,CAAC;YACnD,aAAa,CAAC,OAAO,CAAC,CAAC,WAAW,EAAE,EAAE,CAAC,WAAW,EAAE,CAAC,CAAC;YACtD,aAAa,CAAC,MAAM,GAAG,CAAC,CAAC;YACzB,KAAK,GAAG,EAAE,CAAC;QACb,CAAC;KACF,CAAC;AACJ,CAAC"}
|
package/dist/randomized.d.ts
CHANGED
|
@@ -13,7 +13,7 @@ export interface RandomizedOptions<S extends object = any> {
|
|
|
13
13
|
steps?: number;
|
|
14
14
|
/** What a step may do to a random peer's store, one picked at random. Defaults to `[randomEdit]`. */
|
|
15
15
|
actions?: readonly RandomAction<S>[];
|
|
16
|
-
/** Latency, drops and
|
|
16
|
+
/** Latency, drops, reordering and the lost key check, as for `createNetwork`. */
|
|
17
17
|
network?: Omit<NetworkOptions, "seed">;
|
|
18
18
|
/** Chance that a step cuts or restores a random link instead of changing a store. Defaults to 0.1. */
|
|
19
19
|
linkChangeRate?: number;
|
|
@@ -24,10 +24,16 @@ export interface RandomizedOptions<S extends object = any> {
|
|
|
24
24
|
* Runs random steps against a fresh network, then heals it, settles it and checks that every
|
|
25
25
|
* peer converged. Each step changes a random peer's store with a random action, or cuts or
|
|
26
26
|
* restores a random link, and lets some virtual time pass, so changes cross each other on
|
|
27
|
-
* their way.
|
|
27
|
+
* their way. Throughout, the network checks that no peer's store and document both lose a
|
|
28
|
+
* synced key that no peer deleted, unless `network.checkLostKeys` is `false`.
|
|
28
29
|
*
|
|
29
|
-
*
|
|
30
|
-
*
|
|
30
|
+
* With a `dropRate`, the updates stores write in answer to the healing exchange can be dropped
|
|
31
|
+
* too, so the run heals and settles again until a round drops nothing, at most 10 rounds, as a
|
|
32
|
+
* provider that resyncs would.
|
|
33
|
+
*
|
|
34
|
+
* Rejects with an error that names the seed and step when a step throws, a peer loses a synced
|
|
35
|
+
* key or the peers do not converge; pass that `seed` to replay the same run. When the last of
|
|
36
|
+
* those rounds still dropped updates, the error says so. Give each document a fixed ID for runs
|
|
31
37
|
* that replay exactly, such as `doc.clientID = index + 1` for Yjs.
|
|
32
38
|
*/
|
|
33
39
|
export declare function runRandomized<S extends object>(options: RandomizedOptions<S>): Promise<void>;
|
package/dist/randomized.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"randomized.d.ts","sourceRoot":"","sources":["../src/randomized.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,IAAI,EACT,KAAK,SAAS,EACf,MAAM,cAAc,CAAC;AAEtB,OAAO,EAA4B,KAAK,MAAM,EAAE,MAAM,aAAa,CAAC;AAEpE,mEAAmE;AAEnE,MAAM,MAAM,YAAY,CAAC,CAAC,SAAS,MAAM,GAAG,GAAG,IAAI,CACjD,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC,EACb,MAAM,EAAE,MAAM,KACX,IAAI,CAAC;
|
|
1
|
+
{"version":3,"file":"randomized.d.ts","sourceRoot":"","sources":["../src/randomized.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,IAAI,EACT,KAAK,SAAS,EACf,MAAM,cAAc,CAAC;AAEtB,OAAO,EAA4B,KAAK,MAAM,EAAE,MAAM,aAAa,CAAC;AAEpE,mEAAmE;AAEnE,MAAM,MAAM,YAAY,CAAC,CAAC,SAAS,MAAM,GAAG,GAAG,IAAI,CACjD,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC,EACb,MAAM,EAAE,MAAM,KACX,IAAI,CAAC;AAMV,MAAM,WAAW,iBAAiB,CAAC,CAAC,SAAS,MAAM,GAAG,GAAG;IACvD,kEAAkE;IAClE,UAAU,EAAE,CAAC,KAAK,EAAE,MAAM,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC;IAC5C,iGAAiG;IACjG,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,qBAAqB;IACrB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,uBAAuB;IACvB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,qGAAqG;IACrG,OAAO,CAAC,EAAE,SAAS,YAAY,CAAC,CAAC,CAAC,EAAE,CAAC;IACrC,iFAAiF;IACjF,OAAO,CAAC,EAAE,IAAI,CAAC,cAAc,EAAE,MAAM,CAAC,CAAC;IACvC,sGAAsG;IACtG,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,wFAAwF;IACxF,YAAY,CAAC,EAAE,MAAM,CAAC;CACvB;AAED;;;;;;;;;;;;;;;GAeG;AACH,wBAAsB,aAAa,CAAC,CAAC,SAAS,MAAM,EAClD,OAAO,EAAE,iBAAiB,CAAC,CAAC,CAAC,GAC5B,OAAO,CAAC,IAAI,CAAC,CA8Df;AA+ED;;;;;;;;GAQG;AACH,eAAO,MAAM,UAAU,EAAE,YAyBxB,CAAC"}
|
package/dist/randomized.js
CHANGED
|
@@ -1,13 +1,21 @@
|
|
|
1
1
|
import { createNetwork, } from "./network.js";
|
|
2
2
|
import { createRandom, randomSeed } from "./random.js";
|
|
3
|
+
/** Most rounds of `heal()` and `settle()` at the end of a run. */
|
|
4
|
+
const HEAL_ROUNDS = 10;
|
|
3
5
|
/**
|
|
4
6
|
* Runs random steps against a fresh network, then heals it, settles it and checks that every
|
|
5
7
|
* peer converged. Each step changes a random peer's store with a random action, or cuts or
|
|
6
8
|
* restores a random link, and lets some virtual time pass, so changes cross each other on
|
|
7
|
-
* their way.
|
|
9
|
+
* their way. Throughout, the network checks that no peer's store and document both lose a
|
|
10
|
+
* synced key that no peer deleted, unless `network.checkLostKeys` is `false`.
|
|
8
11
|
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
12
|
+
* With a `dropRate`, the updates stores write in answer to the healing exchange can be dropped
|
|
13
|
+
* too, so the run heals and settles again until a round drops nothing, at most 10 rounds, as a
|
|
14
|
+
* provider that resyncs would.
|
|
15
|
+
*
|
|
16
|
+
* Rejects with an error that names the seed and step when a step throws, a peer loses a synced
|
|
17
|
+
* key or the peers do not converge; pass that `seed` to replay the same run. When the last of
|
|
18
|
+
* those rounds still dropped updates, the error says so. Give each document a fixed ID for runs
|
|
11
19
|
* that replay exactly, such as `doc.clientID = index + 1` for Yjs.
|
|
12
20
|
*/
|
|
13
21
|
export async function runRandomized(options) {
|
|
@@ -16,6 +24,8 @@ export async function runRandomized(options) {
|
|
|
16
24
|
const random = createRandom(seed ^ 0x5bd1e995);
|
|
17
25
|
const network = createNetwork({ ...networkOptions, seed });
|
|
18
26
|
let step = 0;
|
|
27
|
+
/** Whether every round of healing dropped updates, the last included. */
|
|
28
|
+
let stillDropping = false;
|
|
19
29
|
try {
|
|
20
30
|
const peers = Array.from({ length: peerCount }, (_, index) => network.addPeer(createPeer(index)));
|
|
21
31
|
for (; step < steps; step++) {
|
|
@@ -32,14 +42,25 @@ export async function runRandomized(options) {
|
|
|
32
42
|
}
|
|
33
43
|
await network.advance(random.int(stepInterval + 1));
|
|
34
44
|
}
|
|
35
|
-
|
|
36
|
-
|
|
45
|
+
// Stores may write in answer to the documents a heal exchanges, and those updates can be
|
|
46
|
+
// dropped too: heal again until a round drops nothing, as a provider that resyncs would.
|
|
47
|
+
let dropping = true;
|
|
48
|
+
for (let round = 0; dropping && round < HEAL_ROUNDS; round++) {
|
|
49
|
+
const dropped = network.stats().dropped;
|
|
50
|
+
network.heal();
|
|
51
|
+
await network.settle();
|
|
52
|
+
dropping = network.stats().dropped !== dropped;
|
|
53
|
+
}
|
|
54
|
+
stillDropping = dropping;
|
|
37
55
|
network.expectConverged();
|
|
38
56
|
}
|
|
39
57
|
catch (error) {
|
|
40
58
|
const when = step < steps ? `at step ${step + 1} of ${steps}` : `after ${steps} steps`;
|
|
41
59
|
const message = error instanceof Error ? error.message : String(error);
|
|
42
|
-
|
|
60
|
+
const dropping = stillDropping
|
|
61
|
+
? `\nUpdates were still being dropped after ${HEAL_ROUNDS} rounds of heal() and settle(): does a store write in answer to every exchange?`
|
|
62
|
+
: "";
|
|
63
|
+
throw Object.assign(new Error(`Randomized run failed ${when}; pass \`seed: ${seed}\` to replay it.\n${message}${dropping}`), { cause: error });
|
|
43
64
|
}
|
|
44
65
|
finally {
|
|
45
66
|
network.destroy();
|
package/dist/randomized.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"randomized.js","sourceRoot":"","sources":["../src/randomized.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,GAId,MAAM,cAAc,CAAC;AAEtB,OAAO,EAAE,YAAY,EAAE,UAAU,EAAe,MAAM,aAAa,CAAC;
|
|
1
|
+
{"version":3,"file":"randomized.js","sourceRoot":"","sources":["../src/randomized.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,aAAa,GAId,MAAM,cAAc,CAAC;AAEtB,OAAO,EAAE,YAAY,EAAE,UAAU,EAAe,MAAM,aAAa,CAAC;AASpE,kEAAkE;AAClE,MAAM,WAAW,GAAG,EAAE,CAAC;AAsBvB;;;;;;;;;;;;;;;GAeG;AACH,MAAM,CAAC,KAAK,UAAU,aAAa,CACjC,OAA6B;IAE7B,MAAM,EACJ,UAAU,EACV,IAAI,GAAG,UAAU,EAAE,EACnB,KAAK,EAAE,SAAS,GAAG,CAAC,EACpB,KAAK,GAAG,GAAG,EACX,OAAO,GAAG,CAAC,UAAU,CAAC,EACtB,OAAO,EAAE,cAAc,EACvB,cAAc,GAAG,GAAG,EACpB,YAAY,GAAG,EAAE,GAClB,GAAG,OAAO,CAAC;IACZ,+EAA+E;IAC/E,MAAM,MAAM,GAAG,YAAY,CAAC,IAAI,GAAG,UAAU,CAAC,CAAC;IAC/C,MAAM,OAAO,GAAG,aAAa,CAAC,EAAE,GAAG,cAAc,EAAE,IAAI,EAAE,CAAC,CAAC;IAC3D,IAAI,IAAI,GAAG,CAAC,CAAC;IACb,yEAAyE;IACzE,IAAI,aAAa,GAAG,KAAK,CAAC;IAE1B,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,SAAS,EAAE,EAAE,CAAC,CAAC,EAAE,KAAK,EAAE,EAAE,CAC3D,OAAO,CAAC,OAAO,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC,CACnC,CAAC;QAEF,OAAO,IAAI,GAAG,KAAK,EAAE,IAAI,EAAE,EAAE,CAAC;YAC5B,IAAI,KAAK,CAAC,MAAM,GAAG,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,cAAc,CAAC,EAAE,CAAC;gBACtD,MAAM,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;gBAC7B,MAAM,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,CAAC;gBAC1D,IAAI,OAAO,CAAC,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC;oBAAE,OAAO,CAAC,UAAU,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;;oBAC9C,OAAO,CAAC,OAAO,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;YAC7B,CAAC;iBAAM,CAAC;gBACN,MAAM,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,MAAM,CAAC,CAAC;YACnD,CAAC;YACD,MAAM,OAAO,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,YAAY,GAAG,CAAC,CAAC,CAAC,CAAC;QACtD,CAAC;QAED,yFAAyF;QACzF,yFAAyF;QACzF,IAAI,QAAQ,GAAG,IAAI,CAAC;QACpB,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,QAAQ,IAAI,KAAK,GAAG,WAAW,EAAE,KAAK,EAAE,EAAE,CAAC;YAC7D,MAAM,OAAO,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC,OAAO,CAAC;YACxC,OAAO,CAAC,IAAI,EAAE,CAAC;YACf,MAAM,OAAO,CAAC,MAAM,EAAE,CAAC;YACvB,QAAQ,GAAG,OAAO,CAAC,KAAK,EAAE,CAAC,OAAO,KAAK,OAAO,CAAC;QACjD,CAAC;QACD,aAAa,GAAG,QAAQ,CAAC;QACzB,OAAO,CAAC,eAAe,EAAE,CAAC;IAC5B,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,MAAM,IAAI,GACR,IAAI,GAAG,KAAK,CAAC,CAAC,CAAC,WAAW,IAAI,GAAG,CAAC,OAAO,KAAK,EAAE,CAAC,CAAC,CAAC,SAAS,KAAK,QAAQ,CAAC;QAC5E,MAAM,OAAO,GAAG,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC;QACvE,MAAM,QAAQ,GAAG,aAAa;YAC5B,CAAC,CAAC,4CAA4C,WAAW,iFAAiF;YAC1I,CAAC,CAAC,EAAE,CAAC;QACP,MAAM,MAAM,CAAC,MAAM,CACjB,IAAI,KAAK,CACP,yBAAyB,IAAI,kBAAkB,IAAI,qBAAqB,OAAO,GAAG,QAAQ,EAAE,CAC7F,EACD,EAAE,KAAK,EAAE,KAAK,EAAE,CACjB,CAAC;IACJ,CAAC;YAAS,CAAC;QACT,OAAO,CAAC,OAAO,EAAE,CAAC;IACpB,CAAC;AACH,CAAC;AAID,MAAM,QAAQ,GAAG,CAAC,KAAc,EAAkB,EAAE,CAClD,KAAK,KAAK,IAAI,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,CAAC;AAEvE,0DAA0D;AAC1D,MAAM,IAAI,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;AAClC,qFAAqF;AACrF,MAAM,UAAU,GAAG,CAAC,GAAG,EAAE,GAAG,EAAE,GAAG,EAAE,IAAI,EAAE,GAAG,CAAC,CAAC;AAE9C,MAAM,UAAU,GAAG,CAAC,MAAc,EAAU,EAAE,CAC5C,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;AAEhF,MAAM,WAAW,GAAG,CAAC,MAAc,EAAE,KAAK,GAAG,CAAC,EAAW,EAAE;IACzD,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,EAAE,CAAC;IAC3B,IAAI,KAAK,GAAG,CAAC,IAAI,IAAI,GAAG,IAAI;QAC1B,OAAO,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,GAAG,EAAE,CAChD,WAAW,CAAC,MAAM,EAAE,KAAK,GAAG,CAAC,CAAC,CAC/B,CAAC;IACJ,IAAI,KAAK,GAAG,CAAC,IAAI,IAAI,GAAG,GAAG;QACzB,OAAO,MAAM,CAAC,WAAW,CACvB,KAAK,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,GAAG,EAAE,CAAC;YAC1C,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC;YACjB,WAAW,CAAC,MAAM,EAAE,KAAK,GAAG,CAAC,CAAC;SAC/B,CAAC,CACH,CAAC;IACJ,IAAI,IAAI,GAAG,IAAI;QAAE,OAAO,MAAM,CAAC,GAAG,CAAC,GAAG,CAAC,CAAC;IACxC,IAAI,IAAI,GAAG,GAAG;QAAE,OAAO,UAAU,CAAC,MAAM,CAAC,CAAC;IAC1C,IAAI,IAAI,GAAG,IAAI;QAAE,OAAO,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;IAC3C,OAAO,IAAI,CAAC;AACd,CAAC,CAAC;AAEF,yFAAyF;AACzF,MAAM,QAAQ,GAAG,CACf,IAAa,EACb,IAAU,EACV,IAAiC,EACxB,EAAE;IACX,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,IAAI,CAAC,IAAI,CAAC,CAAC;IACzC,MAAM,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC;IAC5B,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC;QACxB,MAAM,IAAI,GAAG,IAAI,CAAC,KAAK,EAAE,CAAC;QAC1B,IAAI,CAAC,GAAa,CAAC,GAAG,QAAQ,CAAC,IAAI,CAAC,GAAa,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,CAAC;QAChE,OAAO,IAAI,CAAC;IACd,CAAC;IACD,OAAO;QACL,GAAI,IAAc;QAClB,CAAC,GAAG,CAAC,EAAE,QAAQ,CAAE,IAAc,CAAC,GAAG,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC;KAClD,CAAC;AACJ,CAAC,CAAC;AAEF,oGAAoG;AACpG,MAAM,SAAS,GAAG,CAAC,KAAc,EAAE,MAAc,EAAW,EAAE;IAC5D,IAAI,KAAK,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QAC/C,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,EAAE,CAAC;QAC3B,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC;YACvC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC,CAAC;;YACrC,IAAI,CAAC,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC,EAAE,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,CAAC;QACzE,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,QAAQ,CAAC,KAAK,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QAC1C,MAAM,IAAI,GAAG,EAAE,GAAG,KAAK,EAAE,CAAC;QAC1B,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QAC/B,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC;YAAE,OAAO,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;;YACrE,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,GAAG,WAAW,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;QACtD,OAAO,IAAI,CAAC;IACd,CAAC;IACD,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QACpD,MAAM,UAAU,GAAG,CAAC,GAAG,KAAK,CAAC,CAAC;QAC9B,MAAM,EAAE,GAAG,MAAM,CAAC,GAAG,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC;QAC7C,IAAI,UAAU,CAAC,MAAM,GAAG,CAAC,IAAI,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC;YAAE,UAAU,CAAC,MAAM,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC;;YACrE,UAAU,CAAC,MAAM,CAAC,EAAE,EAAE,CAAC,EAAE,MAAM,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC;QACvD,OAAO,UAAU,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;IAC7B,CAAC;IACD,OAAO,WAAW,CAAC,MAAM,CAAC,CAAC;AAC7B,CAAC,CAAC;AAEF;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,UAAU,GAAiB,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE;IACvD,MAAM,MAAM,GAAG,IAAI,CAAC,WAAW,EAAE,CAAC;IAClC,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC;QAAE,OAAO;IAC9B,MAAM,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;IACjC,IAAI,IAAI,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO;IAE9B,MAAM,IAAI,GAAS,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;IACvC,IAAI,IAAI,GAAY,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC;IACpC,OAAO,MAAM,CAAC,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;QAC1B,IAAI,KAAK,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,IAAI,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC3C,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;YACtC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;YACjB,IAAI,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC;QACrB,CAAC;aAAM,IAAI,QAAQ,CAAC,IAAI,CAAC,IAAI,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAC1D,MAAM,GAAG,GAAG,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,CAAC;YAC3C,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACf,IAAI,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC;QACnB,CAAC;;YAAM,MAAM;IACf,CAAC;IAED,IAAI,CAAC,OAAO,CAAC,QAAQ,CACnB,QAAQ,CAAC,IAAI,CAAC,OAAO,CAAC,QAAQ,EAAE,EAAE,IAAI,EAAE,CAAC,KAAK,EAAE,EAAE,CAChD,SAAS,CAAC,KAAK,EAAE,MAAM,CAAC,CACf,CACZ,CAAC;AACJ,CAAC,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@homeostate/tool-simulator",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Test homeostate peers on a simulated network with latency, drops, reordering and partitions, and check that they converge",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -43,17 +43,17 @@
|
|
|
43
43
|
"access": "public"
|
|
44
44
|
},
|
|
45
45
|
"dependencies": {
|
|
46
|
-
"@homeostate/core": "^0.
|
|
46
|
+
"@homeostate/core": "^0.5.0"
|
|
47
47
|
},
|
|
48
48
|
"devDependencies": {
|
|
49
49
|
"@automerge/automerge": "^3.4.1",
|
|
50
|
-
"loro-crdt": "^1.
|
|
50
|
+
"loro-crdt": "^1.17.1",
|
|
51
51
|
"yjs": "^13.6.23",
|
|
52
52
|
"zustand": "^4.5.2",
|
|
53
|
-
"@homeostate/crdt-
|
|
54
|
-
"@homeostate/store-zustand": "^0.3.
|
|
55
|
-
"@homeostate/crdt-
|
|
56
|
-
"@homeostate/crdt-
|
|
53
|
+
"@homeostate/crdt-automerge": "^0.3.1",
|
|
54
|
+
"@homeostate/store-zustand": "^0.3.1",
|
|
55
|
+
"@homeostate/crdt-yjs": "^0.4.0",
|
|
56
|
+
"@homeostate/crdt-loro": "^0.4.0"
|
|
57
57
|
},
|
|
58
58
|
"scripts": {
|
|
59
59
|
"build": "tsc -b"
|
package/src/index.ts
CHANGED
package/src/lost-keys.ts
ADDED
|
@@ -0,0 +1,164 @@
|
|
|
1
|
+
import { formatPath } from "./compare.js";
|
|
2
|
+
import type { Peer } from "./network.js";
|
|
3
|
+
|
|
4
|
+
/** Thrown by `advance` and `settle` when a peer lost a synced key; see `checkLostKeys`. */
|
|
5
|
+
export class LostKeyError extends Error {
|
|
6
|
+
readonly seed: number;
|
|
7
|
+
/** The peer's name. */
|
|
8
|
+
readonly peer: string;
|
|
9
|
+
readonly key: string;
|
|
10
|
+
/** Virtual time of the checkpoint that found the loss, in milliseconds. */
|
|
11
|
+
readonly at: number;
|
|
12
|
+
|
|
13
|
+
constructor(
|
|
14
|
+
message: string,
|
|
15
|
+
details: { seed: number; peer: string; key: string; at: number },
|
|
16
|
+
) {
|
|
17
|
+
super(message);
|
|
18
|
+
this.name = "LostKeyError";
|
|
19
|
+
this.seed = details.seed;
|
|
20
|
+
this.peer = details.peer;
|
|
21
|
+
this.key = details.key;
|
|
22
|
+
this.at = details.at;
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
/** What the check knows of one peer. */
|
|
27
|
+
interface Tracked {
|
|
28
|
+
/** The store's synced keys at the peer's last checkpoint. */
|
|
29
|
+
store: Set<string>;
|
|
30
|
+
/** The document's keys, while they cannot have changed since they were read. */
|
|
31
|
+
doc: Set<string> | undefined;
|
|
32
|
+
/** The document's keys when it was last read, for telling which ones a change removed. */
|
|
33
|
+
lastDoc: Set<string> | undefined;
|
|
34
|
+
/** Whether the document changed outside a delivery since it was last read. */
|
|
35
|
+
changedOutside: boolean;
|
|
36
|
+
/** Keys a checkpoint saw both in the store and in the document. */
|
|
37
|
+
held: Set<string>;
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
/**
|
|
41
|
+
* Checks that no peer's store and document both lose a synced top-level key that no peer
|
|
42
|
+
* deleted. The network calls it as its documents change and around its deliveries.
|
|
43
|
+
*/
|
|
44
|
+
export interface LostKeyCheck {
|
|
45
|
+
/** Start tracking a peer, before it links and connects. */
|
|
46
|
+
addPeer: (peer: Peer) => void;
|
|
47
|
+
/** The peer's document changed; `outside` when no delivery is running. */
|
|
48
|
+
changed: (peer: Peer, outside: boolean) => void;
|
|
49
|
+
/**
|
|
50
|
+
* Before deliveries: what changed since the last checkpoint was done by the test, so a key
|
|
51
|
+
* that left a store or document is deleted.
|
|
52
|
+
*/
|
|
53
|
+
beforeDeliveries: (peers: readonly Peer[]) => void;
|
|
54
|
+
/** After a delivery to `peer` from `from`, or after the last one; throws a `LostKeyError`. */
|
|
55
|
+
afterDelivery: (peer: Peer, at: number, from?: Peer) => void;
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
/** The keys of `state` that JSON keeps, as `peer.syncedState()` has them. */
|
|
59
|
+
const jsonKeys = (state: object): Set<string> => {
|
|
60
|
+
const keys = new Set<string>();
|
|
61
|
+
for (const [key, value] of Object.entries(state))
|
|
62
|
+
if (
|
|
63
|
+
value !== undefined &&
|
|
64
|
+
typeof value !== "function" &&
|
|
65
|
+
typeof value !== "symbol"
|
|
66
|
+
)
|
|
67
|
+
keys.add(key);
|
|
68
|
+
return keys;
|
|
69
|
+
};
|
|
70
|
+
|
|
71
|
+
const formatTime = (ms: number): string => String(Number(ms.toFixed(2)));
|
|
72
|
+
|
|
73
|
+
export function createLostKeyCheck(seed: number): LostKeyCheck {
|
|
74
|
+
const tracked = new Map<Peer, Tracked>();
|
|
75
|
+
/** Keys a checkpoint outside a delivery saw leave a store or document; never cleared. */
|
|
76
|
+
const deleted = new Set<string>();
|
|
77
|
+
|
|
78
|
+
const storeKeys = (peer: Peer): Set<string> =>
|
|
79
|
+
jsonKeys(peer.engine.getSyncedState());
|
|
80
|
+
|
|
81
|
+
const docKeys = (peer: Peer, state: Tracked): Set<string> => {
|
|
82
|
+
if (!state.doc) {
|
|
83
|
+
state.doc = jsonKeys(peer.backend.read());
|
|
84
|
+
// Only the test's own changes delete; a delivery that removes a key may be a loss.
|
|
85
|
+
if (state.changedOutside && state.lastDoc)
|
|
86
|
+
for (const key of state.lastDoc)
|
|
87
|
+
if (!state.doc.has(key)) deleted.add(key);
|
|
88
|
+
state.lastDoc = state.doc;
|
|
89
|
+
state.changedOutside = false;
|
|
90
|
+
}
|
|
91
|
+
return state.doc;
|
|
92
|
+
};
|
|
93
|
+
|
|
94
|
+
/** Mark the keys that left the store since its last checkpoint as deleted. */
|
|
95
|
+
const noteRemovals = (state: Tracked, store: Set<string>): void => {
|
|
96
|
+
for (const key of state.store) if (!store.has(key)) deleted.add(key);
|
|
97
|
+
state.store = store;
|
|
98
|
+
};
|
|
99
|
+
|
|
100
|
+
const hold = (peer: Peer, state: Tracked, keys: Iterable<string>): void => {
|
|
101
|
+
for (const key of keys) {
|
|
102
|
+
if (state.held.has(key) || deleted.has(key)) continue;
|
|
103
|
+
if (docKeys(peer, state).has(key)) state.held.add(key);
|
|
104
|
+
}
|
|
105
|
+
};
|
|
106
|
+
|
|
107
|
+
const track = (peer: Peer): Tracked => {
|
|
108
|
+
const state = tracked.get(peer);
|
|
109
|
+
if (!state) throw new Error(`${peer.name} is not on the network.`);
|
|
110
|
+
return state;
|
|
111
|
+
};
|
|
112
|
+
|
|
113
|
+
return {
|
|
114
|
+
addPeer: (peer) => {
|
|
115
|
+
tracked.set(peer, {
|
|
116
|
+
store: storeKeys(peer),
|
|
117
|
+
doc: undefined,
|
|
118
|
+
lastDoc: undefined,
|
|
119
|
+
changedOutside: false,
|
|
120
|
+
held: new Set(),
|
|
121
|
+
});
|
|
122
|
+
},
|
|
123
|
+
|
|
124
|
+
changed: (peer, outside) => {
|
|
125
|
+
const state = track(peer);
|
|
126
|
+
state.doc = undefined;
|
|
127
|
+
if (!outside) return;
|
|
128
|
+
state.changedOutside = true;
|
|
129
|
+
// A key added and removed again before the next delivery still counts as deleted.
|
|
130
|
+
noteRemovals(state, storeKeys(peer));
|
|
131
|
+
},
|
|
132
|
+
|
|
133
|
+
beforeDeliveries: (peers) => {
|
|
134
|
+
for (const peer of peers) {
|
|
135
|
+
const state = track(peer);
|
|
136
|
+
const store = storeKeys(peer);
|
|
137
|
+
noteRemovals(state, store);
|
|
138
|
+
if (state.changedOutside) docKeys(peer, state);
|
|
139
|
+
hold(peer, state, store);
|
|
140
|
+
}
|
|
141
|
+
},
|
|
142
|
+
|
|
143
|
+
afterDelivery: (peer, at, from) => {
|
|
144
|
+
const state = track(peer);
|
|
145
|
+
const store = storeKeys(peer);
|
|
146
|
+
for (const key of state.held) {
|
|
147
|
+
if (store.has(key)) continue;
|
|
148
|
+
if (deleted.has(key)) {
|
|
149
|
+
state.held.delete(key);
|
|
150
|
+
continue;
|
|
151
|
+
}
|
|
152
|
+
if (docKeys(peer, state).has(key)) continue;
|
|
153
|
+
const source = from ? `, on an update from ${from.name}` : "";
|
|
154
|
+
throw new LostKeyError(
|
|
155
|
+
`${peer.name} lost the synced key ${formatPath([key])} at ${formatTime(at)} ms (seed ${seed})${source}: its store and document both lack it, and no peer deleted it.`,
|
|
156
|
+
{ seed, peer: peer.name, key, at },
|
|
157
|
+
);
|
|
158
|
+
}
|
|
159
|
+
const appeared = [...store].filter((key) => !state.store.has(key));
|
|
160
|
+
state.store = store;
|
|
161
|
+
hold(peer, state, appeared);
|
|
162
|
+
},
|
|
163
|
+
};
|
|
164
|
+
}
|
package/src/network.ts
CHANGED
|
@@ -8,6 +8,7 @@ import {
|
|
|
8
8
|
type Unsubscribe,
|
|
9
9
|
} from "@homeostate/core";
|
|
10
10
|
import { asJson, firstDifference, formatPath, formatValue } from "./compare.js";
|
|
11
|
+
import { createLostKeyCheck } from "./lost-keys.js";
|
|
11
12
|
import { createRandom, randomSeed } from "./random.js";
|
|
12
13
|
|
|
13
14
|
/**
|
|
@@ -70,6 +71,17 @@ export interface NetworkOptions {
|
|
|
70
71
|
* as two peers writing in a loop. Defaults to 100 000.
|
|
71
72
|
*/
|
|
72
73
|
maxDeliveries?: number;
|
|
74
|
+
/**
|
|
75
|
+
* Whether `advance` and `settle` reject with a `LostKeyError` when a peer's store and
|
|
76
|
+
* document both lose a synced top-level key that no peer deleted. Defaults to `true`.
|
|
77
|
+
*
|
|
78
|
+
* A peer holds a key once a checkpoint sees it in both its store's synced state and its
|
|
79
|
+
* document. A key is deleted for good once it leaves a store or document outside a delivery:
|
|
80
|
+
* in a step, or in a test's own change, between `advance` and `settle` calls. Checkpoints
|
|
81
|
+
* come after each delivery and when `advance` or `settle` returns. A key deleted during a
|
|
82
|
+
* delivery, such as by a store subscriber or a timer, is reported as lost: pass `false`.
|
|
83
|
+
*/
|
|
84
|
+
checkLostKeys?: boolean;
|
|
73
85
|
}
|
|
74
86
|
|
|
75
87
|
export interface NetworkStats {
|
|
@@ -89,6 +101,9 @@ export interface NetworkStats {
|
|
|
89
101
|
* document goes to every peer it is linked to, after the latency, unless it is dropped or its
|
|
90
102
|
* link is cut first. Linking two peers, as `connect` and `heal` do, makes them exchange their
|
|
91
103
|
* whole documents, as providers do on reconnecting, so what was lost on the way arrives.
|
|
104
|
+
*
|
|
105
|
+
* Unless `checkLostKeys` is `false`, `advance` and `settle` also check after each delivery
|
|
106
|
+
* that no peer's store and document both lost a synced key that no peer deleted.
|
|
92
107
|
*/
|
|
93
108
|
export interface Network {
|
|
94
109
|
/** The seed of this network's random choices. */
|
|
@@ -118,11 +133,21 @@ export interface Network {
|
|
|
118
133
|
/**
|
|
119
134
|
* Link every pair of peers and have each exchange documents, as if every peer reconnected,
|
|
120
135
|
* so that updates lost to drops and cut links arrive too.
|
|
136
|
+
*
|
|
137
|
+
* With a `dropRate`, the updates stores write in answer to that exchange, such as a key one
|
|
138
|
+
* derives from synced state, can be dropped too. Repeat `heal()` and `settle()` until
|
|
139
|
+
* `stats().dropped` stops growing before checking convergence.
|
|
121
140
|
*/
|
|
122
141
|
heal: () => void;
|
|
123
|
-
/**
|
|
142
|
+
/**
|
|
143
|
+
* Deliver the messages due within the next `ms` of virtual time, and move time on. Rejects
|
|
144
|
+
* with a `LostKeyError` when a peer lost a synced key; see `checkLostKeys`.
|
|
145
|
+
*/
|
|
124
146
|
advance: (ms: number) => Promise<void>;
|
|
125
|
-
/**
|
|
147
|
+
/**
|
|
148
|
+
* Deliver every message on its way, and those they lead to, until none are left. Rejects
|
|
149
|
+
* with a `LostKeyError` when a peer lost a synced key; see `checkLostKeys`.
|
|
150
|
+
*/
|
|
126
151
|
settle: () => Promise<void>;
|
|
127
152
|
stats: () => NetworkStats;
|
|
128
153
|
/**
|
|
@@ -131,7 +156,10 @@ export interface Network {
|
|
|
131
156
|
* document. Empty once the peers have converged.
|
|
132
157
|
*/
|
|
133
158
|
differences: () => string[];
|
|
134
|
-
/**
|
|
159
|
+
/**
|
|
160
|
+
* Throws a `ConvergenceError` unless `differences()` is empty. Its message also says how many
|
|
161
|
+
* messages are still on their way, and how many updates were dropped since the last `heal()`.
|
|
162
|
+
*/
|
|
135
163
|
expectConverged: () => void;
|
|
136
164
|
/** Disconnect every engine, stop listening to every document and drop what is on its way. */
|
|
137
165
|
destroy: () => void;
|
|
@@ -199,8 +227,12 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
199
227
|
dropRate = 0,
|
|
200
228
|
reorder = false,
|
|
201
229
|
maxDeliveries = 100_000,
|
|
230
|
+
checkLostKeys = true,
|
|
202
231
|
} = options;
|
|
203
232
|
const random = createRandom(seed);
|
|
233
|
+
const lostKeys = checkLostKeys ? createLostKeyCheck(seed) : undefined;
|
|
234
|
+
/** Whether `advance` or `settle` is delivering, so a document's change is not the test's. */
|
|
235
|
+
let delivering = false;
|
|
204
236
|
const peers: Peer[] = [];
|
|
205
237
|
const unsubscribers: Unsubscribe[] = [];
|
|
206
238
|
/** Pairs of peers whose link is cut; every other pair is linked. */
|
|
@@ -211,6 +243,8 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
211
243
|
let now = 0;
|
|
212
244
|
let order = 0;
|
|
213
245
|
const stats = { sent: 0, delivered: 0, dropped: 0, cut: 0 };
|
|
246
|
+
/** `stats.dropped` at the last `heal()`, or `undefined` before the first. */
|
|
247
|
+
let droppedAtHeal: number | undefined;
|
|
214
248
|
|
|
215
249
|
const delay = (): number => {
|
|
216
250
|
if (typeof latency === "number") return latency;
|
|
@@ -275,11 +309,12 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
275
309
|
const deliver = (message: Message): void => {
|
|
276
310
|
now = Math.max(now, message.at);
|
|
277
311
|
stats.delivered++;
|
|
312
|
+
lostKeys?.changed(message.to, false);
|
|
278
313
|
message.to.doc.apply(message.update);
|
|
279
314
|
};
|
|
280
315
|
|
|
281
316
|
/** Deliver messages due by `until`, letting each delivery's work finish before the next. */
|
|
282
|
-
const
|
|
317
|
+
const deliverUntil = async (until: number): Promise<void> => {
|
|
283
318
|
let deliveries = 0;
|
|
284
319
|
for (;;) {
|
|
285
320
|
const next = queue[0];
|
|
@@ -291,6 +326,7 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
291
326
|
queue.shift();
|
|
292
327
|
deliver(next);
|
|
293
328
|
await Promise.resolve();
|
|
329
|
+
lostKeys?.afterDelivery(next.to, now, next.from);
|
|
294
330
|
continue;
|
|
295
331
|
}
|
|
296
332
|
// Asynchronous stores may still send; only stop once a full turn sends nothing.
|
|
@@ -300,6 +336,18 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
300
336
|
}
|
|
301
337
|
};
|
|
302
338
|
|
|
339
|
+
/** `deliverUntil`, with the lost key check's checkpoints before and after. */
|
|
340
|
+
const run = async (until: number): Promise<void> => {
|
|
341
|
+
lostKeys?.beforeDeliveries(peers);
|
|
342
|
+
delivering = true;
|
|
343
|
+
try {
|
|
344
|
+
await deliverUntil(until);
|
|
345
|
+
} finally {
|
|
346
|
+
delivering = false;
|
|
347
|
+
}
|
|
348
|
+
for (const peer of peers) lostKeys?.afterDelivery(peer, now);
|
|
349
|
+
};
|
|
350
|
+
|
|
303
351
|
const view = (peer: Peer) => ({
|
|
304
352
|
store: peer.syncedState(),
|
|
305
353
|
document: asJson(peer.backend.read()),
|
|
@@ -317,7 +365,7 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
317
365
|
|
|
318
366
|
const views = peers.map(view);
|
|
319
367
|
peers.forEach((peer, i) => {
|
|
320
|
-
if (peer.engine.
|
|
368
|
+
if (peer.engine.getStatus().connected)
|
|
321
369
|
describe(
|
|
322
370
|
`${peer.name}'s store differs from its document`,
|
|
323
371
|
views[i].store,
|
|
@@ -347,11 +395,11 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
347
395
|
|
|
348
396
|
addPeer: <S extends object>(setup: PeerSetup<S>): Peer<S> => {
|
|
349
397
|
const index = peers.length;
|
|
350
|
-
const engine = createSyncEngine(
|
|
351
|
-
setup.
|
|
352
|
-
setup.
|
|
353
|
-
setup.
|
|
354
|
-
);
|
|
398
|
+
const engine = createSyncEngine({
|
|
399
|
+
...setup.config,
|
|
400
|
+
backend: setup.backend,
|
|
401
|
+
adapter: setup.adapter,
|
|
402
|
+
});
|
|
355
403
|
const peer: Peer<S> = {
|
|
356
404
|
index,
|
|
357
405
|
name: setup.name ?? `peer-${index}`,
|
|
@@ -362,8 +410,12 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
362
410
|
syncedState: () => asJson(engine.getSyncedState()),
|
|
363
411
|
};
|
|
364
412
|
peers.push(peer);
|
|
413
|
+
lostKeys?.addPeer(peer);
|
|
365
414
|
unsubscribers.push(
|
|
366
|
-
setup.doc.subscribe((update) =>
|
|
415
|
+
setup.doc.subscribe((update) => {
|
|
416
|
+
lostKeys?.changed(peer, !delivering);
|
|
417
|
+
broadcast(peer, update);
|
|
418
|
+
}),
|
|
367
419
|
);
|
|
368
420
|
for (const other of peers) link(peer, other);
|
|
369
421
|
if (setup.connect !== false) engine.connect();
|
|
@@ -393,6 +445,7 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
393
445
|
},
|
|
394
446
|
|
|
395
447
|
heal: () => {
|
|
448
|
+
droppedAtHeal = stats.dropped;
|
|
396
449
|
for (const a of peers)
|
|
397
450
|
for (const b of peers) if (a.index < b.index) link(a, b);
|
|
398
451
|
},
|
|
@@ -413,13 +466,22 @@ export function createNetwork(options: NetworkOptions = {}): Network {
|
|
|
413
466
|
const lines = differences();
|
|
414
467
|
if (lines.length === 0) return;
|
|
415
468
|
const inFlight =
|
|
416
|
-
queue.length
|
|
417
|
-
?
|
|
469
|
+
queue.length === 1
|
|
470
|
+
? "\n1 message is still on its way: settle() delivers it."
|
|
471
|
+
: queue.length > 1
|
|
472
|
+
? `\n${queue.length} messages are still on their way: settle() delivers them.`
|
|
473
|
+
: "";
|
|
474
|
+
const dropped = stats.dropped - (droppedAtHeal ?? 0);
|
|
475
|
+
const droppedHint =
|
|
476
|
+
dropped > 0
|
|
477
|
+
? `\n${dropped === 1 ? "1 update was" : `${dropped} updates were`} dropped${
|
|
478
|
+
droppedAtHeal === undefined ? "" : " since the last heal()"
|
|
479
|
+
}: heal() and settle() until stats().dropped stops growing.`
|
|
418
480
|
: "";
|
|
419
481
|
throw new ConvergenceError(
|
|
420
482
|
`${peers.length} peers did not converge (seed ${seed}):\n${lines
|
|
421
483
|
.map((line) => `- ${line}`)
|
|
422
|
-
.join("\n")}${inFlight}`,
|
|
484
|
+
.join("\n")}${inFlight}${droppedHint}`,
|
|
423
485
|
lines,
|
|
424
486
|
seed,
|
|
425
487
|
);
|
package/src/randomized.ts
CHANGED
|
@@ -14,6 +14,9 @@ export type RandomAction<S extends object = any> = (
|
|
|
14
14
|
random: Random,
|
|
15
15
|
) => void;
|
|
16
16
|
|
|
17
|
+
/** Most rounds of `heal()` and `settle()` at the end of a run. */
|
|
18
|
+
const HEAL_ROUNDS = 10;
|
|
19
|
+
|
|
17
20
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
|
18
21
|
export interface RandomizedOptions<S extends object = any> {
|
|
19
22
|
/** Makes peer `index`; called once per peer as the run starts. */
|
|
@@ -26,7 +29,7 @@ export interface RandomizedOptions<S extends object = any> {
|
|
|
26
29
|
steps?: number;
|
|
27
30
|
/** What a step may do to a random peer's store, one picked at random. Defaults to `[randomEdit]`. */
|
|
28
31
|
actions?: readonly RandomAction<S>[];
|
|
29
|
-
/** Latency, drops and
|
|
32
|
+
/** Latency, drops, reordering and the lost key check, as for `createNetwork`. */
|
|
30
33
|
network?: Omit<NetworkOptions, "seed">;
|
|
31
34
|
/** Chance that a step cuts or restores a random link instead of changing a store. Defaults to 0.1. */
|
|
32
35
|
linkChangeRate?: number;
|
|
@@ -38,10 +41,16 @@ export interface RandomizedOptions<S extends object = any> {
|
|
|
38
41
|
* Runs random steps against a fresh network, then heals it, settles it and checks that every
|
|
39
42
|
* peer converged. Each step changes a random peer's store with a random action, or cuts or
|
|
40
43
|
* restores a random link, and lets some virtual time pass, so changes cross each other on
|
|
41
|
-
* their way.
|
|
44
|
+
* their way. Throughout, the network checks that no peer's store and document both lose a
|
|
45
|
+
* synced key that no peer deleted, unless `network.checkLostKeys` is `false`.
|
|
46
|
+
*
|
|
47
|
+
* With a `dropRate`, the updates stores write in answer to the healing exchange can be dropped
|
|
48
|
+
* too, so the run heals and settles again until a round drops nothing, at most 10 rounds, as a
|
|
49
|
+
* provider that resyncs would.
|
|
42
50
|
*
|
|
43
|
-
* Rejects with an error that names the seed and step when a step throws
|
|
44
|
-
* converge; pass that `seed` to replay the same run.
|
|
51
|
+
* Rejects with an error that names the seed and step when a step throws, a peer loses a synced
|
|
52
|
+
* key or the peers do not converge; pass that `seed` to replay the same run. When the last of
|
|
53
|
+
* those rounds still dropped updates, the error says so. Give each document a fixed ID for runs
|
|
45
54
|
* that replay exactly, such as `doc.clientID = index + 1` for Yjs.
|
|
46
55
|
*/
|
|
47
56
|
export async function runRandomized<S extends object>(
|
|
@@ -61,6 +70,8 @@ export async function runRandomized<S extends object>(
|
|
|
61
70
|
const random = createRandom(seed ^ 0x5bd1e995);
|
|
62
71
|
const network = createNetwork({ ...networkOptions, seed });
|
|
63
72
|
let step = 0;
|
|
73
|
+
/** Whether every round of healing dropped updates, the last included. */
|
|
74
|
+
let stillDropping = false;
|
|
64
75
|
|
|
65
76
|
try {
|
|
66
77
|
const peers = Array.from({ length: peerCount }, (_, index) =>
|
|
@@ -79,16 +90,27 @@ export async function runRandomized<S extends object>(
|
|
|
79
90
|
await network.advance(random.int(stepInterval + 1));
|
|
80
91
|
}
|
|
81
92
|
|
|
82
|
-
|
|
83
|
-
|
|
93
|
+
// Stores may write in answer to the documents a heal exchanges, and those updates can be
|
|
94
|
+
// dropped too: heal again until a round drops nothing, as a provider that resyncs would.
|
|
95
|
+
let dropping = true;
|
|
96
|
+
for (let round = 0; dropping && round < HEAL_ROUNDS; round++) {
|
|
97
|
+
const dropped = network.stats().dropped;
|
|
98
|
+
network.heal();
|
|
99
|
+
await network.settle();
|
|
100
|
+
dropping = network.stats().dropped !== dropped;
|
|
101
|
+
}
|
|
102
|
+
stillDropping = dropping;
|
|
84
103
|
network.expectConverged();
|
|
85
104
|
} catch (error) {
|
|
86
105
|
const when =
|
|
87
106
|
step < steps ? `at step ${step + 1} of ${steps}` : `after ${steps} steps`;
|
|
88
107
|
const message = error instanceof Error ? error.message : String(error);
|
|
108
|
+
const dropping = stillDropping
|
|
109
|
+
? `\nUpdates were still being dropped after ${HEAL_ROUNDS} rounds of heal() and settle(): does a store write in answer to every exchange?`
|
|
110
|
+
: "";
|
|
89
111
|
throw Object.assign(
|
|
90
112
|
new Error(
|
|
91
|
-
`Randomized run failed ${when}; pass \`seed: ${seed}\` to replay it.\n${message}`,
|
|
113
|
+
`Randomized run failed ${when}; pass \`seed: ${seed}\` to replay it.\n${message}${dropping}`,
|
|
92
114
|
),
|
|
93
115
|
{ cause: error },
|
|
94
116
|
);
|