@sovereignbase/sequencer 0.0.1 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/INTERNALS.md ADDED
@@ -0,0 +1,1530 @@
1
+ # How Sequencer works
2
+
3
+ Sequencer gives applications an ordinary indexed sequence while internally maintaining a stable distributed history.
4
+
5
+ An application sees something simple:
6
+
7
+ ```text
8
+ A B C D
9
+ ```
10
+
11
+ and edits it using positions:
12
+
13
+ ```text
14
+ insert at 2
15
+ remove 1..3
16
+ replace 4..6
17
+ ```
18
+
19
+ Those positions are useful locally, but they cannot be replicated directly.
20
+
21
+ Two replicas may edit concurrently. By the time an operation reaches another replica, content may have been inserted or removed before the original position. Position `5` on one replica may therefore refer to something completely different on another.
22
+
23
+ Sequencer solves this by translating every local edit from a temporary visible position into a stable relationship with existing history.
24
+
25
+ The basic model is:
26
+
27
+ ```text
28
+ visible Projection
29
+ ↓
30
+ historical logical point
31
+ ↓
32
+ canonical Insertion
33
+ ↓
34
+ deterministic Structural Order
35
+ ↓
36
+ visible Projection
37
+ ```
38
+
39
+ Replicas do not need to agree on temporary positions, network arrival order, runtime fragmentation, or local indexes.
40
+
41
+ They only need to eventually receive the same canonical history.
42
+
43
+ ---
44
+
45
+ ## Projection is the visible sequence
46
+
47
+ The **Projection** is what the application sees.
48
+
49
+ For example:
50
+
51
+ ```text
52
+ A B C D
53
+ ```
54
+
55
+ contains four visible values, called **Frames**:
56
+
57
+ ```text
58
+ A B C D
59
+ 0 1 2 3
60
+ ```
61
+
62
+ Applications read and edit these visible positions.
63
+
64
+ Insertion positions are boundaries between Frames:
65
+
66
+ ```text
67
+ | A | B | C | D |
68
+ 0 1 2 3 4
69
+ ```
70
+
71
+ This distinction is important.
72
+
73
+ A Frame position identifies visible content.
74
+
75
+ A boundary identifies a place where new history can be attached.
76
+
77
+ Projection positions are deliberately temporary. They describe the current visible result, not the permanent identity of anything inside it.
78
+
79
+ ---
80
+
81
+ ## History must use stable coordinates
82
+
83
+ Suppose two replicas begin with:
84
+
85
+ ```text
86
+ A B
87
+ ```
88
+
89
+ Replica 1 inserts:
90
+
91
+ ```text
92
+ X
93
+ ```
94
+
95
+ between them:
96
+
97
+ ```text
98
+ A X B
99
+ ```
100
+
101
+ At the same time, Replica 2 inserts:
102
+
103
+ ```text
104
+ Y
105
+ ```
106
+
107
+ at the same visible position:
108
+
109
+ ```text
110
+ A Y B
111
+ ```
112
+
113
+ Both operations were authored at position `1`.
114
+
115
+ If Sequencer replicated only:
116
+
117
+ ```text
118
+ insert X at 1
119
+ insert Y at 1
120
+ ```
121
+
122
+ the final result could depend on which packet happened to arrive first.
123
+
124
+ Instead, both edits are translated into a relationship with the same historical boundary between `A` and `B`.
125
+
126
+ That boundary remains meaningful even after the visible sequence changes.
127
+
128
+ Sequencer can then deterministically order the two competing Insertions.
129
+
130
+ ---
131
+
132
+ # Canonical Insertions
133
+
134
+ The fundamental replicated unit in Sequencer is an **Insertion**.
135
+
136
+ Conceptually, every Insertion contains:
137
+
138
+ ```text
139
+ where it belongs
140
+ +
141
+ what history it introduces
142
+ ```
143
+
144
+ Its canonical coordinates are:
145
+
146
+ ```text
147
+ anchorSession
148
+ anchorStart
149
+ anchorDiff
150
+
151
+ insertionSession
152
+ insertionStart
153
+ insertionDiff
154
+ ```
155
+
156
+ These six values describe two independent things.
157
+
158
+ The anchor:
159
+
160
+ ```text
161
+ (anchorSession, anchorStart, anchorDiff)
162
+ ```
163
+
164
+ describes where the Insertion belongs.
165
+
166
+ The identity:
167
+
168
+ ```text
169
+ (insertionSession, insertionStart)
170
+ ```
171
+
172
+ describes which Insertion it is.
173
+
174
+ The effect:
175
+
176
+ ```text
177
+ insertionDiff
178
+ ```
179
+
180
+ describes how many original Frames it contributes.
181
+
182
+ A positive Insertion introduces Frames.
183
+
184
+ A negative Insertion represents removal.
185
+
186
+ ---
187
+
188
+ ## Insertion identity never depends on location
189
+
190
+ A canonical Insertion is identified by:
191
+
192
+ ```text
193
+ (insertionSession, insertionStart)
194
+ ```
195
+
196
+ Its identity does not include its anchor.
197
+
198
+ This means:
199
+
200
+ ```text
201
+ identity
202
+ ```
203
+
204
+ and:
205
+
206
+ ```text
207
+ position in history
208
+ ```
209
+
210
+ are separate concepts.
211
+
212
+ That separation is useful for several reasons.
213
+
214
+ The same operation can be recognized after retransmission.
215
+
216
+ Fragments of one Insertion still belong to the same canonical identity.
217
+
218
+ An Insertion can be found independently from whichever Strip currently represents its anchor.
219
+
220
+ And duplicate delivery does not create duplicate visible content.
221
+
222
+ ---
223
+
224
+ # Logical time
225
+
226
+ Every sequencing Session owns its own logical coordinate space.
227
+
228
+ A newly authored Insertion receives a start position inside that Session.
229
+
230
+ For example:
231
+
232
+ ```text
233
+ Session 12
234
+
235
+ Insertion A starts at 0
236
+ Insertion B starts at 6
237
+ Insertion C starts at 11
238
+ ```
239
+
240
+ These are not wall-clock timestamps.
241
+
242
+ They are logical coordinates used only to identify history.
243
+
244
+ A canonical Insertion normally reserves both:
245
+
246
+ ```text
247
+ its Frame extent
248
+ +
249
+ a following logical boundary
250
+ ```
251
+
252
+ before the next independent Insertion in the same Session.
253
+
254
+ That following boundary gives later history a stable point to reference.
255
+
256
+ ---
257
+
258
+ ## Frames and logical points are different
259
+
260
+ Consider an Insertion containing:
261
+
262
+ ```text
263
+ A B C
264
+ ```
265
+
266
+ It contains three Frames:
267
+
268
+ ```text
269
+ 0 1 2
270
+ ```
271
+
272
+ but four logical boundary points:
273
+
274
+ ```text
275
+ | A | B | C |
276
+ 0 1 2 3
277
+ ```
278
+
279
+ An anchor refers to one of these logical points.
280
+
281
+ For example:
282
+
283
+ ```text
284
+ anchorDiff = 2
285
+ ```
286
+
287
+ means:
288
+
289
+ ```text
290
+ A B | C
291
+ ```
292
+
293
+ inside that specific historical Insertion.
294
+
295
+ It does **not** mean:
296
+
297
+ ```text
298
+ Projection position 2
299
+ ```
300
+
301
+ That distinction is one of the most important ideas in Sequencer.
302
+
303
+ Visible positions move.
304
+
305
+ Historical logical points do not.
306
+
307
+ ---
308
+
309
+ # From a local edit to canonical history
310
+
311
+ Suppose the current Projection is:
312
+
313
+ ```text
314
+ Hello world
315
+ ```
316
+
317
+ and the application inserts:
318
+
319
+ ```text
320
+ big
321
+ ```
322
+
323
+ here:
324
+
325
+ ```text
326
+ Hello | world
327
+ ```
328
+
329
+ The application supplies only the visible boundary.
330
+
331
+ Sequencer resolves that boundary to the historical Insertion that owns it.
332
+
333
+ It then records the edit relative to that original Insertion:
334
+
335
+ ```text
336
+ specific historical Insertion
337
+ +
338
+ specific logical point inside it
339
+ ```
340
+
341
+ The operation therefore means:
342
+
343
+ ```text
344
+ insert here in history
345
+ ```
346
+
347
+ rather than:
348
+
349
+ ```text
350
+ insert at visible index 6
351
+ ```
352
+
353
+ Another replica may have more visible content before that point.
354
+
355
+ The numeric Projection position may differ.
356
+
357
+ The historical relationship does not.
358
+
359
+ ---
360
+
361
+ # Structural Order
362
+
363
+ Canonical history must eventually form one deterministic order.
364
+
365
+ Sequencer calls this **Structural Order**.
366
+
367
+ Structural Order contains everything required to explain the visible sequence, including history that may no longer itself be visible.
368
+
369
+ It may contain:
370
+
371
+ ```text
372
+ positive Insertions
373
+ removal history
374
+ fragments
375
+ concurrent siblings
376
+ causal descendants
377
+ hidden historical boundaries
378
+ ```
379
+
380
+ The Projection is derived from this structure.
381
+
382
+ Conceptually:
383
+
384
+ ```text
385
+ Structural Order
386
+ ↓
387
+ surviving positive Frames
388
+ ↓
389
+ Projection
390
+ ```
391
+
392
+ Structural Order is therefore richer than the visible sequence.
393
+
394
+ The Projection answers:
395
+
396
+ ```text
397
+ what is visible now?
398
+ ```
399
+
400
+ Structural Order answers:
401
+
402
+ ```text
403
+ what historical relationships produced that result?
404
+ ```
405
+
406
+ ---
407
+
408
+ # Concurrent Insertions
409
+
410
+ Two Insertions directly compete when they use exactly the same canonical anchor.
411
+
412
+ For example, both of these may attach to:
413
+
414
+ ```text
415
+ A | B
416
+ ```
417
+
418
+ independently:
419
+
420
+ ```text
421
+ X
422
+ Y
423
+ ```
424
+
425
+ The order between them cannot depend on which replica received which packet first.
426
+
427
+ Sequencer therefore defines a deterministic competitor ordering.
428
+
429
+ Closer to the shared anchor:
430
+
431
+ 1. positive Insertions come before reducing Insertions;
432
+ 2. within the same sign, larger Session identifiers come first;
433
+ 3. within the same Session, larger start coordinates come first.
434
+
435
+ Every replica can calculate this from the canonical history alone.
436
+
437
+ So one replica may receive:
438
+
439
+ ```text
440
+ X
441
+ Y
442
+ ```
443
+
444
+ while another receives:
445
+
446
+ ```text
447
+ Y
448
+ X
449
+ ```
450
+
451
+ but both eventually construct the same Structural Order.
452
+
453
+ ---
454
+
455
+ # Causal subtrees stay together
456
+
457
+ A concurrent competitor may later receive children of its own.
458
+
459
+ Those children form its **causal subtree**.
460
+
461
+ Consider two replicas that independently create root content:
462
+
463
+ ```text
464
+ Document A
465
+ ```
466
+
467
+ and:
468
+
469
+ ```text
470
+ Document B
471
+ ```
472
+
473
+ Before synchronizing, both continue editing their own document.
474
+
475
+ The two roots compete at the same virtual root anchor.
476
+
477
+ Sequencer deterministically decides which root comes first.
478
+
479
+ But the descendants of each root must remain with that root.
480
+
481
+ The final Structural Order may be:
482
+
483
+ ```text
484
+ [Document A and its descendants]
485
+
486
+ [Document B and its descendants]
487
+ ```
488
+
489
+ or the reverse.
490
+
491
+ It must not become:
492
+
493
+ ```text
494
+ part of A
495
+ part of B
496
+ rest of A
497
+ rest of B
498
+ ```
499
+
500
+ Competitor ordering therefore works on causal subtrees, not merely isolated nodes.
501
+
502
+ This is what lets independently developed branches of history remain coherent when they meet.
503
+
504
+ ---
505
+
506
+ # History can arrive in any causal order
507
+
508
+ Distributed delivery may be reordered.
509
+
510
+ Suppose one replica first creates:
511
+
512
+ ```text
513
+ Hello
514
+ ```
515
+
516
+ and later extends it into:
517
+
518
+ ```text
519
+ Hello world
520
+ ```
521
+
522
+ Another replica may receive the history for:
523
+
524
+ ```text
525
+ world
526
+ ```
527
+
528
+ before it receives the history containing:
529
+
530
+ ```text
531
+ Hello
532
+ ```
533
+
534
+ The child relationship is perfectly valid, but its parent does not yet exist locally.
535
+
536
+ Sequencer keeps that history unresolved until the required canonical parent becomes available.
537
+
538
+ When the missing parent later arrives, the relationship becomes resolvable.
539
+
540
+ No retransmission or special causal network ordering is required.
541
+
542
+ The same principle applies when the parent exists but has not yet grown far enough to contain the logical point referenced by the incoming history.
543
+
544
+ ---
545
+
546
+ # Author-time relationships survive receiver-time differences
547
+
548
+ Consider an author producing:
549
+
550
+ ```text
551
+ world
552
+ Hello world
553
+ Well, Hello world
554
+ Well, actually Hello world
555
+ Now, Well, actually Hello world
556
+ Okay, Now, Well, actually Hello world
557
+ ```
558
+
559
+ Another replica might receive the individual pieces in a very different order:
560
+
561
+ ```text
562
+ world
563
+ Hello
564
+ actually
565
+ Now
566
+ Well
567
+ Okay
568
+ ```
569
+
570
+ The operations do not depend on the receiver seeing the same intermediate Projection that the author saw.
571
+
572
+ Each operation describes where it belonged relative to existing canonical history at author time.
573
+
574
+ Once all necessary relationships are available, they reconstruct:
575
+
576
+ ```text
577
+ Okay, Now, Well, actually Hello world
578
+ ```
579
+
580
+ Sequencer therefore preserves:
581
+
582
+ ```text
583
+ author-time relationships
584
+ ```
585
+
586
+ rather than:
587
+
588
+ ```text
589
+ author-time numeric indexes
590
+ ```
591
+
592
+ ---
593
+
594
+ # Canonical history and runtime fragmentation
595
+
596
+ A canonical Insertion is a logical piece of history.
597
+
598
+ It does not need to remain one physical runtime object forever.
599
+
600
+ Suppose one Insertion originally contains:
601
+
602
+ ```text
603
+ A B C D
604
+ ```
605
+
606
+ Later edits may require the runtime to split it:
607
+
608
+ ```text
609
+ [A B] [C D]
610
+ ```
611
+
612
+ These pieces are called **Strips**.
613
+
614
+ The important point is that they are not new canonical Insertions.
615
+
616
+ Both still belong to the same original history:
617
+
618
+ ```text
619
+ same anchor
620
+ same insertion identity
621
+ same original coordinate space
622
+ ```
623
+
624
+ They merely represent different portions of it at runtime.
625
+
626
+ Conceptually:
627
+
628
+ ```text
629
+ canonical Insertion
630
+
631
+ | A | B | C | D |
632
+ 0 1 2 3 4
633
+
634
+ ↓
635
+
636
+ runtime materialization
637
+
638
+ [A B] [C D]
639
+ ```
640
+
641
+ The logical points remain:
642
+
643
+ ```text
644
+ 0 1 2 3 4
645
+ ```
646
+
647
+ even if the current physical representation is split around point `2`.
648
+
649
+ This allows old operations to continue referring to historical locations after arbitrary later fragmentation.
650
+
651
+ ---
652
+
653
+ # Canonical coordinates survive fragmentation
654
+
655
+ This separation between canonical identity and runtime representation is essential.
656
+
657
+ A replica may represent one Insertion as:
658
+
659
+ ```text
660
+ [A B C D]
661
+ ```
662
+
663
+ while another currently represents the same canonical Insertion as:
664
+
665
+ ```text
666
+ [A] [B C] [D]
667
+ ```
668
+
669
+ The two replicas still agree on:
670
+
671
+ ```text
672
+ which Insertion it is
673
+ ```
674
+
675
+ and:
676
+
677
+ ```text
678
+ which original logical point an anchor refers to
679
+ ```
680
+
681
+ They do not need identical runtime layouts.
682
+
683
+ The receiver resolves canonical coordinates through whatever fragmentation exists locally.
684
+
685
+ Runtime fragmentation is therefore invisible to replication.
686
+
687
+ ---
688
+
689
+ # Consecutive editing can extend history
690
+
691
+ Without any coalescing, typing:
692
+
693
+ ```text
694
+ H
695
+ He
696
+ Hel
697
+ Hell
698
+ Hello
699
+ ```
700
+
701
+ could create one canonical Insertion for every character.
702
+
703
+ That would preserve correctness, but history would grow unnecessarily quickly.
704
+
705
+ Sequencer can instead extend a recent positive Insertion when the new edit is a genuine continuation of that same history.
706
+
707
+ Conceptually:
708
+
709
+ ```text
710
+ H
711
+ ↓
712
+ He
713
+ ↓
714
+ Hel
715
+ ↓
716
+ Hell
717
+ ↓
718
+ Hello
719
+ ```
720
+
721
+ may remain one growing canonical Insertion.
722
+
723
+ The key requirement is that extending it must not change previously established structural relationships.
724
+
725
+ If history has already attached to the old endpoint in a way that requires that boundary to remain distinct, a new independent Insertion is created instead.
726
+
727
+ If the canonical Insertion has fragmented at runtime, a valid extension can still become another fragment of the same grown canonical identity.
728
+
729
+ Extension is therefore a history-growth optimization that preserves the same underlying model.
730
+
731
+ ---
732
+
733
+ # Removal is not deletion of history
734
+
735
+ Removing visible content does not immediately erase its original Insertion.
736
+
737
+ Instead, removal is represented as new history.
738
+
739
+ A removal is a reducing Insertion:
740
+
741
+ ```text
742
+ insertionDiff < 0
743
+ ```
744
+
745
+ which targets original Frames belonging to a positive Insertion.
746
+
747
+ This is best understood as **masking**.
748
+
749
+ Suppose the original content is:
750
+
751
+ ```text
752
+ A B C D
753
+ ```
754
+
755
+ and `B C` are removed.
756
+
757
+ The historical positive Insertion still exists.
758
+
759
+ A reducing Insertion records:
760
+
761
+ ```text
762
+ mask original Frames B and C
763
+ ```
764
+
765
+ The Projection then contains:
766
+
767
+ ```text
768
+ A D
769
+ ```
770
+
771
+ but Structural Order still knows what happened.
772
+
773
+ ---
774
+
775
+ # Why removal must target original Frames
776
+
777
+ Consider two replicas starting with:
778
+
779
+ ```text
780
+ The old house
781
+ ```
782
+
783
+ Replica A removes:
784
+
785
+ ```text
786
+ old
787
+ ```
788
+
789
+ while Replica B concurrently inserts:
790
+
791
+ ```text
792
+ small
793
+ ```
794
+
795
+ after it:
796
+
797
+ ```text
798
+ The old small house
799
+ ```
800
+
801
+ Replica A never observed:
802
+
803
+ ```text
804
+ small
805
+ ```
806
+
807
+ Its removal should therefore not remove that concurrent content.
808
+
809
+ Because removal targets original Frames rather than a mutable visible interval, it masks only:
810
+
811
+ ```text
812
+ old
813
+ ```
814
+
815
+ After synchronization the result becomes:
816
+
817
+ ```text
818
+ The small house
819
+ ```
820
+
821
+ regardless of delivery order.
822
+
823
+ This is one of the core consequences of using historical coordinates rather than Projection indexes.
824
+
825
+ ---
826
+
827
+ # A visible removal may cross several histories
828
+
829
+ What looks like one contiguous visible range may internally contain Frames from several canonical Insertions.
830
+
831
+ For example:
832
+
833
+ ```text
834
+ Hello world
835
+ ```
836
+
837
+ might structurally contain:
838
+
839
+ ```text
840
+ [Hello]
841
+ [ ]
842
+ [world]
843
+ ```
844
+
845
+ from three different historical Insertions.
846
+
847
+ Removing:
848
+
849
+ ```text
850
+ lo wo
851
+ ```
852
+
853
+ therefore cannot be represented as one blind operation against a Projection range.
854
+
855
+ Sequencer expresses the removal against the actual original Insertions whose Frames are being consumed.
856
+
857
+ Each reducing Insertion targets only Frames belonging to its own historical parent.
858
+
859
+ This prevents removals from accidentally claiming content they never observed.
860
+
861
+ ---
862
+
863
+ # Removal can also fragment
864
+
865
+ A single canonical removal may encounter an original Insertion that has already fragmented.
866
+
867
+ Suppose original history contains:
868
+
869
+ ```text
870
+ A B C D E F
871
+ ```
872
+
873
+ but the current runtime representation is:
874
+
875
+ ```text
876
+ [A B] [C D] [E F]
877
+ ```
878
+
879
+ A removal of:
880
+
881
+ ```text
882
+ B C D E
883
+ ```
884
+
885
+ may need to materialize across several runtime regions.
886
+
887
+ Those pieces are still one canonical removal.
888
+
889
+ They do not become independent replicated operations merely because the runtime needed several Strips to express their effect.
890
+
891
+ Again:
892
+
893
+ ```text
894
+ canonical history
895
+ ```
896
+
897
+ and:
898
+
899
+ ```text
900
+ runtime representation
901
+ ```
902
+
903
+ remain separate.
904
+
905
+ ---
906
+
907
+ # Overlapping removals
908
+
909
+ Concurrent replicas may independently remove overlapping Frames.
910
+
911
+ For example, from:
912
+
913
+ ```text
914
+ the brown fox jumps
915
+ ```
916
+
917
+ one replica removes:
918
+
919
+ ```text
920
+ brown fox
921
+ ```
922
+
923
+ while another removes:
924
+
925
+ ```text
926
+ fox jumps
927
+ ```
928
+
929
+ Both removals refer to original Frames.
930
+
931
+ The Frames belonging to:
932
+
933
+ ```text
934
+ fox
935
+ ```
936
+
937
+ were targeted twice.
938
+
939
+ They must still disappear only once from the Projection.
940
+
941
+ A reducing Insertion therefore retains its canonical authored meaning while its current runtime effect is resolved against Frames that remain available to consume.
942
+
943
+ This guarantees:
944
+
945
+ ```text
946
+ one original Frame cannot reduce the Projection twice
947
+ ```
948
+
949
+ Delivery order cannot cause repeated subtraction or make deleted content reappear.
950
+
951
+ ---
952
+
953
+ # Replacement is composition
954
+
955
+ Replacement does not require its own structural concept.
956
+
957
+ Replacing:
958
+
959
+ ```text
960
+ Draft
961
+ ```
962
+
963
+ with:
964
+
965
+ ```text
966
+ Final
967
+ ```
968
+
969
+ is simply:
970
+
971
+ ```text
972
+ remove Draft
973
+ insert Final
974
+ ```
975
+
976
+ The important part is the relationship between those two operations.
977
+
978
+ The new content is anchored against the history that exists after the removal relationship has been established.
979
+
980
+ This keeps replacement semantics consistent with ordinary insertion and removal instead of introducing another independent ordering model.
981
+
982
+ ---
983
+
984
+ # Visible content and historical content are deliberately different
985
+
986
+ After enough editing, Structural Order may contain much more history than the Projection shows.
987
+
988
+ For example:
989
+
990
+ ```text
991
+ Structural Order
992
+
993
+ [A B C]
994
+ [mask B]
995
+ [X]
996
+ [mask C]
997
+ [Y]
998
+ ```
999
+
1000
+ could produce a Projection such as:
1001
+
1002
+ ```text
1003
+ A X Y
1004
+ ```
1005
+
1006
+ The hidden history is not accidental garbage.
1007
+
1008
+ It may still be required to understand:
1009
+
1010
+ ```text
1011
+ future delayed operations
1012
+ historical anchor points
1013
+ concurrent relationships
1014
+ removal ownership
1015
+ ordering of descendants
1016
+ ```
1017
+
1018
+ Sequencer therefore cannot simply delete every hidden node as soon as it stops being visible.
1019
+
1020
+ ---
1021
+
1022
+ # Replication transports canonical history
1023
+
1024
+ Replicas exchange canonical Insertions and acknowledgement information.
1025
+
1026
+ The transport itself is not part of the sequencing model.
1027
+
1028
+ History may be delivered:
1029
+
1030
+ ```text
1031
+ immediately
1032
+ later
1033
+ out of order
1034
+ more than once
1035
+ through a central server
1036
+ peer to peer
1037
+ after being persisted
1038
+ ```
1039
+
1040
+ None of these should affect the eventual ordering.
1041
+
1042
+ The important property is that the same canonical history eventually reaches the replicas that need it.
1043
+
1044
+ ---
1045
+
1046
+ # Duplicate delivery is harmless
1047
+
1048
+ Because canonical Insertions have stable identities, receiving the same history multiple times does not duplicate its visible effect.
1049
+
1050
+ This allows replication systems to favor reliability over exactly-once transport.
1051
+
1052
+ A sender can retransmit uncertain history.
1053
+
1054
+ A relay can replay stored history.
1055
+
1056
+ A reconnecting replica can receive data it already has.
1057
+
1058
+ Sequencer recognizes the canonical identity and keeps the resulting history idempotent.
1059
+
1060
+ ---
1061
+
1062
+ # Actors and Sessions have different roles
1063
+
1064
+ A **Session** identifies authored sequencing history.
1065
+
1066
+ An **Actor** participates in acknowledgement of removal history.
1067
+
1068
+ These concepts are intentionally separate.
1069
+
1070
+ One Actor may have several independent Sessions.
1071
+
1072
+ Different replicas belonging to the same logical Actor can therefore create history without sharing one mutable sequencing clock.
1073
+
1074
+ Session identities answer:
1075
+
1076
+ ```text
1077
+ who authored this canonical history stream?
1078
+ ```
1079
+
1080
+ Actor identities answer:
1081
+
1082
+ ```text
1083
+ whose acknowledgement is required before old removal history can be forgotten?
1084
+ ```
1085
+
1086
+ ---
1087
+
1088
+ # Why acknowledgements are needed
1089
+
1090
+ Suppose Replica A removes some content.
1091
+
1092
+ Replica B is currently offline.
1093
+
1094
+ Replica B may still contain unsent history anchored to the content that A removed.
1095
+
1096
+ If A permanently discarded every trace of that historical content immediately, B's delayed operation could later refer to a point that no longer exists anywhere in A's retained state.
1097
+
1098
+ Sequencer therefore separates:
1099
+
1100
+ ```text
1101
+ content is no longer visible
1102
+ ```
1103
+
1104
+ from:
1105
+
1106
+ ```text
1107
+ history is safe to forget
1108
+ ```
1109
+
1110
+ Those are not the same event.
1111
+
1112
+ ---
1113
+
1114
+ # Removal frontiers
1115
+
1116
+ Actors acknowledge received prefixes of removal Sessions.
1117
+
1118
+ Conceptually an acknowledgement says:
1119
+
1120
+ ```text
1121
+ Actor A has observed this removal Session
1122
+ through logical point N
1123
+ ```
1124
+
1125
+ A frontier summarizes a whole contiguous prefix rather than listing every individual removal.
1126
+
1127
+ This keeps acknowledgement state small while still allowing Sequencer to determine which removal history is globally known among the active Actors.
1128
+
1129
+ ---
1130
+
1131
+ # Compaction
1132
+
1133
+ Once all active Actors agree on a removal frontier, the acknowledged removal history becomes eligible for compaction.
1134
+
1135
+ Compaction does not simply delete mask nodes.
1136
+
1137
+ It constructs an equivalent retained history in which obsolete removed Frames and the historical machinery surrounding them can be omitted.
1138
+
1139
+ This may involve:
1140
+
1141
+ ```text
1142
+ removing acknowledged masks
1143
+ removing Footage that can no longer reappear
1144
+ shortening positive Insertions
1145
+ adjusting surviving mask coordinates
1146
+ collapsing obsolete logical regions
1147
+ re-anchoring surviving history
1148
+ ```
1149
+
1150
+ The required invariant is simple:
1151
+
1152
+ ```text
1153
+ Projection before compaction
1154
+ ==
1155
+ Projection reconstructed from compacted history
1156
+ ```
1157
+
1158
+ But preserving that invariant under future editing is the harder part.
1159
+
1160
+ The compacted history must not only look the same now.
1161
+
1162
+ It must also preserve the ordering relationships that remain relevant later.
1163
+
1164
+ ---
1165
+
1166
+ # Compaction can collapse historical space
1167
+
1168
+ Consider surviving children originally attached to two different logical points:
1169
+
1170
+ ```text
1171
+ point 1
1172
+ ↓
1173
+ removed historical region
1174
+ ↓
1175
+ point 2
1176
+ ```
1177
+
1178
+ After the region between them becomes safe to forget, both historical points may collapse onto the same retained boundary.
1179
+
1180
+ At that point the original geometric distinction no longer exists.
1181
+
1182
+ But the children may already have a resolved order.
1183
+
1184
+ Compaction must preserve that order.
1185
+
1186
+ Sequencer can therefore re-anchor surviving positive history according to its already resolved retained Structural Order.
1187
+
1188
+ In other words, once obsolete history is discarded:
1189
+
1190
+ ```text
1191
+ resolved order
1192
+ ```
1193
+
1194
+ can become more important than:
1195
+
1196
+ ```text
1197
+ the exact historical path that originally produced it
1198
+ ```
1199
+
1200
+ This is what lets retained state become smaller without reopening old ordering decisions.
1201
+
1202
+ ---
1203
+
1204
+ # Compaction does not rewrite live history
1205
+
1206
+ Compaction operates on retained state.
1207
+
1208
+ A live Projection does not need to restructure itself every time a snapshot is produced.
1209
+
1210
+ Instead:
1211
+
1212
+ ```text
1213
+ live Structural Order
1214
+ ↓
1215
+ derive equivalent compacted history
1216
+ ↓
1217
+ persist / transfer snapshot
1218
+ ```
1219
+
1220
+ A later Projection can hydrate from that compacted history and produce the same visible state and compatible future ordering.
1221
+
1222
+ This keeps persistence optimization separate from the current editing runtime.
1223
+
1224
+ ---
1225
+
1226
+ # Actor retirement
1227
+
1228
+ A permanently gone Actor would otherwise block removal compaction forever.
1229
+
1230
+ Sequencer therefore allows an Actor to be retired from future acknowledgement consensus.
1231
+
1232
+ Retirement means:
1233
+
1234
+ ```text
1235
+ this Actor is no longer expected to acknowledge future compaction
1236
+ ```
1237
+
1238
+ It does not mean:
1239
+
1240
+ ```text
1241
+ delete history authored by this Actor
1242
+ ```
1243
+
1244
+ Those are unrelated concepts.
1245
+
1246
+ Retirement is an application-level decision because Sequencer cannot know whether a disconnected replica is temporarily offline or permanently abandoned.
1247
+
1248
+ ---
1249
+
1250
+ # Persistence retains history, not runtime state
1251
+
1252
+ A persisted Sequence contains the history needed to reconstruct the Projection.
1253
+
1254
+ It does not need to persist every runtime optimization.
1255
+
1256
+ For example, persistence needs the equivalent of:
1257
+
1258
+ ```text
1259
+ canonical Insertions
1260
+ retained acknowledgement state
1261
+ positive Footage
1262
+ ```
1263
+
1264
+ It does not need to preserve things such as:
1265
+
1266
+ ```text
1267
+ current fragmentation layout
1268
+ temporary traversal indexes
1269
+ recent lookup positions
1270
+ allocation addresses
1271
+ ```
1272
+
1273
+ Those can be reconstructed.
1274
+
1275
+ The persistence boundary therefore follows the same conceptual separation as replication:
1276
+
1277
+ ```text
1278
+ stable history is durable
1279
+ runtime materialization is disposable
1280
+ ```
1281
+
1282
+ ---
1283
+
1284
+ # Hydration reconstructs an equivalent Projection
1285
+
1286
+ When retained history is loaded again, Sequencer reconstructs the structural relationships encoded by that history and derives the visible Projection.
1287
+
1288
+ The important requirement is not that the new runtime representation be byte-for-byte identical to the previous one.
1289
+
1290
+ It only needs to reconstruct the same meaningful state:
1291
+
1292
+ ```text
1293
+ same visible content
1294
+ same retained canonical relationships
1295
+ same deterministic future ordering
1296
+ ```
1297
+
1298
+ Runtime Strip boundaries or indexes may differ.
1299
+
1300
+ They are not part of the distributed identity of the sequence.
1301
+
1302
+ ---
1303
+
1304
+ # The same history can have different runtime shapes
1305
+
1306
+ This is a useful general principle.
1307
+
1308
+ Replica A might internally represent some history as:
1309
+
1310
+ ```text
1311
+ [A B C] [D E]
1312
+ ```
1313
+
1314
+ Replica B might represent the same canonical history as:
1315
+
1316
+ ```text
1317
+ [A] [B C] [D] [E]
1318
+ ```
1319
+
1320
+ A hydrated replica may create yet another local representation.
1321
+
1322
+ As long as all of them agree on:
1323
+
1324
+ ```text
1325
+ canonical identities
1326
+ canonical anchors
1327
+ original Frame coordinates
1328
+ deterministic Structural Order
1329
+ ```
1330
+
1331
+ they represent the same Sequence.
1332
+
1333
+ Convergence is therefore defined over logical history and Projection, not over internal memory layout.
1334
+
1335
+ ---
1336
+
1337
+ # Three coordinate spaces
1338
+
1339
+ Many apparent complexities in Sequencer become much simpler when three different coordinate spaces are kept separate.
1340
+
1341
+ ## Projection positions
1342
+
1343
+ These are the indexes applications use:
1344
+
1345
+ ```text
1346
+ 0, 1, 2, 3, ...
1347
+ ```
1348
+
1349
+ They describe the visible sequence right now.
1350
+
1351
+ They are temporary.
1352
+
1353
+ ---
1354
+
1355
+ ## Canonical logical coordinates
1356
+
1357
+ These identify Frames and boundaries inside original Insertions.
1358
+
1359
+ They survive:
1360
+
1361
+ ```text
1362
+ concurrent edits
1363
+ fragmentation
1364
+ removal
1365
+ reordered delivery
1366
+ restart
1367
+ ```
1368
+
1369
+ These are the coordinates replication cares about.
1370
+
1371
+ ---
1372
+
1373
+ ## Runtime structural positions
1374
+
1375
+ A live implementation also has some local way to identify materialized pieces of Structural Order.
1376
+
1377
+ Those positions are purely runtime state.
1378
+
1379
+ Different replicas do not need to share them.
1380
+
1381
+ ---
1382
+
1383
+ So:
1384
+
1385
+ ```text
1386
+ Projection position
1387
+ !=
1388
+ canonical logical coordinate
1389
+ !=
1390
+ runtime structural position
1391
+ ```
1392
+
1393
+ Keeping those concepts separate is the easiest way to reason about Sequencer.
1394
+
1395
+ ---
1396
+
1397
+ # The model in one example
1398
+
1399
+ Start with an empty Projection.
1400
+
1401
+ Replica A inserts:
1402
+
1403
+ ```text
1404
+ Hello
1405
+ ```
1406
+
1407
+ This creates positive canonical history.
1408
+
1409
+ Replica B later receives that history.
1410
+
1411
+ Both now derive:
1412
+
1413
+ ```text
1414
+ Hello
1415
+ ```
1416
+
1417
+ Replica A inserts:
1418
+
1419
+ ```text
1420
+ world
1421
+ ```
1422
+
1423
+ at the end.
1424
+
1425
+ If eligible, the existing positive history may simply extend.
1426
+
1427
+ Meanwhile Replica B inserts:
1428
+
1429
+ ```text
1430
+ beautiful
1431
+ ```
1432
+
1433
+ at the same historical boundary.
1434
+
1435
+ Now the replicas contain concurrent history.
1436
+
1437
+ They may receive each other's edits in opposite orders.
1438
+
1439
+ Because both operations describe stable historical relationships, Sequencer deterministically resolves them.
1440
+
1441
+ Suppose the resulting Projection is:
1442
+
1443
+ ```text
1444
+ Hello beautiful world
1445
+ ```
1446
+
1447
+ Replica A removes:
1448
+
1449
+ ```text
1450
+ beautiful
1451
+ ```
1452
+
1453
+ while Replica B concurrently inserts:
1454
+
1455
+ ```text
1456
+ wide
1457
+ ```
1458
+
1459
+ inside nearby content.
1460
+
1461
+ The removal masks only Frames actually observed by its author.
1462
+
1463
+ It cannot accidentally consume the concurrent insertion.
1464
+
1465
+ Later, all active Actors acknowledge the relevant removal history.
1466
+
1467
+ The removed Frames and their masks become eligible for compaction.
1468
+
1469
+ A persisted snapshot can then omit the obsolete historical region while retaining the already resolved order of everything that survived.
1470
+
1471
+ A newly hydrated replica reconstructs:
1472
+
1473
+ ```text
1474
+ Hello wide world
1475
+ ```
1476
+
1477
+ and continues editing from equivalent history.
1478
+
1479
+ That lifecycle contains almost the entire Sequencer model:
1480
+
1481
+ ```text
1482
+ visible positions
1483
+ → canonical anchors
1484
+ → deterministic history
1485
+ → concurrency
1486
+ → fragmentation
1487
+ → masking
1488
+ → acknowledgement
1489
+ → compaction
1490
+ → equivalent reconstruction
1491
+ ```
1492
+
1493
+ ---
1494
+
1495
+ # The central idea
1496
+
1497
+ Sequencer works because temporary visible positions are never treated as permanent distributed identities.
1498
+
1499
+ Every edit is translated into stable historical relationships.
1500
+
1501
+ Those relationships can survive:
1502
+
1503
+ ```text
1504
+ concurrent editing
1505
+ different packet orders
1506
+ duplicate delivery
1507
+ offline replicas
1508
+ fragmentation
1509
+ removal
1510
+ restart
1511
+ compaction
1512
+ ```
1513
+
1514
+ The overall transformation is:
1515
+
1516
+ ```text
1517
+ local Projection position
1518
+ ↓
1519
+ canonical historical relationship
1520
+ ↓
1521
+ deterministic Structural Order
1522
+ ↓
1523
+ local Projection
1524
+ ```
1525
+
1526
+ Every replica may have a different temporary view while history is still arriving.
1527
+
1528
+ Once they know the same canonical history, they can independently derive the same result.
1529
+
1530
+ That is the core of Sequencer.