@fluidframework/merge-tree 2.0.0-dev.4.3.0.159619 → 2.0.0-dev.4.4.0.161322

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fluidframework/merge-tree",
-  "version": "2.0.0-dev.4.3.0.159619",
+  "version": "2.0.0-dev.4.4.0.161322",
   "description": "Merge tree",
   "homepage": "https://fluidframework.com",
   "repository": {
@@ -37,27 +37,27 @@
   "dependencies": {
     "@fluidframework/common-definitions": "^0.20.1",
     "@fluidframework/common-utils": "^1.1.1",
-    "@fluidframework/container-definitions": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/container-utils": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/core-interfaces": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/datastore-definitions": "2.0.0-dev.4.3.0.159619",
+    "@fluidframework/container-definitions": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/container-utils": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/core-interfaces": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/datastore-definitions": "2.0.0-dev.4.4.0.161322",
     "@fluidframework/protocol-definitions": "^1.1.0",
-    "@fluidframework/runtime-definitions": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/runtime-utils": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/shared-object-base": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/telemetry-utils": "2.0.0-dev.4.3.0.159619"
+    "@fluidframework/runtime-definitions": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/runtime-utils": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/shared-object-base": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/telemetry-utils": "2.0.0-dev.4.4.0.161322"
   },
   "devDependencies": {
-    "@fluid-internal/stochastic-test-utils": "2.0.0-dev.4.3.0.159619",
-    "@fluid-internal/test-pairwise-generator": "2.0.0-dev.4.3.0.159619",
+    "@fluid-internal/stochastic-test-utils": "2.0.0-dev.4.4.0.161322",
+    "@fluid-internal/test-pairwise-generator": "2.0.0-dev.4.4.0.161322",
     "@fluid-tools/benchmark": "^0.47.0",
     "@fluid-tools/build-cli": "^0.17.0",
     "@fluidframework/build-common": "^1.1.0",
     "@fluidframework/build-tools": "^0.17.0",
     "@fluidframework/eslint-config-fluid": "^2.0.0",
     "@fluidframework/merge-tree-previous": "npm:@fluidframework/merge-tree@2.0.0-internal.4.1.0",
-    "@fluidframework/mocha-test-setup": "2.0.0-dev.4.3.0.159619",
-    "@fluidframework/test-runtime-utils": "2.0.0-dev.4.3.0.159619",
+    "@fluidframework/mocha-test-setup": "2.0.0-dev.4.4.0.161322",
+    "@fluidframework/test-runtime-utils": "2.0.0-dev.4.4.0.161322",
     "@microsoft/api-extractor": "^7.34.4",
     "@types/diff": "^3.5.1",
     "@types/mocha": "^9.1.1",

package/.vscode/launch.json

@@ -1,16 +0,0 @@
-{
-	// Use IntelliSense to learn about possible attributes.
-	// Hover to view descriptions of existing attributes.
-	// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
-	"version": "0.2.0",
-	"configurations": [
-		{
-			"name": "Merge Tree Test",
-			"type": "node",
-			"request": "launch",
-			"program": "${workspaceRoot}/../../../node_modules/mocha/bin/_mocha",
-			"args": ["dist/test", " --recursive", "--no-timeouts", "--exit"],
-			"cwd": "${workspaceRoot}"
-		}
-	]
-}

package/docs/Attribution.md

@@ -1,382 +0,0 @@
-# Attribution
-
-This design document covers a high-level plan for embedding attribution information into merge-tree.
-It attempts to be detailed enough to start fleshing out proposed optimizations into code, though the actual factoring of the code
-(responsibilities of objects, names/semantics) may be subject to change in further refinements of the design.
-
-## Motivation
-
-A common feature in collaborative applications is the ability to attribute pieces of content to a particular user.
-This attribution information generally contains information about who edited the content as well as when the edit occurred.
-
-At the time of writing this document, the Fluid Framework doesn't natively support this kind of functionality,
-though in theory it has all of the data it needs (the op envelope contains both a timestamp and a client id, which can
-be mapped to information about the user using the audience).
-This has forced Fluid consumers that want attribution information to use workaround schemes. For example, in SharedString it's
-straightforward to conceptualize a scheme where each time a client submits an op that edits the string, it waits for that op to
-ack and uses the timestamp on that op to submit an additional op that annotates the edited segments with attribution information.
-
-Besides unnecessarily complicating client code, this has several drawbacks:
-
--   It is noisy on the wire
--   Attribution information can be lost in various cases if the submitting client disconnects
--   In-memory and snapshot size for the SharedString is more bloated than it should be; without binning the timestamps this strategy
-    entirely invalidated the zamboni scheme, and even if the timestamps are binned this will unnecessarily include the same user info
-    many times on different segments
-
-Rather than force this burden on consumers, it makes more sense to bake some attribution capability into the Fluid Framework in an opt-in way.
-Though this document will cover an approach for doing so in merge-tree (primarily targeted at support for attribution in SharedString),
-none of the above concerns are specific to a single DDS.
-It's imaginible that Fluid will eventually want to generalize this to a platform mechanism that's supported by each DDS that wants to opt in to it.
-For that reason, the design is aimed to modularize into areas that are generic to the container runtime and those that are DDS-specific.
-
-## High-level
-
-If one had access to the entire op stream, a lookup from all historical client ids to their user info,
-and every DDS retained information about which sequence number created/modified each part of its data,
-attribution would be straightforward. Ask the DDS for the relevant sequence number, then look at this sequence number's op for a timestamp + clientId
-and use the client id to look up user information.
-
-All of this information is knowable from the Fluid runtime perspective, though not all of it is persisted indefinitely.
-Notably:
-
--   Access to the entire op stream is an unreasonable assumption due to the summarization process
--   User information is only accessible for connected clients
-
-However, this conceptualization of attribution does suggest a reasonable split of concerns that can be individually assessed:
-none of the association between sequence numbers, timestamps, clientIds, and user information is specific to any given DDS.
-Thus, all of this bookkeeping could be generically done by the framework (potential candidates include on container runtime, data store runtime, or channel context),
-and any query-style APIs a DDS might support for retrieving attribution information could be accomplished by asking the runtime for information about a given sequence number.
-
-This leaves two high-level problems:
-
-1. How can the framework manage to associate sequence numbers to attribution information efficiently?
-2. What degrees of freedom should merge-tree expose for attributing its state to different users?
-
-## Sequence Number to Attribution Association
-
-Setting aside the problem of where to put the state for now, there are two primary ways by which associations between sequence numbers and attribution
-can be made practical from a memory perspective.
-First, there needs to be a garbage collection scheme to clean up attribution information on removed content.
-
-Secondly, attribution information needs to be compacted to an efficient format, both in terms of snapshot size (i.e. plain data representation) and desired
-level of granularity (applications don't care about millisecond-accurate timestamps).
-
-Finally, the in-memory data structures that support the necessary APIs are discussed.
-
-### Cleanup of outdated information
-
-Since the semantics of each op is opaque to the runtime, the runtime needs some mechanism to ascertain when an op's attribution is no longer relevant, i.e.
-not referenced.
-
-There are a few general models that could work:
-
-1. Assume that the runtime controls authoring of references to attribution information. It could stamp such information with a unique symbol such that it could
-   later be recognized in serialization to determine if the info was still referenced.
-   This approach is not far off from how `IFluidHandle`s work.
-   Reference counting the created objects could also work, but would likely be messier (responsibility of cleanup will likely end up extending past where we want it).
-2. Demand objects that store attribution information implement a function that exposes all sequence numbers they reference.
-
-Option 1 might look something like
-
-```typescript
-const attributionHandle = Symbol("attribution handle");
-
-class /*Container/DataStore/etc. (TBD)*/ Runtime {
-	public createAttributionHandle(sequenceNumber: number) {
-		return {
-			[attributionHandle]: true,
-			sequenceNumber,
-		};
-	}
-}
-
-// Serialization logic in ISerializer would need to look for attributionHandle symbol usages
-// and serialize it appropriately.
-// Similarly for deserialization.
-// At summary time, the set of sequence numbers that were referenced can be recorded for each data store,
-// and any sequence numbers that are no longer referenced could have their attribution information cleaned up.
-// Incremental summaries make the bookkeeping of this scheme slightly more complicated, but the general idea
-// still works.
-```
-
-The main advantage of option 1 is that it requires less DDS/application code.
-However, it causes larger-sized snapshots, since the serialized form of runtime-minted attribution handles will be more verbose than a simple number.
-It also leads to a potentially nasty bug pit: there's not a practical way to enforce that objects storing attribution information actually call
-`createAttributionHandle` before serializing their data: they could just as easily store the sequence number and only call `createAttributionHandle`
-directly before trying to obtain attribution information.
-This would risk attribution information getting GC'd too early.
-
-Option 2 would look closer to this:
-
-```typescript
-interface IReferenceAttributionInfo {
-	/**
-	 * @returns an iterable over all sequence numbers for which this object references attribution information.
-	 */
-	getReferencedSeqs(): Iterable<number>;
-}
-
-class MergeTree implements IReferenceAttributionInfo {
-	public getReferencedSeqs() {
-		const seqs = new Set();
-		this.walkAllSegments(this.root, (seg) => {
-			seqs.add(seg.seq);
-		});
-		return seqs;
-	}
-}
-```
-
-Though this design forces extra code on users, it's typically not conceptually difficult to implement and enables the serialized format to be more compact.
-
-It's worth noting that both of these models have interesting interactions with partial checkouts / schemes for more incremental summarization at the DDS level (via blob re-use [see #832](https://dev.azure.com/fluidframework/internal/_workitems/edit/832)): each would need to support some notion of reference count deltas from the previous result.
-
-### Compaction of similar information
-
-One primary motivation for supporting attribution information natively in the framework is the potential to reduce redundant attribution information in snapshots.
-There are two obvious ways data is redundant: user information gets repeatedly inlined into `JSON.stringify`d content, and various sets of ops all likely have
-virtually the same attribution information (same user, perhaps a slightly different timestamp).
-Ops that have closer together sequence number are more likely to contain such redundant information, as users tend to edit documents in bursts.
-This suggests a few strategies for keeping a compact format (either only on serialization or in-memory as well):
-
-1. Intern user objects
-2. Intern attribution objects
-3. If a range of sequence numbers all have the same attribution information, store it as such
-4. Allow "equivalent timestamp" policy injection: it's unlikely any app needs millisecond or better accuracy on the server ack timestamp for attribution purposes.
-   There should be a configurable policy for how timestamps get binned. Basic implementations could bin on a fixed cadence, but for even more compact files a dynamic bin size policy with larger bins for less recent data could also give a reasonable user experience
-
-Optimizations 1 through 3 are all things that standard compression algorithms can detect: interning objects is essentially
-[dictionary compression](https://en.wikipedia.org/wiki/Dictionary_coder) and compressing adjacent ranges is
-[run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding), so before going through the trouble of writing bespoke compression code
-we should experiment with things like [LZ4](<https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)>) and [DEFLATE](https://en.wikipedia.org/wiki/Deflate).
-For the purposes of illustration, the following sections will outline how the bespoke code might look.
-
-#### Interning
-
-Rather than repeatedly serialize the same information in the snapshot format, we can internally add a level of indirection to the `user` field, the entire
-attribution object, or both.
-This optimization would be entirely transparent to the public API: whatever snapshot/in-memory format we use, we'd always convert to `AttributionInfo` before
-returning the information for a given seq to the DDS/application.
-
-Interfaces might look like this, with exported properties being those visible to an application:
-
-```typescript
-export interface AttributionInfo {
-	user: IUser;
-	timestamp: number;
-}
-
-export interface IAttributor {
-	getAttributionInfo(seq: number): AttributionInfo;
-}
-
-type InternedRef = number & { readonly InternedRef: "e86840d8-8384-450c-b0e3-9a2855ba2d21" };
-
-interface ObjectInterner {
-	getOrCreateRef(obj: Jsonable): InternedRef;
-	getObject(id: InternedRef): Jsonable;
-	getSerializable(): Jsonable;
-}
-
-interface CompactAttributionInfo {
-	userRef: InternedRef;
-	timestamp: number;
-}
-
-// Concrete types for a particular `Attributor` implementation
-interface SerializedAttributor {
-	interner: Jsonable /* result of calling getSerializable() on an ObjectInterner */;
-	lookup: {
-		[seq: number]: InternedRef /* to CompactAttributionInfo */ | CompactAttributionInfo;
-	};
-}
-```
-
-#### Adjacent Range Coalescing
-
-Typical documents will likely have a number of consecutive ops with the same attribution information.
-This happens for a few reasons: users might make a number of edits in a short period of time (consider a user typing
-out a new paragraph), and ops submitted by a single container are batched under some circumstances.
-
-Rather than end up with a `SerializedAttributor` that resembles this:
-
-```javascript
-{
-  interner: [{ email: "john.doe@contoso.com", id: "f400ddf3-4d04-48e9-8783-4b1db8a45fc3" }, { user: 0, timestamp: 1661974200000 }],
-  lookup: {
-    50: 1,
-    51: 1,
-    52: 1,
-    53: 1,
-    54: 1
-  }
-}
-```
-
-we could instead serialize the lookup table like so:
-
-```javascript
-{
-  interner: [{ email: "john.doe@contoso.com", id: "f400ddf3-4d04-48e9-8783-4b1db8a45fc3" }, { user: 0, timestamp: 1661974200000 }],
-  lookup: [{ key: [50, 54], value: 1 }]
-}
-```
-
-Since objects are distinguishable from numbers, single-number ranges could just have a number key.
-
-```typescript
-interface AttributionEntry {
-	/**
-	 * Either a single `seq` number for this attribution entry, or a consecutive range `[start, end]` (inclusive)
-	 * of `seq` numbers which all have the same attribution information.
-	 */
-	k: number | [number, number];
-	v: InternedRef | CompactAttributionInfo;
-}
-
-// Concrete types for a particular `Attributor` implementation
-interface SerializedAttributor {
-	interner: Jsonable /* result of calling getSerializable() on an ObjectInterner */;
-	lookup: AttributionEntry[];
-}
-```
-
-#### Timestamp Binning
-
-One key aspect of information compaction is the ability to bin the precise timestamps given by the server into more reasonable granularity levels for attribution.
-Unlike the other optimizations to compact information, binning is lossy.
-Binning can simply be dictated by a function that takes in a timestamp and returns a timestamp for the output bin.
-Simple strategies like "bin every 5 minutes" are as simple as `(timestamp: number) => timestamp - (timestamp % (1000 * 60 * 5))`,
-but allowing an arbitrary function here also empowers more advanced users to make partitions of timespace like "5-minute granularity up to a day ago, 1-day granularity up to a month ago, 1-month granularity up to a year ago, yearly granularity otherwise".
-For the simple strategy, running the binning function on initial sequencing of the op would be sufficient.
-To make the second function behave as desired ("old attribution information tends to get coalesced"),
-the runtime would also have to re-bin existing attribution information either every so often or just on document load.
-
-We should apply this optimization last, and only if we need it. It's possible standard time-series compression of numbers will be sufficient here.
-
-### In-memory attribution structure
-
-Attributor bookkeeping needs to efficiently support:
-
--   Lookup of attribution information at a `seq`
--   Adding attribution information for a newly sequenced op
--   Merging consecutive attribution entries that should now be coalesced (depending on other design choices, this one is less important)
-
-One candidate implementation would be to expand the serialized format entirely and use a `Map`. This implementation is viable, but uses
-`O(attributed seq#s)` memory. It would provide `O(1)` lookup.
-Another reasonable candidate would be to keep the overall structure of having coalesced adjacent ranges, putting the serialized form into a
-sorted list that can be binary searched.
-This would give a reasonable memory win at the cost of increasing lookup time to `O(log(attributed seq#s))`.
-
-Putting all of the optimizations together, an `Attributor` implementation might look something like this:
-
-```typescript
-export interface IAttributor {
-  getAttributionInfo(seq: number): AttributionInfo;
-}
-
-export const binByMinutes = (interval: number) => (timestamp: number) => timestamp - (timestamp % (1000 * 60 * interval));
-
-const seqComparator = (a: AttributionEntry, b: AttributionEntry) => {
-  aEnd = typeof a.k === 'number' ? a.k : a.k[1];
-  bEnd = typeof a.k === 'number' ? b.k : b.k[1];
-  return aEnd - bEnd;
-}
-
-class Attributor implements IAttributor {
-  private seqToInfo: SortedList<AttributionEntry> = new SortedList(seqComparator);
-  constructor(
-    runtime: IFluidDataStoreRuntime,
-    serialized?: SerializedAttributor,
-    bin: (timestamp: number) => number = binByMinutes(5)
-  ) {
-    if (serialized) {
-      const interner = new ObjectInterner(serialized.interner);
-      // Note: this implementation doesn't coalesce re-binned attribution entries that are newly equivalent and adjacent.
-      this.seqToInfo.extend(...serialized.lookup.map(({ k, v: internedV }) => {
-        const { timestamp, userRef } = isInternedRef(maybeInternedV) ? interner.getObject(maybeInternedV) : maybeInternedV;
-        const v = {
-          timestamp: bin(timestamp),
-          user: interner.getObject(userRef)
-        };
-        return { k, v };
-      }));
-    }
-
-    const { deltaManager, audience } = runtime;
-    deltaManager.on("op", (message: ISequencedDocumentMessage) => {
-      const attributionInfo = {
-         /* note: for object interning to work, this needs to be a referentially equal user object. If that isn't provided by the
-            Fluid Framework, we probably would want a layer of caching here. For interning of overall attribution info objects,
-            we may want a similar cache. */
-        user: audience.get(message.clientId).user,
-        timestamp: bin(message.timestamp)
-      };
-      const { k, v } = seqToInfo.getAt(seqToInfo.length - 1);
-      const lastEntryStart = typeof k === 'number' ? k : k[0];
-      const lastEntryEnd = typeof k === 'number' ? k : k[1];
-      if (
-        attributionInfosAreEquivalent(attributionInfo, v) &&
-        // Note: this coalescing logic is somewhat unideal since no-ops break it.
-        message.seq === 1 + lastEntryEnd)
-      ) {
-        this.seqToInfo.pop();
-        this.seqToInfo.insert({ k: [lastEntryStart, message.seq], v });
-      } else {
-        this.seqToInfo.insert({ k: message.seq, v: attributionInfo });
-      }
-    });
-  }
-
-  public getAttributionInfo(seq: number): AttributionInfo {
-    const { k, v } = seqToInfo.findAtOrAfter(seq);
-    assert(k === seq || (k.length === 2 && k[0] <= seq && seq <= k[1]));
-    return v;
-  }
-
-  // Unpictured:
-  // - serialization (not interesting; deserialization logic is pictured)
-  // - GC (there are several ways to hook this up, though one can check the data structure should support it in O(n))
-}
-
-```
-
-### Bookkeeping Placement Considerations
-
-There are several levels that the framework could choose to conceptually store "sequence number to attribution" information:
-
--   Container Runtime
--   Data store runtime
--   DDS
-
-The initial attributor implementation will likely be hooked up to only `SharedString` due to current feature asks of partner teams.
-However, it's worth calling out that depending on which layer the runtime places the information, there are consequences with respect
-to GC and how well information compacts.
-
-For GC:
-
--   Determining whether sequence numbers are referenced by any attribution information gets complicated slightly by incremental summarization if
-    information is stored on container runtime
-
-For compaction:
-
--   Compaction schemes potentially get worse for sequences of ops that alter different data stores if attribution information is stored at a
-    fine-grained level (e.g. DDS, Data store runtime).
-
-## Merge-Tree Attribution API
-
-TODO: This section will cover planned extension points for specifying attribution information on merge-tree.
-
-My current thinking is something along the lines of the following:
-
-Segments have a `attribution` field which is an opaque object to merge-tree, but splits/combines/impacts merge behavior a la tracking groups.
-Users of merge-tree are empowered to inject policy into the `attribution` of the segment as they see fit.
-The most basic policy which we should get for free would be to use `clientSeq` as the only tracked attribution state, which corresponds to
-an application that only wants to track who inserted the segment and when they did it.
-
-More advanced users could provide fancier json-serializable state objects such as `{ inserted: number, annotated: number }` and set up proper
-semantics for those fields.
-
-I need to think through if current merge-tree delta operation events are a sufficient entrypoint for managing such state, or if there's a nicer
-way to encapsulate common desires.

package/docs/DEV.md

@@ -1,22 +0,0 @@
-# Developer Notes
-
-## Merge Tree
-
-### Node lengths
-
-If a function reports a node's length as undefined, it means the node has been removed from the perspective of the client and/or reference sequence number.
-Alternately, if a functions reports a nodes length as 0 it means that node is not yet visible from the perspective client and/or reference sequence number.
-Ths distinction is important, as a removed segment with undefined length may not exists on remote clients, as it could have already been zambonied.
-However a not yet visible segment with 0 length may already exist, or will eventually exits on all clients.
-These have implications for eventually consistent conflict resolution. Generally, we ignore removed segments, and special case invisible segments, like in the case
-of conflicting insert as handled in the `breakTie` function
-
-### Zamboni
-
-Zamboni is the garbage collection process in the merge tree. As segment change due to inserts and deletes, we add them to a heap which keeps the segment with the lowest sequence number at the head. These segments drive the zamboni process which is also run on every change. The zamboni process peeks at the heap to determine if the head is below the min sequence, then the segment is eligible. The minimum sequence number is important here, as the minimum sequence number is a sequence seen by all clients, and all clients will specify their reference sequence number as above the minimum sequence number. This mean that no new operations can come in that reference anything at or below the minimum sequence number, so we are safe to clean up anything we would need to applying incoming. Eligible segments are collected, and then a few different operations are done, superficially, merge, remove, and tree rebalance. Zamboni is incremental, and only collects a constant number of segments at each change so as not to introduce performance issues.
-
-Merge is done if two adjacent segments are of the same type like text, that type is mergable (markers are not), neither are deleted, and all the properties match. The merge process reduces the number of segments, which are leaf nodes of the merge tree. For instance a user may type `c`, `a`, and `t` with each character being it's own operation therefore segment. The user could then highlight that range, and set a property on on all the characters indicating that they are bold, `{bold: true}`. At some later point, these segments would move to the top of th heap, and their sequence numbers would move below the minium sequence number. At that point zamboni could take those individual segments, and merge the into a single segment, `cat` with the property `{bold: true}`
-
-Remove is a bit simpler. On removal of a segment, we track it's removed sequence number. When the segment's removed sequence number drops below the minimum sequence number it can be safely removed from the tree.
-
-Rebalance is a bit different from merge and remove, as it has to do with maintaining the tree itself. After merge or removal there are fewer segments aka leaf nodes in the tree. This allows us to more efficiently pack the non-leaf node of the tree, and potentially remove layers from the tree. This keeps the tree compact, which has both memory and cpu performance implications.

package/docs/Obliterate.md

@@ -1,647 +0,0 @@
-# Merge Tree Obliterate
-
-This document covers motivation, spec, and design for the upcoming "obliterate" feature of merge-tree.
-
-## Spec
-
-A concise description of merge-tree's current merge conflict resolution strategy is as follows:
-
--   Insertion of a text segment only conflicts with other insertions at the same location.
-    The conflict is resolved by inserting the segment added later nearer in the string.
-    For example, from an initial state of "abc", if the operations [insert "hi " at 0] from client 1
-    and [insert "bye " at 0] from client 2 are sequenced in that order, the resulting state is "bye hi abc".
--   Range operations (delete, annotate) apply to the range at the time the operation was issued.
-    Specifically, insertion of a segment into a range that is concurrently deleted or annotated
-    will not result in that inserted segment being deleted or annotated. For example, from an initial state "012",
-    the operations [delete the range [1, 3)] from client 1 and [insert "hi" at index 2 (i.e. between "1" and "2")] from client 2,
-    the resulting text is "0hi".
-
-The merge outcomes for ranges are easy to understand, but not always desirable.
-Oftentimes, when consumers want to work with ranges, they may want their operation to apply to concurrently inserted segments.
-In the example above, these semantics would look like so:
-
-```
-// Initial state at seq 0: "012"
-{ seq: 1, refSeq: 0, clientId: 1, op: <insert "hi" at index 2> }
-{ seq: 2, refSeq: 0, clientId: 2, op: <delete the range [1, 3)> }
-// final desired state: "0"
-```
-
-```
-// Initial state at seq 0: "012"
-{ seq: 1, refSeq: 0, clientId: 2, op: <delete the range [1, 3)> }
-{ seq: 2, refSeq: 0, clientId: 1, op: <insert "hi" at index 2> }
-// final desired state: "0"
-```
-
-A `SharedString` feature request for a removal operation with these semantics dubbed them "obliterate".
-
-At an implementation level, these semantics can be viewed in two parts:
-
--   The range specification is resolved at the time the op is sequenced
--   Any subsequent segments inserted into that range concurrently should also be removed
-
-The first clause handles concurrent inserts before the removal is sequenced, and the second clause handles concurrent inserts after the removal is sequenced.
-
-However, there is a way to view obliterate's semantics as a special case of a "move" operation,
-which preserves content identity such that concurrently inserted segments will be inserted to the range at its current location.
-A main motivator here from the app perspective might be the idea that if user 1 cut and pastes an entire paragraph to a different section of the document
-whiler user 2 edits it, the desired merge outcome would likely be for user 2's edit to apply to the paragraph in its new location.
-Roughly, anywhere an application would want obliterate merge semantics on user delete of some content,
-the same application would want move semantics if the user instead cut and pasted the content somewhere else.
-
-There have historically been feature requests for move semantics inside merge-tree (for example [issue 8518](https://github.com/microsoft/FluidFramework/issues/8518)),
-so it makes sense to do forward-thinking on implementing obliterate in a way that we can extend it to cover move semantics in the future.
-
-For that reason, naming choices of fields and semantics for the remainder of the document will be written in terms of obliterate being the special case
-"move this range out of existence".
-This should alleviate any back-compat issues if/when we do decide to implement move (esp. fields that end up in ops or snapshots).
-The current proposal is to use the runtime value "null" to represent "out of existence", but this choice is flexible.
-In prose, for terseness that operation will still be called obliterate.
-After describing obliterate's design, this document [digs into how the design can be extended to work for move](##Move).
-
-Notice that the above examples always insert text at positions strictly inside the removed range.
-If the insert operation was instead before the "1" or after the "2", one can imagine different applications wanting different behavior:
-either the obliterated region should expand to include that text, or it should not.
-This topic will be covered in the [endpoint behavior](#endpoint-behavior) section,
-but for eventual consistency strategy discussion one should assume that the design should generally support both options
-(and either leave it up to merge-tree to restrict degrees of freedom as it seems fit).
-
-## Eventual Consistency Strategy
-
-This section is focused on how one could implement the "obliterate" semantics inside merge tree in an eventually consistent fashion.
-This will constitute the bulk of the complexity of the feature.
-Since obliterate is generally a "different kind of remove," there may be a nice abstraction to introduce at the code level to generalize
-removal information. However, in favor of introducing niceties later this design document will assume fields are inlined and focus on
-the strategy for ensuring eventual consistency. If such an abstraction is introduced, ideally it would enable better "pay-to-play" of
-common code paths based on merge-tree feature usage.
-As an example, `BaseSegment.split()` needs to copy segment properties to the split segment.
-So new properties added to segment will unnecessarily copy undefined values.
-
-There are a few aspects of merge tree's bookkeeping and general feature set that require consideration when designing new op semantics:
-
--   Any changes to direct fields of tree nodes themselves (either new data or changes to bookkeeping of existing data)
--   How the feature interacts with an increasing collab window and zamboni
--   Impact on the partial lengths scheme
--   Bookkeeping and handling of overlapping removals (note some may be obliterates and some may not be)
--   Reconnection
--   Snapshotting impact
-
-We'll first present an overview of a potential scheme for implementing the obliterate op, then comment on these aspects.
-
-### High-level bookkeeping changes
-
-Segments will be augmented with `movedSeq` and `localMovedSeq` fields which generally align with the semantics of `seq, localSeq, removedSeq,` and `localRemovedSeq`.
-When segments are moved and not just obliterated, they will also contain a reference to the destination segment.
-This may look as follows:
-
-```typescript
-/**
- * Tracks information about when and where this segment was moved to.
- * @example - Suppose a merge tree had 3 TextSegments "X", "A", and "B", and
- * received the operation `move({ start: 0, end: 1 }, { dest: 3 }, { seq: 30 })` (moving the "X"
- * after the "A" and the "B").
- * After processing this operation, it would have the segments `[<moved "X" tombstone>, "A", "B", "X"]`.
- * The moved "X" tombstone segment would have the following IMoveInfo: `{ movedSeq: 30, moveDst: <reference to living "X" segment>}`
- */
-export interface IMoveInfo {
-	/**
-	 * Local seq at which this segment was moved if the move is yet-to-be acked. Only set on the tombstone "source" segment of the move.
-	 */
-	localMovedSeq?: number;
-	/**
-	 * Seq at which this segment was moved. Only set on the tombstone "source" segment of the move.
-	 */
-	movedSeq: number;
-	/**
-	 * A reference to the inserted destination segment corresponding to this segment's move.
-	 * If undefined, the move was an obliterate.
-	 */
-	moveDst?: ReferencePosition;
-
-	/**
-	 * List of client IDs that have moved this segment.
-	 * The client that actually moved the segment (i.e. whose move op was sequenced first) is stored as the first
-	 * client in this list. Other clients in the list have all issued concurrent ops to move the segment.
-	 */
-	movedClientIds: number[];
-}
-
-export interface ISegment extends Partial<IRemovalInfo>, Partial<IMoveInfo> {
-	// ...
-}
-```
-
-The `moveDst` reference position functions as a redirection pointer when another client attempts to concurrently insert into the moved range: the usual approach
-for locating a node at some `{ pos, refSeq, clientId }` applies, and if the resulting segment has been moved, one can follow the trail of moves to find the segment's
-current location.
-
-Note that though `movedSeq` and `localMovedSeq` act very similarly to `removedSeq` and `localRemovedSeq` when considering the length of a segment at a given
-perspective: if the perspective is from after the segment was moved, the tombstone segment should have length 0.
-However, these fields need to be independent from `removedSeq` due to the possibility of a removal and a move overlapping, as well as the differences
-in how concurrent inserts are handled into a removed or a moved range.
-
-### Remote perspective
-
-We now move to some lower-level implementation details on how to ensure eventual consistency operating in this model.
-
-First, consider the behavior a client must have when processing an obliterate op it didn't submit.
-For concreteness and ease of explanation, say this op is `{ seq: 50, refSeq: 40, clientId: 2, op: <move the range [10, 15) to null }`.
-The processing client should first mark all segments between the segment `getContainingSegment({ pos: 10, refSeq: 40, clientId: 2 })` and
-`getContainingSegment({ pos: 15, refSeq: 40, clientId: 2 })` that are alive (i.e. inserted, not removed) from the perspective
-`{ seq: 50, clientId: localClientId }` obliterated.
-Note this means that if a segment in the range was concurrently removed, it won't be marked as moved as well.
-The marking process should be roughly equivalent to what happens in a "remove" operation, but instead of updating `removedSeq`/`localRemovedSeq`
-it updates `movedSeq` and `localMovedSeq`.
-
-The other interesting difference between this operation and a normal removal is its inclusion of segments inserted between seq 40 and seq 50.
-The current API on merge tree used for `markRangeRemoved` (which is `mapRange`) doesn't support iterating in this fashion,
-but could easily be extended to do so.
-One way to do that would be to decouple the `refSeq` and length calculations used for locating the positions and the `refSeq` used for
-deciding whether or not to descend and `map` children nodes.
-
-This handles removal of any concurrently inserted segments sequenced before the obliterate op, as well as local ops sequenced after the
-obliterate op (since we use `localClientId`).
-However, the client still needs to ensure concurrently inserted segments sequenced after the obliterate op are immediately removed.
-The insert codepath will therefore need to take into account if the destination is inside of an ongoing moved area.
-Excursions are a good tool for this job, but checking is still easier said than done.
-Concretely, and continuing with the example operations given above, suppose this insertion happens:
-
-```
-{ seq: 60, refSeq: 40, clientId: 3, op: <insert "hello" at index 10> }
-```
-
-After locating the insertion point and updating the merge tree, we need to decide if the resulting segment is inside of a moved region.
-If we happened to know the `seq` of the move we were testing for, this would be easy: the first adjacent segment in each direction from
-the perspective of `{ seq: 50, clientId: localClientId }` can inform us if we're either inside or directly adjacent to that moved range.
-Thus, a naive implementation could check all sequence numbers in the collab window.
-The obvious optimization of only checking seq numbers of move ops would improve this slightly.
-But we can do asymptotically better by leveraging the tree structure.
-It would be ideal if we only needed to perform one commonly short excursion in each direction.
-The only candidate that makes much sense is from the perspective of `{ seq: 60, clientId: 3 }` (i.e. the client submitting the insert op at
-the time the op is sequenced).
-The problem with this perspective is that ops 51 through 59 may have inserted a segment between the inserted "hello"
-and the obliterated range that was submitted by a client which has already acked the obliterate.
-For example, `{ seq: 55, clientId: 5, refSeq: 50, op: <insert "i won't be obliterated" at index 10> }`.
-
-The forward excursion would need to continue past this segment in order to conclude it isn't in an obliterated range.
-If that was the only such concurrent insert, the next segment it would visit would be an obliterated one and we'd decide
-whether or not to include the newly inserted segment as part of the obliterated region based on some endpoint merge strategy.
-
-The key insight is that visiting the segment with seq 55 does provide the excursion with information: since the segment
-was inserted at seq 55 and isn't moved or removed, any move operation must have occurred before seq 55.
-If we keep track of the smallest sequence number of alive segments that we've visited, we therefore have an upper bound
-for any possible adjacent move op.
-Thus, we can halt the excursion as soon as this upper bound falls below the smallest obliterate operation within the collab window.
-If we alternatively reach a segment that has been moved concurrently to the insert we're processing, we can also stop
-and use some endpoint resolution strategy.
-
-The guarantee we get for a removed segment isn't quite as good: we only know that the move must have come either before
-the segment was inserted or after it was removed (since move doesn't impact segments that are removed before its application).
-We _could_ track this as part of our excursion by maintaining a range of disjoint intervals at which an obliterate "might have happened"
-and exiting as soon as we know no obliterate is possible, but this is probably more effort than required: only decreasing our upper bound
-for removed segments if our existing upper bound is below when the segment was removed is a reasonable intermediate approach that uses
-less bookkeeping overhead.
-
-All-in-all, the insert logic modification might look something like this:
-
-```typescript
-function wasRemovedAfter(seg: ISegment, seq: number): boolean {
-	return seg.removedSeq !== UnassignedSequenceNumber && seg.removedSeq > seq;
-}
-
-function insertingWalk(args /* mostly omitted */, op) {
-	/* regular insert logic goes here */
-
-	let moveUpperBound = Number.POSITIVE_INFINITY;
-	let movedSegment: ISegment | undefined = undefined;
-	const smallestSeqMoveOp = this.getSmallestSeqMoveOp();
-	const findAdjacedMovedSegment = (seg) => {
-		if (seg.movedSeq && seg.movedSeq > op.referenceSequenceNumber) {
-			movedSegment = seg;
-			return false;
-		}
-
-		if (!isRemovedAndAcked(seg) || wasRemovedAfter(seg, moveUpperBound)) {
-			moveUpperBound = Math.min(moveUpperBound, seg.seq);
-		}
-		// If we've reached a segment that existed before any of our in-collab-window move ops
-		// happened, no need to continue.
-		return moveUpperBound > smallestSeqMoveOp;
-	};
-	forwardExcursion(insertSegment, findAdjacedMovedSegment);
-	const furtherMovedSegment = movedSegment;
-	currentMin = Number.POSITIVE_INFINITY;
-	movedSeg = undefined;
-	backwardExcursion(insertSegment, findAdjacedMovedSegment);
-	const nearerMovedSegment = movedSegment;
-	if (
-		(nearerMovedSegment && breakEndpointTie(nearerMovedSegment, insertSegment, op)) ||
-		(furtherMovedSegment && breakEndpointTie(insertSegment, furtherMovedSegment, op))
-	) {
-		// These objects will be analogous to return from `toRemovalInfo`.
-		const nearMoveInfo = toMoveInfo(nearerMovedSegment);
-		const farMoveInfo = toMoveInfo(furtherMovedSegment);
-		// The inserted segment could potentially be adjacent to two different moved regions.
-		// We mark it as moved using the info from the earlier such operation.
-		const moveInfo = min(nearMoveInfo, farMoveInfo);
-		markSegmentMoved(insertSegment, moveInfo, op);
-	}
-}
-```
-
-In reality it will be a bit more complicated: this does not properly handle inserting walks performed for local edits (which should never be immediately obliterated),
-nor does it handle local, un-acked obliterates (which will be covered in the next section).
-It's worth noting that removals between the obliterated seq and the inserting op's seq don't complicate things much because excursions visit all segments, regardless of visibility.
-
-This limits the segment excursions to not be longer than the number of consecutive segments adjacent to the insertion
-point that are all within the collaboration window.
-That's probably performant enough, but if we want to optimize further at some memory cost it is probably possible to use the
-partialLengths information to skip over blocks in some cases if the sequence numbers of obliterate ops are stored on
-each merge block.
-
-### Local perspective
-
-Next, we move to the local handling of a move op while it's in flight.
-For consistency with the rest of merge tree's segment state machine, the state transitions of `{ localMovedSeq, movedSeq }` and `{ localRemovedSeq, removedSeq }` should align (`movedSeq` is set to `UnassignedSeqNumber` while the op is in flight with `localMovedSeq` recording the local seq at which the move happened, then on ack of the op `localMovedSeq` is cleared out and `movedSeq` is replaced with the op's seq).
-
-While a move op is in flight, any non-local insertions into a locally moved range need to be immediately moved to the range's current location
-(or removed, if it was obliterated).
-This can be accomplished by tweaking the `findAdjacentMovedSegment` function above to account for `localMovedSeq`:
-
-```typescript
-const findAdjacentMovedSegment = (seg) => {
-	if (
-		(seg.movedSeq && seg.movedSeq > op.referenceSequenceNumber) ||
-		seg.localMovedSeq !== undefined
-	) {
-		movedSegment = seg;
-		return false;
-	}
-
-	if (!isRemovedAndAcked(seg) || wasRemovedAfter(seg, moveUpperBound)) {
-		moveUpperBound = Math.min(moveUpperBound, seg.seq);
-	}
-	// If we've reached a segment that existed before any of our in-collab-window move ops
-	// happened, no need to continue.
-	return moveUpperBound > smallestSeqMoveOp;
-};
-```
-
-We don't need to worry about the analogous problem of extending the excursion as a result of segments between the insert location and a local move
-because any such segments would have also been marked as locally moved when they were inserted into the merge tree.
-In the sample code written for the remote segment, this will also necessitate `markSegmentMoved` to tolerate marking segments with local obliteration info.
-
-Much of the same logic that goes into conflicting local + remote removal will need to be applied for move.
-Nothing stands out as a conceptual issue or hurdle in this realm, though. Just tricky conditionals.
-
-Once the op is acked, the behavior in the [Remote perspective](#remote-perspective) section suffices for any further concurrent segments.
-
-### Other aspects
-
-#### Zamboni
-
-Zamboni will need updating to account for the new bookkeeping fields, but there aren't any conceptual issues in this realm since zamboni cleans up unnecessary data for segments outside of the collaboration window and the only difference between remove and obliterate happens within the collab window.
-
-#### Snapshot
-
-Segments in the snapshot will need to serialize and rehydrate the newly added properties.
-Most of the types are plain-old data and JSON.serialize with no issue.
-When move is implemented (and so `moveDst` can actually be a local reference rather than undefined), that field will need some special handling.
-Several schemes are possible, but in the end it should convert to either a `pos` within some view of the merge-tree or an index+offset into the array
-of serialized segments.
-
-#### Reconnection
-
-When a move op is rebased, there will need to be local fixup of the range marked moved locally, since the resulting range may expand with different semantics (different ops
-will be concurrent to the rebased version). Since locally applying a move doesn't impact any sequenced segment state (and merge policy is to override pending local moves
-with any remote ones just like the remove merge policy), at worst this can be done unperformantly by walking the range, resetting state, and re-applying.
-
-The methods necessary for interpreting where the new range should be in the rebased view of the local merge-tree already exist and are used for regular reconnect (e.g.
-to remove a range of content), so should not present additional trouble.
-
-#### Partial Lengths
-
-One key capability of merge-tree is its ability to resolve the information `{ pos, clientId, refSeq }` (and potentially `localSeq` if the local client) into a particular
-segment + offset in the merge-tree's leaves.
-It does this efficiently by storing indexing structures at each internal node that allow querying for that node's length at any such perspective within the collab window,
-then leveraging those structures in an efficient tree walk.
-
-Adding additional tree operations that any client can undertake means that all other clients must be able to reason about their peers' current states.
-For example, `movedSeq` and `localMovedSeq` will need to be considered when calculating the length of a node/range from a given perspective.
-If the duplicated segment that's inserted as the result of a move is given the `clientId` of the moving client (as opposed to the originating client)
-and `seq` of the move operation, generally existing partial lengths logic will work correctly for non-concurrently inserted segments if
-`movedSeq` and `localMovedSeq` on the tombstoned segment are interpreted analogously to `removedSeq` and `localRemovedSeq`.
-Note that this would require updating the description of the `clientId` field, and for attribution purposes we may want to track the clientId that originally
-created the segment separately from the clientId that most recently caused the segment to be where it is (via move).
-
-Things get more complicated when considering resolution of node lengths for concurrently inserted segments.
-The remainder of this section assumes the content is obliterated rather than moved; there are additional difficulties for partial lengths when dealing with
-overlapping moves not covered in this document (they are probably solvable, but may require changes to the representation of the partial lengths indexing
-structure rather than just its data).
-
-Concretely, let's consider how partial lengths might look for a segment concurrently inserted into a moved region.
-
-Suppose:
-
-```
-// Initial state at seq 0: "0123456789"
-{ seq: 1, refSeq: 0, clientId: 1, op: <move [0, 5) out of existence> }
-{ seq: 2, refSeq: 0, clientId: 2, op: <insert "hi" at 2> }
-{ seq: 3, refSeq: 0, clientId: 2, op: <insert "hello" at 7> }
-```
-
-The desired final state in this case would be "56hello789". After seq 2, client 0 (an observer) has segments that look like so (clientIds that aren't relevant are omitted):
-
-```
-[
-  { seq: 0, movedSeq: 1, text: "01", movedClientIds: [1] },
-  { seq: 2, movedSeq: 1, clientId: 2, text: "hi", movedClientIds: [1] },
-  { seq: 0, movedSeq: 1, text: "234", movedClientIds: [1] },
-  { seq: 0, text: "56789" }
-]
-```
-
-If these segments are all in a single block and the minimum sequence number is 0, their parent's partial lengths resembles the following:
-
-```
-{
-  minLength: 10 // length of "0123456789"
-  partialLengths: [{ seq: 1, seglen: -5 }, { seq: 2, seglen: 0 }],
-  clientSeqNumbers: [[], [{ seq: 1, seglen: -5 }], /* client 2 */[ ?? ]]
-}
-```
-
-This data reflects the fact that the subsequence starts at length 10 at seq 0, an observer client sees the length of the subsequence shrink by 5 at seq 1,
-and doesn't see the length change afterward (note such a client hasn't yet received seq 3). It also looks correct for resolving client 1's perspective: even if
-the refSeq isn't at least 1, `clientSeqNumbers[1]` will still cause the current client's interpretation of client 1's view to include the removal of the
-range `[0, 5)`. Client 2 is the tricky one: the length of the block from client 2's perspective should be 12 at refSeq 0, but 5 at either refSeq 1 or 2.
-It looks odd, but this can be accomplished by adding a `{ seq: 1 /* comes from movedSeq */, seglen: 2 }` entry to `clientSeqNumbers[2]`.
-The intuition is that client 2 counts the length of the segment unless `seq >= movedSeq`, and the method used in partial lengths computes the length of
-a subsequence using
-
-(length at min seq) + (any deltas between minSeq and refSeq) + (any deltas for ops submitted by remote client between refSeq and now),
-
-so the last term counts this entry precisely when it's desired.
-
-What happens if the insert and the obliterate are concurrent but sequenced in the other order?
-
-```
-// Initial state at seq 0: "0123456789"
-{ seq: 1, refSeq: 0, clientId: 2, op: <insert "hi" at 2> }
-{ seq: 2, refSeq: 0, clientId: 1, op: <move [0, 5) out of existence> }
-{ seq: 3, refSeq: 0, clientId: 2, op: <insert "hello" at 7> }
-```
-
-The segment state after seq 2 from client 0's perspective will look mostly the same:
-
-```
-[
-  { seq: 0, movedSeq: 2, text: "01", movedClientIds: [1] },
-  { seq: 1, movedSeq: 2, clientId: 2, text: "hi", movedClientIds: [1] },
-  { seq: 0, movedSeq: 2, text: "234", movedClientIds: [1] },
-  { seq: 0, text: "56789" }
-]
-```
-
-And the partial lengths object might look like this:
-
-```
-{
-  minLength: 10 // length of "0123456789"
-  partialLengths: [{ seq: 1, seglen: 2 }, { seq: 2, seglen: -7 }],
-  clientSeqNumbers: [[], [{ seq: 1, seglen: 2 }, { seq: 2, seglen: -7 }], /* client 2 */[{ seq: 1, seglen: 2 }]]
-}
-```
-
-Note that in this case, client 1's `clientSeqNumbers` needed to be fixed up to include an entry for the concurrently inserted segment.
-Thus, when an obliterate/move affects a concurrently inserted segment, it's generally possible to modify the generated partial lengths'
-`clientSeqNumbers` for the client that sequenced its concurrent op later using the information on the inserted segment to interpret
-correct values.
-
-This strategy is also consistent with the existing strategy for overlapping delete: see the following snippet from `addClientSeqNumberFromPartial`:
-
-```typescript
-if (partialLength.overlapRemoveClients) {
-	partialLength.overlapRemoveClients.map((oc: Property<number, IOverlapClient>) => {
-		// Original client entry was handled above
-		if (partialLength.clientId !== oc.data.clientId) {
-			this.addClientSeqNumber(oc.data.clientId, partialLength.seq, oc.data.seglen);
-		}
-		return true;
-	});
-}
-```
-
-The other interesting case to go through is when an obliterate/move conflicts with another obliterate/move.
-
-##### Review of overlapping removal
-
-This section illustrates the basic existing handling for overlapping removal.
-It can probably be skipped by readers familiar with the scheme, but is here to help the reader determine where assumptions may break down or go wrong
-for overlapping obliterate/move.
-
-Overlapping removal of a segment is tracked using the `removedClientIds` field, which is used in partial lengths to add adjustment entries to avoid double-counting
-removal.
-For example, suppose client 1 and client 2 concurrently remove the range `[0, 5)` and each performs some more ops before acking the others' remove.
-That might look something like this:
-
-```
-// Initial state at seq 0: "0123456789"
-{ seq: 1, refSeq: 0, clientId: 1, op: <remove [0, 5)> }
-{ seq: 2, refSeq: 0, clientId: 2, op: <remove [0, 5)> }
-{ seq: 3, refSeq: 0, clientId: 2, op: <insert "hi" at 2> }
-```
-
-The correct final state is "56hi789". Consider what happens when a listener client (say, client 0) attempts to interpret the insertion of "hi" by client 2.
-Before processing, its merge tree segment state would look like so:
-
-```
-[
-  { seq: 0, removedSeq: 1, removedClientIds: [1, 2], text: "01234" },
-  { seq: 0, text: "56789" }
-]
-```
-
-The constructed partial lengths object for the root of the merge tree would then be:
-
-```
-{
-  minLength: 10,
-  partialLengths: [{ seq: 1, seglen: -5 }],
-  clientSeqNumbers: [[], /* client 1 */[{ seq: 1, seglen: -5 }], /* client 2 */[{ seq: 1, seglen: -5 }]]
-}
-```
-
-Note that client 2's delta applies from seq 1 onward rather than seq 2, since it's constructed using the `seq` and `removedClientIds` on the removed segment.
-
-Client 0 would determine where to insert the op with seq 3 by:
-
-1. Asking the root for its length at `{ clientId: 2, refSeq: 0 }`
-
--   This calculation is based on (length at min seq) + (any deltas between minSeq and refSeq) + (any deltas for ops submitted by client 2) - (deltas submitted by client 2 before refSeq)
--   From the above bookkeeping, it would compute 10 + 0 + (-5) - 0 = 5
-
-2. Asking for the length of the first child at `{ clientId: 2, refSeq: 0 }`
-
--   Conditionals here are a bit tedious, but we'd see that clientId 2 is in the segment's removedClientId list, so it has length 0
-
-3. Asking for the length of the second child `{ clientId: 2, refSeq: 0 }`
-
--   The segment is inserted and not removed, so it has length 5.
-    Since the search is looking for an accumulated position of 2, it determines that the correct insertion point is amidst this segment.
-
-##### Overlapping obliterate
-
-The same general strategy used for overlapping removes should be sufficient for tracking overlapping obliteration of segments.
-It relies only on information about when and by who a segment was removed, and the main difference between remove and obliterate comes
-from which segments they affect rather than how the segments are affected.
-
-Note also that because `movedSeq` is distinct from `removedSeq`, the corresponding partial lengths entry for `movedClientIds[0]` obliterating the segment can be entered
-distinctly from the partial lengths entry for `removedClientIds[0]` removing the segment.
-
-Again, the interesting case to check is if two separate clients issue obliterate ops amidst a concurrent insert (otherwise it is functionally identical to the remove case).
-
-```
-// Initial state at seq 0: "0123456789"
-{ seq: 1, refSeq: 0, clientId: 1, op: <obliterate [0, 5)> }
-{ seq: 2, refSeq: 0, clientId: 2, op: <obliterate [0, 5)> }
-{ seq: 3, refSeq: 0, clientId: 3, op: <insert "hi" at 2> }
-```
-
-The segment state of some observing client after seq 3 is essentially the same as in the non-overlapping example:
-
-```
-[
-  { seq: 0, movedSeq: 1, text: "01", movedClientIds: [1, 2] },
-  { seq: 3, movedSeq: 1, clientId: 3, text: "hi", movedClientIds: [1, 2] },
-  { seq: 0, movedSeq: 1, text: "234", movedClientIds: [1, 2] },
-  { seq: 0, text: "56789" }
-]
-```
-
-From the observing client perspective, the interpretation of each client's text if they were to submit an op with refSeq 0 through 3 is as follows:
-
-| refSeq | client 1 | client 2 | client 3 |
-| 0 | 56789 | 56789 | 01hi23456789 |
-| 1 | 56789 | 56789 | 56789 |
-| 1 | 56789 | 56789 | 56789 |
-| 1 | 56789 | 56789 | 56789 |
-
-The corresponding lengths table is exactly what's achieved by combining the overlapping remove strategy with the strategy for bookkeeping concurrently inserted segments:
-
-```
-{
-  minLength: 10,
-  partialLengths: [{ seq: 1, seglen: -5 }, { seq: 3, seglen: 0 }],
-  clientSeqNumbers: [
-    [],
-    [{ seq: 1, seglen: -5 }],
-    [{ seq: 1, seglen: -5 }], /* comes from adding clientSeqNumber to all entries in removedClientIds */
-    [{ seq: 1, seglen: 2 }]   /* comes from the inserted "hi" segment which has movedSeq <= seq */
-  ]
-}
-```
-
-This approach works if the operations are sequenced in the other order or intermediately as well.
-
-## Endpoint Behavior
-
-One important consideration is what happens near the endpoints of the removed range.
-There are two general possibilities: either the obliterate expands to include segments inserted
-adjacent to the endpoint, or it doesn't.
-
-In the initial implementation, we should pick some fixed endpoint behavior analogous to how insertion merge policy is fixed.
-Each option is roughly equivalent in difficulty.
-
-If applications request more degrees of freedom in this area, the framework for merge outcomes described in [Move](##Move) is a good starting point.
-
-## Public API
-
-The public API of sequence will need to be updated for users to leverage the obliterate operation. The most obvious way to extend it would be to align the API shape with
-`removeRange`:
-
-```typescript
-class SharedSegmentSequence<TInterval extends IInterval> {
-	public obliterateRange(start: number, end: number);
-}
-```
-
-One interesting alternative is to align the public API of sequence with the idea that there are two conceptual kinds of ranges: slice ranges and set ranges (see the next section
-for details).
-If we did this, we might instead unify `removeRange` and `obliterateRange` into a single method taking in such a range object.
-This would have the nice property of naturally extending to annotate operations, if we anticipate wanting to be able to annotate slice ranges.
-
-## Move
-
-There are several different possible options for defining merge outcomes for the "move" operation.
-The upcoming SharedTree DDS has done a lot of thinking in this area and landed on a relatively simple set of semantics that give reasonable
-outcomes in most cases (see [issue 9658](https://github.com/microsoft/FluidFramework/issues/9658) for some very detailed reading).
-
-These semantics are implementable in merge-tree, are compatible with feature requests for obliterate, and generally seem like a good direction to take
-that we can later extend if applications request.
-
-There are a few primitive concepts that all of the merge outcomes depend on.
-
-First, a sequence of length `N` is conceptualized as an interleaving set of `N+1` _gaps_ and `N` nodes.
-Nodes in the sequence may move, but the gaps between the nodes do not.
-
-Insertion into the sequence is performed by specifying a gap to insert in as well as a direction that the inserted content prefers to tend toward
-in case other content is inserted/moved concurrently into the same gap.
-
-> Merge-tree already conceptualizes insert locations similarly: it names the gaps `0` through `N`. It does not permit app-level specification of concurrent merges,
-> but that degree of freedom doesn't need to be exposed.
-
-Next, there are two types of range specifications: _set ranges_ and _slice ranges_.
-
-A _set range_ targets exactly the objects in a given range at the time it was specified. In merge-tree terms, the segments that the range affects are
-resolved from the perspective of the submitting client at its refSeq, and only those segments undergo whatever operation applies (move, annotate, remove).
-
-> Merge-tree's `remove` operation has set range semantics, since it doesn't cause removal of any concurrently inserted segments.
-> It's worth noting that a move operation with set range semantics is conceivable inside this framework, and not something merge-tree currently implements.
-> E.g., if the set range "CDE" inside a string "ABCDEF" was moved to the end of the string, and someone concurrently moved "B" and "C" to the start,
-> the string may end up "BAFCDE" or "BCAFDE" depending on the sequencing order of the moves.
-
-Finally, a _slice range_ specifies a start location and an end location, where a location has the same object shape as an insert destination: a gap plus a merge direction.
-The range of nodes that the operation affects is interpreted at the time the operation applies, and any concurrent insertions/moves of content _into_ that range
-are also affected. The merge direction should be interpreted as relative to a "phantom segment" in the gap specifying the slice endpoint.
-For example, in the string "ABCDE", the slice range
-`[{ pos: 0, merge: <concurrent segments merge nearer> }, { pos: 3, merge: <concurrent segments merge further> })` referring to "ABC" would
-not expand at either endpoint whereas if the merge options were flipped, it would expand at both endpoints.
-
-> These semantics align with the proposed merge-tree `move` operation. Like insert, we can fix the direction things should merge
-> (in this case it instead affects which way the move "expands") if consumers don't need the extra degrees of freedom.
-
-Notice that because gaps don't move, this set of outcomes doesn't suffer from problems like a range specification becoming invalid (which happens with
-how the legacy shared-tree assigns semantics to its ops, where each is relative to an id).
-It also gives reasonable merge outcomes which basically amount to "first move wins."
-Consider the following two troublesome cases of overlapping move.
-
-#### Move within a move
-
-```
-// Initial state: "12345 AB CD"
-{ seq: 1, refSeq: 0, clientId: 1, op: <move 2 through 4 to after "A"> } // (all of the op specification would actually be in terms of indices)
-{ seq: 2, refSeq: 0, clientId: 2, op: <move 3 to after paragraph "C"> }
-```
-
-One can see with this order of sequencing, we'd end up with "15 A234B CD".
-With the other order, we'd get "15 A24B C3D".
-
-Both outcomes are reasonable; clients 1 and 2 effectively expressed opposing desires on where the 3 should go.
-
-#### Move of a single endpoint past the other
-
-```
-// Initial state: "Paragraph 1<br>Paragraph 2<br>Paragraph 3<br>Paragraph 4<br>Paragraph 5"
-{ seq: 1, refSeq: 0, clientId: 1, op: <move paragraphs 2 through 3 to the gap after paragraph 5> } // (all of the op specification would actually be in terms of indices)
-{ seq: 2, refSeq: 0, clientId: 2, op: <move paragraphs 3 through 4 to the gap after paragraph 5> }
-```
-
-Client 1's op succeeds without conflict, giving intermediate state order of the paragraphs "14523".
-Then client 2's op has a start endpoint targetting a tombstoned segment for paragraph 3, so it only affects paragraph 4.
-The final state is "15423" since merge-tree chose near-merge-later.
-
-If the ops are sequenced in the other order, the final state would instead be "15234".
-
-Both of these outcomes are again generally plausible.

package/docs/REFERENCEPOSITIONS.md

@@ -1,199 +0,0 @@
-# ReferencePosition Documentation
-
-ReferencePositions are used to indicates a MergeTree position which is stable as operations are performed. There are two
-types:
-
-1. LocalReferences refer to a segment and offset within that segment
-2. Markers are actual segments in the Merge Tree
-
-The function `Client.localReferencePositionToPosition` returns the numerical position of a reference in the client's
-current view.
-
-## LocalReference behavior on Remove
-
-By default, LocalReferences become detached when the segment they reference is removed.
-The ReferenceTypes SlideOnRemove, StayOnRemove, and Transient change this behavior.
-They are only valid for LocalReferences.
-They are exclusive - a reference may be at most one of these types.
-
-### SlideOnRemove
-
-The reference will slide to the next farthest segment when the segment is removed and the remove has been acknowledged.
-Sliding will look for the next valid segment.
-A valid segment is one whose creation has been acknowledged and either hasn't been removed
-or the remove is pending (not acknowledged).
-If a farther segment is found, then the LocalReference will be changed to refer to that segment and have offset 0.
-In the event that the slide is happening on the acknowledgement of a remove, the slide to a farther segment will not
-change the numerical position of the reference.
-If there is no there is no valid segment farther in the tree, then the slide will place the reference on the last valid segment.
-The offset will be set to the last position in that segment.
-In the event that the slide is happening on the acknowledgement of a remove, the reference would have been on the removed
-segment. This slide from the removed segment to a nearer segment does change the numerical position of the reference.
-If there is no valid position (all segments removed and acknowledged) then the reference is detached.
-
-### StayOnRemove
-
-The reference will stay on removed segments.
-This behavior is only defined until the removed segment is cleaned up by Zamboni.
-This is intended to be used only while collaborating (see below) while waiting for an acknowledgement.
-
-### Transient
-
-The reference is not tracked by the MergeTree.
-It will continue to reference removed segments.
-This behavior is only defined until the removed segment is cleaned up by Zamboni.
-This is intended to be used to create transient references which may be compared with other references.
-
-### Detached LocalReferences
-
-A detached LocalReference does not reference a segment in the MergeTree.
-It's position is defined to be `LocalReference.DetachedPosition` (-1).
-
-### LocalReferences on Removed Segments
-
-LocalReferences may reference removed segments:
-
--   SlideOnRemove references may reference a removed segment which is pending (not acknowledged)
--   StayOnRemove references may reference removed segments
--   Transient references may reference removed segments
-
-The numerical position of a reference which is on a removed segment will be one more than the previous (nearer) segment.
-If there is a farther segment that is not removed, this will be the same as the position of the start of that segment.
-If there is no farther segment, then the reference position will be the length of the tree (one more than the last valid
-position in the tree).
-
-## Eventually Consistent References
-
-Markers are segments in the MergeTree and are eventually consistent.
-LocalReferences may be used as part of an eventually consistent feature.
-For example, SharedIntervals are built using LocalReferences.
-
-### Implementing Eventually Consistent LocalReferences
-
-To implement an operation which creates LocalReferences which will have an eventually consistent position:
-
-1. Locally create the reference as StayOnRemove
-2. Send the reference numerical position in an op
-3. On acknowledgement of the local create:
-    1. set the `refType` of the reference to include `SlideOnRemove`
-    2. call `Client.getSlideToSegment` with the references current segment and offset to get the proper new location
-    3. Delete the old reference and create a new one with the returned values
-4. Remote clients, on receiving the op, call `Client.getContainingSegment` followed by `Client.getSlideToSegment`
-   on the result. Call `Client.createLocalReferencePosition` with the result to create a `SlideOnRemove` reference.
-5. If there is a dependency on the comparison of reference positions (such as the index in IntervalCollections)
-   must listen to the `beforeSlide` and `afterSlide` events on `IReferencePositionEvents`. When slide occurs the
-   relative position of references may have changed.
-
-### Implementation Notes
-
-This is the state diagram for the implementation of Eventually Consistent References.
-
-```mermaid
-flowchart LR
-    subgraph StayOnRemove
-    localCreate[Local Create Ref]
-    pendingRef(("Ref:StayOnRemove\nSegment:Pending|Normal"))
-    pendingRefPendingRemove((Ref:StayOnRemove\nSegment:Pending Remove))
-    pendingRefRemoved((Ref:StayOnRemove\nSegment:Removed))
-    localCreate-->pendingRef
-    pendingRef--local remove-->pendingRefPendingRemove
-    pendingRefPendingRemove--remote remove-->pendingRefRemoved
-    pendingRef--remote remove-->pendingRefRemoved
-    end
-
-    subgraph SlideOnRemove
-    remoteCreate[Remote Create Ref]
-    remoteChoice{Segment Removed?}
-    ref((Ref:SlideOnRemove\nSegment:Normal))
-    refPendingRemove((Ref:SlideOnRemove\nSegment:Pending Remove))
-
-    remoteCreate-->remoteChoice
-    remoteChoice--no-->ref
-    remoteChoice--locally-->refPendingRemove
-    ref--local remove-->refPendingRemove
-    end
-
-    slide{Slide To}
-    detached((Ref:Detached))
-
-    pendingRef--create ack-->ref
-    pendingRefPendingRemove--create ack-->refPendingRemove
-    pendingRefRemoved--create ack-->slide
-    remoteChoice--yes-->slide
-    ref--remote remove-->slide
-    refPendingRemove--remove ack-->slide
-    refPendingRemove--remote remove-->slide
-    slide--segment-->ref
-    slide--locally removed segment-->refPendingRemove
-    slide--no segment-->detached
-```
-
-This algorithm works because it ensures that slid reference slide to the same segment.
-The slide only happens when both the creation of the reference and removal of the segment have been acknowledged.
-When sliding we do not consider any local (unacknowledged) ops.
-
-Keeping references on removed segments until they can be slid works well in most cases because of these properties:
-
-1. Interval positions on removed segments appear as if they were on the following position in the string.
-   If the removed segment is between positions 5 and 6, the interval positions on the removed segment appear to be at
-   position 6. This matches where they will eventually slide, so slide will not cause a change in position as long as
-   segments are not slid over and it is not necessary to slide to the near end of the string.
-2. Text inserted at the same location as the removed segment is inserted before the removed segment.
-   So if the removed segment is between 0 and 1 (“A[removed]B”), insertText(1, “X”) inserts before the removed segment
-   (“AX[removed]B”). This makes it hard to end up with local only segments to be slid over, which will mean it is rare
-   that slide visibly changes the interval position. It can still happen if there is a conflicting remove, but that is
-   much less likely.
-
-#### Conflict Scenarios
-
-Considering Create Interval / Remove Range conflicts, here are the scenarios
-(before indicates the relative sequence order):
-
-1. Local create before local remove. Interval position needs to slide on ack of the local remove.
-2. Remote create before remote remove. Slide on receiving the remove.
-3. Local remove before local create. This is impossible – once the segment is removed locally an
-   interval position can’t be created on it.
-4. Remote remove before remote create. (Possible if ops are from different remote clients).
-   Slide on receiving the remote create.
-5. Remote create before local remove. Slide on the ack of the local remove.
-6. Local create before remote remove. Slide on receiving the remove.
-7. Local remove before remote create. Slide on receiving the create.
-8. Remote remove before local create. Slide on receiving the ack of the create.
-
-### Why Eventually Consistent References Can Not Have Stable Order
-
-In an ideal system reference positions would have stable order. Specifically:
-
-1. If in any client state the position of a reference is less than the position of a specific item in the sequence,
-   then the position of that reference would always be less than or equal to the position of that item.
-2. If in any client state the position of reference A is less than the position of reference B,
-   then the position of reference A would always be less than or equal to the position of reference B.
-
-Neither of these properties is true for SlideOnRemove references. This is a result of them sliding over local
-only segments. This could change the relative positions of the sliding reference at items that are slide over,
-as well as any references on those items. Note that these properties do hold for items and references
-once the creation has been acknowledged (sequenced by the server).
-
-Supporting stable order is not possible in the current system because:
-
-1. Removing a range may cause an multiple references that had been at different positions to all be at the same
-   position.
-2. To preserve stable ordering, an insert that conflicts with that remove would need to be after some of those
-   references and before others.
-3. Insertion position is specified as a numerical offset in the sequence, so can't specify where in the set
-   of references at a position to be inserted. (Technically there is enough information to do this within the
-   collab window. But that information is lost if reconnect/resubmit is required.)
-
-Therefore implementing eventually consistent references with stable order would require adding additional
-information to insert ops.
-
-## Tests
-
--   `packages\dds\merge-tree\src\test\client.localReference.spec.ts`
-    unit tests for LocalReferences
--   `packages\dds\sequence\src\test\intervalCollection.spec.ts`
-    test LocalReferences as used in interval collections (including eventual consistency)
--   `packages\test\test-end-to-end-tests\src\test\sharedInterval.spec.ts`
-    end-to-end tests using LocalReferences for interval collections.
-    These tests have only been minimally updated to reflect this implementation,
-    so they do not comprehensively test LocalReferences.