@fluidframework/tree 2.0.0-dev-rc.1.0.0.224419

package/docs/main/cell-model-of-collaborative-editing.md

@@ -0,0 +1,225 @@
+# The Cell Model of Collaborative Editing
+
+This document presents a conceptual model that provides a foundation upon which the semantics of SharedTree editing can be understood.
+
+The model itself is not tied to SharedTree or SharedTree's data model.
+It is instead concerned with providing an abstraction for what it means to edit a document.
+
+The model only concerns itself with the editing of a single field.
+More complex structures (notably, trees) can be constructed by allowing the contents of the field to represent more fields.
+
+## Summary
+
+Previously we were trying to build the notion of tree locations (anchors) on top of nodes that are moving and popping in/out of existence.
+
+It's simpler to start with a foundation of tree locations (cells) that never move/disappear, and then layer the idea of moveable/removable nodes on top of that.
+
+The impedance mismatch between our internal model and the semantics desired by users was making things unnecessarily complicated: forcing the internal model to be excessively general, and then requiring higher layers to compensate.
+The primary example being the use of constraints and/or hierarchical edits to constrain the semantics of insert/remove to fit the desired semantics of setting a fixed-sized field.
+
+## Motivation
+
+We present here an example that portrays the challenges encountered by editing models based on insertion and removal.
+
+Consider a collaborative editing session between three participants (all acting concurrently), over a field populated with elements [A, B, C, D], where the edits are sequenced as follows:
+
+1. User 1: remove B, C (local state: [A, D])
+
+2. User 2: insert Y after C (local state: [A, B, C, Y, D])
+
+3. User 3: insert X after B (local state: [A, B, X, C, D])
+
+4. User 1: undo the removal of B, C (local state: [A, B, C, D])
+
+The outcome we expect from such a session is [A, B, X, C, Y, D].
+
+Traditional models based on insertion and removal may fail to produce this outcome in two ways:
+
+-   The order of X relative to Y may not be correct (i.e., Y may appear before X in the list of elements).
+
+-   The order the inserted element relative to the temporarily removed elements may not be correct (yielding either [A, X, Y, B, C, D] or [A, B, C, X, Y, D]).
+
+The first failure stems from the fact that, as the insertion of X and Y are made relative to removed content, information is lost as to their precise insertion point.
+
+The second failure stems from the fact that the undo of the removal of B, C by user 1 is modeled as an insertion.
+
+This second failure can also occur when elements are moved-out and the move is then undone (and the undo is modeled as a move in the opposite direction).
+
+These failures can be addressed by introducing new features:
+
+-   Tombstones, which represent removed or moved-out elements
+
+-   The "revive" operation which restores elements in the place of their tombstones
+
+-   The "return" operation which moves elements back in the place of their tombstone.
+
+These additions, while effective, have the following drawbacks:
+
+-   They increase the complexity of the core editing model
+
+-   They lead to a set of operations that is not orthogonalized:
+
+    -   Insert, revive, and return seem to share some common traits, with revive and return being closer to each other.
+
+    -   It seems like it should be possible to support a "replace" operation that is similar to revive but with new elements instead of resurrecting removed ones, but that too needs to be added as separate operation)
+
+*   They do not alleviate the complexity of modeling an optional field (clearing the field requires a slice-delete operation over the field, overwriting the field requires that same slice-delete and an insert whose exact position is meaningless).
+
+The cell model can be thought of as a refactoring of both the traditional editing primitives and the above additions such that the listed drawbacks are avoided.
+
+## The Model
+
+### Fields
+
+In this model, a field is a possibly empty sequence of cells.
+The sequence makes cells totally ordered.
+It is this ordering which serves as the foundation upon which the appropriate ordering of content is ensured.
+
+### Cells
+
+In this model, a cell is a unit of storage.
+A cell cannot be subdivided.
+It may either be empty or full.
+The contents of a (full) cell may be arbitrarily large or small (though a specific data model may constrain this in practice).
+
+> Note: the model does not prescribe for the cells to be explicitly reified.
+> While some cells may be represented at runtime some of the time, they are primarily a conceptual artifact.
+> An application would not have explicit access to the cell, and is likely not aware of the concept in the first place.
+
+A cell may be annotated with forwarding information that specifies a target destination cell.
+A single cell may bear any number of such forwarding annotations.
+This forwarding information is used to represent move information: the source cell is annotated with forwarding information that points to the destination cell.
+
+In addition to their storage role, cells act as markers relative to which an edit may be performed (e.g., "Before/after cell foo").
+
+### Operations
+
+The model admits the following operations:
+
+> WIP: An alternative set of operations based on hiding cells instead of clearing them is being considered.
+> The general structure of the model is unaffected.
+
+-   <u>Allocate</u> a new cell at a specific location in the sequence
+
+-   <u>Fill</u> a cell with content (overwriting existing content if any)
+
+-   <u>Clear</u> content from a cell, leaving it empty
+
+-   <u>Add a forwarding annotation</u> to a cell
+
+-   <u>Remove a forwarding annotation</u> from a cell
+
+Note that cells cannot be deallocated (or moved).
+This reflects the fact that a client may perform an edit relative to some content's location at a point in time, even if that content has since been deleted or moved: the client is performing the edit relative to the cell which contained that content.
+
+## Building on the Model
+
+The model can be used to describe low-level edits that a client may perform on a field in a collaborative environment and by so doing imply some of their merge semantics.
+
+We provide here two lists of such low-level edits, one for fields whose number of cells is fixed, and one for fields whose number of cells is dynamic.
+
+Note that the fixed and dynamic characteristics here apply only to the number of cells, not to the number of elements.
+For example, an optional field, which can contain zero or one element, is a fixed-sized field composed of exactly one cell.
+That cell exists even when the field is not populated with an element.
+
+The dichotomy has no basis in the cell model but reflects what we think is a sensible separation and matches industry-standard patterns of editing.
+
+### Edits on Dynamically-Sized Fields
+
+Dynamically-sized fields can be thought of as having list-like behavior: elements can be added anywhere in the list and removed at will which makes the list contents dynamically grow and shrink (while the number of cells in the list only grows).
+Dynamically-sized fields typically start out empty of any cell.
+
+The low-level edits that such fields might support can be decomposed as follows:
+
+-   Insert: allocate and fill a cell
+
+-   Delete/Remove: clear a cell
+
+-   Revive: fill a (cleared) cell with the contents it contained before
+
+-   Replace: fill a (filled) cell
+
+-   Move content from A to B:
+
+    -   A: clear a (filled) cell and add a forwarding annotation to it
+
+    -   B: allocate and fill a cell with the content being moved
+
+-   Return content to A from B:
+
+    -   A: remove the forwarding annotation from a cell and fill it with the returned content
+
+    -   B: clear the (filled) cell
+
+### Edits on Fixed-Sized Fields
+
+Fixed-sized fields can be used to model required (i.e., unary) fields, optional fields, fixed-sized arrays (not to be confused with JavaScript arrays which behave like dynamically-sized lists), and tuples.
+Fixed-sized fields are populated with their cells (and possibly content) from the time they are created.
+
+The low-level edits that such fields might support can be decomposed as follows:
+
+-   Upsert: fill a (potentially empty) cell
+
+-   Populate if empty: fill a cell if the cell is empty
+
+-   Load-linked store: fill a cell if the cell has not been concurrently filled or cleared
+
+-   Clear: clear a cell
+
+-   Move content from A to B:
+
+    -   A: clear a (filled) cell
+
+    -   B: fill a cell with the content being moved
+
+-   Return content to A from B:
+
+    -   A: fill a cell with the content being moved
+
+    -   B: clear the (filled) cell
+
+The Move/Return operations described here assume both the source and the destination of the move are in the same kind of trait. This is not a requirement, though some combinations may be questionable.
+
+Note that no further cell allocations are made after the field is created.
+
+## Implications For SharedTree
+
+### Data Model
+
+The SharedTree data model should differentiate between fixed-sized and dynamically-sized fields.
+Users of SharedTree, when writing a schema, implicitly choose whether to use fixed-size or dynamically-sized field.
+This is not a new choice (they're still choosing from the same set of options) but their choice is now reflected at the data model level.
+
+### Editing API
+
+New editing operations need to be exposed for fixed-sized fields.
+
+The editing operations offered by SharedTree, even for unschematized data, need not be the ones prescribed by the model.
+Instead, we see value in offering two sets of higher-level operations: one for fixed-sized fields and one for dynamically-sized fields.
+This allows the editing API to be more specialized for each kind of field, which in turn allows the API to be more expressive and more familiar.
+
+### Merge Semantics
+
+The merge resolution logic can now exclude the possibility of having to merge operations from both sets within a single field (outside of schema migration scenarios).
+
+Since the merge resolution logic is specialized to each field kind, merge resolution logic becomes extensible: we can introduce new kind of fields with associated merge logic.
+This allows us to support new kinds of fields in the future, but more importantly it means that we can migrate from a less desirable set of merge semantics to a more desirable one.
+This dramatically reduces the negative impact of starting out with suboptimal merge semantics in early SharedTree releases because we can implement better ones later on and offer them as new kinds of fields.
+
+### Changeset Format
+
+The changeset format should not be forced to adopt the primitive operations defined by the model as this would likely lead to bloat (such as specifying both "allocate" and "fill" for the insert edit).
+
+The changeset format may however be constructed in such a way that it could be translated to the primitive operations defined by the model.
+
+The changeset format also needs to represent the new kind of edits supported by fixed-sized fields.
+
+### Change Application Logic
+
+While there is no imperative to do so, the change application logic of SharedTree, as an implementation detail, may be factored in a way that resembles the primitive operations defined by the format.
+
+### Cost of Tombstone Data
+
+By allowing some usage patterns to leverage fixed-sized fields, we are able to offer adequate merge semantics without incurring the cost of managing a potentially unbounded number of tombstones.
+This is particularly welcome because we conjecture that fixed-sized fields are more likely to see a lot of repeated overwrites.
+Dynamically sized fields typically contain more data, but the amount of deleted content tends to remain proportional to the amount of tip-state content.

package/docs/main/compatibility.md

@@ -0,0 +1,149 @@
+# Compatibility
+
+This document provides concrete guidelines and strategies for organizing code that impacts SharedTree's persisted format.
+
+## Prerequisites
+
+[This document](../../../../../packages/dds/SchemaVersioning.md) provides general "best practices" for working with persisted data within the Fluid Framework.
+It's strongly recommended to read through and understand its rationale before continuing with this document,
+as most of the concrete recommendations presented henceforth fall out of those best practices.
+
+## What State is Persisted?
+
+A DDS's persisted format encompasses the format it uses for its summaries as well as its ops (due to [trailing ops](../../../README.md))
+including transitively referenced structured blob data.
+
+Since documents are stored outside of Fluid control (i.e. no type of central data migration is possible),
+DDSes necessarily commit to backwards compatibility of their format for all time.
+
+## Format Management
+
+The persisted format version should be a configuration option an application author can specify using `SharedTreeFactory`:
+this ensures applications can control rollout of configuration changes which require code saturation of some prior version.
+It also empowers the container author (rather than the host--if they differ) to control their data model.
+
+In the SharedTree MVP, there is currently no mechanism for safely changing the persisted format version.
+However, it is feasible to add such a mechanism in the future, and specifying the persisted format explicity in configuration sets us up to easily do so.
+One example of prior art in the space is `@fluid-experimental/tree`'s [format-breaking migration strategy](../../../tree/docs/Breaking-Change-Migration.md),
+though we would likely want to make the mechanism usable across the Fluid Framework.
+
+## Code Organization
+
+Each part of SharedTree which contributes to the persisted format should define:
+
+1. Types defining the _in-memory format_ needed to load or work with its data
+1. A set of versioned _persisted formats_ which encompass all supported formats in the current and past.
+1. EITHER An `IJsonCodec` capable of transcoding between the in-memory format and a particular persisted format, OR an `ICodecFamily` of such `IJsonCodec`s (for different persisted format versions)
+
+Split the above components into files as is reasonable.
+Current organizational standards put:
+
+-   The in-memory format into a file ending in `Types.ts`
+-   The persisted format into a file ending in `Format.ts`
+-   Codecs into a file ending in `Codec.ts` / `Codecs.ts`
+
+Having consistent conventions for these files helps to make changes to persisted formats obvious at review time.
+Schemas for primitive types which are used in persisted formats but don't intrinsically define formats (such as branded strings) can be defined where convenient.
+Codec logic should generally be self-contained: all imports should either be of the form `import type`, or should import from another persisted format file.
+Importing Fluid Framework libraries that have the same guarantees (e.g. `SummaryTreeBuilder`) is also acceptable.
+Codecs should expose the minimal necessary set of types.
+Encoding should take care to only include necessary object properties. **In particular, avoid constructs like object spread**.
+Decoding should validate that the data is not malformed: see [encoding validation](#encoding-validation) for more details.
+
+With the exception of primitives, storage format types should never be exposed in the public API.
+
+> Note: due to API-extractor implementation details, the typebox schemas for primitive types _cannot_ share a name with the primitive type,
+> as it exposes _both_ the value and the type exported under the same name, even if the export is specified via `export type`.
+> For example, the typebox schema for `ChangesetLocalId` is named `ChangesetLocalIdSchema`.
+
+Using this structure, SharedTree will have access to a library of codecs capable of encoding/decoding between
+the in-memory format and some persisted format.
+
+## Encoding Validation
+
+Validating that an encoded format thoroughly matches what the code expects and failing fast otherwise is valuable to reduce the risk of data corruption:
+it's a lot easier to investigate and deploy fixes for documents when incompatible clients haven't also changed the contents of a file.
+
+For this reason, encoded data formats should declare JSON schema for the purpose of runtime validation.
+
+> In the near term, we're using [typebox](https://github.com/sinclairzx81/typebox) to declare these schemas.
+> This choice is a matter of convenience: its API naturally matches the expressiveness of typescript types.
+
+Since format validation does incur runtime and bundle-size cost to obtain additional safety,
+whether or not to perform it should ultimately be left as a policy choice to the user of shared-tree.
+This choice will probably also be made in the shared-tree factory by providing a `JsonValidator`,
+but it doesn't need to be persisted and can be changed at will
+(there's no issue with collaboration between clients that have different policies around how much
+of the persisted data should be validated).
+
+An out-of-the-box implementation of `JsonValidator` based on Typebox's JSON validator is provided,
+but application authors may feel free to implement their own.
+
+## Test Strategy
+
+This section covers types of tests to include when adding new persisted configuration.
+
+There are a couple different dimensions to consider with respect to testing:
+
+1. SharedTree works correctly for all configurations it can be initialized in when collaborating with SharedTrees with similar configuration
+1. SharedTree is compatible with clients using different source code versions of SharedTree (and the documents those clients may create)
+1. Once supported, SharedTree can correctly execute document upgrade processes (changes to persisted configuration such as write format)
+
+### Configuration Unit Tests
+
+Each codec family should contain a suite of unit tests which verify the in-memory representation can be round-tripped through encoding and decoding.
+When adding a new codec version, the test data for this suite should be augmented if existing data doesn't yield 100% code coverage on the new
+codec version.
+
+If the persisted configuration impacts more than just the data encoding step,
+appropriate unit tests should be added for whatever components that configuration impacts.
+As a simple example, a persisted configuration flag which controls whether SharedTree stores attribution information
+should have unit tests which verify processing ops of various sorts yield reasonable attribution on the parts of the tree they affect.
+
+Example: [experimental/dds/tree2/src/test/feature-libraries/editManagerCodecs.spec.ts](../../src/test/feature-libraries/editManagerCodecs.spec.ts)
+
+### Multiple-configuration Functional Tests
+
+Once SharedTree supports multiple persisted formats, we should modify a small set of functional acceptance tests
+(e.g. `sharedTree.spec.ts`) to run for larger sets of configurations.
+Using `generatePairwiseOptions` will help mitigate the combinatorial explosion concern.
+
+These tests in aggregate will verify that SharedTree works when initialized with some particular configuration
+and collaborates with other SharedTree instances initialized with the same configuration.
+They would reasonably detect basic defects in codecs or problems unrelated to backwards compatibility or any upgrade process.
+
+In the same vein, fuzz tests should cover a variety of valid configurations.
+
+### Snapshot Tests
+
+The last dimension of compatibility concerns direct or indirect collaboration between clients using different versions of SharedTree source code.
+This is a vast area that could use more well-established framework testing support, but snapshot testing is a relatively effective category for
+catching regressions.
+
+The idea behind snapshot testing is to verify a document produced using one version of the code is still usable using another version of the code.
+It's typically implemented by writing some code to generate a set of fixed documents "from scratch," then source-controlling the serialized form
+of those documents after summarization.
+Since the serialized form of the documents correspond to documents produced by an older version of the code, this enables writing a test suite that verifies:
+
+1. The current version of the code serializes each document to exactly match how the older version of the code serialized each document.
+1. The current version of the code is capable of loading documents written using older versions of the code.
+
+A few examples (which may not be exhaustive) of snapshot tests are:
+
+-   [Legacy SharedTree](../../../../../experimental/dds/tree/src/test/Summary.tests.ts)
+-   [Sequence / SharedString](../../../sequence/src/test/snapshotVersion.spec.ts)
+-   [e2e Snapshot tests](../../../../test/snapshots/README.md)
+
+The first two examples generate their "from scratch" documents by directly calling DDS APIs on a newly created document.
+The e2e snapshot tests accomplish "from scratch" generation by serializing the op stream alongside the snapshots and replaying it.
+In addition to verifying serialized states line up between old and current version of the code, it can also be helpful to
+verify equivalence at runtime, which typically gives more friendly error messages.
+
+> Aside: this approach is also a bit more flexible: it's possible that different snapshots can load to produce logically equivalent DDSes.
+> Matrix is an example of this: it uses a pair of permutation vectors mapping logical indices (i.e. "row 5, column 3") to in-memory indices for the contents of cells.
+> Thus, permuting both the permutation vectors and the cell data contained within a snapshot would not logically change its data.
+
+Snapshot tests are effective at catching changes which inadvertently modify the document format over time.
+
+Tree2's full-scale snapshot tests can be found at [experimental/dds/tree2/src/test/snapshots/summary.spec.ts](../../src/test/snapshots/summary.spec.ts),
+with smaller-scale snapshot tests (e.g. snapshot testing just the SchemaIndex format) nearby.

package/docs/main/data-model.md

@@ -0,0 +1,191 @@
+# Data Model
+
+## Introduction
+
+This document discusses how tree data is conceptually organized within the SharedTree DDS.
+It is primarily of interest to Fluid developers working on SharedTree.
+Most SharedTree users will interact with tree data through a pre-existing API layer.
+However, the data model may be of interest to advanced users who are implementing a specialized API (see [#8989](https://github.com/microsoft/FluidFramework/issues/8989)).
+
+The SharedTree data model is closely related to, but not precisely the same as:
+
+-   The internal in-memory representation of tree data
+-   The serialized tree data sent across the network
+-   The format used to persist tree data in storage
+
+While each of the above involve specific encodings of the SharedTree data model, the SharedTree data model itself is not concerned with the byte-level representation (see [#10034](https://github.com/microsoft/FluidFramework/issues/10034)).
+
+## Requirements
+
+### JSON Interoperability
+
+We take it as a given that the underlying data model for the SharedTree is tree-structured data.
+We also agree that JSON is the modern lingua franca of the web and services.
+We therefore begin with the following requirements:
+
+1. JSON can efficiently and losslessly be encoded by the underlying SharedTree data model, with a small number of possible caveats (see below).
+2. Deviations from JSON in the underlying SharedTree data model must be well justified.
+
+Consequently, we require that the underlying SharedTree data model can express the following in a natural way:
+
+-   Object-like records of named properties
+-   Array-like sequences of consecutive items (non-sparse)
+-   null, true/false, finite numbers (f64), and strings
+
+#### Potential JSON Caveats
+
+We currently know of two potential caveats regarding JSON interoperability.
+
+##### Arbitrary Precision Numbers
+
+As defined, the JSON specification allows numbers to be of arbitrary precision.
+Some systems honor this by parsing JSON into a representation that uses an alternative numeric encoding than IEEE 64b floating point format.
+However, JavaScript's built-in `JSON.parse()` does not.
+Therefore, we do not currently see a need to preserve arbitrary precision numbers in M1.
+
+##### Reserved Keys
+
+Projecting arbitrary SharedTree data to JSON requires tradeoffs due to the fact that the SharedTree data model is a superset of JSON.
+Specifically, the SharedTree data model includes special fields for ids and types.
+Preserving these special fields in the JSON encoding will require the use of reserved keys or a reserved key prefix.
+
+##### Primitives with Ids and Augmentations
+
+A second complication when projecting SharedTree data to JSON is that the JSON encoding does not support fields on primitive types, such as numbers and booleans.
+In order to preserve ids and augmentations on primitives, we will need to wrap the primitives in objects using the reserved type field to identify which objects represent encoded primitives.
+
+### Durable References
+
+The data model must support creating inexpensive durable references to nodes within the tree.
+This includes scenarios like creating "share link" URLs or building graph-like relationships within the tree.
+
+### Schema
+
+While the data model is schema agnostic, the data model must encode sufficient information that a system opting into a schema-on-write policy can efficiently implement schema validation on top of the underlying data model (see #9282).
+
+### Augmentation
+
+The data model must allow a subset of collaborators to unobtrusively attach extra-schema data to the tree in a way that is ignored (but preserved) by clients that are unaware of these augmentations.
+
+## Model
+
+### Node
+
+In the SharedTree data model, each addressable piece of data is represented as a tree **_node_**.
+There is an implicit node at the root of the SharedTree that serves as the initial insertion point for trees constructed by the application..
+
+<figure align="center">
+  <img src="../.attachments/root.drawio.svg" alt=""/>
+  <figcaption>Figure: Implicit root node</figcaption>
+</figure>
+
+The root node may not be moved/removed and has a well-known identity, but otherwise is indistinguishable from other tree nodes.
+
+### Value
+
+Each tree node has an optional **_value_**.
+Values are used to store scalar data, such as numbers and booleans.
+
+<figure align="center">
+  <img src="../.attachments/scalar-nodes.drawio.svg" alt=""/>
+  <figcaption>Figure: Nodes with values</figcaption>
+</figure>
+
+From the perspective of the SharedTree data model, values are [**_opaque_**](https://en.wikipedia.org/wiki/Opaque_data_type), immutable byte sequences.
+
+In practice, node values are Fluid [**_serializable_**](https://github.com/microsoft/FluidFramework/blob/main/packages/runtime/datastore-definitions/src/serializable.ts) types and can be interpreted as such using schema information.
+
+### Field
+
+Each tree node has a set of zero or more fields.
+Fields are used to model map-like composite types, where each **_field_** represents one relationship between a parent node and an ordered **_sequence_** of one or more children.
+
+<figure align="center">
+  <img src="../.attachments/children.drawio.svg" alt=""/>
+  <figcaption>Figure: A parent node with two fields</figcaption>
+</figure>
+
+The fields of a node are distinguished by a field **_key_**.
+From the data model's perspective, field keys are opaque byte sequences.
+In practice, dynamic keys are string literals and static keys are identifiers that correspond to a schema field declaration.
+
+### Sequences
+
+In the SharedTree data model abstraction, there is no distinction between a field that always contain a single child and a field that can contain multiple children.
+Instead, SharedTree treats all fields as ordered collections and uses schema to restrict which fields are optional, must contain a single value, or may contain multiple values.
+
+#### Implicit Sequences
+
+When mapping the SharedTree data model to conventional programming languages, it is sometimes helpful to think of the sequence as being distinct from the field, as if each field points to an **_implicit sequence_** object.
+
+<figure align="center">
+  <img src="../.attachments/implicit-sequence.drawio.svg" alt=""/>
+  <figcaption>Figure: Visualizing implicit sequence of 'key' field</figcaption>
+</figure>
+
+However, it is important to remember that sequences are not tree nodes.
+This means that sequences are not directly addressable.
+The combination of the field + sequences is implicitly created when the first item is inserted, implicitly deleted when the last item is removed, and may only be referenced indirectly via the combination of parent node + field key.
+
+#### Explicit Arrays
+
+Most languages represent sequences with an explicit array object. As opposed to an implicit sequence, these explicit array objects have identity, are directly addressable, and can be moved within the tree. To model explicit array objects in the SharedTree data model, one introduces a node between the parent and items.
+
+<figure align="center">
+  <img src="../.attachments/explicit-sequence.drawio.svg" alt=""/>
+  <figcaption>Figure: 'key' field points to an explicit array node</figcaption>
+</figure>
+
+### Special Fields
+
+This section covers fields that receive special treatment in the SharedTree data model.
+These fields are special because:
+
+-   They have reserved keys
+-   They are universally available on all nodes (regardless of schema)
+-   They can not be targeted by normal tree operations.
+
+#### Type
+
+The SharedTree data model optionally supports nominal typing via an optional special _type_ field.
+The value of the _type_ field is the unique identifier of the corresponding schema type.
+
+In the data model abstraction, the value of the type field is an opaque byte sequence.
+However, there is a set of well-known types (_boolean_, _number_, etc.) that are transparent to the underlying implementation.
+
+#### Id
+
+The SharedTree data model optionally supports durable references to tree nodes via a special _id_ field.
+When present, the _id_ field is used by SharedTree to provide a bidirectional _id_ ⟷ _node_ index that provides efficient lookup of nodes by their id.
+This is the underlying building block for features like "share link" URLs and graph-like references within the tree.
+Because there is a cost to maintaining with the _id_ ⟷ _node_ index, the application may choose which nodes are assigned ids.
+
+## JSON Comparison
+
+The below diagram highlights the differences between the JSON data model and the SharedTree data model using the following snippet:
+
+```json
+{
+	"visible": true,
+	"text": "cat", // String uses an implicit sequence
+	"dashStyle": [0.5, 3] // Array uses an explicit tree node
+}
+```
+
+<figure align="center">
+  <img src="../.attachments/data-model.drawio.svg" alt=""/>
+  <figcaption>Figure: JSON and SharedTree</figcaption>
+</figure>
+
+Of note:
+
+-   Scalar values are represented by tree nodes and consequently have a durable identity.
+-   The 'visible' field is a sequence, even though it is constrained by schema to only contain a single value.
+-   The 'text' field leverages its implicit sequence to represent the letters of "cat" as individual nodes, allowing the text to be collaboratively edited in the same way as array.
+-   The 'dashStyle' field points to an extra node that represents the array object, which in turn contains the array items.
+
+# Appendix A: Notes
+
+-   We chose the term 'field' because it is convenient to have a term that is distinct from the 'properties' that result when the domain model is projected to an API
+    ('field' also happens to align with GraphQL).
+-   We chose the term 'key' because it sounds more opaque than 'name' or 'label', both of which conjure the notion of something that is human readable.