@fluidframework/container-runtime 1.4.0-121020 → 2.0.0-dev-rc.1.0.0.225277

package/src/opLifecycle/README.md

@@ -0,0 +1,321 @@
+# Configs and feature gates for solving the 1MB limit.
+
+## Table of contents
+
+-   [Introduction](#introduction)
+    -   [How batching works](#how-batching-works)
+-   [Compression](#compression)
+-   [Grouped batching](#grouped-batching)
+    -   [Risks](#risks)
+-   [Chunking for compression](#chunking-for-compression)
+-   [Disabling in case of emergency](#disabling-in-case-of-emergency)
+-   [Example configs](#example-configs)
+-   [Note about performance and latency](#note-about-performance-and-latency)
+-   [How it works](#how-it-works)
+-   [How grouped batching works](#how-grouped-batching-works)
+
+## Introduction
+
+There is a current limitation regarding the size of the payload a Fluid client can send and receive. [The limit is 1MB per payload](https://github.com/microsoft/FluidFramework/issues/9023) and it is currently enforced explicitly with the `BatchTooLarge` error which closes the container.
+
+There are two features which can be used to work around this size limit, batch compression and compressed batch chunking. This document describes how to enable/disable them, along with a brief description of how they work. The features are enabled by default.
+
+By default, the runtime is configured with a max batch size of `716800` bytes, which is lower than the 1MB limit. The reason for the lower value is to account for possible overhead from the op envelope and metadata.
+
+### How batching works
+
+Batching in the context of Fluid ops is a way in which the framework accumulates and applies ops. A batch is a group of ops accumulated within a single JS turn, which will be broadcasted in the same order to all the other connected clients and applied synchronously. Additional logic and validation ensure that batches are never interleaved, nested or interrupted and they are processed in isolation without interleaving of ops from other clients.
+
+The way batches are formed is governed by the `FlushMode` setting of the `ContainerRuntimeOptions` and it is immutable for the entire lifetime of the runtime and subsequently the container.
+
+```
+export enum FlushMode {
+    /**
+     * In Immediate flush mode the runtime will immediately send all operations to the driver layer.
+     */
+    Immediate,
+
+    /**
+     * When in TurnBased flush mode the runtime will buffer operations in the current turn and send them as a single
+     * batch at the end of the turn. The flush call on the runtime can be used to force send the current batch.
+     */
+    TurnBased,
+}
+```
+
+What this means is that `FlushMode.Immediate` will send each op in its own payload to the server, while `FlushMode.TurnBased` will accumulate all ops in a single JS turn and send them together in the same payload. Technically, `FlushMode.Immediate` can be simulated with `FlushMode.TurnBased` by interrupting the JS turn after producing only one op (for example by pausing the execution to wait on a promise). Therefore, for all intents and purposes, `FlushMode.Immediate` enables all batches to have only one op.
+
+**By default, Fluid uses `FlushMode.TurnBased`** as:
+
+-   it is more efficient from an I/O perspective (batching ops overall decrease the number of payloads sent to the server)
+-   reduces concurrency related bugs, as it ensures that all ops generated within the same JS turn are also applied by all other clients within a single JS turn. Clients using the same pattern can safely assume ops will be applied exactly as they are observed locally. The alternative would be for ops to be both produced and applied with interruptions (which may involve processing input or rendering), invalidating the state based off which the changes were produced.
+
+As `FlushMode.TurnBased` accumulates ops, it is the most vulnerable to run into the 1MB socket limit.
+
+## Compression
+
+**Compression targets payloads which exceed the max batch size and it is enabled by default.**. The `IContainerRuntimeOptions.compressionOptions` property, of type `ICompressionRuntimeOptions` is the configuration governing how compression works.
+
+`ICompressionRuntimeOptions` has two properties:
+
+-   `minimumBatchSizeInBytes` – the minimum size of the batch for which compression should kick in. If the payload is too small, compression may not yield too many benefits. To target the original 1MB issue, a good value here would be to match the default maxBatchSizeInBytes (972800), however, experimentally, a good lower value could be at around 614400 bytes. Setting this value to `Number.POSITIVE_INFINITY` will disable compression.
+-   `compressionAlgorithm` – currently, only `lz4` is supported.
+
+Compression is relevant for both `FlushMode.TurnBased` and `FlushMode.Immediate` as it only targets the contents of the ops and not the number of ops in a batch. Compression is opaque to the server and implementations of the Fluid protocol do not need to alter their behavior to support this client feature.
+
+## Grouped batching
+
+**Note: This feature is currently considered experimental and is not ready for production usage.**
+
+The `IContainerRuntimeOptions.enableGroupedBatching` option has been added to the container runtime layer and is **off by default**. This option will group all batch messages under a new "grouped" message to be sent to the service. Upon receiving this new "grouped" message, the batch messages will be extracted and given the sequence number of the parent "grouped" message.
+
+The purpose for enabling grouped batching on top of compression is that regular compression won't include the empty messages in the chunks. Thus, if we have batches with many messages (i.e. more than 4k), we will go over the batch size limit just on empty op envelopes alone.
+
+See [below](#how-grouped-batching-works) for an example.
+
+### Risks
+
+This option is experimental and should not be enabled yet in production. This option should **ONLY** be enabled after observing that 99.9% of your application sessions contains these changes (runtime version "2.0.0-internal.4.1.0" or later). Containers created with this option may not open in future versions of the framework.
+
+This option will change a couple of expectations around message structure and runtime layer expectations. Only enable this option after testing
+and verifying that the following expectation changes won't have any effects:
+
+-   batch messages observed at the runtime layer will not match messages seen at the loader layer (i.e. grouped form at loader layer, ungrouped form at runtime layer)
+-   messages within the same batch will have the same sequence number
+-   client sequence numbers on batch messages can only be used to order messages with the same sequenceNumber
+-   requires all ops to be processed by runtime layer (version "2.0.0-internal.1.2.0" or later https://github.com/microsoft/FluidFramework/pull/11832)
+
+Grouped batching may become problematic for batches which contain reentrant ops. This is the case when changes are made to a DDS inside a DDS 'onChanged' event handler. This means that the reentrant op will have a different reference sequence number than the rest of the ops in the batch, resulting in a different view of the state of the data model.
+
+Therefore, when grouped batching is enabled, all batches with reentrant ops are rebased to the current reference sequence number and resubmitted to the data stores so that all ops are in agreement about the state of the data model and ensure eventual consistency.
+
+### How to enable
+
+**This feature is disabled by default, currently considered experimental and not ready for production usage.**
+
+If all prerequisites in the previous section are met, enabling the feature can be done via the `IContainerRuntimeOptions` as following:
+
+```
+    const runtimeOptions: IContainerRuntimeOptions = {
+        (...)
+        enableGroupedBatching: true,
+        (...)
+    }
+```
+
+In case of emergency grouped batching can be disabled at runtime, using feature gates. If `"Fluid.ContainerRuntime.DisableGroupedBatching"` is set to `true`, it will disable grouped batching if enabled from `IContainerRuntimeOptions` in the code.
+
+Grouped batching is only relevant for `FlushMode.TurnBased` as it only targets the number of ops in a batch. Grouped batching is opaque to the server and implementations of the Fluid protocol do not need to alter their behavior to support this client feature.
+
+## Chunking for compression
+
+**Op chunking for compression targets payloads which exceed the max batch size after compression.** So, only payloads which are already compressed. By default, the feature is enabled.
+
+The `IContainerRuntimeOptions.chunkSizeInBytes` property is the only configuration for chunking and it represents the size of the chunked ops, when chunking is necessary. When chunking is performed, the large op is split into smaller ops (chunks). This config represents both the size of the chunks and the threshold for the feature to activate. The value enables a trade-off between large chunks / few ops and small chunks / many ops. A good value for this would be at around 204800. Setting this value to `Number.POSITIVE_INFINITY` will disable chunking.
+
+This config would govern chunking compressed batches only. We will not be enabling chunking across all types of ops/batches but **only when compression is enabled and when the batch is compressed**, and its payload size is more than `IContainerRuntimeOptions.chunkSizeInBytes`.
+
+Chunking is relevant for both `FlushMode.TurnBased` and `FlushMode.Immediate` as it only targets the contents of the ops and not the number of ops in a batch. Chunking is opaque to the server and implementations of the Fluid protocol do not need to alter their behavior to support this client feature.
+
+## Disabling in case of emergency
+
+If the features are enabled using the configs, they can be disabled at runtime via feature gates as following:
+
+-   `Fluid.ContainerRuntime.CompressionDisabled` - if set to true, will disable compression (this has a side effect of also disabling chunking, as chunking is invoked only for compressed payloads).
+-   `Fluid.ContainerRuntime.DisableGroupedBatching` - if set to true, will disable grouped batching.
+-   `Fluid.ContainerRuntime.CompressionChunkingDisabled` - if set to true, will disable chunking for compression.
+
+## Example configs
+
+By default, the runtime is configured with the following values related to compression and chunking:
+
+```
+    const runtimeOptions: IContainerRuntimeOptions = {
+        compressionOptions: {
+            minimumBatchSizeInBytes: 614400,
+            compressionAlgorithm: CompressionAlgorithms.lz4,
+        },
+        chunkSizeInBytes: 204800,
+        maxBatchSizeInBytes: 716800,
+    }
+```
+
+To enable grouped batching, use the following property:
+
+```
+    const runtimeOptions: IContainerRuntimeOptions = {
+        enableGroupedBatching: true,
+    }
+```
+
+## Note about performance and latency
+
+In terms of performance and impact on latency, the results greatly depend on payload size, payload structure, network speed and CPU speed. Therefore, customers must perform the required measurements and adjust the settings according to their scenarios.
+
+In general, compression offers a trade-off between higher compute costs, lower bandwidth consumption and lower storage requirements, while chunking slightly increases latency due to the overhead of splitting an op, sending the chunks and reconstructing them on each client. Grouped batching heavily decreases the number of ops observed by the server and slightly decreases the bandwidth requirements as it merges all the ops in a batch into a single op and also eliminates the op envelope overhead.
+
+## How it works
+
+Compression currently works as a runtime layer over the regular op sending/receiving pipeline.
+
+If we have a batch with a size larger than the configured minimum required for compression (in the example let’s say it’s 850 bytes), as following:
+
+```
++-----------+-----------+-----------+-----------+
+| Op 1      | Op 2      | Op 3      | Op 4      |
+| SeqNum: 1 | SeqNum: 2 | SeqNum: 3 | SeqNum: 4 |
+| Size: 100 | Size: 150 | Size: 200 | Size: 400 |
++-----------+-----------+-----------+-----------+
+```
+
+The total size of the batch is 850 bytes. The client which needs to send the batch would compress the batch to a smaller size (200 bytes) and will send a new batch like the following:
+
+```
++--------------------+-----------+-----------+-----------+
+| Op 1               | Op 2      | Op 3      | Op 4      |
+| SeqNum: 1          | SeqNum: 2 | SeqNum: 3 | SeqNum: 4 |
+| Size: 200          | Size: 0   | Size: 0   | Size: 0   |
+| Compression: 'lz4' |           |           |           |
++--------------------+-----------+-----------+-----------+
+```
+
+The first op in the batch is the only one with content (which is opaque due to it being compressed), the rest of the ops serve only to reserve the sequence numbers so that the state machine which rebuilds the original batch on the receiving client can reconstruct the original batch.
+
+When the batch is received by a client, it will detect the first op as being compressed, it will decompress it and store it in memory. For each empty op subsequently received, it will fetch the uncompressed content from memory and rebuild the original ops. The original ops are then processed by the runtime and applied accordingly.
+So, compression virtualizes the batch.
+
+After compression, the first op in the batch can exceed 1MB, therefore it would still be rejected. In this case, another layer of virtualization is added after compression (and before decompression, symmetrically on the receiving end).
+
+The first op in the compressed batch can be chunked into smaller ops which can be sent outside the original batch. However, to conveniently maintain the batch semantics, the last chunk (the chunk which triggers rebuilding the original op) is the first op in the new batch.
+
+To illustrate, let’s take the large batch below:
+
+```
++--------------------+-----------+-----------+-----------+
+| Op 1               | Op 2      | Op 3      | Op 4      |
+| SeqNum: 1          | SeqNum: 2 | SeqNum: 3 | SeqNum: 4 |
+| Size: 900          | Size: 0   | Size: 0   | Size: 0   |
+| Compression: 'lz4' |           |           |           |
++--------------------+-----------+-----------+-----------+
+```
+
+This will produce the following batches:
+
+```
++-----------+
+| Chunk 1/3 |
+| SeqNum: 1 |
+| Size: 300 |
++-----------+
+
+```
+
+```
++-----------+
+| Chunk 2/3 |
+| SeqNum: 2 |
+| Size: 300 |
++-----------+
+
+```
+
+```
++-----------+-----------+-----------+-----------+
+| Chunk 3/3 | Op 2      | Op 3      | Op 4      |
+| SeqNum: 3 | SeqNum: 4 | SeqNum: 5 | SeqNum: 6 |
+| Size: 300 | Size: 0   | Size: 0   | Size: 0   |
++-----------+-----------+-----------+-----------+
+```
+
+The first 2 chunks are sent in their own batches, while the last chunk is the first op in the last batch which contains the ops reserving the required sequence numbers.
+
+Notice that the sequence numbers don’t matter here, as all ops will be based off the same reference sequence number, so the sequence number will be recalculated for all, without additional work.
+
+Additionally, as compression preserves the original uncompressed batch layout in terms of the number of ops by using empty ops to reserve the sequence numbers, this ensures that the clients will always receive the exact count of ops to rebuild the uncompressed batch sequentially.
+
+On the receiving end, the client will accumulate chunks 1 and 2 and keep them in memory. When chunk 3 is received, the original large, decompressed op will be rebuilt, and the runtime will then process the batch as if it is a compressed batch.
+
+## How grouped batching works
+
+Given the following baseline batch:
+
+```
++---------------+---------------+---------------+---------------+---------------+
+| Op 1          | Op 2          | Op 3          | Op 4          | Op 5          |
+| Contents: "a" | Contents: "b" | Contents: "c" | Contents: "d" | Contents: "e" |
++---------------+---------------+---------------+---------------+---------------+
+```
+
+Grouped batch:
+
+```
++---------------------------------------------------------------------------------------------------------------------+
+| Op 1                   Contents: +----------------+---------------+---------------+---------------+---------------+ |
+| Type: "groupedBatch"             | Op 1           | Op 2          | Op 3          | Op 4          | Op 5          | |
+|                                  | Contents: "a"  | Contents: "b" | Contents: "c" | Contents: "d" | Contents: "e" | |
+|                                  +----------------+---------------+---------------+---------------+---------------+ |
++---------------------------------------------------------------------------------------------------------------------+
+```
+
+Compressed batch:
+
+```
++-------------------------------------------------------------------------------------------------------------------------+
+| Op 1                   Contents: +------------------------------------------------------------------------------------+ |
+| Compression: 'lz4'               | Type: "groupedBatch"                                                               | |
+|                                  | +----------------+---------------+---------------+---------------+---------------+ | |
+|                                  | | Op 1           | Op 2          | Op 3          | Op 4          | Op 5          | | |
+|                                  | | Contents: "a"  | Contents: "b" | Contents: "c" | Contents: "d" | Contents: "e" | | |
+|                                  | +----------------+---------------+---------------+---------------+---------------+ | |
+|                                  +------------------------------------------------------------------------------------+ |
++-------------------------------------------------------------------------------------------------------------------------+
+```
+
+Can produce the following chunks:
+
+```
++------------------------------------------------+
+| Chunk 1/2    Contents: +---------------------+ |
+|                        | +-----------------+ | |
+|                        | | Contents: "abc" | | |
+|                        | +-----------------+ | |
+|                        +---------------------+ |
++------------------------------------------------+
+```
+
+```
++-----------------------------------------------+
+| Chunk 2/2    Contents: +--------------------+ |
+|                        | +----------------+ | |
+|                        | | Contents: "de" | | |
+|                        | +----------------+ | |
+|                        +--------------------+ |
++-----------------------------------------------+
+```
+
+-   Send to service
+-   Service acks ops sent
+-   Receive chunks from service
+-   Recompile to the compression step
+
+Decompressed batch:
+
+```
++---------------------------------------------------------------------------------------------------------------------+
+| Op 1                   Contents: +----------------+---------------+---------------+---------------+---------------+ |
+| SeqNum: 2                        | Op 1           | Op 2            | Op 3        | Op 4          | Op 5          | |
+| Type: "groupedBatch"             | Contents: "a"  | Contents: "b" | Contents: "c" | Contents: "d" | Contents: "e" | |
+|                                  +----------------+---------------+---------------+---------------+---------------+ |
++---------------------------------------------------------------------------------------------------------------------+
+```
+
+Ungrouped batch:
+
+```
++-----------------+-----------------+-----------------+-----------------+-----------------+
+| Op 1            | Op 2            | Op 3            | Op 4            | Op 5            |
+| Contents: "a"   | Contents: "b"   | Contents: "c"   | Contents: "d"   | Contents: "e"   |
+| SeqNum: 2       | SeqNum: 2       | SeqNum: 2       | SeqNum: 2       | SeqNum: 2       |
+| ClientSeqNum: 1 | ClientSeqNum: 2 | ClientSeqNum: 3 | ClientSeqNum: 4 | ClientSeqNum: 5 |
++-----------------+-----------------+-----------------+-----------------+-----------------+
+```

package/src/opLifecycle/batchManager.ts

@@ -0,0 +1,179 @@
+/*!
+ * Copyright (c) Microsoft Corporation and contributors. All rights reserved.
+ * Licensed under the MIT License.
+ */
+
+import { ICompressionRuntimeOptions } from "../containerRuntime";
+import { BatchMessage, IBatch, IBatchCheckpoint } from "./definitions";
+
+export interface IBatchManagerOptions {
+	readonly hardLimit: number;
+	readonly softLimit?: number;
+	readonly compressionOptions?: ICompressionRuntimeOptions;
+}
+
+export interface BatchSequenceNumbers {
+	referenceSequenceNumber?: number;
+	clientSequenceNumber?: number;
+}
+
+/**
+ * Estimated size of the stringification overhead for an op accumulated
+ * from runtime to loader to the service.
+ */
+const opOverhead = 200;
+
+/**
+ * Helper class that manages partial batch & rollback.
+ */
+export class BatchManager {
+	private pendingBatch: BatchMessage[] = [];
+	private batchContentSize = 0;
+	private hasReentrantOps = false;
+
+	public get length() {
+		return this.pendingBatch.length;
+	}
+	public get contentSizeInBytes() {
+		return this.batchContentSize;
+	}
+
+	public get sequenceNumbers(): BatchSequenceNumbers {
+		return {
+			referenceSequenceNumber: this.referenceSequenceNumber,
+			clientSequenceNumber: this.clientSequenceNumber,
+		};
+	}
+
+	private get referenceSequenceNumber(): number | undefined {
+		return this.pendingBatch.length === 0
+			? undefined
+			: this.pendingBatch[this.pendingBatch.length - 1].referenceSequenceNumber;
+	}
+
+	private clientSequenceNumber: number | undefined;
+
+	constructor(public readonly options: IBatchManagerOptions) {}
+
+	public push(
+		message: BatchMessage,
+		reentrant: boolean,
+		currentClientSequenceNumber?: number,
+	): boolean {
+		const contentSize = this.batchContentSize + (message.contents?.length ?? 0);
+		const opCount = this.pendingBatch.length;
+		this.hasReentrantOps = this.hasReentrantOps || reentrant;
+
+		// Attempt to estimate batch size, aka socket message size.
+		// Each op has pretty large envelope, estimating to be 200 bytes.
+		// Also content will be strigified, and that adds a lot of overhead due to a lot of escape characters.
+		// Not taking it into account, as compression work should help there - compressed payload will be
+		// initially stored as base64, and that requires only 2 extra escape characters.
+		const socketMessageSize = contentSize + opOverhead * opCount;
+
+		// If we were provided soft limit, check for exceeding it.
+		// But only if we have any ops, as the intention here is to flush existing ops (on exceeding this limit)
+		// and start over. That's not an option if we have no ops.
+		// If compression is enabled, the soft and hard limit are ignored and the message will be pushed anyways.
+		// Cases where the message is still too large will be handled by the maxConsecutiveReconnects path.
+		if (
+			this.options.softLimit !== undefined &&
+			this.length > 0 &&
+			socketMessageSize >= this.options.softLimit
+		) {
+			return false;
+		}
+
+		if (socketMessageSize >= this.options.hardLimit) {
+			return false;
+		}
+
+		if (this.pendingBatch.length === 0) {
+			this.clientSequenceNumber = currentClientSequenceNumber;
+		}
+
+		this.batchContentSize = contentSize;
+		this.pendingBatch.push(message);
+		return true;
+	}
+
+	public get empty() {
+		return this.pendingBatch.length === 0;
+	}
+
+	public popBatch(): IBatch {
+		const batch: IBatch = {
+			content: this.pendingBatch,
+			contentSizeInBytes: this.batchContentSize,
+			referenceSequenceNumber: this.referenceSequenceNumber,
+			hasReentrantOps: this.hasReentrantOps,
+		};
+
+		this.pendingBatch = [];
+		this.batchContentSize = 0;
+		this.clientSequenceNumber = undefined;
+		this.hasReentrantOps = false;
+
+		return addBatchMetadata(batch);
+	}
+
+	/**
+	 * Capture the pending state at this point
+	 */
+	public checkpoint(): IBatchCheckpoint {
+		const startPoint = this.pendingBatch.length;
+		return {
+			rollback: (process: (message: BatchMessage) => void) => {
+				for (let i = this.pendingBatch.length; i > startPoint; ) {
+					i--;
+					const message = this.pendingBatch[i];
+					this.batchContentSize -= message.contents?.length ?? 0;
+					process(message);
+				}
+
+				this.pendingBatch.length = startPoint;
+			},
+		};
+	}
+}
+
+const addBatchMetadata = (batch: IBatch): IBatch => {
+	if (batch.content.length > 1) {
+		batch.content[0].metadata = {
+			...batch.content[0].metadata,
+			batch: true,
+		};
+		batch.content[batch.content.length - 1].metadata = {
+			...batch.content[batch.content.length - 1].metadata,
+			batch: false,
+		};
+	}
+
+	return batch;
+};
+
+/**
+ * Estimates the real size in bytes on the socket for a given batch. It assumes that
+ * the envelope size (and the size of an empty op) is 200 bytes, taking into account
+ * extra overhead from stringification.
+ *
+ * @param batch - the batch to inspect
+ * @returns An estimate of the payload size in bytes which will be produced when the batch is sent over the wire
+ */
+export const estimateSocketSize = (batch: IBatch): number => {
+	return batch.contentSizeInBytes + opOverhead * batch.content.length;
+};
+
+export const sequenceNumbersMatch = (
+	seqNums: BatchSequenceNumbers,
+	otherSeqNums: BatchSequenceNumbers,
+): boolean => {
+	return (
+		(seqNums.referenceSequenceNumber === undefined ||
+			otherSeqNums.referenceSequenceNumber === undefined ||
+			seqNums.referenceSequenceNumber === otherSeqNums.referenceSequenceNumber) &&
+		(seqNums.clientSequenceNumber === undefined ||
+			otherSeqNums.clientSequenceNumber === undefined ||
+			seqNums.clientSequenceNumber === otherSeqNums.clientSequenceNumber)
+	);
+};

package/src/opLifecycle/definitions.ts

@@ -0,0 +1,89 @@
+/*!
+ * Copyright (c) Microsoft Corporation and contributors. All rights reserved.
+ * Licensed under the MIT License.
+ */
+
+import { IBatchMessage } from "@fluidframework/container-definitions";
+import { ISequencedDocumentMessage, MessageType } from "@fluidframework/protocol-definitions";
+import { CompressionAlgorithms } from "../containerRuntime";
+import { ContainerMessageType } from "../messageTypes";
+
+/**
+ * Batch message type used internally by the runtime
+ */
+export type BatchMessage = IBatchMessage & {
+	localOpMetadata: unknown;
+	type: ContainerMessageType;
+	referenceSequenceNumber: number;
+	compression?: CompressionAlgorithms;
+};
+
+/**
+ * Batch interface used internally by the runtime.
+ */
+export interface IBatch {
+	/**
+	 * Sum of the in-memory content sizes of all messages in the batch.
+	 * If the batch is compressed, this number reflects the post-compression size.
+	 */
+	readonly contentSizeInBytes: number;
+	/**
+	 * All the messages in the batch
+	 */
+	readonly content: BatchMessage[];
+	/**
+	 * The reference sequence number for the batch
+	 */
+	readonly referenceSequenceNumber: number | undefined;
+	/**
+	 * Wether or not the batch contains at least one op which was produced as the result
+	 * of processing another op. This means that the batch must be rebased before
+	 * submitted, to ensure that all ops have the same reference sequence numbers and a
+	 * consistent view of the data model. This happens when the op is created within a
+	 * 'changed' event handler of a DDS and will have a different reference sequence number
+	 * than the rest of the ops in the batch, meaning that it has a different view of the
+	 * state of the data model, therefore all ops must be resubmitted and rebased to the current
+	 * reference sequence number to be in agreement about the data model state.
+	 */
+	readonly hasReentrantOps?: boolean;
+}
+
+export interface IBatchCheckpoint {
+	rollback: (action: (message: BatchMessage) => void) => void;
+}
+
+/**
+ * @internal
+ */
+export interface IChunkedOp {
+	chunkId: number;
+	totalChunks: number;
+	contents: string;
+	originalType: MessageType | ContainerMessageType;
+	originalMetadata?: Record<string, unknown>;
+	originalCompression?: string;
+}
+
+/**
+ * The state of remote message processing:
+ * `Processed` - the message can be considered processed
+ * `Skipped` - the message was ignored by the processor
+ * `Accepted` - the message was processed partially. Eventually, a message
+ * will make the processor return `Processed`.
+ */
+export type ProcessingState = "Processed" | "Skipped" | "Accepted";
+
+/**
+ * Return type for functions which process remote messages
+ */
+export interface IMessageProcessingResult {
+	/**
+	 * A shallow copy of the input message if processing happened, or
+	 * the original message otherwise
+	 */
+	readonly message: ISequencedDocumentMessage;
+	/**
+	 * Processing result of the input message.
+	 */
+	readonly state: ProcessingState;
+}

package/src/opLifecycle/index.ts

@@ -0,0 +1,19 @@
+/*!
+ * Copyright (c) Microsoft Corporation and contributors. All rights reserved.
+ * Licensed under the MIT License.
+ */
+
+export { BatchManager, estimateSocketSize, BatchSequenceNumbers } from "./batchManager";
+export {
+	BatchMessage,
+	IBatch,
+	IBatchCheckpoint,
+	IChunkedOp,
+	IMessageProcessingResult,
+} from "./definitions";
+export { Outbox, getLongStack } from "./outbox";
+export { OpCompressor } from "./opCompressor";
+export { OpDecompressor } from "./opDecompressor";
+export { OpSplitter, splitOp } from "./opSplitter";
+export { RemoteMessageProcessor, unpackRuntimeMessage } from "./remoteMessageProcessor";
+export { OpGroupingManager } from "./opGroupingManager";