@streamsy/storage 0.0.0-stage → 0.4.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025-2026 Gareth Andrew
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,135 @@
-# Temporary Holding Version
+# @streamsy/storage
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+SQL storage for `@streamsy/core`, built on Effect SQL. One entry for Bun
+SQLite, one for SQLite Durable Objects, and a driver-free root for your own
+SQLite or Postgres `SqlClient`. Version 0.4.0 requires `effect@4.0.1`.
+
+```sh
+bun add @streamsy/storage @streamsy/core effect
+```
+
+## Entries
+
+| Entry                              | Use it for                                                                       |
+| ---------------------------------- | -------------------------------------------------------------------------------- |
+| `@streamsy/storage/bun`            | A retained SQLite file on Bun (`@effect/sql-sqlite-bun`, `@effect/platform-bun`) |
+| `@streamsy/storage/durable-object` | SQLite inside a Cloudflare Durable Object (`@effect/sql-sqlite-do`)              |
+| `@streamsy/storage`                | A SQLite (3.42 or newer), Postgres or MySQL `SqlClient` you provide              |
+
+The Bun and Durable Object entries export `BunStorage` and
+`DurableObjectStorage`, each with `layer` and `layerProtocol`.
+`layerProtocol` exposes `StreamsReader`, `StreamsWriter`, `Storage`,
+`CommitBoundary`, `SqlClient`, and SQL reactivity on both hosts. Your own SQL
+can share the connection and commit with a stream mutation. `layer` exposes
+the same storage services without the protocol reader and writer. The root entry
+exports `SqlStorage`, with `layer` over a `SqlClient` you supply and
+`clientLayer` to provide one.
+`CommitBoundary` comes from `@streamsy/core`.
+
+## Use your own client
+
+`SqlStorage.clientLayer` builds a client from its `make` function and provides it with the
+`Reactivity` it captured, so the client and the storage share one change graph.
+The root entry imports no driver. Add the one you use, at `4.0.1`.
+
+```ts
+import { Layer, Redacted } from "effect";
+import { PgClient } from "@effect/sql-pg";
+import { SqlStorage } from "@streamsy/storage";
+
+const storage = SqlStorage.layer(streams).pipe(
+  Layer.provide(SqlStorage.clientLayer(PgClient.make({ url: Redacted.make(url) }))),
+);
+```
+
+On Postgres and MySQL, every mutation transaction first locks the rows of the
+streams it writes (`SELECT ... FOR UPDATE`), so writers on many connections or
+processes apply one after the other.
+
+## Serve a SQLite-backed host on Bun
+
+```ts
+import { Layer } from "effect";
+import { HttpRouter } from "effect/http";
+import { BunRuntime } from "@effect/platform-bun";
+import { Http, Streamsy } from "@streamsy/core";
+import { listener } from "@streamsy/serve/bun";
+import { BunStorage } from "@streamsy/storage/bun";
+
+const streams = Streamsy.group(Streamsy.Wildcard);
+const Server = HttpRouter.serve(Http.routes(streams, { prefix: "/streams", access: "full" })).pipe(
+  Layer.provide(BunStorage.layer(streams, { client: { filename: "./streams.sqlite" } })),
+  Layer.provide(listener({ port: 3000 })),
+);
+BunRuntime.runMain(Layer.launch(Server));
+```
+
+## Commit your SQL with a stream write
+
+`CommitBoundary.withTransaction` from `@streamsy/core` runs application SQL and Streamsy mutations
+in one transaction. Nested calls join the outer boundary. A rejected mutation
+fails with `MutationRejected` and rolls the transaction back; catch it outside
+the boundary. The [Layers and storage guide](https://streamsy.dev/docs/hosting)
+shows the complete pattern.
+
+On a server database, the stream row lock is taken by the first stream mutation
+inside the transaction, not when the transaction starts. Your SQL before that
+mutation runs before the lock. Storage's reads inside a mutation are safe on
+every dialect: on MySQL they are locking reads (`FOR UPDATE` or `FOR SHARE`),
+so they see the latest committed rows even when your SQL fixed the
+transaction's snapshot first. A stale plan is rejected, never written.
+
+The protocol plans an append from `storage.record` and `storage.producer`.
+Inside a transaction on MySQL, these reads are `FOR UPDATE` too, so each replan
+sees the latest tail and the stream row stays locked until your transaction
+ends. Other writers to that stream wait for your commit. Outside a transaction
+they are plain reads and take no lock.
+
+Your reads of your own tables are your job. On Postgres each statement reads
+the latest committed rows. On MySQL at the default REPEATABLE READ, a plain
+read uses the snapshot of the transaction's first plain read, which can be
+older than the stream lock. If a decision depends on a row another writer can
+change, read it with `FOR UPDATE` or `FOR SHARE` on MySQL.
+
+For a projection on that same connection:
+
+```ts
+import { Layer } from "effect";
+import { CheckpointStore, Transaction } from "@streamsy/projection";
+import { BunStorage } from "@streamsy/storage/bun";
+
+const host = Layer.mergeAll(CheckpointStore.layer, Transaction.layer).pipe(
+  Layer.provideMerge(BunStorage.layerProtocol({ client: { filename: "./streamsy.sqlite" } })),
+);
+```
+
+`CheckpointStore.layer` keeps checkpoints in the storage table
+`streamsy_owner_records`. On Postgres and MySQL, a checkpoint save locks the
+record it read, and a first save is a plain insert, so two concurrent saves of
+one projection never both win. `Transaction.layer` gives the SQL-only `Transaction`
+service. `Commit.transaction` uses them to commit SQL, same-owner stream
+appends, and the checkpoint together.
+
+In a Durable Object, use `DurableObjectStorage.layerProtocol` with
+`client: { storage: ctx.storage }` in the same composition.
+
+## Good to know
+
+- The Bun entry creates the parent folder of a new file. A read-only open, or
+  `create: false`, creates nothing.
+- Only files in the current storage format are accepted. Older Streamsy files,
+  including those from earlier 0.4 builds, and newer unknown schemas are
+  rejected without being changed. Use a fresh database path.
+- Stream ids are limited to 512 characters, and producer ids to 255. A longer
+  id fails with a non-retryable `StorageFault` before anything is written.
+- Bun defaults the client `busyTimeout` to zero and retries storage-owned
+  transactions with a bounded yielding policy. Tune `client.busyTimeout`,
+  `transactionRetryAttempts`, and `transactionRetryDelayMs` if you need to.
+- Long-poll reads default to 30 seconds on Bun and 25 seconds in a Durable
+  Object.
+- Change notifications are process-local. A repair pass every `repairIntervalMs`
+  (default 1,000 ms) bounds staleness across processes.
+
+## License
+
+MIT

package/dist/bun.d.ts

@@ -0,0 +1,20 @@
+import { r as SqlStorageOptions } from "./storage-CrMmTyyG.js";
+import { Layer } from "effect";
+import { StorageFault, Streamsy } from "@streamsy/core";
+import { Protocol } from "@streamsy/core/protocol";
+import { SqliteClient } from "@effect/sql-sqlite-bun";
+declare namespace bun_storage_d_exports {
+  export { Options, layer, layerProtocol };
+}
+interface BunStorageOptions extends SqlStorageOptions {
+  readonly client: Parameters<typeof SqliteClient.make>[0];
+}
+/** Options for `layer` and `layerProtocol`. */
+interface Options extends BunStorageOptions, Protocol.ProtocolOptions {}
+/** Official Bun SQLite host layer with one shared SqlClient/Reactivity graph. */
+declare const storageLayer: (options: BunStorageOptions) => Layer.Layer<import("@streamsy/core").CommitBoundary | import("@streamsy/core/internal").OwnerRecords | import("effect/reactivity/Reactivity").Reactivity | import("effect/sql/SqlClient").SqlClient | import("@streamsy/core").Storage, StorageFault, never>;
+/** Complete persistent protocol Layer for an HttpRouter application. */
+declare const layerProtocol: (options: Options) => Layer.Layer<import("@streamsy/core").CommitBoundary | import("@streamsy/core/internal").OwnerRecords | import("effect/reactivity/Reactivity").Reactivity | import("effect/sql/SqlClient").SqlClient | import("@streamsy/core").Storage | import("@streamsy/core/protocol").StreamsReader | import("@streamsy/core/protocol").StreamsWriter, StorageFault, never>;
+declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(group: Streamsy.Group<Keys>, options: Options): Layer.Layer<Layer.Success<ReturnType<typeof storageLayer>> | Streamsy.Host<Streamsy.Group<Keys>>, Layer.Error<ReturnType<typeof storageLayer>>>;
+//#endregion
+export { bun_storage_d_exports as BunStorage };

package/dist/bun.js

@@ -0,0 +1,109 @@
+import { t as __exportAll } from "./rolldown-runtime-D7D4PA-g.js";
+import { a as storageLayer$1, d as retryWithPolicy, f as transactionRetryPolicy, o as inspectFormat, s as newerVersionFault, u as sharedSqlClientLayer } from "./storage-CpnFxwO5.js";
+import { Cause, Effect, FileSystem, Layer, Option, Path, Result, Schema } from "effect";
+import { StorageFault } from "@streamsy/core";
+import { Protocol } from "@streamsy/core/protocol";
+import * as BunFileSystem from "@effect/platform-bun/BunFileSystem";
+import * as BunPath from "@effect/platform-bun/BunPath";
+import { SqliteClient } from "@effect/sql-sqlite-bun";
+import { Database } from "bun:sqlite";
+//#region src/sqlite-file.ts
+const openFault$1 = (message) => new StorageFault({
+	operation: "migration.open",
+	message,
+	retryable: false
+});
+const decodeTableNames = Schema.decodeUnknownSync(Schema.Array(Schema.Struct({ name: Schema.String })));
+const decodeVersions = Schema.decodeUnknownSync(Schema.Array(Schema.Struct({ version: Schema.NullOr(Schema.Finite) })));
+const readFormat = (reader) => {
+	const names = new Set(decodeTableNames(reader.all("SELECT name FROM sqlite_master WHERE type='table'")).map(({ name }) => name));
+	return {
+		names,
+		version: names.has("streamsy_schema") ? decodeVersions(reader.all(`SELECT MAX(version) version FROM streamsy_schema`)).at(0)?.version ?? 0 : 0
+	};
+};
+/** The same format rule as the SQL migration runner, read before the client opens the file. */
+const preflight = (filename, driver, retry) => Effect.gen(function* () {
+	const state = yield* retryWithPolicy(driver.read(filename, readFormat), retry, (error) => error.retryable);
+	yield* inspectFormat(state.names);
+	if (state.version > 1) return yield* newerVersionFault(state.version);
+});
+/**
+* Prepare a SQLite file before the client opens it: create the parent folder,
+* refuse a foreign format, check the open mode, and set WAL with busy timeout 0.
+* A read-only or no-create open never creates a folder or a file.
+*/
+const prepare = (driver, options, retry) => Effect.gen(function* () {
+	const { filename } = options;
+	if (filename === ":memory:") return;
+	const fs = yield* FileSystem.FileSystem;
+	const path = yield* Path.Path;
+	const readonly = options.readonly === true;
+	const mode = {
+		readonly,
+		readwrite: readonly ? false : options.readwrite ?? true,
+		create: readonly ? false : options.create ?? true
+	};
+	if (mode.readwrite && mode.create) yield* fs.makeDirectory(path.dirname(filename), { recursive: true }).pipe(Effect.mapError((error) => openFault$1(`Cannot create the folder for SQLite file ${filename}: ${error.message}`)));
+	if (yield* fs.exists(filename).pipe(Effect.mapError((error) => openFault$1(`Cannot check SQLite file ${filename}: ${error.message}`)))) yield* preflight(filename, driver, retry);
+	const retried = (step) => retryWithPolicy(step, retry, (error) => error.retryable);
+	yield* retried(driver.run(filename, mode, []));
+	if (!readonly && options.disableWAL !== true) yield* retried(driver.run(filename, {
+		readonly: false,
+		readwrite: options.readwrite ?? true,
+		create: options.create ?? true
+	}, ["PRAGMA busy_timeout=0", "PRAGMA journal_mode=WAL"]));
+});
+//#endregion
+//#region src/bun-storage.ts
+var bun_storage_exports = /* @__PURE__ */ __exportAll({
+	layer: () => layer,
+	layerProtocol: () => layerProtocol
+});
+const SqliteOpenError = Schema.Struct({
+	code: Schema.optional(Schema.String),
+	errno: Schema.optional(Schema.Finite)
+});
+const decodeSqliteOpenError = Schema.decodeUnknownOption(SqliteOpenError);
+const isBusy = ({ code, errno }) => code === "SQLITE_BUSY" || code === "SQLITE_LOCKED" || errno === 5 || errno === 6;
+const isSqliteOperationalError = ({ code, errno }) => (code?.startsWith("SQLITE_") ?? false) || errno !== void 0;
+const openFault = (error, rendered) => new StorageFault({
+	operation: "migration.open",
+	message: rendered,
+	retryable: isBusy(error)
+});
+const normalizeSqliteDefect = (acquisition) => acquisition.pipe(Effect.catch((error) => Effect.die(error.reason.cause)), Effect.catchCause((cause) => Result.match(Cause.findDefect(cause), {
+	onFailure: () => Effect.failCause(cause),
+	onSuccess: (defect) => Option.match(decodeSqliteOpenError(defect), {
+		onNone: () => Effect.failCause(cause),
+		onSome: (error) => isSqliteOperationalError(error) ? Effect.fail(openFault(error, String(defect))) : Effect.failCause(cause)
+	})
+})));
+/** Raw `bun:sqlite` opens for the shared SQLite file step. */
+const bunDriver = {
+	read: (filename, use) => Effect.acquireUseRelease(normalizeSqliteDefect(Effect.sync(() => new Database(filename, { readonly: true }))), (database) => normalizeSqliteDefect(Effect.sync(() => use({ all: (sql) => database.query(sql).all() }))), (database) => Effect.sync(() => database.close(false))),
+	run: (filename, mode, statements) => Effect.acquireUseRelease(normalizeSqliteDefect(Effect.sync(() => new Database(filename, mode))), (database) => normalizeSqliteDefect(Effect.sync(() => {
+		for (const statement of statements) database.run(statement);
+	})), (database) => Effect.sync(() => database.close(false)))
+};
+const BunPlatform = Layer.mergeAll(BunFileSystem.layer, BunPath.layer);
+/** Official Bun SQLite host layer with one shared SqlClient/Reactivity graph. */
+const storageLayer = (options) => {
+	const retryPolicy = transactionRetryPolicy(options);
+	const preparesJournal = options.client.filename !== ":memory:" && options.client.readonly !== true && options.client.disableWAL !== true;
+	const clientLayer = Layer.effectContext(sharedSqlClientLayer(normalizeSqliteDefect(SqliteClient.make({
+		busyTimeout: 0,
+		...options.client,
+		disableWAL: preparesJournal ? true : options.client.disableWAL
+	}))));
+	return Layer.unwrap(prepare(bunDriver, options.client, retryPolicy).pipe(Effect.as(storageLayer$1(options).pipe(Layer.provideMerge(clientLayer))))).pipe(Layer.provide(BunPlatform));
+};
+/** Complete persistent protocol Layer for an HttpRouter application. */
+const layerProtocol = (options) => Protocol.layer(options).pipe(Layer.provideMerge(storageLayer(options)));
+function layer(group, options) {
+	const storage = storageLayer(options);
+	const protocol = Protocol.layer(options).pipe(Layer.provideMerge(storage));
+	return Protocol.bind(protocol, group).pipe(Layer.provideMerge(storage));
+}
+//#endregion
+export { bun_storage_exports as BunStorage };

package/dist/durable-object.d.ts

@@ -0,0 +1,24 @@
+import { r as SqlStorageOptions } from "./storage-CrMmTyyG.js";
+import { Layer } from "effect";
+import { Streamsy } from "@streamsy/core";
+import { Protocol } from "@streamsy/core/protocol";
+import { SqliteClient } from "@effect/sql-sqlite-do";
+declare namespace durable_object_storage_d_exports {
+  export { DEFAULT_LONG_POLL_TIMEOUT_MS, Options, layer, layerProtocol };
+}
+interface DurableObjectStorageOptions extends SqlStorageOptions {
+  /** Full DurableObjectStorage is required because Storage mutations are transactional. */
+  readonly client: Parameters<typeof SqliteClient.make>[0] & {
+    readonly storage: NonNullable<Parameters<typeof SqliteClient.make>[0]["storage"]>;
+  };
+}
+declare const DEFAULT_LONG_POLL_TIMEOUT_MS = 25000;
+/** Options for `layer` and `layerProtocol`. */
+interface Options extends DurableObjectStorageOptions, Protocol.ProtocolOptions {}
+/** Official local/host Durable Object SQLite layer with commit-boundary support. */
+declare const storageLayer: (options: DurableObjectStorageOptions) => Layer.Layer<import("@streamsy/core").CommitBoundary | import("@streamsy/core/internal").OwnerRecords | import("effect/reactivity/Reactivity").Reactivity | import("effect/sql/SqlClient").SqlClient | import("@streamsy/core").Storage, import("@streamsy/core").StorageFault, never>;
+/** Protocol over the official Durable Object SQLite layer. */
+declare const layerProtocol: (options: Options) => Layer.Layer<import("@streamsy/core").CommitBoundary | import("@streamsy/core/internal").OwnerRecords | import("effect/reactivity/Reactivity").Reactivity | import("effect/sql/SqlClient").SqlClient | import("@streamsy/core").Storage | import("@streamsy/core/protocol").StreamsReader | import("@streamsy/core/protocol").StreamsWriter, import("@streamsy/core").StorageFault, never>;
+declare function layer<const Keys extends ReadonlyArray<Streamsy.Any>>(group: Streamsy.Group<Keys>, options: Options): Layer.Layer<Layer.Success<ReturnType<typeof storageLayer>> | Streamsy.Host<Streamsy.Group<Keys>>, Layer.Error<ReturnType<typeof storageLayer>>>;
+//#endregion
+export { durable_object_storage_d_exports as DurableObjectStorage };

package/dist/durable-object.js

@@ -0,0 +1,33 @@
+import { t as __exportAll } from "./rolldown-runtime-D7D4PA-g.js";
+import { i as layerWithDurableObjectTransactions, u as sharedSqlClientLayer } from "./storage-CpnFxwO5.js";
+import { Layer } from "effect";
+import "@streamsy/core";
+import { Protocol } from "@streamsy/core/protocol";
+import { SqliteClient } from "@effect/sql-sqlite-do";
+//#region src/durable-object-storage.ts
+var durable_object_storage_exports = /* @__PURE__ */ __exportAll({
+	DEFAULT_LONG_POLL_TIMEOUT_MS: () => DEFAULT_LONG_POLL_TIMEOUT_MS,
+	layer: () => layer,
+	layerProtocol: () => layerProtocol
+});
+const DEFAULT_LONG_POLL_TIMEOUT_MS = 25e3;
+/** Official local/host Durable Object SQLite layer with commit-boundary support. */
+const storageLayer = (options) => {
+	const clientLayer = Layer.effectContext(sharedSqlClientLayer(SqliteClient.make(options.client)));
+	return layerWithDurableObjectTransactions(options).pipe(Layer.provideMerge(clientLayer));
+};
+/** Protocol over the official Durable Object SQLite layer. */
+const layerProtocol = (options) => Protocol.layer({
+	...options,
+	longPollTimeoutMs: options.longPollTimeoutMs ?? 25e3
+}).pipe(Layer.provideMerge(storageLayer(options)));
+function layer(group, options) {
+	const storage = storageLayer(options);
+	const protocol = Protocol.layer({
+		...options,
+		longPollTimeoutMs: options.longPollTimeoutMs ?? 25e3
+	}).pipe(Layer.provideMerge(storage));
+	return Protocol.bind(protocol, group).pipe(Layer.provideMerge(storage));
+}
+//#endregion
+export { durable_object_storage_exports as DurableObjectStorage };

package/dist/index.d.ts

@@ -0,0 +1,6 @@
+import { a as layerProtocol, c as DEFAULT_TRANSACTION_RETRY_DELAY_MS, i as layer, n as Options, o as clientLayer, s as DEFAULT_TRANSACTION_RETRY_ATTEMPTS, t as DEFAULT_REPAIR_INTERVAL_MS } from "./storage-CrMmTyyG.js";
+declare namespace sql_storage_d_exports {
+  export { DEFAULT_REPAIR_INTERVAL_MS, DEFAULT_TRANSACTION_RETRY_ATTEMPTS, DEFAULT_TRANSACTION_RETRY_DELAY_MS, Options, clientLayer, layer, layerProtocol };
+}
+//#endregion
+export { sql_storage_d_exports as SqlStorage };

package/dist/index.js

@@ -0,0 +1,13 @@
+import { t as __exportAll } from "./rolldown-runtime-D7D4PA-g.js";
+import { l as clientLayer, n as layer, r as layerProtocol, t as DEFAULT_REPAIR_INTERVAL_MS } from "./storage-CpnFxwO5.js";
+//#region src/sql-storage.ts
+var sql_storage_exports = /* @__PURE__ */ __exportAll({
+	DEFAULT_REPAIR_INTERVAL_MS: () => DEFAULT_REPAIR_INTERVAL_MS,
+	DEFAULT_TRANSACTION_RETRY_ATTEMPTS: () => 16,
+	DEFAULT_TRANSACTION_RETRY_DELAY_MS: () => 25,
+	clientLayer: () => clientLayer,
+	layer: () => layer,
+	layerProtocol: () => layerProtocol
+});
+//#endregion
+export { sql_storage_exports as SqlStorage };

package/dist/rolldown-runtime-D7D4PA-g.js

@@ -0,0 +1,13 @@
+//#region \0rolldown/runtime.js
+var __defProp = Object.defineProperty;
+var __exportAll = (all, no_symbols) => {
+	let target = {};
+	for (var name in all) __defProp(target, name, {
+		get: all[name],
+		enumerable: true
+	});
+	if (!no_symbols) __defProp(target, Symbol.toStringTag, { value: "Module" });
+	return target;
+};
+//#endregion
+export { __exportAll as t };