@livestore/livestore 0.4.0-dev.2 → 0.4.0-dev.20

package/docs/reference/syncing/sync-provider/custom/index.md

@@ -0,0 +1,65 @@
+# Build your own sync provider
+
+It's very straightforward to implement your own sync provider. A sync provider implementation needs to do the following:
+
+## Client-side
+
+Implement the `SyncBackend` interface (running in the client) which describes the protocol for syncing events between the client and the server.
+
+```ts
+// Slightly simplified API (see packages/@livestore/common/src/sync/sync.ts for the full API)
+export type SyncBackend = {
+  pull: (cursor: EventSequenceNumber) => Stream<{ batch: LiveStoreEvent[] }, InvalidPullError>
+  push: (batch: LiveStoreEvent[]) => Effect<void, InvalidPushError>
+}
+
+// my-sync-backend.ts
+const makeMySyncBackend = (args: { /* ... */ }) => {
+  return {
+    pull: (cursor) => {
+      // ...
+    },
+    push: (batch) => {
+      // ...
+    }
+  }
+}
+
+// my-app.ts
+const adapter = makeAdapter({
+  sync: {
+    backend: makeMySyncBackend({ /* ... */ })
+  }
+})
+```
+
+The actual implementation of those methods is left to the developer and mostly depends on the network protocol used to communicate between the client and the server.
+
+Ideally this implementation considers the following:
+
+- Network connectivity (offline, unstable connection, etc.)
+- Ordering of events in case of out-of-order delivery
+- Backoff and retry logic
+
+## Server-side
+
+Implement the actual sync backend protocol (running in the server). At minimum this sync backend needs to do the following:
+
+  - For client `push` requests:
+    - Validate the batch of events
+      - Ensure the batch sequence numbers are in ascending order and larger than the sync backend head
+      - Further validation checks (e.g. schema-aware payload validation)
+    - Persist the events in the event store (implying a new sync backend head equal to the sequence number of the pushed last event)
+    - Return a success response
+    - It's important that the server only processes one push request at a time to ensure a total ordering of events.
+
+  - For client `pull` requests:
+    - Validate the cursor
+    - Query the events from the database
+    - Return the events to the client
+    - This can be done in a batch or streamed to the client
+    - `pull` requests can be handled in parallel by the server
+
+## General recommendations
+
+It's recommended to study the existing sync backend implementations for inspiration.

package/docs/reference/syncing/sync-provider/electricsql/index.md

@@ -0,0 +1,159 @@
+# ElectricSQL
+
+The `@livestore/sync-electric` package lets you sync LiveStore with ElectricSQL.
+
+- Package: `pnpm add @livestore/sync-electric`
+- Protocol: HTTP push/pull with long-polling support
+
+## Architecture
+
+```mermaid
+graph LR
+    subgraph Browser
+        LS[LiveStore Client]
+    end
+    
+    subgraph "Your Server"
+        AP[API Proxy<br/>'/api/electric']
+    end
+    
+    subgraph "Infrastructure"
+        ES[Electric Server<br/>':30000']
+        PG[(Postgres DB)]
+    end
+    
+    LS -->|"GET (pull)"| AP
+    LS -->|"POST (push)"| AP
+    AP -->|"Pull requests<br/>(proxied)"| ES
+    AP -->|"Push events<br/>(direct write)"| PG
+    ES -->|"Listen"| PG
+    
+    style LS fill:#e1f5fe
+    style AP fill:#fff3e0
+    style ES fill:#f3e5f5
+    style PG fill:#e8f5e9
+```
+
+The API proxy has dual responsibilities:
+- **Push Events**: Writes events directly to Postgres tables (bypasses Electric)
+- **Pull Requests**: Proxies to Electric server for reading events
+- **Authentication**: Implements your custom auth logic
+- **Database Management**: Initializes tables and manages connections
+
+## Client Setup
+
+Basic usage in your worker/server code:
+
+## `reference/syncing/sync-provider/electricsql/client-setup.ts`
+
+```ts filename="reference/syncing/sync-provider/electricsql/client-setup.ts"
+
+const _backend = makeSyncBackend({
+  endpoint: '/api/electric', // Your API proxy endpoint
+  ping: { enabled: true },
+})
+```
+
+## API Proxy Implementation
+
+ElectricSQL requires an API proxy on your server to handle authentication and database operations. Your proxy needs two endpoints:
+
+### Minimal Implementation Example
+
+## `reference/syncing/sync-provider/electricsql/api-proxy.ts`
+
+```ts filename="reference/syncing/sync-provider/electricsql/api-proxy.ts"
+
+const electricHost = 'http://localhost:30000' // Your Electric server
+
+/** Placeholder for your database factory function */
+declare const makeDb: (storeId: string) => {
+  migrate: () => Promise<void>
+  disconnect: () => Promise<void>
+  createEvents: (batch: (typeof ApiSchema.PushPayload.Type)['batch']) => Promise<void>
+}
+
+// ---cut---
+
+// GET /api/electric - Pull events (proxied through Electric)
+export async function GET(request: Request) {
+  const searchParams = new URL(request.url).searchParams
+  const { url, storeId, needsInit } = makeElectricUrl({
+    electricHost,
+    searchParams,
+    apiSecret: 'your-electric-secret',
+  })
+
+  // Add your authentication logic here
+  // if (!isAuthenticated(request)) {
+  //   return new Response('Unauthorized', { status: 401 })
+  // }
+
+  // Initialize database tables if needed
+  if (needsInit) {
+    const db = makeDb(storeId)
+    await db.migrate()
+    await db.disconnect()
+  }
+
+  // Proxy pull request to Electric server for reading
+  return fetch(url)
+}
+
+// POST /api/electric - Push events (direct database write)
+export async function POST(request: Request) {
+  const payload = await request.json()
+  const parsed = Schema.decodeUnknownSync(ApiSchema.PushPayload)(payload)
+
+  // Write events directly to Postgres table (bypasses Electric)
+  const db = makeDb(parsed.storeId)
+  await db.createEvents(parsed.batch)
+  await db.disconnect()
+
+  return Response.json({ success: true })
+}
+```
+
+### Important Considerations
+
+- **Database Setup**: Ensure your Postgres database is configured for Electric
+- **Authentication**: Implement proper auth checks in your proxy
+- **Error Handling**: Add robust error handling for database operations
+- **Connection Management**: Properly manage database connections
+
+## Example
+
+See the
+[todomvc-sync-electric](https://github.com/livestorejs/livestore/tree/main/examples/web-todomvc-sync-electric)
+example for a complete implementation.
+
+## How the sync provider works
+
+The initial version of the ElectricSQL sync provider will use the server-side
+Postgres DB as a store for the mutation event history.
+
+Events are stored in a table following the pattern
+`eventlog_${PERSISTENCE_FORMAT_VERSION}_${storeId}` where
+`PERSISTENCE_FORMAT_VERSION` is a number that is incremented whenever the
+`sync-electric` internal storage format changes.
+
+## F.A.Q.
+
+### Can I use my existing Postgres database with the sync provider?
+
+Unless the database is already modelled as a eventlog following the
+`@livestore/sync-electric` storage format, you won't be able to easily use your
+existing database with this sync backend implementation.
+
+We might support this use case in the future, you can follow the progress
+[here](https://github.com/livestorejs/livestore/issues/286). Please share any
+feedback you have on this use case there.
+
+### Why do I need an API proxy in front of the ElectricSQL server?
+
+The API proxy is used to handle pull/push requests between LiveStore and ElectricSQL,
+allowing you to implement custom logic such as:
+- Authentication and authorization
+- Rate limiting and quota management
+- Database initialization and migration
+- Custom business logic and validation

package/docs/reference/syncing/sync-provider/s2/index.md

@@ -0,0 +1,230 @@
+# S2
+
+export const CODE = {
+  recordStructureExample: recordStructureExampleCode,
+};
+
+The `@livestore/sync-s2` package lets you sync LiveStore with the official S2 backend (s2.dev).
+
+- Package: `pnpm add @livestore/sync-s2`
+- Protocol: HTTP push/pull, live pull via SSE
+
+## Architecture
+
+```mermaid
+graph LR
+    subgraph Browser
+        LS[LiveStore Client]
+    end
+    
+    subgraph "Your Server"
+        AP[API Proxy<br/>'/api/s2']
+    end
+    
+    subgraph "S2 Cloud"
+        S2[S2 Backend<br/>'*.s2.dev']
+    end
+    
+    LS -->|"GET (pull)<br/>POST (push)<br/>HEAD (ping)"| AP
+    AP -->|"Authenticated<br/>Requests"| S2
+    
+    style LS fill:#e1f5fe
+    style AP fill:#fff3e0
+    style S2 fill:#f3e5f5
+```
+
+The API proxy handles:
+- **Business logic**: Any kind of business logic that is specific to your application (e.g. rate limiting, auth, logging, etc.)
+- **S2 Stream Management**: Creates basins and streams as needed
+- **S2 Request Translation**: Converts LiveStore sync operations to authenticated S2 API calls
+
+## Client Setup
+
+Basic usage in your worker/server code:
+
+## `reference/syncing/s2/client-setup.ts`
+
+```ts filename="reference/syncing/s2/client-setup.ts"
+
+const _backend = makeSyncBackend({
+  endpoint: '/api/s2', // Your API proxy endpoint
+  // more options...
+})
+```
+
+## API Proxy Implementation
+
+S2 requires authentication and stream management that can't be handled directly from the browser. You'll need to implement an API proxy on your server that:
+
+1. **Handles authentication** with S2 using your access token
+2. **Manages basins and streams** (creates them if they don't exist)
+3. **Proxies requests** between LiveStore and S2
+
+Your proxy needs three endpoints:
+
+### Using Helper Functions
+
+The `@livestore/sync-s2` package provides helper functions to simplify the proxy implementation:
+
+## `reference/syncing/s2/api-proxy-implementation.ts`
+
+```ts filename="reference/syncing/s2/api-proxy-implementation.ts"
+
+// Configure S2 connection
+const s2Config: S2Helpers.S2Config = {
+  basin: process.env.S2_BASIN ?? 'your-basin',
+  token: process.env.S2_ACCESS_TOKEN!, // Your S2 access token
+}
+
+// HEAD /api/s2 - Health check/ping
+export async function HEAD() {
+  return new Response(null, { status: 200 })
+}
+
+// GET /api/s2 - Pull events
+export async function GET(request: Request) {
+  const url = new URL(request.url)
+  const args = S2.decodePullArgsFromSearchParams(url.searchParams)
+  const streamName = S2.makeS2StreamName(args.storeId)
+
+  // Ensure basin and stream exist
+  await S2Helpers.ensureBasin(s2Config)
+  await S2Helpers.ensureStream(s2Config, streamName)
+
+  // Build request with appropriate headers and URL
+  // Note: buildPullRequest handles cursor+1 conversion internally
+  const { url: pullUrl, headers } = S2Helpers.buildPullRequest({ config: s2Config, args })
+
+  const res = await fetch(pullUrl, { headers })
+
+  // For live pulls (SSE), proxy the response
+  if (args.live === true) {
+    if (!res.ok) {
+      return S2Helpers.sseKeepAliveResponse()
+    }
+    return new Response(res.body, {
+      status: 200,
+      headers: { 'content-type': 'text/event-stream' },
+    })
+  }
+
+  // For regular pulls
+  if (!res.ok) {
+    return S2Helpers.emptyBatchResponse()
+  }
+
+  const batch = await res.text()
+  return new Response(batch, {
+    headers: { 'content-type': 'application/json' },
+  })
+}
+
+// POST /api/s2 - Push events
+export async function POST(request: Request) {
+  const requestBody = await request.json()
+  const parsed = Schema.decodeUnknownSync(S2.ApiSchema.PushPayload)(requestBody)
+  const streamName = S2.makeS2StreamName(parsed.storeId)
+
+  // Ensure basin and stream exist
+  await S2Helpers.ensureBasin(s2Config)
+  await S2Helpers.ensureStream(s2Config, streamName)
+
+  // Build push request with proper formatting
+  const pushRequests = S2Helpers.buildPushRequests({
+    config: s2Config,
+    storeId: parsed.storeId,
+    batch: parsed.batch,
+  })
+
+  for (const pushRequest of pushRequests) {
+    const res = await fetch(pushRequest.url, {
+      method: 'POST',
+      headers: pushRequest.headers,
+      body: pushRequest.body,
+    })
+
+    if (!res.ok) {
+      return S2Helpers.errorResponse('Push failed', 500)
+    }
+  }
+
+  return S2Helpers.successResponse()
+}
+```
+
+### Cursor Semantics
+
+The S2 sync provider uses a cursor that represents the **last processed record**:
+- The cursor points to the last S2 sequence number we've seen
+- S2's `seq_num` parameter expects where to start reading from (inclusive)
+- The helper functions automatically handle the `+1` conversion: `seq_num = cursor + 1`
+- When starting from the beginning, cursor is `'from-start'` which maps to `seq_num = 0`
+
+### Important Considerations
+
+- **Stream provisioning**: The helper functions provide `ensureBasin()` and `ensureStream()` to handle creation automatically.
+- **Error handling**: The helpers include fallback responses (`emptyBatchResponse()`, `sseKeepAliveResponse()`) to maintain stream continuity during errors.
+- **Authentication**: Store your S2 access token securely (e.g., environment variables).
+- **Rate limiting**: Consider implementing rate limiting to protect your S2 quota.
+- **Response helpers**: Use the provided response helpers (`successResponse()`, `errorResponse()`) for consistent API responses.
+
+## Live Pull (SSE)
+
+S2 provider supports live pulls over Server-Sent Events (SSE). When `live: true` is passed to `pull`, the client:
+- Immediately emits one page (possibly empty) with `pageInfo: NoMore`.
+- Parses SSE frames robustly (multi-line `data:` support) and reacts to typed events:
+  - `event: batch` → parses `data` as S2 `ReadBatch` and emits items.
+  - `event: ping` → ignored; keeps the stream alive.
+  - `event: error` → mapped to `InvalidPullError`.
+
+## Implementation Notes
+
+### Data Storage & Encoding
+
+LiveStore leverages S2 streams for durable event storage. Understanding the mapping between LiveStore concepts and S2 primitives helps developers comprehend the persistence layer, though direct manipulation is discouraged.
+
+#### LiveStore → S2 Mapping
+
+**Store to Stream**: Each LiveStore `storeId` maps to exactly one S2 stream. The stream name is derived from the `storeId` after sanitization to meet S2 naming requirements.
+
+**Event Encoding**: LiveStore events (`AnyEncodedGlobal`) are JSON-serialized and stored as the `body` field of S2 records. Each event contains:
+- `name`: Event type identifier
+- `args`: Event-specific payload data
+- `seqNum`: LiveStore's global event sequence number
+- `parentSeqNum`: Previous event's sequence number for ordering
+- `clientId`: Origin client identifier
+- `sessionId`: Session that created the event
+
+**Record Structure**: When pushed to S2, each LiveStore event becomes one S2 record:
+
+<Code lang="json" code={CODE.recordStructureExample} title="S2 Record Example" />
+
+#### Sequence Number Handling
+
+**LiveStore and S2 maintain completely independent sequence numbering systems**:
+
+- **LiveStore's `seqNum`**: Stored inside the JSON event payload (starts at 0). Used for logical event ordering and cursor management within LiveStore.
+- **S2's `seq_num`**: Assigned by S2 to each record in the stream (also starts at 0). Used solely for stream positioning when reading records.
+
+These are **two separate numbering systems** that happen to both start at 0. While they often align numerically (first event is LiveStore seqNum 0, stored in S2 record with seq_num 0), this is coincidental rather than a direct mapping. The sync provider:
+- Preserves LiveStore's sequence numbers unchanged in the event payload
+- Uses S2's seq_num only for querying records from the stream (e.g., "read from position X")
+- Never relies on S2's seq_num for LiveStore's logical event ordering
+
+#### Technical Details
+
+**Format**: The provider uses `s2-format: raw` when communicating with S2, treating record bodies as UTF-8 JSON strings.
+
+**Headers**: S2 record headers are not utilized; all LiveStore metadata is contained within the JSON body.
+
+**Batch Operations**: Multiple events can be pushed in a single batch, with each event becoming a separate S2 record while maintaining order.
+
+#### Important Note
+
+**Direct stream manipulation is strongly discouraged**. Always interact with S2 streams through LiveStore's sync provider to ensure:
+- Proper event encoding/decoding
+- Sequence number integrity
+- Cursor management consistency
+- Compatibility with LiveStore's sync protocol
+
+Bypassing LiveStore to modify S2 streams directly may corrupt the event log and break synchronization.

package/docs/tutorial/0-welcome/index.md

@@ -0,0 +1,48 @@
+# Overview and prerequisites
+
+## Welcome to the LiveStore tutorial
+
+This tutorial will guide you through the process of building a simple todo app with React, Vite & Tailwind, using LiveStore as its data layer.
+
+It is focused on building a _minimalistic_ application to gradually introduce the main concepts of LiveStore. That being said, LiveStore itself has been specifically designed for large, complex applications, and shines especially when used in these contexts.
+
+:::note[Goal of this tutorial: Education]
+The goal of the tutorial is to _educate_. It reduces as much noise as possible so you can focus on the parts that actually matter for building an application with LiveStore. 
+
+**If you just want to see LiveStore in action and play around with it, consider setting up the [starter project](/getting-started/react-web/) directly.**
+:::
+
+## Prerequisites
+
+### Useful knowledge
+
+While the tutorial is aimed at LiveStore newcomers, it will be helpful to have some knowledge in basic areas of web development, such as of [React](https://react.dev/), [TypeScript](https://www.typescriptlang.org/) and [event-driven architectures](/evaluation/how-livestore-works/#event-sourcing).
+
+### Technical setup
+
+- The tutorial assumes that you're using a Unix-like shell, e.g. by using commands like `mkdir` and `touch` for creating directories and files.
+- It gives you the option to use either [Bun](https://bun.sh/) or [`pnpm`](https://pnpm.io/) as the package manager.
+- You'll deploy the application to Cloudflare. If you don't have an account there yet, you can sign up for a free one [here](https://dash.cloudflare.com/sign-up) (or skip the deployment steps in the tutorial).
+
+## What you'll do
+
+Here's an overview of the steps you'll take in the tutorial:
+
+1. Set up a starter project with React, Vite & Tailwind
+    - This project will be used as a starting point for the tutorial and already comes with basic functionality for a todo app. 
+    - It uses local React state to manage the todo list. Throughout the tutorial you'll gradually replace the ephemeral, local state with LiveStore persistent storage.
+1. Deploy the application to Cloudflare 
+    - This enables you to observe the evolution in behaviour as you're introducing LiveStore.
+1. Add LiveStore to the project
+    - You'll add LiveStore dependencies to the project and implement persisting the todos so that they survive a page refresh.
+1. Automatically sync data to Cloudflare
+    - You'll use LiveStore to set up syncing of the todo list data via Cloudflare Workers and Durable Objects in the background. 
+    - Now your todos will not only survive a page refresh, but also automatically sync across browser sessions, and even across devices.
+1. Expand the business logic with more LiveStore events
+    - You'll learn how to use LiveStore events to expand the business logic of the todo app.
+1. Persist UI state
+    - LiveStore can also be used to persist UI state, such as the text from an input field or a filter selection.
+
+## Credits
+
+This tutorial has been written by [Nikolas Burk](https://x.com/nikolasburk).

package/docs/tutorial/1-setup-starter-project/index.md

@@ -0,0 +1,105 @@
+# 1. Set up starter project with React, Vite & Tailwind
+
+## Set up a starter project with React, Vite & Tailwind
+
+We have prepared a [starter project](https://github.com/livestorejs/livestore/tree/dev/examples/tutorial-starter) for you that you can use as a starting point for the tutorial. Download it via the following command:
+
+<Tabs syncKey="package-manager">
+
+  <TabItem label="bun">
+
+    <Code code={`bunx @livestore/cli@dev create \\\n  --example tutorial-starter livestore-todo-app`} lang="sh" />
+
+  </TabItem>
+
+  <TabItem label="pnpm">
+
+    <Code code={`pnpm dlx @livestore/cli@dev create \\\n  --example tutorial-starter livestore-todo-app`} lang="sh" />
+
+  </TabItem>
+
+</Tabs>
+
+Once you've downloaded the project, you can navigate to the project directory and install the dependencies:
+
+The project currently is set up as follows:
+- Minimal project created via [`vite create`](https://vite.dev/guide/#scaffolding-your-first-vite-project) using React and TypeScript.
+- Using [Tailwind CSS](https://tailwindcss.com/) for styling.
+- Has basic functionality for adding and deleting todos via local [`React.useState`](https://react.dev/learn/state-a-components-memory).
+
+## Understand the current project state
+
+Run the app with:
+
+<Tabs syncKey="package-manager">
+
+  <TabItem label="bun">
+
+    <Code code={`bun dev`} lang="sh" />
+
+  </TabItem>
+
+  <TabItem label="pnpm">
+
+    <Code code={`pnpm dev`} lang="sh" />
+
+  </TabItem>
+
+</Tabs>
+
+Here's the UI you're going to see after adding a few todos:
+
+![](../../../assets/tutorial/chapter-1/0-screenshot-todo-list.png)
+
+Let's take a quick moment to understand how the app is currently implemented:
+
+All relevant code lives in `App.tsx`. Here's a simplified version of it:
+
+```ts
+interface Todo {
+  id: number
+  text: string
+}
+
+function App() {
+  const [todos, setTodos] = useState<Todo[]>([])
+  const [input, setInput] = useState('')
+
+  const addTodo = () => {
+    const newTodo: Todo = {
+      id: Date.now(),
+      text: input
+    }
+    setTodos([...todos, newTodo])
+    setInput('')
+  }
+
+  const deleteTodo = (id: number) => {
+    setTodos(todos.filter(todo => todo.id !== id))
+  }
+
+  return (
+    // Render input text field and todo list ...
+    // ... and invoke `addTodo` and `deleteTodo` 
+    // ... when the buttons are clicked. 
+  )
+}
+```
+
+For any React developer, this is a very familiar setup:
+
+<StateAndUiDiagram class="my-8" />
+
+You have two pieces of state:
+- application state: `todos: Todo[]` → manipulated by the `addTodo` and `deleteTodo` functions.
+- UI state: `input: string` → manipulated when the text in the input field changes.
+
+The "problem" with this code is that the todo items are not _persisted_, meaning they vanish when:
+- the page is refreshed in the browser.
+- the development server is restarted.
+
+In the next chapters, you'll learn how to persist the todos in the list, so that they'll "survive" both actions. 
+
+Even more: They will not only persist, they will automatically sync across multiple browsers tabs/windows, and even across devices—without you needing to think about the syncing logic and managing remote state. 
+
+That's the power of LiveStore!