toiljs 0.0.125 → 0.0.126
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +10 -0
- package/build/cli/.tsbuildinfo +1 -1
- package/build/cli/index.js +2 -2
- package/build/compiler/.tsbuildinfo +1 -1
- package/build/compiler/toil-docs.generated.js +3 -3
- package/build/devserver/.tsbuildinfo +1 -1
- package/build/devserver/daemon/catalog.d.ts +2 -0
- package/build/devserver/daemon/catalog.js +13 -4
- package/build/devserver/daemon/index.js +1 -1
- package/package.json +3 -3
|
@@ -10,9 +10,9 @@ export const TOIL_DOCS = {
|
|
|
10
10
|
"backend/README.md": "# Backend overview\n\nYour backend is TypeScript that toilscript compiles into a small, sandboxed WebAssembly program. You can self-host it with `toiljs build` and `toiljs start`, or run the compiled program on the Dacely edge.\n\n## What the backend is\n\nIn a toiljs project, everything under `server/` is your backend. You write it in TypeScript, the same language as your frontend. But it does not run in a browser and it does not run in Node. Instead, a compiler called **toilscript** turns it into **WebAssembly** (often shortened to **WASM**): a compact, fast, portable binary format that many kinds of servers can run safely.\n\nThat compiled program can be self-hosted or deployed to the **Dacely edge**. Two pieces of jargon to unpack there:\n\n- **The edge** means a fleet of servers spread across many cities around the world. When a user makes a request, it is served by the edge node physically closest to them. Close means fast: less distance for the data to travel, so lower latency. You do not pick a region or manage servers; your one compiled backend runs everywhere at once.\n- **Sandboxed** means your WASM program runs inside a locked box. It cannot open files, reach the operating system, or make raw network connections on its own. The only way it can touch the outside world is through a small, fixed set of **host functions** that toiljs provides (read the request, build a response, query the database, send an email, and so on). Every one of those calls is metered and bounded, so a buggy or hostile backend cannot crash the node or read another app's data. This is what makes it safe to run thousands of different apps on the same shared edge.\n\nYou never call the WASM boundary by hand. You write normal TypeScript classes and functions, tag them with decorators (like `@rest` or `@service`), and the compiler wires everything up.\n\nOther servers can communicate with your backend by sending HTTP requests to private REST endpoints. Protect those endpoints with authentication and shared secrets, and enforce an IP allowlist at your firewall or reverse proxy. An integration server can call external services and send the results to ToilJS through those endpoints.\n\nThe hot request path keeps outbound network I/O out by design so it stays responsive under load. Servers have finite I/O capacity; slow external calls can tie up connections and in-flight work until that capacity is exhausted, even with async code. Put external calls in a separate integration server or [background daemon](../background/daemons.md#outbound-http) so the hot path does not wait on external services.\n\n> New to decorators? A **decorator** is the `@name` you write just above a class or method. It attaches meaning to that code without changing what the code does line by line. toiljs uses decorators to say \"this class is an HTTP controller\" or \"this method is callable from the browser.\" See [Decorators](../concepts/decorators.md).\n\n## The request lifecycle\n\nHere is what happens, end to end, when a browser talks to your backend.\n\n```mermaid\nflowchart TD\n U[\"User's browser\"] -->|\"HTTP request\"| E[\"Nearest Dacely edge node\"]\n E --> S{\"Static file<br/>for this path?\"}\n S -->|\"Yes (GET/HEAD)\"| F[\"Serve the file<br/>(HTML, JS, images)\"]\n S -->|\"No\"| W[\"Your compiled backend<br/>(server.wasm handle)\"]\n W --> H[\"Your handler runs<br/>and returns a Response\"]\n H --> E\n F --> E\n E -->|\"HTTP response\"| U\n```\n\nStep by step:\n\n1. The request lands on the closest edge node.\n2. The edge first checks whether the path is a **static file** it can serve directly (your built frontend: HTML pages, JavaScript bundles, images). If so, it serves the file and never wakes your code. This is fast and free.\n3. Otherwise the edge hands the request to your compiled backend by calling its single WASM export, `handle`.\n4. Inside, toiljs decodes the raw bytes into a friendly [`Request`](./rest.md#the-request-object) object and calls your handler's `handle(req)` method.\n5. Your handler returns a [`Response`](./rest.md#building-a-response). toiljs encodes it back into bytes.\n6. The edge sends that response to the browser.\n\nThe key mental model: your backend is a pure function of the request. Bytes in, bytes out, one request at a time.\n\n## Stateless by default\n\nA **fresh copy** of your handler serves each request. Any fields you set on a controller do not survive to the next request, and the request might even be served by a different edge node on the other side of the world. This is called being **stateless**.\n\nThat is a feature, not a limitation: it is what lets your backend scale to the whole planet with no coordination. When you need data to persist between requests (a user account, a counter, a list of posts), you store it in the built-in global database, **ToilDB**. See [the database section](../database/README.md).\n\n```mermaid\nflowchart LR\n R1[\"Request 1\"] --> B1[\"Handler copy A<br/>(fields reset)\"]\n R2[\"Request 2\"] --> B2[\"Handler copy B<br/>(fields reset)\"]\n B1 --> DB[(\"ToilDB<br/>(shared, persistent)\")]\n B2 --> DB\n```\n\n## The three surfaces\n\nYour backend can expose three different kinds of endpoint. Each is opted into with a decorator, and each has its own page:\n\n| Surface | Decorator | What it is | When to use it |\n| --- | --- | --- | --- |\n| **HTTP REST** | `@rest` + `@get`/`@post`/... | Plain HTTP routes with paths, methods, and status codes. | A public API, webhooks, anything a browser, `curl`, or a third party calls directly. See [REST](./rest.md). |\n| **Typed RPC** | `@service` / `@remote` | Server functions your own frontend calls like local async functions, fully type-checked end to end. | Talking from your own React app to your own backend. See [RPC](./rpc.md). |\n| **Realtime** | `@stream` | A long-lived connection where the server keeps state per connected client. | Chat, live cursors, notifications, anything push-based. See [Realtime](../realtime/README.md). |\n\nREST and RPC are the everyday tools. Most apps use both: REST for anything the outside world calls, RPC for your own frontend. They are not exclusive; you can use all three in one project.\n\nAll of these share the same building block for their data: **`@data` classes**, which are typed structs that travel safely between the browser and your WASM backend. See [Data types](./data.md).\n\n## Where your handler lives\n\nREST, RPC, and streams self-register: you tag a class and the compiler adds it to the right dispatcher. A tiny amount of glue lives in `server/main.ts`, which imports your route files and names your top-level handler class. In a typical project you rarely touch `main.ts`; you add route and service files and they are discovered automatically.\n\nYour handler class extends `ToilHandler` and overrides `handle`. The common pattern is to try the REST dispatcher first, then fall back to your own logic:\n\n```ts\nimport { Method, Request, Response, Rest, ToilHandler } from 'toiljs/server/runtime';\n\nexport class AppHandler extends ToilHandler {\n public handle(req: Request): Response {\n // Try every @rest controller. Returns the first match, or null.\n const hit = Rest.dispatch(req);\n if (hit != null) return hit;\n\n // Your own hand-written endpoints can go here.\n if (req.path == '/health') return Response.text('ok\\n');\n\n // \"I have no answer for this path\": let the edge serve it (a static\n // file, the client app) instead of returning a hard 404.\n return Response.unhandled();\n }\n}\n```\n\nRPC calls (to the reserved path `/__toil_rpc`) are handled by the framework before your `handle` runs, so you do not dispatch them yourself.\n\n### Per-request hooks: `onRequestStarted` and `onRequestCompleted`\n\n`ToilHandler` gives you two optional hooks that run around every request, so you can add cross-cutting logic (logging, timing, metrics) once instead of repeating it in every route, and without re-implementing `handle`:\n\n- **`onRequestStarted(req: Request): void`** runs just **before** `handle(req)`. Override it for per-request setup: start a timer, log the incoming method and path, read a header you need everywhere.\n- **`onRequestCompleted(req: Request, resp: Response): void`** runs just **after** `handle` returns, and it is handed the `Response` that is about to go out. It **also runs when your handler throws**, after the runtime has turned that throw into a `500`, so it is the right place for teardown you always want to happen (record the outcome, stop the timer, emit a metric).\n\nBoth are empty by default, so override only the one you need. The framework calls them around every request; you never call them yourself.\n\n```ts\nimport { Request, Response, Rest, ToilHandler } from 'toiljs/server/runtime';\n\nexport class AppHandler extends ToilHandler {\n public handle(req: Request): Response {\n const hit = Rest.dispatch(req);\n return hit != null ? hit : Response.unhandled();\n }\n\n // Runs before every handle().\n public onRequestStarted(req: Request): void {\n // per-request setup, e.g. note the path you are about to serve\n }\n\n // Runs after every handle(), including after a throw became a 500.\n public onRequestCompleted(req: Request, resp: Response): void {\n // always-run teardown: record the outcome, stop a timer, emit a metric\n }\n}\n```\n\nIf your project is REST-only, you do not even need a custom handler; toiljs ships a ready-made one. See [REST](./rest.md#dispatch-and-the-404-fallback).\n\n## Compute tiers\n\nThe request/response backend described here is the default and most common tier, called **L1**. toiljs also has higher tiers for long-lived connections (streams) and scheduled background work (daemons), each compiled into its own WASM artifact from the same project. You opt into a tier just by adding its entry file and surface decorator. For the full picture, see [Compute tiers](../concepts/tiers.md).\n\n## Related\n\n- [HTTP routes (`@rest`)](./rest.md): paths, methods, params, and responses.\n- [Typed RPC (`@service`/`@remote`)](./rpc.md): calling the server from your frontend with end-to-end types.\n- [Data types (`@data`)](./data.md): the serializable structs everything uses.\n- [The database (ToilDB)](../database/README.md): where persistent state lives.\n- [Compute tiers](../concepts/tiers.md): L1 request, L2/L3 stream, L4 daemon.\n- [Realtime streams](../realtime/README.md): the `@stream` surface.\n",
|
|
11
11
|
"backend/rest.md": "# HTTP routes (`@rest`)\n\nExpose a real HTTP API by tagging a class `@rest` and its methods with an HTTP verb; toiljs generates the router for you.\n\n## Why and when\n\nUse `@rest` when something outside your own frontend needs to call your backend over plain HTTP: a webhook from another service, a public API, a mobile app, a `curl` script, or a browser hitting a URL directly. REST endpoints are ordinary URLs with ordinary HTTP methods and status codes, so any HTTP client understands them.\n\nIf instead you are calling your own backend from your own React frontend, [typed RPC](./rpc.md) is usually nicer (you get end-to-end types and skip URL wrangling). The two live happily side by side.\n\n## The shape of a route\n\nA REST route is a method on a class. The class decorator says where the class is mounted; the method decorator says which HTTP method and path it answers.\n\n```ts\nimport { Response, RouteContext } from 'toiljs/server/runtime';\n\n@rest('players') // this controller is mounted at /players\nclass Players {\n @get('/:id') // answers GET /players/:id\n public get(ctx: RouteContext): Response {\n const id = ctx.param('id');\n return Response.json(`{\"id\":\"${id}\"}`);\n }\n}\n```\n\nSave that file, import it once (see [How routes are discovered](#how-routes-are-discovered)), and `GET /players/42` returns `{\"id\":\"42\"}`. That is a working endpoint.\n\n## `@rest`: mounting a controller\n\n`@rest` marks a class as a route **controller** (a group of related routes) and mounts it at a URL prefix.\n\n```ts\n@rest('api') // mounted at /api\n@rest('/') // or @rest('') : mounted at the root\n@rest({ stream: DataStream.Binary }) // root mount, binary body codec by default\n```\n\n- The string is the mount prefix. `\"api\"`, `\"/api\"`, and `\"api/\"` all normalize to `/api`. `\"\"` and `\"/\"` mean the root.\n- The object form sets class-wide defaults. `stream: DataStream.Binary` makes every route in the class use the binary body codec instead of JSON (more on that in [Request and response bodies](#request-and-response-bodies)).\n\nThe full URL of a route is the controller prefix joined with the method path. With `@rest('api')` and `@get('/todos/:id')`, the route is `GET /api/todos/:id`.\n\n## Verb decorators\n\nEach HTTP method has a decorator that takes the route path:\n\n```ts\n@get('/path') @post('/path') @put('/path') @patch('/path')\n@del('/path') @head('/path') @options('/path')\n```\n\nNote the delete decorator is `@del`, not `@delete` (the word `delete` is reserved in TypeScript, so it cannot be a decorator name).\n\n### `@route`: the explicit form\n\n`@route` is the general form. Reach for it when you want to set the body codec per route, or you just prefer an object:\n\n```ts\n@route({ method: Methods.POST, path: '/upload', stream: DataStream.Binary })\npublic upload(body: FileData): FileResult { /* ... */ }\n```\n\n`method` (from the `Methods` enum) and `path` are required; `stream` is optional and overrides the controller default. `Methods` and `DataStream` are global enums (like the decorators themselves, they need no import).\n\n## Path parameters\n\nA path segment written as `:name` is a **path parameter**: it captures whatever the request has in that position. Read it with `ctx.param('name')`.\n\n```ts\n@get('/todos/:id/items/:itemId')\npublic getItem(ctx: RouteContext): Response {\n const id = ctx.param('id'); // \"42\" for /todos/42/items/9\n const itemId = ctx.param('itemId'); // \"9\"\n return Response.json(`{\"todo\":\"${id}\",\"item\":\"${itemId}\"}`);\n}\n```\n\nMatching is **segment-exact**: the request path must have the same number of `/`-separated segments, the static segments must match literally, and each `:param` captures one segment. The query string is removed before matching. A captured param is always a string; convert it yourself if you need a number (for example `u64.parse(ctx.param('id'))`).\n\n## Method parameters: reading the request\n\nA route method may declare zero, one, or two parameters. toiljs looks at their types to decide what to pass:\n\n- a parameter typed `RouteContext` receives the [request context](#the-routecontext-object) (path params, query, headers, raw body);\n- any other type is treated as the **request body**, decoded into that [`@data`](./data.md) type.\n\n```ts\n@get('/status')\npublic status(): StatusResponse { /* no body, no context */ }\n\n@get('/user/:id')\npublic getUser(ctx: RouteContext): User { /* context only */ }\n\n@post('/create')\npublic create(input: NewTodo): Todo { /* body only */ }\n\n@post('/user/:id/score')\npublic addScore(input: ScoreDelta, ctx: RouteContext): Player {\n const id = ctx.param('id'); // body AND context\n /* ... */\n}\n```\n\nThe order of the two parameters does not matter; toiljs classifies them by type.\n\n## Return types\n\nYou return one of two things from a route, and toiljs encodes it for you:\n\n| You return | toiljs sends |\n| --- | --- |\n| a [`@data`](./data.md) value | The value serialized (JSON by default, binary if the route is in binary mode). Status `200`. |\n| a `Response` | Exactly that response: your status, headers, cookies, and body, untouched. |\n| `void` (nothing) | `204 No Content`. |\n\nReturning a `@data` value is the short path when you just want to send the data. Returning a `Response` gives you full control: a custom status, extra headers, cookies, caching. Use whichever fits.\n\n```ts\n// Short path: return the data, let toiljs serialize it as JSON.\n@get('/me')\npublic me(): Player { return currentPlayer(); }\n\n// Full control: custom 404, a header, and the body serialized by hand.\n@get('/:id')\npublic get(ctx: RouteContext): Response {\n const id = u64.parse(ctx.param('id'));\n if (!store.has(id)) return Response.notFound();\n return Response.json(store.get(id).toJSON().toString())\n .setHeader('cache-control', 'no-store');\n}\n```\n\n## Request and response bodies\n\nEvery route is either a **JSON route** (the default) or a **Binary route**. This decides how request bodies are decoded and response values are encoded:\n\n- **JSON**: the request body is parsed as JSON and revived into your `@data` type; the response value is serialized to JSON. Best for endpoints a browser or a third party calls directly, because JSON is universally readable.\n- **Binary**: the request body and response use toiljs's compact binary codec (`DataWriter`/`DataReader`). Smaller and faster, and exact for very large integers. Best for app-to-app traffic and anything performance sensitive.\n\nSet the mode on the whole controller with `@rest({ stream: DataStream.Binary })`, or per route with `@route({ ..., stream: DataStream.Binary })`. Full detail on both codecs is in [Data types](./data.md).\n\n> Large integers and JSON: numbers of 64 bits or more (`u64`, `i64`, `u128`, `u256`, and friends) are sent over JSON as decimal strings, so they stay exact at any size (plain JSON numbers would lose precision). The generated client turns them back into `bigint`. See [Types](../concepts/types.md).\n\n## The `RouteContext` object\n\n`RouteContext` is your window into the incoming request. toiljs builds one and passes it to any route method that declares a `RouteContext` parameter.\n\n| Member | Signature | What it gives you |\n| --- | --- | --- |\n| `request` | `Request` | The raw request (method, path, headers, body). |\n| `param` | `param(name: string): string` | A captured path param, or `\"\"` if absent. |\n| `query` | `query(name: string): string` | A query-string value (`?q=hi`), or `\"\"` if absent. Not URL-decoded in v1. |\n| `header` | `header(name: string): string \\| null` | A request header, case-insensitive, or `null`. |\n| `text` | `text(): string` | The raw request body decoded as UTF-8 text. |\n| `clientIp` | `clientIp(): string` | The connecting client's IP, or `\"\"` if unavailable. |\n\n`clientIp()` is the real socket address the edge observed, not a forgeable header like `X-Forwarded-For`, so it is safe to key rate limits, geo lookups, or audit logs on it.\n\n## The `Request` object\n\nThe raw request. You get it as `ctx.request`, or directly as the argument to a hand-written `handle(req)`.\n\n**Fields:**\n\n| Field | Type | Notes |\n| --- | --- | --- |\n| `method` | `Method` | `GET`, `POST`, `PUT`, `DELETE`, `PATCH`, `HEAD`, `OPTIONS`, or `UNKNOWN`. |\n| `path` | `string` | The path, including the query string. |\n| `headers` | `Array<Header>` | Ordered list; a `Header` is `{ name, value }`. |\n| `body` | `Uint8Array` | The raw request body bytes. |\n\n**Methods:**\n\n| Method | Signature | Notes |\n| --- | --- | --- |\n| `header` | `header(name: string): string \\| null` | Case-insensitive lookup, `null` if absent. |\n| `cookies` | `cookies(): CookieMap` | Parses the `Cookie` header (values percent-decoded); cached per request. |\n| `cookie` | `cookie(name: string): string \\| null` | A single cookie value, or `null`. |\n\nThe `Method` enum and the `Header` class are exported from `toiljs/server/runtime`.\n\n## Building a response\n\n`Response` is what you return. Create one with a static factory, then chain instance methods to add headers, cookies, and caching. Every instance method returns the same `Response`, so calls chain.\n\n### Static factories\n\n| Factory | Signature | Status | Content-Type |\n| --- | --- | --- | --- |\n| `Response.text` | `text(body: string, status = 200)` | 200 | `text/plain; charset=utf-8` |\n| `Response.html` | `html(body: string, status = 200)` | 200 | `text/html; charset=utf-8` |\n| `Response.json` | `json(body: string, status = 200)` | 200 | `application/json; charset=utf-8` |\n| `Response.bytes` | `bytes(body: Uint8Array, status = 200)` | 200 | `application/octet-stream` |\n| `Response.empty` | `empty(status)` | custom | (none) |\n| `Response.notFound` | `notFound()` | 404 | text |\n| `Response.badRequest` | `badRequest(msg = 'bad request')` | 400 | text |\n| `Response.internalError` | `internalError(msg = 'internal error')` | 500 | text |\n| `Response.unhandled` | `unhandled()` | 404 | text, plus a marker header (see below) |\n\n`Response.json` takes an **already-serialized** string, not an object. Build it from a `@data` value with `value.toJSON().toString()`, or return the `@data` value directly and let toiljs serialize it (usually simpler).\n\n### Instance methods\n\n| Method | Signature | What it does |\n| --- | --- | --- |\n| `setHeader` | `setHeader(name, value): Response` | Appends a header (call again to add more). |\n| `setCookie` | `setCookie(cookie: Cookie): Response` | Appends a `Set-Cookie`. |\n| `setCookieKV` | `setCookieKV(name, value): Response` | Shorthand for a cookie with no attributes. |\n| `clearCookie` | `clearCookie(name, path = '/', domain = ''): Response` | Emits a deletion cookie (empty value, `Max-Age=0`). |\n| `cache` | `cache(edgeTtlMinutes, browserTtlSeconds = 0, privateScope = false, allowAuth = false): Response` | Marks the response cacheable. See [Caching](../services/caching.md). |\n| `cacheFor` | `cacheFor(minutes): Response` | Shorthand for edge-caching for N minutes. |\n\n```ts\nreturn Response.json('{\"id\":42}')\n .setHeader('x-trace', traceId)\n .setCookieKV('seen', '1')\n .cacheFor(5);\n```\n\nSee [Cookies](../services/cookies.md) for the full cookie builder and [Caching](../services/caching.md) for the caching rules.\n\n## How dispatch works\n\nEvery `@rest` controller registers itself into a global `Rest` registry when its module loads. At request time, `Rest.dispatch(req)` walks the controllers and tries their routes.\n\n```mermaid\nflowchart TD\n A[\"Request arrives\"] --> B[\"Rest.dispatch(req)\"]\n B --> C{\"Any controller<br/>route match<br/>the method + path?\"}\n C -->|\"Yes\"| D[\"Run that method\"]\n D --> E[\"Return its Response\"]\n C -->|\"No\"| F[\"dispatch returns null\"]\n F --> G[\"Your fallback logic<br/>(or Response.unhandled)\"]\n```\n\nMatching order is deterministic: controllers are tried in the order their modules load, and routes within a controller in declaration order. The **first match wins**, so put more specific routes before catch-all ones.\n\n## How routes are discovered\n\n`toiljs build` scans every file under `server/` and finds your decorated classes on its own, so a route file is picked up as soon as it exists. To keep a plain `toilscript` run (and your editor) seeing the same set, projects also `import` each route file once in `server/main.ts`:\n\n```ts\n// server/main.ts (excerpt)\nimport './routes/Players';\nimport './routes/Todos';\n```\n\nAdding a new controller is two steps: write the file, add the one-line import.\n\n## Dispatch and the 404 fallback\n\nInside your handler, the usual pattern is: try REST first, then fall through to anything else.\n\n```ts\nconst hit = Rest.dispatch(req); // Response, or null if nothing matched\nif (hit != null) return hit;\nreturn Response.unhandled(); // no route matched here\n```\n\nThere are two different 404s, and the difference matters:\n\n- **`Response.notFound()`** means \"I looked, and that resource does not exist.\" It is sent to the client as a plain `404`.\n- **`Response.unhandled()`** means \"this server has no route for that path.\" It is a `404` carrying a marker header (`x-toil-unhandled`). The edge (and the dev server) reads that marker and tries to serve the path another way: a static file, or the client-side app. The marker is stripped before anything reaches the browser.\n\nRule of thumb: return `unhandled()` when a path is simply not yours to handle, and `notFound()` when the path is yours but the specific thing is missing.\n\nIf your project is REST-only, you do not need a custom handler at all. toiljs ships `RestHandler`, which does exactly the dispatch-then-`unhandled` dance:\n\n```ts\nimport { Server, RestHandler } from 'toiljs/server/runtime';\nServer.handler = () => new RestHandler();\n```\n\n## Guards: auth, rate limits, caching\n\nYou can stack extra decorators on a route (or a whole controller) to protect or cache it. They compose with the verb decorators:\n\n```ts\n@rest('admin')\nclass Admin {\n @get('/stats')\n @auth // reject with 401 if there is no valid session\n @ratelimit(RateLimit.SlidingWindow, 30, 60) // at most 30 requests per 60 seconds\n public stats(): Stats { /* ... */ }\n}\n```\n\n- `@auth` requires a valid signed session, else the request is rejected with `401` before your method runs. See the [Auth guide](../auth/README.md).\n- `@ratelimit` caps how often a caller may hit the route. See [Rate limiting](../services/ratelimit.md).\n- Response caching is opt-in per response with `.cache(...)` / `.cacheFor(...)`. See [Caching](../services/caching.md).\n\n## A complete CRUD example\n\nA small in-memory players API showing create, read, update, and delete. In a real app you would store players in [ToilDB](../database/README.md) instead of a local map, because each request gets a fresh handler (see [statelessness](./README.md#stateless-by-default)); this example keeps it in memory to stay focused on routing.\n\n```ts\n// server/models/NewPlayer.ts\n@data\nexport class NewPlayer {\n name: string = '';\n}\n```\n\n```ts\n// server/models/Player.ts\n@data\nexport class Player {\n id: u64 = 0;\n name: string = '';\n score: i64 = 0;\n}\n```\n\n```ts\n// server/routes/Players.ts\nimport { Response, RouteContext } from 'toiljs/server/runtime';\nimport { NewPlayer } from '../models/NewPlayer';\nimport { Player } from '../models/Player';\n\n// A stand-in store. Real apps use ToilDB (see the database section).\nconst store = new Map<u64, Player>();\nlet nextId: u64 = 1;\n\n@rest('players')\nclass Players {\n // CREATE: POST /players (JSON body -> Player)\n @post('/')\n public create(input: NewPlayer): Player {\n const p = new Player();\n p.id = nextId++;\n p.name = input.name;\n p.score = 0;\n store.set(p.id, p);\n return p; // 200 with the player as JSON\n }\n\n // READ: GET /players/:id\n @get('/:id')\n public get(ctx: RouteContext): Response {\n const id = u64.parse(ctx.param('id'));\n if (!store.has(id)) return Response.notFound();\n return Response.json(store.get(id).toJSON().toString());\n }\n\n // UPDATE: PUT /players/:id (JSON body -> Player)\n @put('/:id')\n public update(input: NewPlayer, ctx: RouteContext): Response {\n const id = u64.parse(ctx.param('id'));\n if (!store.has(id)) return Response.notFound();\n const p = store.get(id);\n p.name = input.name;\n return Response.json(p.toJSON().toString());\n }\n\n // DELETE: DELETE /players/:id\n @del('/:id')\n public remove(ctx: RouteContext): Response {\n const id = u64.parse(ctx.param('id'));\n if (!store.has(id)) return Response.notFound();\n store.delete(id);\n return Response.empty(204); // 204 No Content\n }\n}\n```\n\nCalling it from anywhere:\n\n```sh\ncurl -X POST localhost:5173/players -d '{\"name\":\"Ada\"}'\ncurl localhost:5173/players/1\ncurl -X PUT localhost:5173/players/1 -d '{\"name\":\"Ada Lovelace\"}'\ncurl -X DELETE localhost:5173/players/1\n```\n\nBecause `@rest` also generates a typed fetch client, your React frontend can call the same routes without writing URLs:\n\n```ts\nawait Server.REST.players.create({ body: new NewPlayer('Ada') });\nawait Server.REST.players.get({ params: { id: 1 } });\n```\n\nSee [RPC and the generated client](./rpc.md#the-rest-fetch-client) for that client.\n\n## Gotchas\n\n- **Fields do not persist.** A fresh controller instance serves each request, so instance fields reset every time and are never shared between requests or edge nodes. Persist to [ToilDB](../database/README.md).\n- **`Response.json` wants a string.** Pass an already-serialized JSON string (or return the `@data` value directly). Passing an object will not do what you expect.\n- **`@del`, not `@delete`.** `delete` is a reserved word, so the decorator is `@del`.\n- **Query values are not URL-decoded in v1.** `ctx.query('q')` returns the raw value; decode it yourself if it may contain percent-encoding.\n- **Path matching is exact on segment count.** `/todos/:id` does not match `/todos/1/extra`. Add a route for the longer path.\n- **`notFound()` vs `unhandled()`.** Returning `notFound()` from your top handler stops the edge from falling through to static files or the client app. Use `unhandled()` for \"not my path.\"\n\n## Related\n\n- [Data types (`@data`)](./data.md): the request and response body structs, and the JSON vs binary codecs.\n- [Typed RPC](./rpc.md): call your backend from your frontend with end-to-end types (and the generated REST fetch client).\n- [Backend overview](./README.md): the request lifecycle and handler model.\n- [Route execution and transport limits](../concepts/config.md#route-execution-and-transport-limits): handler budgets, HTTP timeouts, and upload size limits.\n- [Cookies](../services/cookies.md), [Caching](../services/caching.md), [Rate limiting](../services/ratelimit.md): response helpers and guards.\n- [Auth](../auth/README.md): protecting routes with `@auth`.\n- [The database](../database/README.md): persisting data between requests.\n",
|
|
12
12
|
"backend/rpc.md": "# Typed RPC (`@service` / `@remote`)\n\nWrite a server function, tag it `@remote`, and call it from your React frontend like a local async function, with the argument and return types checked end to end.\n\n## What RPC is\n\n**RPC** stands for Remote Procedure Call. The idea is old and simple: call a function that actually runs somewhere else (here, on the edge) as if it were a normal function in your own code. You write a method on the server, and on the client you `await` it. No URLs, no `fetch`, no manual JSON. Just a function call.\n\ntoiljs makes that call **fully typed**. When you build the server, it generates a TypeScript file (`shared/server.ts`) describing every callable function, its arguments, and its return type. Your frontend imports nothing extra: a global object called `Server` is available with all of it typed. If you change a server function's signature and rebuild, your frontend code stops type-checking until you fix the call. The client can never drift from the server.\n\n## Why and when\n\nUse RPC when **your own frontend** talks to **your own backend**. It is the most ergonomic and safest way to do that:\n\n- End-to-end types: rename a field on the server, and the client sees it immediately.\n- No plumbing: no route paths to invent, no request or response shapes to hand-write.\n- Exact numbers: 64-bit-and-larger integers arrive as `bigint`, never rounded.\n\nUse [`@rest`](./rest.md) instead when the caller is **not** your frontend: a webhook, a third-party integration, a public API, a mobile client, or anything that speaks in plain URLs and HTTP methods. RPC uses one internal endpoint and a binary wire format, so it is not meant to be called by hand.\n\nYou can use both in the same project. A common split: RPC for your app's own screens, REST for everything public.\n\n## Declaring callable functions\n\nTwo decorators expose server code to the client:\n\n- **`@remote`** on a top-level function makes it directly callable.\n- **`@service`** on a class groups related `@remote` methods under a namespace.\n\n```ts\n// server/services/Stats.ts\nimport { store } from '../core/store';\n\n@service\nclass Stats {\n @remote\n public playerCount(): i32 {\n return store.size;\n }\n}\n```\n\n```ts\n// server/services/remotes.ts\n@remote\nfunction ping(n: i32): i32 {\n return n + 1;\n}\n```\n\nThat is the whole server side. Build the server and the client can call them.\n\n## Calling from the frontend\n\nThe generated client surfaces everything on a global `Server`. A `@service` becomes a namespace keyed by the class name with a lowercase first letter (`Stats` becomes `stats`). A free `@remote` sits directly on `Server`. Every call returns a `Promise`.\n\n```ts\n// anywhere in your React app, no import needed\nconst count = await Server.stats.playerCount(); // number\nconst next = await Server.ping(41); // number -> 42\n```\n\nAutocomplete, argument checking, and the return type all come from `shared/server.ts`, which the build regenerates every time (and `toiljs dev` regenerates on save), so it is always in sync.\n\n## The round trip\n\nHere is what actually happens under that innocent-looking `await`.\n\n```mermaid\nsequenceDiagram\n participant C as Browser (Server.stats.playerCount)\n participant E as Dacely edge\n participant W as Your backend (WASM)\n C->>C: encode args with DataWriter\n C->>E: POST /__toil_rpc<br/>header dacely-rpc = method id<br/>body = encoded args\n E->>W: dispatch to the matching @remote\n W->>W: run playerCount(), encode the result\n W->>E: encoded result bytes\n E->>C: 200 application/octet-stream\n C->>C: decode into the typed value\n```\n\nA few facts worth knowing:\n\n- Every callable has a stable numeric **method id** (a hash of `\"Service.method\"` or the function name). The client sends it in the `dacely-rpc` header; the server dispatches on it.\n- Arguments and results travel in the compact **binary `@data` codec** (see [Data types](./data.md)), so large integers are exact and payloads are small.\n- The endpoint is a single reserved path, `/__toil_rpc`. You never route it yourself; the framework handles it before your `handle` runs.\n\n## RPC is stateless too\n\nJust like a REST controller, a fresh service instance serves each call. Fields you set on a `@service` class do not survive between calls. If two calls need to share data, that data lives in [ToilDB](../database/README.md), not in an instance field. See [statelessness](./README.md#stateless-by-default).\n\n## Argument and return types\n\nArguments and return values may be scalars, arrays, or [`@data`](./data.md) classes, in both directions. Here is how toilscript's types map to what you see on the TypeScript client:\n\n| ToilScript type | TypeScript type |\n| --- | --- |\n| `u8`, `u16`, `u32`, `i8`, `i16`, `i32`, `f32`, `f64` | `number` |\n| `u64`, `i64`, `u128`, `i128`, `u256`, `i256` | `bigint` |\n| `bool` | `boolean` |\n| `string` | `string` |\n| a `@data` class `T` | `T` (the generated class) |\n| `T[]` | `T[]` |\n\nIntegers of 64 bits or more become `bigint` on the client, so they are exact at any magnitude. See [Types](../concepts/types.md) for the full number story.\n\nPassing a `@data` value is just as easy as a scalar. Construct it on the client and pass it in:\n\n```ts\nimport { NewPlayer } from './shared/server';\n\nconst created = await Server.roster.add(new NewPlayer('Ada')); // returns a typed Player\n```\n\nThe `@data` classes in `shared/server.ts` share a byte-for-byte identical codec with the server, so values round-trip exactly.\n\n## Reading and writing the database from a `@remote`\n\nA `@remote` can use the database, but with a safety default: **a plain `@remote` is read-only.** If it tries to write to ToilDB, the compiler rejects it. To let a `@remote` write, add `@action`:\n\n```ts\n@service\nclass Roster {\n @remote\n public count(): i32 { // read-only: fine to just read\n return db.players.count();\n }\n\n @remote\n @action // opts into writes\n public add(input: NewPlayer): Player {\n return db.players.create(/* ... */);\n }\n}\n```\n\n`@query` is the explicit opposite of `@action`: it marks a function read-only on purpose (the default for a `@remote`, so you rarely need to write it). These are ToilDB **function kinds**; the full rules, including what each kind may and may not do, are in the [database docs](../database/README.md). The takeaway for RPC: reads work out of the box, and a write needs `@action`, so a read-only endpoint can never silently mutate your data.\n\n## Guarding a `@remote`\n\nGuards stack on a `@remote` exactly as they do on a REST route:\n\n```ts\n@service\nclass Stats {\n @remote\n @auth // reject with 401 when there is no valid session\n public secretCount(): i32 {\n return store.size;\n }\n}\n```\n\nThe RPC dispatcher enforces `@auth` (and `@ratelimit`) the same way the REST router does: the guard runs first, and an unauthenticated call gets a `401` before your method body executes. See the [Auth guide](../auth/README.md) and [Rate limiting](../services/ratelimit.md).\n\n## The generated `Server` surface\n\n`shared/server.ts` declares `Server` as a global with a shape like this (schematic):\n\n```ts\ndeclare global {\n const Server: {\n // free @remote functions\n ping(n: number): Promise<number>;\n\n // @service classes, keyed by lowercased name\n readonly stats: {\n playerCount(): Promise<number>;\n secretCount(): Promise<number>;\n };\n\n // @rest controllers get a fetch client under REST (see below)\n readonly REST: { /* ... */ };\n\n // @stream classes get a client under Stream (see the realtime docs)\n readonly Stream: { /* ... */ };\n };\n}\n```\n\nYou never edit this file; the build regenerates it. If a `Server` method throws that it is \"unavailable,\" the generated client has not loaded yet: run the server build (or `toiljs dev`, which does it on save).\n\n## The REST fetch client\n\n`@rest` controllers also get a typed client, under `Server.REST.<controller>.<route>`. It is real `fetch` code (because REST is just HTTP), and it is handy when you want your frontend to call a route you also expose publicly:\n\n```ts\n// controller @rest('players') with a create route taking a NewPlayer body\nconst player = await Server.REST.players.create({\n body: new NewPlayer('Ada'), // present only if the route takes a body\n // params: { id: 7 }, // present only if the path has :params\n query: { ref: 'home' }, // optional\n headers: { 'x-trace': id }, // optional\n});\n```\n\nThe wrapper builds the URL, substitutes `:params`, appends `query`, sends the request, throws on a non-2xx status, and decodes the response into the route's return type. A route declared to return `Response` resolves to the raw `fetch` `Response` so you can inspect headers or stream it yourself. See [REST](./rest.md) for the routes themselves.\n\n## RPC vs REST at a glance\n\n| | RPC (`@service` / `@remote`) | REST (`@rest`) |\n| --- | --- | --- |\n| Caller | Your own frontend | Anyone (browser, webhook, third party, `curl`) |\n| Client | `Server.svc.method(args)` | `Server.REST.ctrl.route(args)`, or plain `fetch` |\n| URL shape | One internal endpoint | Real paths and HTTP methods you design |\n| Wire format | Compact binary `@data` | JSON (or binary), your choice |\n| Types | End to end, automatic | End to end for the generated client |\n| Best for | App-internal calls | Public APIs, integrations, webhooks |\n\n## Gotchas\n\n- **RPC is not a public API.** It uses one reserved endpoint and a binary format meant for the generated client. If an outside system needs to call in, expose a [`@rest`](./rest.md) route.\n- **Instance fields do not persist.** A fresh service instance serves every call. Shared state belongs in [ToilDB](../database/README.md).\n- **Writes need `@action`.** A plain `@remote` is read-only; the compiler rejects a database write unless the method is `@action`.\n- **Rebuild after signature changes.** `shared/server.ts` is generated. If autocomplete looks stale, rebuild the server (`toiljs dev` does this on save).\n- **`bigint`, not `number`, for 64-bit values.** A `u64`/`i64`/`u256` argument or return is a `bigint` on the client. Pass `10n`, not `10`.\n\n## Related\n\n- [Data types (`@data`)](./data.md): the structs your RPC arguments and results are made of, and the binary codec they travel in.\n- [HTTP routes (`@rest`)](./rest.md): the public-facing alternative, and the `Server.REST` fetch client.\n- [Types](../concepts/types.md): `u64`, `u256`, and how they map to `number` / `bigint`.\n- [The database](../database/README.md): `@action` vs `@query`, and persisting state.\n- [Fetching data on the frontend](../frontend/data-fetching.md): using `Server.*` from your React components.\n- [Auth](../auth/README.md): guarding a `@remote` with `@auth`.\n",
|
|
13
|
-
"background/daemons.md": "# Daemons (`@daemon` / `@scheduled`)\n\nA `@daemon` is a single, long-lived background worker for your whole app. You mark a class `@daemon`, add `@scheduled` methods that fire on a timer, and the Dacely edge keeps exactly **one** copy of it running worldwide, restarting it elsewhere if the machine it is on fails.\n\n## What a daemon is\n\nThe word \"daemon\" (say \"DEE-mon\") is an old computing term for a program that runs quietly in the background, not tied to any single user. That is exactly what this is.\n\nCompare the three ways your server code can run:\n\n| Kind | How many run | Lives for |\n| ------------------------------- | ------------------------------------ | ----------------------------- |\n| [Request handler](../backend/rest.md) (`@rest`) | a fresh one per request | one request |\n| [Stream box](../realtime/streams.md) (`@stream`) | one per open connection | one connection |\n| **Daemon** (`@daemon`) | **exactly one for the whole app** | as long as it holds the lease |\n\nBecause it is a single, resident instance, its fields persist across scheduled runs (a request handler forgets everything after each request; a daemon does not). It is the right home for work that must happen **once globally on a cadence**, not once per user and not once per server.\n\n```ts\n@daemon\nclass Jobs {\n @scheduled('1h')\n hourly(): void {\n // Runs once an hour, on the one elected worker. Put recurring background\n // work here: rollups, cleanup, polling an upstream, and so on.\n }\n}\n```\n\n## `@scheduled`: run on a cadence\n\nA `@scheduled` method fires on a schedule. The single string argument is the cadence, and it comes in two flavours.\n\n### Interval schedules\n\nAn **interval** fires every fixed span of time. Write a number followed by a unit: `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). The number must be at least 1, and the span may not exceed 7 days.\n\n```ts\n@scheduled('30s') everyHalfMinute(): void { /* ... */ }\n@scheduled('5m') everyFiveMinutes(): void { /* ... */ }\n@scheduled('1h') hourly(): void { /* ... */ }\n@scheduled('1d') daily(): void { /* ... */ }\n```\n\n### Cron schedules\n\nA **cron** expression fires at wall-clock times (\"every weekday at 9:15\", \"midnight on the first of the month\"). Use it when you care about the actual time of day, not just a repeating gap. A cron spec is five fields separated by spaces, in this order:\n\n```\nminute hour day-of-month month day-of-week\n```\n\ntoiljs recognises a cron spec by the spaces in it (an interval has none).\n\n```ts\n@scheduled('15 9 * * 1-5') // 09:15, Monday to Friday\nweekdayMorning(): void { /* ... */ }\n\n@scheduled('0 0 1 * *') // 00:00 on the 1st of every month\nmonthlyReset(): void { /* ... */ }\n```\n\nCron times are evaluated in **UTC** and are **minute-granular** (the smallest cron step is one minute). A `*` means \"every value\" for that field.\n\n### Rules\n\n- A `@scheduled` method takes **no arguments and returns `void`**.\n- A daemon class may have **several** `@scheduled` methods, each on its own cadence.\n- Because only the one elected worker fires them, a task runs **once per tick for the whole app**, never once per server.\n\n### `onStart`: run once at boot\n\nA daemon may also declare a plain `onStart(): void` method (not decorated). It runs **once**, when the daemon box first starts on the elected worker. Use it to set up state or kick off a long-running loop.\n\nBoth `onStart` and scheduled ticks execute database calls with **Action** permissions on the edge and in dev. Point reads and record writes, event appends, counters, membership mutations, and unique claims are allowed. Scans (`Events.latest`, `Events.since`, `Membership.list`) and `View.publish` are rejected. The `@job` permissions of a separately executed job do not transfer to a helper called by a daemon. Each cold call gets a fresh Action budget: 64 database operations, 32 writes, 64 keys per `getMany`, 128 KiB per database result, and 2 MiB of result bytes in total. Operators may change operation counts and deadlines. Startup has a separate 512,000,000 gas allowance; scheduled ticks use the host's tick gas setting. A startup trap or nonzero `init`/`daemon_start` result is reported as a failed start and tasks are not registered.\n\nUse `Log.info('cleanup started')`, `Log.warn(...)`, or `Log.error(...)` for server logging. These work in request handlers, streams, and daemons. Messages pass UTF-8 bytes to the host and include the tenant identity in edge logs. A message may contain at most 4096 UTF-8 bytes; at most 32 messages are emitted per call. Each method returns `false` when the message allowance is spent. Invalid arguments abort the call. Avoid secrets and personal data in log messages. `console` imports are rejected when compiling a cold daemon artifact.\n\n```ts\n@daemon\nclass Jobs {\n onStart(): void {\n // one-time setup when this daemon becomes active\n }\n\n @scheduled('1h')\n hourly(): void { /* ... */ }\n}\n```\n\n## No backfill: missed runs are skipped, not replayed\n\nThis is the single most important thing to understand about scheduling.\n\nIf the daemon is down when a tick was due (say the leader failed and a standby is still taking over), or if the clock jumps forward, toiljs does **not** go back and run all the ticks you missed. It simply fires the **next** due run and moves on. This is called a **no-backfill** policy.\n\nTwo practical consequences:\n\n1. **Design tasks to be safe to skip.** \"Recompute the summary\" is fine to miss (the next run fixes it). \"Charge every user once\" is not, unless you make it idempotent.\n2. **Make tasks idempotent where a missed run matters.** Idempotent means running it twice (or catching up later) has the same effect as running it once. For example, \"set yesterday's total to X\" is idempotent; \"add 1 to a counter\" is not.\n\n## One global worker, with safe failover\n\nThere is exactly **one** daemon running for your app at any moment. A second machine keeps a **warm standby** ready but idle. If the active worker's hold on the job expires (it crashed, lost the network, or was shut down), the standby takes over and fires the following runs.\n\nThe mechanism is a **lease**. Think of the lease as a \"who is in charge\" token that only one worker can hold at a time, and that has to be renewed to keep. Only the worker holding the lease (the **leader**) runs the schedule.\n\n```mermaid\nsequenceDiagram\n participant A as Worker A (leader)\n participant L as Lease\n participant B as Worker B (standby)\n A->>L: hold the lease, renew it\n Note over A: A fires @scheduled ticks\n Note over B: B stays idle, watching\n A--xL: A crashes, stops renewing\n Note over L: lease expires\n B->>L: acquire the lease\n Note over B: B is now leader, fires the next ticks\n```\n\nThe important guarantee: **two workers never run the same tick at the same time.** This is called **at-most-once** scheduling. The trade-off is the no-backfill behaviour above: to be sure a tick is never run twice, the edge would rather skip the in-flight tick when a leader is lost than risk running it on two machines. You never start, stop, or place the daemon yourself; the edge elects the leader and drives it.\n\n## Leadership fencing: side effects only run on the leader\n\nA subtle risk with a warm standby is a \"split brain\": for a brief moment, two workers might both think they are the leader. To make that harmless, toiljs **fences** every side effect behind a leadership check. A **side effect** is anything that changes the outside world:\n\n- **Database writes** (creating, patching, deleting rows, adding to counters, appending events, publishing views).\n- **Outbound HTTP calls** (`http_call`, described below).\n\nThese run **only** on the confirmed leader. If code that is not the leader tries one, the edge refuses it (a \"not leader\" error) rather than let it happen twice. Plain **reads** and computation are not fenced (they are safe to do anywhere). So even if two workers briefly overlap, only one of them can actually write or call out. You do not write the fencing yourself; it is automatic. The upshot: put your writes and outbound calls in `@scheduled` methods freely, and trust that they happen once.\n\n## `daemon.*` host calls\n\nA daemon has a small set of host abilities beyond ordinary computation:\n\n- **Database access.** A daemon reads and writes [ToilDB](../database/README.md) with the same collection handles you use in a route or a derive (`.get`, `.add`, `.append`, `.publish`, and so on). Writes are leader-fenced as described above.\n- **Outbound HTTP (`http_call`).** A daemon can call an external service (to poll an API, post to a webhook). This is the one place your server code reaches out to the internet, so it is deliberately restricted:\n - It is **leader-only** (fenced, like any side effect).\n - It is **SSRF-bounded**. SSRF (server-side request forgery) is an attack where code is tricked into calling internal addresses it should not reach. The edge resolves the target host and blocks private or internal addresses, so a daemon cannot use `http_call` to poke around inside the network.\n - It is **metered**: making many calls or pulling huge responses costs budget, which caps abuse.\n- **Leadership info** (`is_leader`, `current_epoch`). A daemon can check whether it is currently the leader, which is useful for guarding a long `onStart` loop.\n\n> **Note:** In `toiljs dev` (the single-process local emulator), the daemon is always the leader (there is nothing to fail over to), and `http_call` is stubbed to return a \"call failed\" result rather than make real network requests. Everything else, including the schedule and your database writes, runs exactly as it does on the edge.\n\n### The `Daemon` global\n\nInside a `@daemon` class, `Daemon` is an ambient global (no import), like `Analytics`. It needs `toilscript` 0.1.60 or newer.\n\n| Member | Description |\n| --- | --- |\n| `Daemon.isLeader(): bool` | Whether this worker currently holds the lease. A snapshot: re-check it inside a long task rather than trusting the value it started with. |\n| `Daemon.epoch(): i64` | The monotonic fencing token, bumped on every (re)acquire, or `-1` when this worker is not the leader. Stamp it into work that must not outlive the lease that authorized it. |\n| `Daemon.taskCount(): i32` | How many `@scheduled` tasks are registered. |\n| `Daemon.nextFireMs(taskId: i32): i64` | The next fire time for a task, in epoch milliseconds, or `-1` when the id is unknown or the cron never fires again. |\n| `Daemon.yieldNow(): DaemonError` | Give the edge a chance to observe a lost lease. `DaemonError.None` means you are still the leader. |\n| `Daemon.sleep(ms: i64): DaemonError` | Park the task. The edge clamps this to 3 seconds. `LeaseLost` means the lease went away while you slept, and the task should stop. |\n| `Daemon.httpCall(request, responseCap?)` | Make an outbound HTTP request. Returns a `DaemonHttpResponse`, or `null` on failure. |\n| `Daemon.lastError(): DaemonError` | The failure recorded by the most recent `Daemon` call. |\n\n### Outbound HTTP\n\n`Daemon.httpCall` performs the leader-fenced, SSRF-bounded, metered call described above. Its usage shows up in the `Analytics` counters `daemonHttpCallAttempts` / `daemonHttpCallFailures`.\n\n```ts\n@daemon\nclass Jobs {\n @scheduled('5m')\n poll(): void {\n const req = new DaemonHttpRequest('POST', 'https://api.example.com/events');\n req.header('content-type', 'application/json');\n req.body = Uint8Array.wrap(String.UTF8.encode('{\"ping\":true}'));\n\n const res = Daemon.httpCall(req);\n if (res == null) {\n // Daemon.lastError(): NotLeader, CallFailed, ResponseTooLarge, BadEnvelope\n return;\n }\n if (res.status == 200) App.events.add(new Event(res.text()));\n }\n}\n```\n\nA `null` result is never ambiguous: read `Daemon.lastError()`.\n\n| `DaemonError` | Meaning |\n| --- | --- |\n| `NotLeader` | This worker does not hold the lease. The edge refused without touching the network. |\n| `LeaseLost` | The lease went away mid-call. |\n| `CallFailed` | The request did not complete: blocked by the SSRF guard, timed out, or the transport failed. |\n| `ResponseTooLarge` | The response did not fit your buffer. **The call already happened**, so do not blindly retry it: pass a larger `responseCap` instead. |\n| `BadEnvelope` | Your request broke a host cap (nothing was sent), or the response could not be parsed. |\n\nThe caps the edge enforces, which `httpCall` checks in the guest so an over-cap request never costs you a round trip: the method is 1 to 16 bytes, the URL at most 8 KiB, at most 64 headers of at most 8 KiB each, and a request body of at most 256 KiB. Responses are truncated at 1 MiB; `responseCap` defaults to 64 KiB.\n\nIn `toiljs dev` the emulator returns `CallFailed` for every `httpCall` rather than reaching the network, so a daemon that polls an external service does nothing locally. Everything else behaves as it does on the edge.\n\n## The `main.daemon.ts` file (a separate tier)\n\nLike streams, daemons live in their **own entry file**, `server/main.daemon.ts`, and compile into their **own artifact**, `build/server/release-cold.wasm`. Importing your `@daemon` module there pulls it into that artifact.\n\n```ts\n// server/main.daemon.ts\nimport { revertOnError } from 'toiljs/server/runtime/abort/abort';\n\nimport './daemon/Jobs'; // add each @daemon module here\n\n// NOTE: unlike main.ts / main.stream.ts, the daemon entry does NOT re-export the\n// request runtime. A daemon artifact exposes its schedule hooks, not the request handler.\nexport function abort(message: string, fileName: string, line: u32, column: u32): void {\n revertOnError(message, fileName, line, column);\n}\n```\n\n`toiljs build` produces `release-cold.wasm` automatically when your project declares a `@daemon` surface:\n\n```sh\n$ ls build/server/*.wasm\nbuild/server/release.wasm # L1 request (@rest / @service)\nbuild/server/release-stream.wasm # L2/L3 stream (@stream)\nbuild/server/release-cold.wasm # L4 daemon (@daemon)\n```\n\nThere is **at most one** `@daemon` class per project, and it is a compile error to put a `@daemon` in the request build. See [Compute tiers](../concepts/tiers.md) for how one source tree becomes three artifacts.\n\n## Worked example: a periodic rollup\n\nA common daemon job: every hour, read a running total and store a small summary so pages can show it with one cheap read. This reads a [counter](../database/counters.md) and writes a summary [document](../database/documents.md); the write only takes effect on the leader.\n\n```ts\n// server/models/StatKey.ts and Summary.ts (@data types)\n@data\nclass StatKey {\n name: string = 'signups';\n constructor(name: string = 'signups') { this.name = name; }\n}\n\n@data\nclass Summary {\n total: u64 = 0;\n updatedAt: u64 = 0;\n}\n\n// server/data/StatsDb.ts\n@database\nclass StatsDb {\n @collection static signups: Counter<StatKey>;\n @collection static summary: Documents<StatKey, Summary>;\n}\n\n// server/daemon/Jobs.ts\n@daemon\nclass Jobs {\n @scheduled('1h')\n rollup(): void {\n const key = new StatKey('signups');\n const total = StatsDb.signups.get(key); // a read (allowed anywhere)\n\n const s = new Summary();\n s.total = total;\n s.updatedAt = <u64>(Date.now() / 1000);\n StatsDb.summary.patch(key, s); // a write: runs only on the leader\n }\n}\n```\n\nNow any request handler can serve the summary with a single keyed read, and it is never more than an hour stale. Because the write is fenced to the leader and the rollup is idempotent (it *sets* the total rather than adding to it), a missed or failed-over tick is harmless: the next hourly run simply refreshes it.\n\n## When not to use a daemon\n\n- **When a user is waiting for the result.** Do that work in a [route](../backend/rest.md) or [RPC](../backend/rpc.md), on the request path.\n- **When you need a fast read-view kept in sync with data changes.** That is a [`@derive`](./derive.md), which runs on every write rather than on a timer.\n- **When you need exactly-once, never-skipped execution.** Scheduling is at-most-once with no backfill, so make tasks idempotent or safe to skip. A daemon is not a guaranteed job queue.\n\n## Related\n\n- [Background overview](./README.md): daemons versus `@derive`, and which to reach for.\n- [Derived views (`@derive`)](./derive.md): keep a read-view in sync on every write.\n- [Compute tiers (L1 to L4)](../concepts/tiers.md): the daemon runs on the L4 global tier.\n- [Counters](../database/counters.md), [Documents](../database/documents.md), [Events](../database/events.md): the data a daemon reads and writes.\n",
|
|
13
|
+
"background/daemons.md": "# Daemons (`@daemon` / `@scheduled`)\n\nA `@daemon` is a single, long-lived background worker for your whole app. You mark a class `@daemon`, add `@scheduled` methods that fire on a timer, and the Dacely edge keeps exactly **one** copy of it running worldwide, restarting it elsewhere if the machine it is on fails.\n\n## What a daemon is\n\nThe word \"daemon\" (say \"DEE-mon\") is an old computing term for a program that runs quietly in the background, not tied to any single user. That is exactly what this is.\n\nCompare the three ways your server code can run:\n\n| Kind | How many run | Lives for |\n| ------------------------------- | ------------------------------------ | ----------------------------- |\n| [Request handler](../backend/rest.md) (`@rest`) | a fresh one per request | one request |\n| [Stream box](../realtime/streams.md) (`@stream`) | one per open connection | one connection |\n| **Daemon** (`@daemon`) | **exactly one for the whole app** | as long as it holds the lease |\n\nBecause it is a single, resident instance, its fields persist across scheduled runs (a request handler forgets everything after each request; a daemon does not). It is the right home for work that must happen **once globally on a cadence**, not once per user and not once per server.\n\n```ts\n@daemon\nclass Jobs {\n @scheduled('1h')\n hourly(): void {\n // Runs once an hour, on the one elected worker. Put recurring background\n // work here: rollups, cleanup, polling an upstream, and so on.\n }\n}\n```\n\n## `@scheduled`: run on a cadence\n\nA `@scheduled` method fires on a schedule. The single string argument is the cadence, and it comes in two flavours.\n\n### Interval schedules\n\nAn **interval** fires every fixed span of time. Write a number followed by a unit: `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). The number must be at least 1, and the span may not exceed 7 days.\n\n```ts\n@scheduled('30s') everyHalfMinute(): void { /* ... */ }\n@scheduled('5m') everyFiveMinutes(): void { /* ... */ }\n@scheduled('1h') hourly(): void { /* ... */ }\n@scheduled('1d') daily(): void { /* ... */ }\n```\n\n### Cron schedules\n\nA **cron** expression fires at wall-clock times (\"every weekday at 9:15\", \"midnight on the first of the month\"). Use it when you care about the actual time of day, not just a repeating gap. A cron spec is five fields separated by spaces, in this order:\n\n```\nminute hour day-of-month month day-of-week\n```\n\ntoiljs recognises a cron spec by the spaces in it (an interval has none).\n\n```ts\n@scheduled('15 9 * * 1-5') // 09:15, Monday to Friday\nweekdayMorning(): void { /* ... */ }\n\n@scheduled('0 0 1 * *') // 00:00 on the 1st of every month\nmonthlyReset(): void { /* ... */ }\n```\n\nCron times are evaluated in **UTC** and are **minute-granular** (the smallest cron step is one minute). A `*` means \"every value\" for that field.\n\n### Rules\n\n- A `@scheduled` method takes **no arguments and returns `void`**.\n- A daemon class may have **several** `@scheduled` methods, each on its own cadence.\n- Because only the one elected worker fires them, a task runs **once per tick for the whole app**, never once per server.\n\n### `onStart`: run once at boot\n\nA daemon may also declare a plain `onStart(): void` method (not decorated). It runs **once**, when the daemon box first starts on the elected worker. Use it to set up state or kick off a long-running loop.\n\n`onStart` and untagged scheduled ticks execute database calls with **Action** permissions on the edge and in dev. Point reads and record writes, event appends, counters, membership mutations, and unique claims are allowed. Scans (`Events.latest`, `Events.since`, `Membership.list`) and `View.publish` are rejected. A scheduled method tagged `@job` instead runs with **Job** permissions: scans, arbitrary record writes, and `View.publish` are allowed, with the Job database budget and deadline. This requires toilscript 0.1.65, toiljs 0.0.126, and an edge host supporting daemon catalog v2. Older edge hosts reject a Job catalog at load time. Tagging a helper `@job` does not elevate startup or an untagged tick that calls it. Each Action call gets a fresh budget: 64 database operations, 32 writes, 64 keys per `getMany`, 128 KiB per database result, and 2 MiB of result bytes in total. Each Job tick gets a fresh background budget: 65,535 operations and writes, 1024 keys per `getMany`, 1 MiB per database result, and 16 MiB of result bytes in total. Operators may change operation counts and deadlines. Startup has a separate 512,000,000 gas allowance; scheduled ticks use the host's tick gas setting. A startup trap or nonzero `init`/`daemon_start` result is reported as a failed start and tasks are not registered.\n\nUse `Log.info('cleanup started')`, `Log.warn(...)`, or `Log.error(...)` for server logging. These work in request handlers, streams, and daemons. Messages pass UTF-8 bytes to the host and include the tenant identity in edge logs. A message may contain at most 4096 UTF-8 bytes; at most 32 messages are emitted per call. Each method returns `false` when the message allowance is spent. Invalid arguments abort the call. Avoid secrets and personal data in log messages. `console` imports are rejected when compiling a cold daemon artifact.\n\n```ts\n@daemon\nclass Jobs {\n onStart(): void {\n // one-time setup when this daemon becomes active\n }\n\n @scheduled('1h')\n hourly(): void { /* ... */ }\n}\n```\n\n## No backfill: missed runs are skipped, not replayed\n\nThis is the single most important thing to understand about scheduling.\n\nIf the daemon is down when a tick was due (say the leader failed and a standby is still taking over), or if the clock jumps forward, toiljs does **not** go back and run all the ticks you missed. It simply fires the **next** due run and moves on. This is called a **no-backfill** policy.\n\nTwo practical consequences:\n\n1. **Design tasks to be safe to skip.** \"Recompute the summary\" is fine to miss (the next run fixes it). \"Charge every user once\" is not, unless you make it idempotent.\n2. **Make tasks idempotent where a missed run matters.** Idempotent means running it twice (or catching up later) has the same effect as running it once. For example, \"set yesterday's total to X\" is idempotent; \"add 1 to a counter\" is not.\n\n## One global worker, with safe failover\n\nThere is exactly **one** daemon running for your app at any moment. A second machine keeps a **warm standby** ready but idle. If the active worker's hold on the job expires (it crashed, lost the network, or was shut down), the standby takes over and fires the following runs.\n\nThe mechanism is a **lease**. Think of the lease as a \"who is in charge\" token that only one worker can hold at a time, and that has to be renewed to keep. Only the worker holding the lease (the **leader**) runs the schedule.\n\n```mermaid\nsequenceDiagram\n participant A as Worker A (leader)\n participant L as Lease\n participant B as Worker B (standby)\n A->>L: hold the lease, renew it\n Note over A: A fires @scheduled ticks\n Note over B: B stays idle, watching\n A--xL: A crashes, stops renewing\n Note over L: lease expires\n B->>L: acquire the lease\n Note over B: B is now leader, fires the next ticks\n```\n\nThe important guarantee: **two workers never run the same tick at the same time.** This is called **at-most-once** scheduling. The trade-off is the no-backfill behaviour above: to be sure a tick is never run twice, the edge would rather skip the in-flight tick when a leader is lost than risk running it on two machines. You never start, stop, or place the daemon yourself; the edge elects the leader and drives it.\n\n## Leadership fencing: side effects only run on the leader\n\nA subtle risk with a warm standby is a \"split brain\": for a brief moment, two workers might both think they are the leader. To make that harmless, toiljs **fences** every side effect behind a leadership check. A **side effect** is anything that changes the outside world:\n\n- **Database writes** (creating, patching, deleting rows, adding to counters, appending events, publishing views).\n- **Outbound HTTP calls** (`http_call`, described below).\n\nThese run **only** on the confirmed leader. If code that is not the leader tries one, the edge refuses it (a \"not leader\" error) rather than let it happen twice. Plain **reads** and computation are not fenced (they are safe to do anywhere). So even if two workers briefly overlap, only one of them can actually write or call out. You do not write the fencing yourself; it is automatic. The upshot: put your writes and outbound calls in `@scheduled` methods freely, and trust that they happen once.\n\n## `daemon.*` host calls\n\nA daemon has a small set of host abilities beyond ordinary computation:\n\n- **Database access.** A daemon reads and writes [ToilDB](../database/README.md) with the same collection handles you use in a route or a derive (`.get`, `.add`, `.append`, `.publish`, and so on). Writes are leader-fenced as described above.\n- **Outbound HTTP (`http_call`).** A daemon can call an external service (to poll an API, post to a webhook). This is the one place your server code reaches out to the internet, so it is deliberately restricted:\n - It is **leader-only** (fenced, like any side effect).\n - It is **SSRF-bounded**. SSRF (server-side request forgery) is an attack where code is tricked into calling internal addresses it should not reach. The edge resolves the target host and blocks private or internal addresses, so a daemon cannot use `http_call` to poke around inside the network.\n - It is **metered**: making many calls or pulling huge responses costs budget, which caps abuse.\n- **Leadership info** (`is_leader`, `current_epoch`). A daemon can check whether it is currently the leader, which is useful for guarding a long `onStart` loop.\n\n> **Note:** In `toiljs dev` (the single-process local emulator), the daemon is always the leader (there is nothing to fail over to), and `http_call` is stubbed to return a \"call failed\" result rather than make real network requests. Everything else, including the schedule and your database writes, runs exactly as it does on the edge.\n\n### The `Daemon` global\n\nInside a `@daemon` class, `Daemon` is an ambient global (no import), like `Analytics`. It needs `toilscript` 0.1.60 or newer.\n\n| Member | Description |\n| --- | --- |\n| `Daemon.isLeader(): bool` | Whether this worker currently holds the lease. A snapshot: re-check it inside a long task rather than trusting the value it started with. |\n| `Daemon.epoch(): i64` | The monotonic fencing token, bumped on every (re)acquire, or `-1` when this worker is not the leader. Stamp it into work that must not outlive the lease that authorized it. |\n| `Daemon.taskCount(): i32` | How many `@scheduled` tasks are registered. |\n| `Daemon.nextFireMs(taskId: i32): i64` | The next fire time for a task, in epoch milliseconds, or `-1` when the id is unknown or the cron never fires again. |\n| `Daemon.yieldNow(): DaemonError` | Give the edge a chance to observe a lost lease. `DaemonError.None` means you are still the leader. |\n| `Daemon.sleep(ms: i64): DaemonError` | Park the task. The edge clamps this to 3 seconds. `LeaseLost` means the lease went away while you slept, and the task should stop. |\n| `Daemon.httpCall(request, responseCap?)` | Make an outbound HTTP request. Returns a `DaemonHttpResponse`, or `null` on failure. |\n| `Daemon.lastError(): DaemonError` | The failure recorded by the most recent `Daemon` call. |\n\n### Outbound HTTP\n\n`Daemon.httpCall` performs the leader-fenced, SSRF-bounded, metered call described above. Its usage shows up in the `Analytics` counters `daemonHttpCallAttempts` / `daemonHttpCallFailures`.\n\n```ts\n@daemon\nclass Jobs {\n @scheduled('5m')\n poll(): void {\n const req = new DaemonHttpRequest('POST', 'https://api.example.com/events');\n req.header('content-type', 'application/json');\n req.body = Uint8Array.wrap(String.UTF8.encode('{\"ping\":true}'));\n\n const res = Daemon.httpCall(req);\n if (res == null) {\n // Daemon.lastError(): NotLeader, CallFailed, ResponseTooLarge, BadEnvelope\n return;\n }\n if (res.status == 200) App.events.add(new Event(res.text()));\n }\n}\n```\n\nA `null` result is never ambiguous: read `Daemon.lastError()`.\n\n| `DaemonError` | Meaning |\n| --- | --- |\n| `NotLeader` | This worker does not hold the lease. The edge refused without touching the network. |\n| `LeaseLost` | The lease went away mid-call. |\n| `CallFailed` | The request did not complete: blocked by the SSRF guard, timed out, or the transport failed. |\n| `ResponseTooLarge` | The response did not fit your buffer. **The call already happened**, so do not blindly retry it: pass a larger `responseCap` instead. |\n| `BadEnvelope` | Your request broke a host cap (nothing was sent), or the response could not be parsed. |\n\nThe caps the edge enforces, which `httpCall` checks in the guest so an over-cap request never costs you a round trip: the method is 1 to 16 bytes, the URL at most 8 KiB, at most 64 headers of at most 8 KiB each, and a request body of at most 256 KiB. Responses are truncated at 1 MiB; `responseCap` defaults to 64 KiB.\n\nIn `toiljs dev` the emulator returns `CallFailed` for every `httpCall` rather than reaching the network, so a daemon that polls an external service does nothing locally. Everything else behaves as it does on the edge.\n\n## The `main.daemon.ts` file (a separate tier)\n\nLike streams, daemons live in their **own entry file**, `server/main.daemon.ts`, and compile into their **own artifact**, `build/server/release-cold.wasm`. Importing your `@daemon` module there pulls it into that artifact.\n\n```ts\n// server/main.daemon.ts\nimport { revertOnError } from 'toiljs/server/runtime/abort/abort';\n\nimport './daemon/Jobs'; // add each @daemon module here\n\n// NOTE: unlike main.ts / main.stream.ts, the daemon entry does NOT re-export the\n// request runtime. A daemon artifact exposes its schedule hooks, not the request handler.\nexport function abort(message: string, fileName: string, line: u32, column: u32): void {\n revertOnError(message, fileName, line, column);\n}\n```\n\n`toiljs build` produces `release-cold.wasm` automatically when your project declares a `@daemon` surface:\n\n```sh\n$ ls build/server/*.wasm\nbuild/server/release.wasm # L1 request (@rest / @service)\nbuild/server/release-stream.wasm # L2/L3 stream (@stream)\nbuild/server/release-cold.wasm # L4 daemon (@daemon)\n```\n\nThere is **at most one** `@daemon` class per project, and it is a compile error to put a `@daemon` in the request build. See [Compute tiers](../concepts/tiers.md) for how one source tree becomes three artifacts.\n\n## Worked example: a periodic rollup\n\nA common daemon job: every hour, read a running total and store a small summary so pages can show it with one cheap read. This reads a [counter](../database/counters.md) and writes a summary [document](../database/documents.md); the write only takes effect on the leader.\n\n```ts\n// server/models/StatKey.ts and Summary.ts (@data types)\n@data\nclass StatKey {\n name: string = 'signups';\n constructor(name: string = 'signups') { this.name = name; }\n}\n\n@data\nclass Summary {\n total: u64 = 0;\n updatedAt: u64 = 0;\n}\n\n// server/data/StatsDb.ts\n@database\nclass StatsDb {\n @collection static signups: Counter<StatKey>;\n @collection static summary: Documents<StatKey, Summary>;\n}\n\n// server/daemon/Jobs.ts\n@daemon\nclass Jobs {\n @scheduled('1h')\n rollup(): void {\n const key = new StatKey('signups');\n const total = StatsDb.signups.get(key); // a read (allowed anywhere)\n\n const s = new Summary();\n s.total = total;\n s.updatedAt = <u64>(Date.now() / 1000);\n StatsDb.summary.patch(key, s); // a write: runs only on the leader\n }\n}\n```\n\nNow any request handler can serve the summary with a single keyed read, and it is never more than an hour stale. Because the write is fenced to the leader and the rollup is idempotent (it *sets* the total rather than adding to it), a missed or failed-over tick is harmless: the next hourly run simply refreshes it.\n\n## When not to use a daemon\n\n- **When a user is waiting for the result.** Do that work in a [route](../backend/rest.md) or [RPC](../backend/rpc.md), on the request path.\n- **When you need a fast read-view kept in sync with data changes.** That is a [`@derive`](./derive.md), which runs on every write rather than on a timer.\n- **When you need exactly-once, never-skipped execution.** Scheduling is at-most-once with no backfill, so make tasks idempotent or safe to skip. A daemon is not a guaranteed job queue.\n\n## Related\n\n- [Background overview](./README.md): daemons versus `@derive`, and which to reach for.\n- [Derived views (`@derive`)](./derive.md): keep a read-view in sync on every write.\n- [Compute tiers (L1 to L4)](../concepts/tiers.md): the daemon runs on the L4 global tier.\n- [Counters](../database/counters.md), [Documents](../database/documents.md), [Events](../database/events.md): the data a daemon reads and writes.\n",
|
|
14
14
|
"background/derive.md": "# Derived views (`@derive`)\n\nA `@derive` keeps a fast, precomputed **View** of your data in sync automatically. You write a method that reads your source data and publishes a summary; toiljs re-runs it whenever the source data changes, so your pages can read the summary with one cheap lookup instead of doing the work on every request.\n\n## The problem it solves\n\nSome reads are expensive. \"Show the 10 newest comments\" means scanning an [events](../database/events.md) log. \"Show the leaderboard\" means totalling up scores. A **scan** (walking many rows to build a result) can fan out across an unbounded amount of data, and that is too slow and too unpredictable to do while a user waits.\n\nSo toiljs **bars scans on the request path**. A [route](../backend/rest.md) handler runs under a restricted mode:\n\n- A `@get` runs as a **query** (reads only, no scans).\n- A `@post` / `@put` / `@patch` / `@del` runs as an **action** (keyed reads and writes, still no scans).\n\nIf you cannot scan in a route, how do you show \"the latest 10\"? You precompute it. A `@derive` does the scan **off** the request path, folds the result into a [View](../database/views.md) (a read-optimized snapshot stored by key), and your route reads that View with a single keyed lookup, which is not a scan and so is allowed.\n\n```mermaid\nflowchart LR\n W[\"POST /guestbook<br/>(action: append + count)\"] --> S[(Events + Counter<br/>source data)]\n S -->|write triggers| D[\"@derive recompute()<br/>scans, builds, publishes\"]\n D --> V[(View<br/>precomputed snapshot)]\n G[\"GET /guestbook<br/>(query: one keyed read)\"] --> V\n```\n\n## A worked example: a guestbook\n\nHere is the whole pattern in one database class: an [events](../database/events.md) log of signatures, a [counter](../database/counters.md) of how many there are, and a [View](../database/views.md) that holds the ready-to-serve page.\n\n```ts\n@data\nclass GuestKey {\n room: string = 'main';\n constructor(room: string = 'main') { this.room = room; }\n}\n\n@database\nclass GuestbookDb {\n @collection static entries: Events<GuestKey, GuestEntry>; // a source: the log\n @collection static totals: Counter<GuestKey>; // a source: the count\n @collection static book: View<GuestKey, GuestbookView>; // the view we publish\n\n // Recompute the view from the sources. It MAY scan and publish; a route may not.\n @derive\n recompute(): void {\n const key = new GuestKey('main');\n const view = new GuestbookView();\n view.total = GuestbookDb.totals.get(key); // a keyed read\n view.entries = GuestbookDb.entries.latest(key, 10); // a scan: allowed here\n GuestbookDb.book.publish(key, view); // publish the snapshot\n }\n}\n```\n\nThe route then writes to the sources and reads the view:\n\n```ts\n@rest('guestbook')\nclass Guestbook {\n @get('/')\n list(): GuestbookView {\n const key = new GuestKey('main');\n const view = GuestbookDb.book.get(key); // one keyed read, not a scan\n return view == null ? new GuestbookView() : view; // empty until first publish\n }\n\n @post('/')\n sign(input: NewMessage): GuestbookView {\n const key = new GuestKey('main');\n GuestbookDb.entries.append(key, new GuestEntry(input.author, input.message, 0));\n GuestbookDb.totals.add(key, 1);\n // The @derive republishes `book` right after this action returns, so GET\n // serves the new entry. The action just acks with the new total (a counter\n // read is allowed here; scanning the entries list is not).\n const view = new GuestbookView();\n view.total = GuestbookDb.totals.get(key);\n return view;\n }\n}\n```\n\nSign the guestbook twice and the total climbs across requests, because the data lives in the database (and its view), not in module memory.\n\n## What a `@derive` may do\n\nA `@derive` runs under a special **derive** mode that is more powerful than a route:\n\n| Ability | Query (`@get`) | Action (`@post` ...) | Derive |\n| -------------------------------------------- | :------------: | :------------------: | :----------: |\n| Keyed reads (`.get`, counter total) | yes | yes | yes |\n| Writes (`create`, `patch`, `add`, `append`) | no | yes | yes |\n| **Scans** (`events.latest`, membership list) | **no** | **no** | **yes** |\n| `view.publish` / `view.append` | no | no | yes |\n\nSo a derive is exactly the place to do the reads and scans a route cannot, and to publish the result.\n\n## Declaring a derive\n\nA `@derive` is a method on your [`@database`](../database/setup.md) class, next to the collections it reads and the View it writes.\n\n```ts\n@database\nclass MyDb {\n @collection static events: Events<Key, Fact>; // a source\n @collection static home: View<Key, HomePage>; // the materialized view\n\n @derive\n rebuild(): void {\n // read sources, build the value, publish it\n }\n}\n```\n\nRules:\n\n- A `@derive` method takes **no arguments and returns `void`**.\n- A database may declare **multiple** `@derive` methods; each runs independently.\n- The View value and its key are ordinary [`@data`](../backend/data.md) types, so they round-trip through the codec like any other stored value.\n\n## When a derive runs\n\nYou never call a derive yourself. The runtime runs it for you at two moments:\n\n1. **After a write to a source.** When a request, job, or daemon writes one of the database's source collections (an `events.append`, a `counter.add`, a document `create` or `patch`), that database's derives run **right after the response is produced**, so the view reflects the new data on the next read. Cold artifacts include the shared databases' materializers and run them after startup and scheduled writes without needing an HTTP request. Rebuild cold artifacts with the updated compiler to get this behavior. Older cold artifacts without materializers retain their deferred hot-request fallback. Many writes to one database in a single request are **coalesced** into one recompute (it does not run once per write).\n\n2. **On box load.** When a server box starts, hot-reloads, or notices the underlying data changed out of band, the views are rebuilt from their sources **before the first read is served**. This is also where a value type's [`@migrate`](../backend/data.md) runs against old stored events, as the derive re-reads and republishes them.\n\nA derive's own `view.publish` never re-triggers it, so there is no infinite loop.\n\nA failed cold fold retains its pending work and leaves its checkpoint unchanged.\nThe daemon scheduler retries pending folds on its next iteration, including idle\niterations when no scheduled task is due, while it holds the leadership lease.\nDev retries once per second. No HTTP request is needed to trigger a retry.\n\nThe same code runs under `toiljs dev` (the in-process emulator) and on the production edge, with no flags or wiring to change.\n\n## Folding a growing log incrementally\n\nThe guestbook above uses `events.latest` and **recomputes** the view from scratch on every change. That is simple, correct, and the right default for a **bounded** read: the latest N, a counter total, a small set.\n\nSome views instead fold an **unbounded** log: a running total over every event ever, an activity rollup, an audit summary. Rescanning the whole log on every change gets slower as it grows. For those, read the source with [`events.since`](../database/events.md) instead of `latest`. `since` hands you only the events you have **not folded yet**, so the derive folds forward incrementally.\n\n```ts\n@derive\nrollup(): void {\n const key = new StatsKey('all');\n const view = StatsDb.summary.get(key) ?? new Summary(); // the running view so far\n let batch = StatsDb.events.since(key, 500);\n while (batch.length > 0) { // drain the new events in bounded batches\n for (let i = 0; i < batch.length; i++) view.apply(batch[i]);\n batch = StatsDb.events.since(key, 500);\n }\n StatsDb.summary.publish(key, view);\n}\n```\n\nThe host owns the cursor, you never manage it: it seeds `since` from a durable checkpoint, advances it as it hands you events, and saves it **only after** your `publish` lands (so a crash re-folds the batch instead of skipping it). On the next trigger the derive resumes exactly where it left off.\n\n**One rule: the fold must be idempotent per event.** A rare crash-recovery case (an event that \"heals\" at an older position after the derive already passed it) makes the host re-read the whole log that one run, so applying an event twice must not change the result. Use set-style updates keyed by the event's id, not blind `count += 1` accumulation. If your fold cannot be idempotent, stay on the simple `latest` recompute path.\n\n## Guarantees and limitations\n\n**Guarantees**\n\n- **It converges to a correct snapshot.** Publishes are last-writer-wins (the host versions each publish so a later one always supersedes an earlier one), and a derive recomputes from the source of truth, so the view always ends up matching its sources.\n- **Reads stay cheap.** A route serves the view with a single keyed lookup, never a scan.\n\n**Limitations (read these)**\n\n- **By default it recomputes from scratch each time.** A derive re-reads its sources and republishes on every trigger. That is the simple path and a great fit for a **bounded** read: the latest N, a counter total, a small set. To fold an **unbounded, ever-growing** log efficiently, read it with [`events.since`](../database/events.md) instead of `latest` (see [Folding a growing log incrementally](#folding-a-growing-log-incrementally) above); it folds only the new events since a checkpoint.\n- **It is eventually consistent, by a moment.** The view is republished right after the writing request finishes, so there is a tiny window where a reader could see the pre-write view. For most pages (a feed, a leaderboard) that is invisible and fine.\n- **`view.publish` is derive-only.** Routes cannot publish; they can only read the view. That is the whole point: the expensive build happens off the request path.\n\n## When not to use a derive\n\n- **When the read is already cheap.** If a route can answer with a plain keyed `.get`, you do not need a view at all.\n- **When you need it on a timer, not on a data change.** Recomputing \"yesterday's report\" at 2am is a [daemon](./daemons.md) job, not a derive (a derive is triggered by writes, not by the clock).\n- **When the source is an unbounded full-history fold.** See the limitation above; keep derives to bounded reads.\n\n## Related\n\n- [Views](../database/views.md): the `View<K, V>` family a derive publishes into, and how to read it.\n- [Events](../database/events.md): the append-only log a derive commonly folds into a view.\n- [Counters](../database/counters.md): running totals a derive can read.\n- [Background overview](./README.md): `@derive` versus `@daemon`, and which to reach for.\n- [Data types (`@data`)](../backend/data.md): the value and key types a view stores.\n",
|
|
15
|
-
"background/README.md": "# Background work\n\nBackground work is code that runs **without a user waiting for it**: on a timer, once globally, or automatically after your data changes. toiljs gives you two tools for it, `@daemon` and `@derive`, and this page helps you pick the right one.\n\n## Why background work\n\nMost of your server code runs **because a user asked**: a browser hits a [route](../backend/rest.md) or calls an [RPC](../backend/rpc.md), your handler runs, and a response goes back. But some work does not belong on that path:\n\n- It should happen **on a schedule** (every hour, every night at 2am), not when a request happens to arrive.\n- It would be **too slow** to do inside a request, so you want it precomputed and ready.\n- It must happen **exactly once across the whole world**, not once per user and not once per server.\n\nThat is background work. In toiljs there are two kinds, and they solve two different problems.\n\n## The two tools\n\n```mermaid\nflowchart TD\n Q{What do you need?}\n Q -->|Run on a timer, or once globally| D[\"@daemon + @scheduled<br/>a single background worker\"]\n Q -->|Keep a fast read-view in sync with your data| V[\"@derive<br/>maintains a materialized View\"]\n D --> DD[\"e.g. nightly cleanup, hourly poll,<br/>periodic rollup\"]\n V --> VV[\"e.g. a leaderboard, a 'latest 10' feed,<br/>a home-page summary\"]\n```\n\n### `@daemon`: a scheduled, global worker\n\nA [daemon](./daemons.md) is one long-lived background worker for your whole app. It runs on a **schedule** you set (an interval like every 5 minutes, or a cron time like \"9:15 on weekdays\"), and there is exactly **one** of it worldwide at any moment (a \"singleton\"). Use it for work that is driven by **time** or that must run **once globally**:\n\n- clean up stale rows every night;\n- poll an external API every few minutes;\n- send a daily digest email;\n- roll up yesterday's numbers into a summary.\n\n### `@derive`: keep a read-view up to date\n\nA [derive](./derive.md) is not on a timer. It runs **automatically whenever the data it depends on changes**, and its job is to keep a precomputed **View** (a read-optimized copy of your data) fresh. Use it when a page needs data that is **expensive to compute on every read**, like a leaderboard or a \"latest 10 comments\" list, so the read itself stays a single cheap lookup:\n\n- fold an [events](../database/events.md) log into a \"latest N\" list;\n- total up [counters](../database/counters.md) into a scoreboard;\n- assemble a home-page summary from several sources.\n\n## Which one do I reach for?\n\n| Question | Use |\n| ----------------------------------------------------- | ------------ |\n| \"Run this every hour / at midnight.\" | `@daemon` |\n| \"Do this once for the whole app, not per server.\" | `@daemon` |\n| \"Poll or call an outside service on a schedule.\" | `@daemon` |\n| \"This page's data is too slow to compute per request.\"| `@derive` |\n| \"Keep a leaderboard / feed in sync as data changes.\" | `@derive` |\n\nA simple rule of thumb: if the trigger is **the clock**, use a `@daemon`. If the trigger is **a change to your data** and the goal is a fast read, use a `@derive`.\n\nThey also combine well. A `@daemon` might do a heavy nightly aggregation and write a summary row, while a `@derive` keeps a small live view fresh on every write. They are different tiers of the edge and are covered on their own pages.\n\n## `@job`: the widest database surface for background work\n\n`@daemon` and `@derive` decide **where and when** background code runs. `@job` answers a different question: **what a function is allowed to do to the database.** It is one of the ToilDB **function kinds** (the same family as `@query`, `@action`, and `@derive`), and it grants the **widest** data surface of them all.\n\nA **function kind** is a label the compiler puts on a backend function to gate which database operations it may issue. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path. The kinds line up from narrowest to widest:\n\n| Kind | Typical trigger | Point reads | **Scan** (`latest`, membership `list`) | Writes | `publish` a View |\n| ---------- | ------------------------- | ----------- | -------------------------------------- | ----------------- | ---------------- |\n| `@query` | a `@get` / plain `@remote`| yes | no | no | no |\n| `@action` | a `@post` / `@action` | yes | no | yes (bounded) | no |\n| `@derive` | your data changed | yes | yes | append / counter add only | yes |\n| `@job` | you drive it (background) | yes | yes | yes (all) | yes |\n\nA **scan** is a read that can fan out across many rows (like \"the newest 50 events\" via `events.latest`, or \"every member of this set\" via `membership.list`). Scans are **barred from request handlers** because a request must stay fast and bounded. `@derive` and `@job` run **off the request path**, so they are the only kinds allowed to scan.\n\n### When to reach for `@job`\n\nUse `@job` when a piece of background work needs **more** database power than the default:\n\n- it must **scan** (fold `events.latest`, walk `membership.list`), and/or\n- it must **publish a View** *while also* doing arbitrary writes (`create`, `patch`, `delete`). A `@derive` can publish a View but cannot `patch` or `delete` a record; a `@job` can do everything.\n\
|
|
15
|
+
"background/README.md": "# Background work\n\nBackground work is code that runs **without a user waiting for it**: on a timer, once globally, or automatically after your data changes. toiljs gives you two tools for it, `@daemon` and `@derive`, and this page helps you pick the right one.\n\n## Why background work\n\nMost of your server code runs **because a user asked**: a browser hits a [route](../backend/rest.md) or calls an [RPC](../backend/rpc.md), your handler runs, and a response goes back. But some work does not belong on that path:\n\n- It should happen **on a schedule** (every hour, every night at 2am), not when a request happens to arrive.\n- It would be **too slow** to do inside a request, so you want it precomputed and ready.\n- It must happen **exactly once across the whole world**, not once per user and not once per server.\n\nThat is background work. In toiljs there are two kinds, and they solve two different problems.\n\n## The two tools\n\n```mermaid\nflowchart TD\n Q{What do you need?}\n Q -->|Run on a timer, or once globally| D[\"@daemon + @scheduled<br/>a single background worker\"]\n Q -->|Keep a fast read-view in sync with your data| V[\"@derive<br/>maintains a materialized View\"]\n D --> DD[\"e.g. nightly cleanup, hourly poll,<br/>periodic rollup\"]\n V --> VV[\"e.g. a leaderboard, a 'latest 10' feed,<br/>a home-page summary\"]\n```\n\n### `@daemon`: a scheduled, global worker\n\nA [daemon](./daemons.md) is one long-lived background worker for your whole app. It runs on a **schedule** you set (an interval like every 5 minutes, or a cron time like \"9:15 on weekdays\"), and there is exactly **one** of it worldwide at any moment (a \"singleton\"). Use it for work that is driven by **time** or that must run **once globally**:\n\n- clean up stale rows every night;\n- poll an external API every few minutes;\n- send a daily digest email;\n- roll up yesterday's numbers into a summary.\n\n### `@derive`: keep a read-view up to date\n\nA [derive](./derive.md) is not on a timer. It runs **automatically whenever the data it depends on changes**, and its job is to keep a precomputed **View** (a read-optimized copy of your data) fresh. Use it when a page needs data that is **expensive to compute on every read**, like a leaderboard or a \"latest 10 comments\" list, so the read itself stays a single cheap lookup:\n\n- fold an [events](../database/events.md) log into a \"latest N\" list;\n- total up [counters](../database/counters.md) into a scoreboard;\n- assemble a home-page summary from several sources.\n\n## Which one do I reach for?\n\n| Question | Use |\n| ----------------------------------------------------- | ------------ |\n| \"Run this every hour / at midnight.\" | `@daemon` |\n| \"Do this once for the whole app, not per server.\" | `@daemon` |\n| \"Poll or call an outside service on a schedule.\" | `@daemon` |\n| \"This page's data is too slow to compute per request.\"| `@derive` |\n| \"Keep a leaderboard / feed in sync as data changes.\" | `@derive` |\n\nA simple rule of thumb: if the trigger is **the clock**, use a `@daemon`. If the trigger is **a change to your data** and the goal is a fast read, use a `@derive`.\n\nThey also combine well. A `@daemon` might do a heavy nightly aggregation and write a summary row, while a `@derive` keeps a small live view fresh on every write. They are different tiers of the edge and are covered on their own pages.\n\n## `@job`: the widest database surface for background work\n\n`@daemon` and `@derive` decide **where and when** background code runs. `@job` answers a different question: **what a function is allowed to do to the database.** It is one of the ToilDB **function kinds** (the same family as `@query`, `@action`, and `@derive`), and it grants the **widest** data surface of them all.\n\nA **function kind** is a label the compiler puts on a backend function to gate which database operations it may issue. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path. The kinds line up from narrowest to widest:\n\n| Kind | Typical trigger | Point reads | **Scan** (`latest`, membership `list`) | Writes | `publish` a View |\n| ---------- | ------------------------- | ----------- | -------------------------------------- | ----------------- | ---------------- |\n| `@query` | a `@get` / plain `@remote`| yes | no | no | no |\n| `@action` | a `@post` / `@action` | yes | no | yes (bounded) | no |\n| `@derive` | your data changed | yes | yes | append / counter add only | yes |\n| `@job` | you drive it (background) | yes | yes | yes (all) | yes |\n\nA **scan** is a read that can fan out across many rows (like \"the newest 50 events\" via `events.latest`, or \"every member of this set\" via `membership.list`). Scans are **barred from request handlers** because a request must stay fast and bounded. `@derive` and `@job` run **off the request path**, so they are the only kinds allowed to scan.\n\n### When to reach for `@job`\n\nUse `@job` when a piece of background work needs **more** database power than the default:\n\n- it must **scan** (fold `events.latest`, walk `membership.list`), and/or\n- it must **publish a View** *while also* doing arbitrary writes (`create`, `patch`, `delete`). A `@derive` can publish a View but cannot `patch` or `delete` a record; a `@job` can do everything.\n\nWith **toilscript 0.1.65**, **toiljs 0.0.126**, and an edge host that supports daemon catalog v2, the compiler records `@job` on a `@daemon`'s `@scheduled` method and the host executes that tick with Job permissions and budgets. Tag a scheduled task `@job` when it needs that full surface (say, a nightly repair that scans a log, fixes rows, and republishes a summary View). A plain, **untagged** `@scheduled` method runs with the **`@action`** surface instead: point reads plus bounded writes, which is all most rollups and cleanups need.\n\n```ts\n@daemon\nclass Jobs {\n // A plain scheduled method: point reads + bounded writes (the @action surface).\n @scheduled('1h')\n rollup(): void { /* get a counter, patch a summary row */ }\n\n // A scheduled method that also needs SCANS and to PUBLISH a View: tag it @job.\n @scheduled('1d')\n @job\n nightlyRepair(): void {\n // @job unlocks scan-class reads (events.latest / membership.list)\n // and publishing a View, on top of ordinary reads and writes.\n }\n}\n```\n\n`onStart` always runs as Action. A helper tagged `@job` inherits its caller's runtime permissions; calling it from startup or an untagged scheduled method does not elevate the call. Job ticks carry a v2 daemon catalog, which older edge hosts reject at load time. Action-only daemons retain the v1 catalog and remain compatible.\n\n### `@job` versus `@derive`\n\nThey overlap (both run off the request path, both may scan, both may publish a View), but they are triggered differently and sized differently:\n\n- **`@derive`** is **change-triggered**: it re-runs automatically whenever its source data changes, and its narrow job is to keep one **View** in sync. It cannot `create` / `patch` / `delete` records (only append, counter-add, and publish). Reach for it when a read is too slow and you want it kept fresh on every write.\n- **`@job`** is **you-drive-it** background work (typically a `@scheduled` daemon method) with the **full** write surface. Reach for it when the work is clock-driven or one-off and needs to both scan and mutate freely.\n\n> **Rule of thumb.** Clock-driven and needs the full database surface: a `@job` (usually inside a `@daemon`). Change-driven and only maintains a View: a `@derive`. Clock-driven with modest reads and writes: a plain `@scheduled` method (no `@job` needed).\n\nFor the complete permission grid see the [function-kind matrix](../database/setup.md#how-access-is-gated-query-action-and-friends), and for the decorator itself see [every decorator](../concepts/decorators.md#database-function-kinds-data-access-policy).\n\n## Related\n\n- [Daemons and scheduled jobs](./daemons.md): `@daemon`, `@scheduled`, interval vs cron, and how a single global worker fails over safely.\n- [Derived views (`@derive`)](./derive.md): keeping a materialized View in sync with its source data.\n- [Compute tiers (L1 to L4)](../concepts/tiers.md): where daemons and request handlers each run on the edge.\n- [Views](../database/views.md) and [Events](../database/events.md): the database families a `@derive` reads from and writes to.\n",
|
|
16
16
|
"cli/README.md": "# The toiljs CLI\n\nThe `toiljs` command is how you scaffold, run, build, self-host, and diagnose a toiljs app. This page lists every command and every flag, with copy-pasteable examples.\n\n## What it is\n\nWhen you install the `toiljs` package, it adds one executable to your project: `toiljs`. Everything you do day to day (start the dev server, produce a production build, check your setup) goes through it.\n\nYou almost never type the full path. A freshly scaffolded project already has npm scripts that call it, so you run `npm run dev` and `npm run build`. When you want a command that has no script (like `doctor` or `db`), run it with `npx`:\n\n```bash\nnpx toiljs doctor\n```\n\n`toiljs` needs **Node.js 24 or newer** (older versions will not run it).\n\n## Command overview\n\n| Command | What it does |\n| --- | --- |\n| `toiljs create [name]` | Scaffold a brand new toiljs app in a new folder. |\n| `toiljs dev` | Start the local development server with hot reload. |\n| `toiljs test` | Run Vitest unit or real-browser tests, installing missing tooling automatically. |\n| `toiljs build` | Produce the optimized production build (client bundle + server WebAssembly). |\n| `toiljs start` | Self-host the built app on a fast production HTTP server. |\n| `toiljs configure` | Turn styling features (Sass/Less/Stylus, Tailwind) on or off in an existing project. |\n| `toiljs doctor` | Diagnose your project setup and (with `--fix`) repair common wiring. |\n| `toiljs update` | Check npm for newer dependency versions and apply the ones you pick. |\n| `toiljs db <action>` | Inspect, reset, snapshot, or restore the local dev database. |\n| `toiljs generate auth` | Generate private auth keys and the matching client public key, preserving existing credentials. |\n| `toiljs help` | Print the built-in help. Also `--help` or `-h`. |\n| `toiljs --version` | Print the installed toiljs version. Also `-v`. |\n\nRun `toiljs help` any time to see this list in your terminal.\n\n## Global behavior\n\nA few things apply to every command.\n\n- **`--root <dir>`**: run the command against a project in another directory instead of the current one. Every command accepts it.\n- **Automatic update check**: on every run (including `npm run dev`, which calls the CLI under the hood), toiljs quietly asks the npm registry whether a newer `toiljs` exists and prints a one-line notice on stderr if you are behind. It never blocks or slows the command in a meaningful way (the answer is cached for an hour and the network call is capped at two seconds). To turn it off, set the environment variable `TOILJS_NO_UPDATE_CHECK=1`. It also respects the common `NO_UPDATE_NOTIFIER` and `CI` variables.\n\n```bash\n# Run any command against a project in another folder.\ntoiljs build --root ./apps/marketing\n\n# Silence the \"newer version available\" notice.\nTOILJS_NO_UPDATE_CHECK=1 toiljs dev\n```\n\n## `toiljs generate auth`\n\n```bash\ntoiljs generate auth\ntoiljs generate auth --root ./apps/marketing\n```\n\nSaves private authentication keys in `.env.secrets` (mode `0600`) and\n`VITE_AUTH_KEM_PUBLIC_KEY` in `.env`, and adds both files to `.gitignore`. It preserves\nunrelated settings and existing credentials, and can recover a missing public key from the\nexisting private key. It refuses invalid or mismatched keys rather than rotating them.\nNo secret values are printed. Use this for initial setup or completing an existing key set.\n\nPin the public key in your client's auth options, build with that `.env` on the build machine,\nand install the private settings in your backend's secure env store. See\n[auth configuration](../auth/configuration.md) for the client snippet and toil-backend deployment steps.\n\n## `toiljs create`\n\nScaffolds a new project into a new folder. By default it is interactive: it asks a short series of questions (project name, template, styling, and so on), then writes the files and installs dependencies. Pass flags to skip questions, or `-y` to accept every default and run with no questions at all.\n\n```bash\n# Interactive: answer the prompts.\nnpx toiljs create\n\n# Give it a name up front.\nnpx toiljs create my-app\n\n# Fully non-interactive (great for scripts and CI).\nnpx toiljs create my-app --yes --template app --style css\n```\n\n### What it sets up\n\nEvery new project comes wired for you: the enforced TypeScript 7, Oxlint + tsgolint, Prettier, and Vitest unit/browser/coverage presets, file-based routing, a `toil.config.ts`, a `toilconfig.json` (the server compiler settings), a `.gitignore`, and the native TypeScript 7 editor settings. It also scaffolds a `server/migrations/` folder (where ToilDB schema migrations live) and, unless you opt out, a set of AI assistant helper files.\n\nThe scaffolded `package.json` includes these scripts:\n\n| Script | Runs |\n| --- | --- |\n| `npm run dev` | `toiljs dev` |\n| `npm run build` | `toiljs build` |\n| `npm run build:server` | `toiljs build --server` |\n| `npm run lint` | `oxlint --type-aware client` |\n| `npm run typecheck` | `tsc --noEmit` |\n| `npm test` | `toiljs test` |\n| `npm run test:watch` | `toiljs test --watch` |\n| `npm run test:browser` | `toiljs test --browser` |\n| `npm run test:all` | `toiljs test --all` |\n| `npm run test:coverage` | `toiljs test --all --coverage` |\n| `npm run format` | `prettier --write ...` |\n\n### Generated docs and AI-assistant pointers\n\nEvery project carries a full copy of this documentation set at `.toil/docs/`. You do not maintain it: toiljs regenerates it from the installed toiljs version on every `toiljs dev` and `toiljs build`, so it always matches the version you are on. Do not edit those files by hand (your changes are overwritten on the next dev or build).\n\nUnless you opt out, `toiljs create` also writes small **pointer files** at the project root that tell AI coding assistants to read `.toil/docs/` before touching the project: `CLAUDE.md` (Claude Code), `AGENTS.md` (Codex and others), `.cursor/rules/toiljs.mdc` (Cursor), and `.github/copilot-instructions.md` (GitHub Copilot). These are written once, committed, and yours to edit. Control them with `--ai` / `--no-ai` (when you pass neither, `create` asks).\n\n### create options\n\n| Flag | Meaning |\n| --- | --- |\n| `[name]` | The project folder name (a positional argument). If omitted, you are asked. |\n| `-t, --template <app\\|minimal>` | `app` is the full starter (landing page, layout, styles, demo routes). `minimal` is just a layout and a home route. Default `app`. |\n| `--style <css\\|sass\\|less\\|stylus>` | Which CSS flavor to set up. Default `css` (plain CSS). |\n| `--tailwind` / `--no-tailwind` | Add or skip Tailwind CSS (v4). Off by default. |\n| `--ai` / `--no-ai` | Include or skip AI assistant files (like `CLAUDE.md`). When omitted, you are asked. |\n| `--images` / `--no-images` | Enable or skip build-time image optimization. On by default. |\n| `--git` / `--no-git` | Initialize a git repository. When omitted, you are asked (default yes). |\n| `--install` / `--no-install` | Install dependencies after scaffolding. When omitted, you are asked (default yes). |\n| `--pm <npm\\|pnpm\\|yarn\\|bun>` | Which package manager to install with. Default `npm`. |\n| `-y, --yes` | Accept all defaults and skip every prompt. |\n\nFor a walkthrough, see [Create a project](../getting-started/create-project.md).\n\n## `toiljs test`\n\nRuns unit tests by default. All templates include the dependencies, configuration,\nand runnable unit and React browser examples. Existing projects get missing tooling\ninstalled automatically; custom configs, scripts, and existing files are preserved.\n\n```bash\nnpx toiljs test\nnpx toiljs test --browser\nnpx toiljs test --all --coverage\nnpx toiljs test --watch\nnpx toiljs test Greeting --reporter=verbose\n```\n\n| Flag | Meaning |\n| --- | --- |\n| `--browser` | Run the native browser project with Chrome and WebdriverIO. |\n| `--all` | Run both native projects. Cannot be combined with `--browser`. |\n| `--watch`, `-w` | Watch for changes instead of running once. |\n| `--coverage` | Collect V8 coverage; reports go to `coverage/`. |\n| `--root <dir>` | Run against a project in another directory. |\n\nOther options and file filters pass through to Vitest. An explicit `--project`\noverrides native selection. Custom configs keep their own project selection, with\n`--browser` forwarded to Vitest. Test failures preserve a nonzero shell exit status.\nSee [Testing and coverage](../frontend/testing.md) for setup, browser binaries,\nframework globals, coverage reports, and configuration.\n\n## `toiljs dev`\n\nStarts the local development server with hot reload, so you edit a file and the browser updates in place. This is the command you leave running while you build.\n\n```bash\nnpm run dev\n# or, to pick a port:\nnpx toiljs dev --port 4000\n```\n\n### What the dev server emulates\n\nThe whole point of the dev server is to run your app locally the same way the real Dacely edge runs it in production, so what you see locally is what you ship. It emulates three things:\n\n1. **The edge.** For a project with a server (any project with a `toilconfig.json`), a small local HTTP server takes the public port and dispatches incoming requests into your compiled `server.wasm` using the exact same request envelope the production edge uses. Anything your server does not claim (page routes, static assets, the hot-reload websocket) is proxied to Vite behind the scenes. So your React frontend and your WebAssembly backend run together, on one URL, exactly like production.\n2. **The database.** ToilDB runs in-process as a local emulator. Every family (documents, views, unique, events, counters, membership, capacity) works, and the data is written to `.toil/devdata.json` so it survives restarts. You manage that file with [`toiljs db`](#toiljs-db).\n3. **The host functions.** The platform services your server code calls (email, environment variables and secrets, time, crypto, rate limiting, auth) are wired up locally so they behave like the edge. Email actually sends if you configure a provider (see [Email](../services/email.md)).\n\nA **client-only** project (no `toilconfig.json`) just gets the plain Vite dev server on your port, unchanged.\n\n### How hot reload works\n\n```mermaid\nflowchart TD\n E[\"You edit a file\"] --> Q{\"Which file?\"}\n Q -->|\"client/ (React, CSS)\"| V[\"Vite hot-swaps it in the browser\"]\n Q -->|\"server/ (a @rest / @data / @service file)\"| S[\"toilscript recompiles the server\"]\n S --> G[\"shared/server.ts is regenerated\"]\n G --> V2[\"Vite hot-swaps the new typed client\"]\n S --> W[\"the dev server hot-swaps the new server.wasm\"]\n V --> B[\"Browser updates, no full reload\"]\n V2 --> B\n W --> B\n```\n\nClient edits go straight through Vite's hot module replacement. Server edits trigger a toilscript rebuild: toiljs recompiles your backend, regenerates `shared/server.ts` (the typed client the browser imports to call your server), and hot-swaps the recompiled WebAssembly, all without you touching the browser. Rebuilds are debounced (grouped over about 150 milliseconds) so a \"save all\" or a formatter pass does not trigger a storm of builds.\n\nIf your project has an `emails/` folder, the dev server also prints an email-preview URL at `/__toil/emails`.\n\n### dev options\n\n| Flag | Meaning |\n| --- | --- |\n| `--port <n>` | Port to listen on. Default `3000` (or `client.port` from your config). |\n| `--root <dir>` | Run against a project in another directory. |\n\nPress `Ctrl+C` to stop. toiljs restores your terminal and force-exits even if a native listener is slow to close, so you never end up with an orphaned dev server rebuilding in the background.\n\n## `toiljs build`\n\nProduces the optimized production build. Run this before you deploy or before `toiljs start`.\n\n```bash\nnpm run build\n# or build only the server:\nnpx toiljs build --server\n```\n\n### What it produces\n\nA full build runs in a careful order so the pieces line up:\n\n```mermaid\nflowchart LR\n A[\"Your project\"] --> S[\"1. Compile the server<br/>(toilscript)\"]\n S --> W[\"build/server/release.wasm\"]\n S --> R[\"shared/server.ts<br/>(typed client, regenerated)\"]\n R --> C[\"2. Bundle the client<br/>(Vite)\"]\n C --> O[\"build/client/<br/>(HTML, JS, CSS, assets)\"]\n C --> X[\"3. Prerender + SEO<br/>(SSG pages, robots.txt,<br/>sitemap.xml, llms.txt)\"]\n```\n\n1. **The server is built first.** toilscript compiles every decorated server file (not just the entry) into `build/server/release.wasm`, and regenerates `shared/server.ts`. Doing this first means the client always bundles against a current, correct typed server client. A project that also declares `@stream` or `@daemon` surfaces compiles those into their own artifacts (`release-stream.wasm`, `release-cold.wasm`).\n2. **The client is bundled** by Vite into `build/client/` (your HTML, JavaScript, CSS, and optimized assets). The dev toolbar and error overlay are stripped out of the production bundle.\n3. **Static pages and SEO files are generated**: any route that opts into static generation is prerendered to HTML, and if you configured `client.seo`, toiljs writes `robots.txt`, `sitemap.xml`, and `llms.txt`. Routes that opt into server-side rendering get their HTML-with-holes templates baked for the edge.\n\n### build options\n\n| Flag | Meaning |\n| --- | --- |\n| `--server` | Build **only** the server (recompile the wasm and regenerate `shared/server.ts`), and skip the client bundle. Fast when you only touched backend code. This is what `npm run build:server` runs. |\n| `--root <dir>` | Run against a project in another directory. |\n\nA client-only project (no `toilconfig.json`) skips step 1 and just bundles the client.\n\n## `toiljs start`\n\nSelf-hosts the app you just built, on a fast production HTTP server (hyper-express, backed by uWebSockets.js). It serves your static client, runs your `server.wasm` for dynamic requests, does server-side rendering, supports daemons, and exposes a `/_toil` websocket channel. Use it to run your app on your own machine or server instead of deploying to the Dacely edge.\n\n```bash\nnpm run build # start needs a build to serve\nnpx toiljs start\nnpx toiljs start --port 8080 --host 0.0.0.0 --threads 4\n```\n\n`start` fails fast if there is no build yet (it looks for `build/client/index.html`), so run `toiljs build` first.\n\n### start options\n\n| Flag | Meaning |\n| --- | --- |\n| `--port <n>` | Port to listen on. Default `3000` (or `client.port`). |\n| `--host <host>` | Address to bind. Default `127.0.0.1` (loopback only). Pass `0.0.0.0` to accept connections from other machines. |\n| `--threads <n>` | Number of HTTP worker processes. Default is automatic (one per available CPU). Pass `1` to disable the worker pool. `--workers` is an accepted alias. This can also be set as `server.threads` in your config. |\n| `--root <dir>` | Run against a project in another directory. |\n\n## `toiljs configure`\n\nToggles a project's client styling features (the CSS preprocessor and Tailwind) after the fact, on an existing app. It detects your current setup, asks what you want, then rewrites the stylesheets and your app entry's imports, edits `package.json`, and syncs `node_modules` so removed packages are actually uninstalled.\n\n```bash\n# Interactive.\nnpx toiljs configure\n\n# Non-interactive: switch to Sass and turn Tailwind on.\nnpx toiljs configure --style sass --tailwind\n```\n\n### configure options\n\n| Flag | Meaning |\n| --- | --- |\n| `--style <css\\|sass\\|less\\|stylus>` | Switch the CSS preprocessor. |\n| `--tailwind` / `--no-tailwind` | Turn Tailwind on or off. |\n| `--images` / `--no-images` | Turn build-time image optimization on or off (sets `client.images` in your config). |\n| `--no-install` | Edit the files but do not run the package manager. You then run install yourself. |\n| `--root <dir>` | Run against a project in another directory. |\n\nPassing any of `--style`, `--tailwind`, or `--images` makes the command non-interactive (it skips the prompts you did not answer with a flag). See [Styling](../frontend/styling.md) for the full picture.\n\n## `toiljs doctor`\n\nRead-only project diagnostics. It gathers facts from disk (your `package.json`, lockfiles, the resolved config, your app entry, `index.html`, your routes, and the server target), runs a set of checks, and prints a grouped report. It never changes anything unless you pass `--fix`, and it never crashes on a partial or non-toiljs project (missing pieces just become warnings or failures). It exits with a non-zero status when any check **fails** (warnings do not fail), so it is safe to run in CI.\n\n```bash\n# Human-readable report.\nnpx toiljs doctor\n\n# Machine-readable, for CI.\nnpx toiljs doctor --json\n\n# Auto-repair the common wiring.\nnpx toiljs doctor --fix\n```\n\n### What it checks\n\nThe report is grouped:\n\n| Group | Example checks |\n| --- | --- |\n| **Environment** | Node.js version, that `toiljs` and its peer dependencies (React and so on) are installed and new enough, that TypeScript 7 is installed and its declared range stays within 7.x, that Oxlint and tsgolint are configured for type-aware linting, that a lockfile exists, and that your scripts do not wrap `toiljs` in a stray `npx`. |\n| **Project + routing** | The `client/` and `routes/` folders exist, `index.html` has a `<div id=\"root\">`, your app entry calls `mount(...)` with the `slots` argument, at least one route exists, no two routes collide on the same URL, and no asset paths are written in a way that 404s on nested routes. |\n| **Config + assets** | Your `toil.config` loads, the base path is well formed, `client.seo` has a `url` if SEO is configured, and your styling packages are actually installed. |\n| **Testing** | Actual installed Vitest, browser provider, WebdriverIO, and V8 coverage versions; compatible declared ranges; matching Vitest/coverage releases; config presence and unit/browser/watch/coverage scripts. Missing testing is a warning for legacy apps, while broken configured tooling fails. |\n| **Server / WASM** | The `toilconfig.json` and its entry files exist, `toilscript` is installed, a compiled `.wasm` exists, the typed-RPC wiring is in place, your `@rest` controllers are actually dispatched, the Prettier and editor plugins are wired, and a `migrations/` folder exists. |\n| **Security** | If your server uses auth, whether `AUTH_SESSION_SECRET` is set (an unset secret means sessions fall back to a published dev key, which is forgeable). |\n\n### What `--fix` repairs\n\n`--fix` migrates unsupported or missing TypeScript declarations to `^7.0.2`, enables the native TypeScript editor, and repairs testing setup in any project. Testing repairs add missing packages and scripts, migrate unsupported registry versions, align Vitest and coverage, add the preset and examples when there is no custom config, include tests in plain JSON TypeScript configs, and ignore test reports. Existing custom configs, nonempty scripts, explicit package sources, and existing files are preserved; commented TypeScript configs get a manual repair note. It does not install packages or launch a browser. The rest only touch a server project (one with a `toilconfig.json`), repairing the wiring that is easy to get wrong or that older projects predate:\n\n- adds `--rpcModule shared/server.ts` to your server build scripts,\n- adds `shared` and the `shared/*` path alias to `tsconfig.json`,\n- adds `shared/server.ts` to `.gitignore`,\n- lifts the `toilscript` version floor if it is too old,\n- adds the `toiljs/prettier-plugin` to your Prettier config (so Prettier does not choke on server decorators),\n- removes the legacy toilscript JavaScript language-service plugin, which cannot run in the TypeScript 7 native language server; `toiljs build --server` provides the WebAssembly dialect diagnostics,\n- refreshes the editor-only server globals declaration file.\n\nIt is idempotent: it only writes files it actually needs to change, and it tells you which ones changed and which need a manual edit (for example a `tsconfig.json` that contains comments). If it changed `package.json`, run your installer afterward.\n\n### doctor options\n\n| Flag | Meaning |\n| --- | --- |\n| `--json` | Emit machine-readable JSON instead of the human report (the banner is suppressed so stdout stays valid JSON). |\n| `--fix` | Repair testing, TypeScript, editor, and server wiring in place, as above. |\n| `--root <dir>` | Run against a project in another directory. |\n\n## `toiljs update`\n\nA friendly wrapper over `npm-check-updates`. It checks the registry for newer versions of your dependencies, groups them by how big the jump is (major, minor, patch), lets you pick which to apply (or `-y` to apply all), bumps `package.json`, and runs your package manager's install. It also makes sure your `server/migrations/` folder exists and adds missing testing setup when updates are applied. `--yes` repairs and installs missing testing even when no dependency upgrades are available. Custom configs and scripts are preserved; declining updates leaves testing untouched. Native test updates stay within supported versions, and selecting either Vitest or V8 coverage updates both to a matching release. `npm-check-updates` runs via `npx`, so it never becomes a permanent dependency of your project.\n\nTypeScript updates must stay within supported **7.x** releases. Older and future compiler majors,\nunbounded ranges, and prereleases are withheld. An older project is offered the latest 7.x migration\neven with `--target patch`. Select that migration before applying other updates. The command writes\nthe exact reviewed versions before installing, so a second registry query cannot substitute a\ndifferent compiler major.\n\n```bash\n# Interactive picker.\nnpx toiljs update\n\n# Apply everything, non-interactively.\nnpx toiljs update --yes\n\n# Only patch-level updates.\nnpx toiljs update --target patch\n```\n\n### update options\n\n| Flag | Meaning |\n| --- | --- |\n| `-y, --yes` | Apply all available updates without the picker. |\n| `--target <latest\\|minor\\|patch\\|newest\\|greatest>` | How far to bump. Default `latest`. |\n| `--root <dir>` | Run against a project in another directory. |\n\n## `toiljs db`\n\nManages the local dev database: the on-disk ToilDB store your dev server writes to `.toil/devdata.json`. Use it to inspect data, wipe a corrupt state, save a snapshot to share as a fixture, or restore one. The snapshot is exactly the JSON the dev database uses, so an exported file imports cleanly.\n\n```bash\n# See what is stored.\nnpx toiljs db status\n\n# Wipe all dev data.\nnpx toiljs db reset\n\n# Save a snapshot (to a file, or to stdout if you omit the file).\nnpx toiljs db export fixture.json\n\n# Restore a snapshot.\nnpx toiljs db import fixture.json\n\n# Print the on-disk path (scriptable).\nnpx toiljs db path\n```\n\n### db actions\n\n| Action | What it does |\n| --- | --- |\n| `status` (alias `info`) | Show the database path, its size, and per-family row counts. |\n| `reset` (alias `purge`) | Delete all dev data (removes `devdata.json`). |\n| `export [file]` | Write a formatted snapshot to `file`, or to stdout if you omit it (pipe-friendly). |\n| `import <file>` | Replace the dev database with the snapshot in `file`. It refuses a file that is not a valid snapshot. |\n| `path` | Print the `devdata.json` path and nothing else. |\n\nThe dev database is per-project and lives under `.toil/`, which is gitignored. See [Database setup](../database/setup.md) for what actually populates it.\n\n## The dev, build, and deploy flow\n\nHere is how the commands fit together across your workflow.\n\n```mermaid\nflowchart LR\n subgraph Develop\n D[\"toiljs dev<br/>(hot reload, local edge + DB)\"]\n end\n subgraph Ship\n B[\"toiljs build<br/>(client bundle + server.wasm)\"]\n end\n subgraph Run\n ST[\"toiljs start<br/>(self-host locally)\"]\n DP[\"deploy to the Dacely edge\"]\n end\n D --> B\n B --> ST\n B --> DP\n```\n\n- **Develop** with `toiljs dev`. Everything runs locally and reloads as you type.\n- **Ship** with `toiljs build`. This produces the artifacts: the client bundle in `build/client/` and the server WebAssembly in `build/server/`.\n- **Run** the build two ways: `toiljs start` self-hosts it on your own machine, or you deploy the same build to the Dacely edge to serve it worldwide.\n\n## Gotchas\n\n- **`start` needs a build.** `toiljs start` serves what is in `build/`. Run `toiljs build` first, or it exits with an error.\n- **Run `toiljs`, not `npx toiljs`, inside npm scripts.** Under `npm run`, `node_modules/.bin` is already on your PATH, so an extra `npx` layer is redundant and can leave your terminal in a broken input mode after `Ctrl+C`. `doctor` warns about this; the scaffold gets it right.\n- **`--host` and `--threads` are `start`-only.** `toiljs dev` always binds locally and does not take `--host` or `--threads`.\n- **`--server` builds the backend only.** Use it while iterating on server code, but run a full `toiljs build` before you deploy so the client bundle is current.\n- **The update notice is not an error.** The \"newer toiljs available\" line is informational and prints to stderr. Set `TOILJS_NO_UPDATE_CHECK=1` to hide it.\n\n## Related\n\n- [Installation](../getting-started/installation.md) and [Create a project](../getting-started/create-project.md)\n- [Project structure](../getting-started/project-structure.md)\n- [Configuration reference (`toil.config.ts`)](../concepts/config.md)\n- [Styling](../frontend/styling.md)\n- [Database setup](../database/setup.md) and the [database overview](../database/README.md)\n- [Compute tiers (L1 to L4)](../concepts/tiers.md)\n",
|
|
17
17
|
"concepts/ai-guide.md": "# Writing toiljs correctly\n\nThis is a fast, high-signal cheat-sheet for writing toiljs and toilscript code that compiles and behaves. It is aimed at AI assistants generating code (and at humans in a hurry): the patterns to follow, the mistakes to avoid, and a link to the deep page whenever you need more. It does not re-teach the framework; each section points you at the authoritative doc.\n\n## Project shape\n\nA toiljs project has three top-level folders, and knowing which one a file belongs to tells you which rules apply:\n\n- **`client/`**: your React app that runs in the browser. Pages live under `client/routes/`, one file per URL, and each route file `export default`s its component. See [Frontend](../frontend/README.md) and [Routing](../frontend/routing.md).\n- **`shared/`**: a generated typed bridge (`shared/server.ts`) that lets the browser call the backend with full types. It is regenerated on every build, so **never hand-edit it**.\n- **`server/`**: your backend, written in TypeScript but compiled by **toilscript** to WebAssembly. It runs on the Dacely edge, not in Node or the browser. See [Backend](../backend/README.md).\n\nMost code runs on the L1 request tier. Long-lived connections (`@stream`) and scheduled work (`@daemon`) are opt-in tiers in their own entry files. See [Compute tiers](./tiers.md).\n\n## Client rules (the browser, React side)\n\n**Call the backend only through the generated `Server` client.** This is the number one rule. Use `Server.REST.<controller>.<route>(args)` for `@rest` HTTP controllers, or `Server.<service>.<method>(args)` / `Server.<remote>(args)` for RPC. Raw `fetch` to your own backend throws away the type safety, the argument and result decoding, and the binary wire codec, and it silently breaks when a route is renamed. `fetch` is only for third-party URLs (an external API, a CDN). See [Fetching data](../frontend/data-fetching.md).\n\n```ts\n// WRONG: raw fetch to your own backend\nawait fetch('/account/session', { method: 'POST' });\n\n// RIGHT: the typed client (renames become compile errors; args + result typed)\nawait Server.REST.account.session();\n```\n\n**Use the ambient `Toil.*` globals with no import.** `Toil` is the `toiljs/client` package exposed as a global and typed via a generated `toil-env.d.ts`, so it autocompletes with no import: `Toil.Link` / `Toil.NavLink` for navigation, `Toil.useParams`, `Toil.useLoaderData`, `Toil.Image`, `Toil.useHead` / `Toil.Head`, `Toil.Form` / `Toil.useAction`, and more. `Server` and `parseError` are also bare globals; `FastMap`, `FastSet`, `DataWriter`, and `DataReader` are bare globals too (write `new DataWriter()`, not `new Toil.DataWriter()`). Full index: [The Toil global](../frontend/toil-global.md).\n\n**Load page data with a `loader`, not a `useEffect` fetch.** A route file exports a `loader`, and the component reads its result with `Toil.useLoaderData`. The loader runs on navigation in parallel with the route chunk, integrates with `loading.tsx`, caching, and SSR hydration. A `useEffect` fetch runs only after mount (slower, invisible to the server).\n\n```tsx\n// client/routes/blog/[id].tsx\nexport const loader = async ({ params }: Toil.LoaderArgs) => {\n return Server.REST.blog.get({ params: { id: params.id } });\n};\n\nexport default function BlogPost() {\n const post = Toil.useLoaderData<typeof loader>();\n return <article><h1>{post.title}</h1></article>;\n}\n```\n\n**Mutate with `Toil.useAction` or `<Toil.Form>`, then revalidate.** Both track pending and error state and refetch the affected loader data on success, so the page updates without a manual refetch. Use `<Toil.Form>` for form submits, `useAction` for anything else (a delete button, a toggle). Reach for `Toil.revalidate()` / `router.revalidate()` after a write that these do not cover.\n\n**Per-route SEO with `export const metadata`** (or `Toil.useHead` / `<Toil.Head>` from inside a component). Opt a route into edge SSR with `export const ssr = true`. See [Metadata and SEO](../frontend/metadata.md) and [Rendering and SSR](../frontend/rendering.md).\n\n**Typed hrefs.** `Toil.Link` `href`, `Toil.navigate`, and `router.push` are all type-checked against your real routes, so a typo is a compile error. Use `Toil.href(str)` only when a URL is assembled from data and TypeScript cannot prove it is a real route. See [Navigation](../frontend/navigation.md).\n\n**`getUser()` on the client is display-only.** It reads a readable session cookie with no network call, so it is instant but **forgeable**. Use it to render \"logged in as ...\", never to gate real access. The authoritative check is a server route guarded with `@auth`. See [Auth usage](../auth/usage.md).\n\n## Server rules (toilscript, compiles to WASM)\n\nThe decorators, each doing one job (full list in [Decorators](./decorators.md)):\n\n- **`@data`** on a class: generates the binary and JSON codec so the value can cross the wire and the database. Every field needs a default; field order **is** the binary layout, so add new fields at the end. See [Data types](../backend/data.md).\n- **`@rest('name')`** on a class mounts an HTTP controller at `/name`; **`@get`/`@post`/`@put`/`@del`/`@patch`** on its methods declare routes (`@del`, not `@delete`, which is reserved). See [REST](../backend/rest.md).\n- **`@service`** + **`@remote`** expose typed RPC callable from your own frontend as `Server.<service>.<method>()`. See [RPC](../backend/rpc.md).\n- **`@user`** declares the authenticated-user shape and enables `AuthService.getUser()`. **Exactly one `@user` per project.** **`@auth`** guards a route or a whole `@rest` class. See [Auth](../auth/README.md).\n- **`@ratelimit(strategy, limit, window)`** caps how often a caller may hit a route, rejected at the edge before your code runs. See [Security](./security.md).\n- **`@database`** + **`@collection`** declare a ToilDB schema (below).\n\n```ts\n@rest('players')\nclass Players {\n @get('/:id')\n public get(ctx: RouteContext): Response {\n const id = ctx.param('id');\n return Response.json(`{\"id\":\"${id}\"}`);\n }\n}\n```\n\n**`@auth` rejects with 401 before your handler runs.** So inside an `@auth`-guarded handler, `AuthService.getUser()!` is guaranteed non-null and safe to assert. Without `@auth`, `getUser()` can be null, so null-check it instead. Either way the server re-verifies the signed session, so it (not the client cookie) is the real authorization boundary.\n\n```ts\n@auth\n@get('/settings')\npublic settings(): Response {\n const user = AuthService.getUser()!; // safe: @auth guarantees a session\n return Response.text('hi ' + user.username);\n}\n```\n\n**Server code is a strict, WASM-targeted dialect (AssemblyScript), not full TypeScript.** The real constraints, verified against the toilscript standard library:\n\n- **Use explicit value types**, never `number`: `i32` / `u32` / `i64` / `u64` / `f64` (and `i8`/`u8`/`i16`/`u16`/`f32`, plus `u128`..`u256`), `bool`, `string`, `Uint8Array`. Plain `number` resolves to `f64`, which is wrong for ids and counts. Integer math **wraps** on overflow (it never throws), and integer `/` truncates. See [The type system](./types.md).\n- **No `any`.** Every value has a concrete type; that is what lets it compile to WASM.\n- **No arbitrary npm.** Only the toilscript standard library plus the toiljs host APIs.\n- **Use `null`, not `undefined`** for \"no value\" (`T | null`, narrowed with `if (x != null)` or a `!` assertion).\n- **No usable built-in `RegExp`** (it is a host stub that throws). Parse strings by hand.\n- **Structured values are `@data` classes, not object literals.** Encode and decode raw binary with `DataWriter` / `DataReader` (imported on the server from the `data` module: `import { DataWriter, DataReader } from 'data';`). For dynamic JSON, use the ambient `JSON` value tree.\n- **Read config with `Environment.get(key)` and secrets with `Environment.getSecure(key)`** (each returns `string | null`; the two buckets are disjoint so a secret can never leak through `get`). See [Environment](../services/environment.md).\n- **Return a runtime `Response`** (`Response.json` / `.text` / `.html` / `.bytes` / `.notFound` / ...), or return a `@data` value and let toiljs serialize it.\n\n**ToilDB has seven collection families; pick the one that matches the job** (do not force everything into one). Declare each as a `static` `@collection` field on a `@database` class, typed by its family, and reach it statically (`AppDb.users.get(...)`). See [The database](../database/README.md) and [Setup](../database/setup.md).\n\n- **`Documents<K,V>`**: the default record store, looked up by id (users, posts, orders).\n- **`Unique<K,V>`**: a globally one-of-a-kind claim (usernames, emails, slugs).\n- **`Counter<K>`**: a running total many callers bump at once (likes, views); `add` a delta only.\n- **`Events<K,V>`**: an append-only log kept in order (feeds, audit trails).\n- **`Capacity<K>`**: a limited quantity handed out without overselling (tickets, seats).\n- **`Membership<K,M>`**: sets of who belongs to what (followers, tags, room members).\n- **`View<K,V>`**: a precomputed read-optimized snapshot (leaderboards, home pages).\n\n**Data access is gated by function kind.** A `@get` route is a read-only **Query**; a `@post` route is an **Action** (may write); a plain `@remote` defaults to read-only Query, so add **`@action`** to let it write. Scans (`events.latest`, `membership.list`) are barred from request handlers: do them in a `@derive` and have the request read the resulting `View`. The compiler enforces this and the edge re-checks it.\n\n## Common mistakes\n\nA short do-not list. Every one of these is a real, avoidable error:\n\n- **Raw `fetch` to your own backend.** Go through `Server.*` instead.\n- **Importing `Toil.*`.** They are ambient globals; importing them is wrong.\n- **Fetching page data in a `useEffect`.** Use a route `loader` + `Toil.useLoaderData`.\n- **A plain `<a>` for in-app links.** Use `Toil.Link` (a bare `<a>` triggers a full reload).\n- **More than one `@user`.** Exactly one per project.\n- **Trusting client `getUser()` for authorization.** It is display-only and forgeable; enforce on the server with `@auth`.\n- **`any`, `number`, or `RegExp` in server code.** Use explicit value types; there is no usable `RegExp`.\n- **Reordering `@data` fields** (or changing a field type) on a stored type. Field order is the layout; add at the end and use `@migrate` to evolve. See [Data types](../backend/data.md).\n- **Writing from a plain `@remote` or `@get`.** Reads are the default; a write needs `@action`.\n- **Calling a scan (`events.latest`, `membership.list`) from a request handler.** Do it in a `@derive`.\n- **Hand-editing `shared/server.ts` or `.toil/`.** They are generated; rebuild instead.\n\n## Where to go deeper\n\n- Concepts: [Decorators](./decorators.md), [Type system](./types.md), [Security](./security.md), [Compute tiers](./tiers.md).\n- Backend: [Overview](../backend/README.md), [REST](../backend/rest.md), [RPC](../backend/rpc.md), [Data types](../backend/data.md).\n- Frontend: [Overview](../frontend/README.md), [Fetching data](../frontend/data-fetching.md), [Routing](../frontend/routing.md), [Navigation](../frontend/navigation.md), [The Toil global](../frontend/toil-global.md).\n- Database: [ToilDB overview](../database/README.md), [Setup](../database/setup.md).\n- Auth: [Overview](../auth/README.md), [Usage](../auth/usage.md).\n- Services: [Environment](../services/environment.md), [Rate limiting](../services/ratelimit.md).\n</content>\n</invoke>\n",
|
|
18
18
|
"concepts/config.md": "# Configuration (`toil.config.ts`)\n\n`toil.config.ts` is the one file that configures your whole toiljs app: the client (React/Vite) side and the server (WebAssembly) side. Every field is optional and has a sensible default, so most projects keep it tiny.\n\n## The shortest possible config\n\nA scaffolded project ships something like this:\n\n```ts\n// toil.config.ts\nimport { defineConfig } from 'toiljs/compiler';\n\nexport default defineConfig({\n client: {\n // Optimize images at build time (resize and compress imported images).\n images: true,\n },\n});\n```\n\n`defineConfig` does not do anything at runtime: it is an identity helper that gives you full editor autocomplete and type-checking for the config shape. Always wrap your config in it.\n\nAn empty config is valid too. `export default defineConfig({})` gives you a working app with all defaults.\n\n## Where the file lives\n\ntoiljs looks in your project root for the first file that matches, in this order:\n\n```\ntoil.config.ts toil.config.mts toil.config.js toil.config.mjs\ntoiljs.config.ts toiljs.config.mts toiljs.config.js toiljs.config.mjs\n```\n\nUse whichever you like. `toil.config.ts` is the convention.\n\n## Config vs. environment variables\n\nThere are two very different places settings live, and it matters which one you use.\n\n| | `toil.config.ts` (this page) | Environment variables |\n| --- | --- | --- |\n| **When it applies** | Build time and dev time | Runtime, per request |\n| **What it holds** | Framework and build options (routing, styling, SEO, which features to compile) | Values and secrets your running server reads (API keys, feature flags, connection info) |\n| **Committed to git?** | Yes, it is source | No. Local values go in `.env` / `.env.secrets` (both gitignored); production values live on your deploy target |\n| **Read in code with** | Not read from your app code | `Environment.get(...)` / `Environment.getSecure(...)` on the server |\n\nRule of thumb: if it is a **secret** (a password, an API key, a session key), it does **not** go in `toil.config.ts`. It goes in the environment. See [Environment and secrets](../services/environment.md) for the full story.\n\nThere is also a **third** file, `toilconfig.json`, which is a different thing entirely. See [`toil.config.ts` is not `toilconfig.json`](#toilconfigts-is-not-toilconfigjson) at the bottom.\n\n## The top-level shape\n\n```ts\ninterface ToilConfig {\n root?: string; // project root (defaults to the current working directory)\n client?: ClientConfig; // the React / Vite frontend\n server?: ServerConfig; // the toilscript / WebAssembly backend\n}\n```\n\nEverything else lives under `client` or `server`.\n\n## `client` reference\n\nConfigures the frontend: source folders, the dev server, and build-time optimizations.\n\n| Field | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `srcDir` | `string` | `\"client\"` | Your frontend source directory, relative to the project root. |\n| `routesDir` | `string` | `\"routes\"` | Your file-based routes directory, relative to `srcDir`. |\n| `publicDir` | `string` | `\"<srcDir>/public\"` | Static assets directory. Holds `index.html` (which you own) plus files served as-is (favicons, images). |\n| `outDir` | `string` | `\"build/client\"` | Where the production client bundle is written. |\n| `base` | `string` | `\"/\"` | The public base path. Use this if you serve the app under a sub-path (a non-root base should start and end with `/`, like `\"/app/\"`). |\n| `port` | `number` | `3000` | The dev server port. `--port` on the CLI overrides it. |\n| `images` | `boolean` | `true` | Optimize imported images at build time (resize and convert). See [Images](../frontend/images.md). |\n| `fonts` | `boolean` | `true` | Preload bundled fonts (inject `<link rel=\"preload\">` for each `@font-face`) so text paints faster. |\n| `viewTransitions` | `boolean` | `false` | Animate page navigations with the browser View Transitions API (a crossfade by default). Respects `prefers-reduced-motion`. |\n| `transitions` | `boolean` | `false` | Wrap navigations in a React transition, keeping the current page visible while the next route's loader runs (instead of showing its loading state immediately). |\n| `devtools` | `boolean` or object | `true` | The floating dev toolbar (route/build info, errors, live controls). It is dev-only and never ships in production. Set `false` to disable, or pass an object to configure its AI integration. |\n| `seo` | object | (off) | Build-time SEO: bakes site-level metadata into the HTML and generates `robots.txt`, `sitemap.xml`, `llms.txt`. See [`client.seo`](#clientseo) below. |\n| `vite` | Vite `InlineConfig` | `{}` | An escape hatch: raw Vite options, deep-merged over the framework's own Vite setup. toiljs owns the Vite config; use this only to override specific options. |\n\n### Example\n\n```ts\nimport { defineConfig } from 'toiljs/compiler';\n\nexport default defineConfig({\n client: {\n images: true,\n fonts: true,\n viewTransitions: true,\n // Serve the app under https://example.com/app/\n base: '/app/',\n },\n});\n```\n\n### `client.devtools`\n\nThe dev toolbar is on by default. To turn it off, or to give it an AI provider (so its \"explain this error\" helpers can call a model), pass an object:\n\n```ts\nimport { defineConfig, AiProvider } from 'toiljs/compiler';\n\nexport default defineConfig({\n client: {\n devtools: {\n ai: {\n provider: AiProvider.Anthropic, // 'anthropic' or 'openai'\n model: 'claude-sonnet-4-6',\n // The name of the env var holding the API key. It is read\n // server-side by the dev process and never sent to the browser.\n apiKeyEnv: 'ANTHROPIC_API_KEY',\n // Optional: a custom POST endpoint ({ prompt } in, { text } out)\n // that overrides `provider` entirely.\n // endpoint: 'http://localhost:5000/ai',\n },\n },\n },\n});\n```\n\n| `devtools.ai` field | Type | What it does |\n| --- | --- | --- |\n| `provider` | `AiProvider` | Built-in provider: `AiProvider.Anthropic` (`'anthropic'`) or `AiProvider.OpenAI` (`'openai'`). |\n| `model` | `string` | The model id, for example `'claude-sonnet-4-6'` or `'gpt-4o'`. |\n| `apiKeyEnv` | `string` | The **name** of the environment variable that holds the API key. The key stays server-side and never reaches the browser. |\n| `endpoint` | `string` | A custom POST endpoint. When set, it takes precedence over `provider`. |\n\nThe toolbar always offers hand-off links to Claude and ChatGPT even without any AI config; this only enables the in-toolbar helpers.\n\n### `client.seo`\n\n`client.seo` turns on build-time search-engine and social metadata. It bakes tags into your HTML `<head>` (so crawlers and link-preview bots that do not run JavaScript still see real metadata) and generates `robots.txt`, `sitemap.xml`, and `llms.txt`. This is a large section with its own guide; see [Metadata and SEO](../frontend/metadata.md) for the complete field list and examples. The one thing worth knowing here: set `seo.url` to your absolute site URL (like `\"https://example.com\"`), because the sitemap and canonical links need it.\n\n```ts\nexport default defineConfig({\n client: {\n seo: {\n url: 'https://example.com',\n title: 'My App',\n description: 'A toiljs app.',\n },\n },\n});\n```\n\n## `server` reference\n\nConfigures the backend: which platform features to compile in, and how the dev server and self-host behave.\n\n| Field | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `auth` | `boolean` | `false` | Opt into the framework's built-in post-quantum login. See [`server.auth`](#serverauth) below. |\n| `email` | object | (off) | The non-secret email backend config for the dev server and self-host. See [`server.email`](#serveremail) below. |\n| `daemon` | object | (defaults) | Background-job (L4 daemon) settings for dev and self-host. See [`server.daemon`](#serverdaemon) below. |\n| `nodeMode` | string | `\"all\"` | Which compute layer the single dev/self-host process emulates. See [`server.nodeMode`](#servernodemode) below. |\n| `threads` | `number` or `\"auto\"` | `\"auto\"` | HTTP worker count for `toiljs start`. `\"auto\"` uses one per CPU; `1` disables the worker pool. `--threads` on the CLI overrides it. |\n| `srcDir` | `string` | `\"server\"` | Declarative: your server source directory. See the note below. |\n| `outDir` | `string` | `\"build/server\"` | Declarative: the server build output directory. See the note below. |\n\n> **Note on `server.srcDir` / `server.outDir`.** These fields exist in the config type, but the actual location of your server source and its compiled output is driven by `toilconfig.json` (the toilscript compiler config), not by these two fields. Change your server paths in `toilconfig.json`. These fields are currently declarative and left for forward compatibility. See [not `toilconfig.json`](#toilconfigts-is-not-toilconfigjson) below.\n\n### `server.auth`\n\nSet `auth: true` to get a complete post-quantum login system with no boilerplate. The build appends a shipped `@rest('auth')` controller and its `@user` shape to your server, giving you `/auth/register`, `/auth/login`, `/auth/me`, and `/auth/logout` plus sessions.\n\n```ts\nexport default defineConfig({\n server: {\n auth: true,\n },\n});\n```\n\nTwo things to know:\n\n- If you opt in, your app must **not** declare its own `@user` type. The built-in auth owns the single per-program one.\n- There is an escape hatch: adding `import 'toiljs/server/auth'` in `server/main.ts` turns on the same built-in auth surface without this flag.\n\nAuth has its own configuration (session secrets, the OPRF and KEM keys) that lives in the **environment**, not here. See [Auth configuration](../auth/configuration.md).\n\n### `server.email`\n\nThe **non-secret** part of your email setup: which provider, the \"from\" address, and send caps. The dev server and self-host read it. The API key or SMTP password is a **secret** and never goes here; it comes from `.env.secrets` (`TOIL_EMAIL_API_KEY`). Any `TOIL_EMAIL_*` environment variable overrides the matching field here.\n\n```ts\nexport default defineConfig({\n server: {\n email: {\n provider: 'resend',\n from: 'hello@example.com',\n maxPerMin: 60,\n },\n },\n});\n```\n\n| Field | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `provider` | `'resend'` \\| `'gmail'` \\| `'smtp'` | `'resend'` | Which email backend to use. |\n| `from` | `string` | (none) | The \"from\" address. Validated (single address, no line breaks). |\n| `maxPerMin` | `number` | `60` | Per-process send ceiling per minute (rolling). `0` means unlimited. |\n| `maxPerDay` | `number` | `0` | Per-process send ceiling per day (rolling). `0` means unlimited. |\n| `maxPerRecipientPerHour` | `number` | `5` | Per-recipient hourly cap (anti-abuse). |\n| `smtp` | object | (none) | Connection details for the `gmail` / `smtp` providers: `host`, `port` (defaults to 587 STARTTLS; 465 is implicit TLS), and `user` (defaults to `from`). |\n\nSee the full guide at [Email and 2FA](../services/email.md).\n\n### `server.daemon`\n\nSettings for the daemon (L4) background layer, used by the dev process and self-host. In dev, the local process is always the leader, so region fields are informational.\n\n```ts\nexport default defineConfig({\n server: {\n daemon: {\n defaultIntervalMs: 60000,\n maxTasks: 64,\n },\n },\n});\n```\n\n| Field | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `region` | `string` | (none) | Region the daemon is pinned to (informational in dev). |\n| `standbyRegion` | `string` | (none) | Warm standby region (informational in dev). |\n| `defaultIntervalMs` | `number` | `60000` | Default interval for a `@scheduled` task that declares none. Values below `1000` are clamped up to `1000` (a sub-second loop would flood the console). |\n| `tickBudgetMs` | `number` | `30000` | Per-tick wall-clock budget before the dev scheduler logs an overrun. |\n| `gasTick` | `number` | `0` | Per-tick gas cap (a dev stub: charged then ignored). |\n| `maxTasks` | `number` | `64` | Maximum number of `@scheduled` tasks. Clamped to the range 1 to 1024. |\n\nSee [Daemons and scheduled jobs](../background/daemons.md).\n\n### `server.nodeMode`\n\nWhich compute layer the single local process emulates. This is a dev and self-host knob only; in production the Dacely edge decides each server's role. Valid values are `hot`, `regional`, `continental`, `daemon`, and `all`. The default, `all`, runs every surface (requests, streams, and daemons) in one process, which is what you want for a full local run. An invalid value falls back to `all` with a warning rather than failing. See [Compute tiers (L1 to L4)](../concepts/tiers.md) for what each layer means.\n\n```ts\nexport default defineConfig({\n server: {\n nodeMode: 'all', // run everything locally (the default)\n },\n});\n```\n\n## Route execution and transport limits\n\n`server.daemon.tickBudgetMs` applies only to scheduled daemon ticks. It is an overrun warning emitted after a tick returns; it does not cancel the tick or set a REST/RPC route deadline. There is no route execution timeout setting in `toil.config.ts`.\n\nFor `toiljs dev` and the Node self-host (`toiljs start`):\n\n| Limit | Default | What it bounds |\n| --- | --- | --- |\n| HTTP request body | 8 MiB | The body accepted by the front server, before handler dispatch. |\n| HTTP idle timeout | 60 seconds | An idle connection. |\n| HTTP response timeout | 120 seconds | A response stalled while sending under backpressure. |\n| WASM handler execution | No elapsed-time cutoff or gas metering | Synchronous `handle`/`render` execution on its worker. |\n\nThe HTTP timeouts do not interrupt a WASM handler that is still running. A busy loop can occupy its worker indefinitely, so a browser abort or reverse-proxy timeout must not be treated as proof that a write was cancelled. The low-level `startDevServer` / `startBuiltServer` APIs accept `maxBodyLength`; the CLI uses the default above.\n\nThe Dacely production edge meters execution with a per-request **gas budget** (a charge for WASM instructions and host operations). The current public free-plan HTTP and scheduled-tick budget is **128,000,000 gas**; Pro uses **256,000,000 gas**. Module start has a separate **20,000,000 gas** allowance, and daemon startup has **512,000,000 gas**. Host plans and operator configuration determine the deployed values. Gas exhaustion traps the call. Gas is a work budget rather than a guaranteed number of milliseconds; Node hosts do not emulate WASM instruction gas.\n\nFor optional database budget checks on the local Node hosts, set `TOIL_DEV_CHECK_DB_LIMITS=1`. This checks the native defaults for operation/write counts, write payloads, and per-result/cumulative bytes. See [database size limits](../database/documents.md#size-limits-and-file-uploads). It does not emulate storage latency, wait deadlines, or internal unique-field operations.\n\nProduction asynchronous database waits have their own deadlines: the current defaults are **50 ms for a Query**, **250 ms for an Action**, and **5 seconds for a Job or Derive**. The edge operator configures these through `db_quota`, separately from `toil.config.ts`. They bound DB waits rather than the whole route's execution time, and the local emulator does not enforce them. Deployed transport and database budgets are controlled by the host; there is no universal route time limit in seconds to infer from the daemon setting.\n\nKeep each upload request bounded by bytes, chunk count, and database operations. Continue large jobs across requests or background tasks rather than looping until a wall-clock deadline. See [Documents size limits and file uploads](../database/documents.md#size-limits-and-file-uploads) and [Time](../services/time.md#upload-expiry-versus-elapsed-deadlines).\n\n## Defaults at a glance\n\nIf you write nothing, this is what you get.\n\n| Setting | Default |\n| --- | --- |\n| `client.srcDir` | `\"client\"` |\n| `client.routesDir` | `\"routes\"` |\n| `client.publicDir` | `\"client/public\"` |\n| `client.outDir` | `\"build/client\"` |\n| `client.base` | `\"/\"` |\n| `client.port` | `3000` |\n| `client.images` | `true` |\n| `client.fonts` | `true` |\n| `client.viewTransitions` | `false` |\n| `client.transitions` | `false` |\n| `client.devtools` | on |\n| `client.seo` | off |\n| `server.auth` | `false` |\n| `server.email` | off |\n| `server.nodeMode` | `\"all\"` |\n| `server.threads` | `\"auto\"` |\n| `server.daemon.defaultIntervalMs` | `60000` |\n| `server.daemon.tickBudgetMs` | `30000` |\n| `server.daemon.maxTasks` | `64` |\n\n## A fuller example\n\n```ts\nimport { defineConfig, AiProvider } from 'toiljs/compiler';\n\nexport default defineConfig({\n client: {\n images: true,\n fonts: true,\n viewTransitions: true,\n seo: {\n url: 'https://example.com',\n title: 'Example',\n description: 'Built with toiljs.',\n },\n devtools: {\n ai: { provider: AiProvider.Anthropic, model: 'claude-sonnet-4-6', apiKeyEnv: 'ANTHROPIC_API_KEY' },\n },\n },\n server: {\n auth: true,\n email: { provider: 'resend', from: 'hello@example.com' },\n threads: 'auto',\n },\n});\n```\n\n## `toil.config.ts` is not `toilconfig.json`\n\nThese two look almost the same and are easy to confuse. They are not the same file.\n\n| File | What it is |\n| --- | --- |\n| `toil.config.ts` | **This page.** The framework config: client and server options, styling, SEO, which features to build. You edit it often. |\n| `toilconfig.json` | The **toilscript compiler** config: which server files are entry points, where the `.wasm` is written, and low-level WebAssembly options (optimization level, memory layout, enabled wasm features). It is scaffolded for you and you rarely touch it. Its presence is also what tells toiljs \"this project has a server.\" |\n\nIf you ever need to change where your server source lives or what the compiled artifact is named, that is `toilconfig.json`, not `toil.config.ts`.\n\n## The `toilconfig.json` reference\n\n`toilconfig.json` is the **toilscript compiler** config. toilscript is the compiler that turns your `server/` TypeScript into a `.wasm` file (WebAssembly, the sandboxed binary your backend ships as). This file tells toilscript which server files to compile, where to write the output, and which low-level WebAssembly options to use.\n\n`toiljs create` scaffolds it for you and most projects never touch it. You only edit it if you want to rename the compiled artifact, move your server entry, or hand-tune the WebAssembly codegen. Its presence at the project root is also the signal toiljs uses to decide \"this project has a server\" (a project with no `toilconfig.json` is a client-only app).\n\nA scaffolded file looks like this:\n\n```json\n{\n \"entries\": [\"server/main.ts\"],\n \"targets\": {\n \"release\": {\n \"outFile\": \"build/server/release.wasm\",\n \"textFile\": \"build/server/release.wat\"\n }\n },\n \"options\": {\n \"sourceMap\": false,\n \"optimizeLevel\": 3,\n \"shrinkLevel\": 1,\n \"converge\": true,\n \"noAssert\": false,\n \"enable\": [\n \"sign-extension\",\n \"mutable-globals\",\n \"nontrapping-f2i\",\n \"bulk-memory\",\n \"simd\",\n \"reference-types\",\n \"multi-value\"\n ],\n \"runtime\": \"stub\",\n \"lib\": [\"node_modules/toiljs/server/globals\"],\n \"memoryBase\": 65536,\n \"initialMemory\": 4,\n \"debug\": false,\n \"trapMode\": \"allow\"\n }\n}\n```\n\n### `entries`\n\nAn array of your server entry files: the toilscript starting points for the compile. The scaffold lists just `server/main.ts`, and `main.ts` imports your other surface modules so they all get pulled in. (Under `toiljs build`, toiljs compiles every decorated server file it finds, not only the entries, so a `@rest` or `@data` file you drop in is picked up even if `main.ts` does not import it.)\n\nA project that also has a streams tier or a daemon tier lists their entry files here too, so each tier can compile into its own artifact:\n\n```json\n\"entries\": [\"server/main.ts\", \"server/main.stream.ts\", \"server/main.daemon.ts\"]\n```\n\n### `targets`\n\nA map of named build targets. Each target names its output files. The scaffold has one target, `release`.\n\n| Field | Type | What it does |\n| --- | --- | --- |\n| `outFile` | `string` | Where the compiled `.wasm` is written. The scaffold uses `build/server/release.wasm`. |\n| `textFile` | `string` | Where the `.wat` (WebAssembly **text** format, the human-readable text form of the same module) is written. Handy for inspecting the output; not needed at runtime. |\n\n### `options`\n\nLow-level WebAssembly codegen options passed straight to toilscript. The defaults are already tuned for production, so change these only if you know you need to.\n\n| Field | Type | Scaffold value | What it controls |\n| --- | --- | --- | --- |\n| `sourceMap` | `boolean` | `false` | Emit a source map alongside the `.wasm` so a debugger can map machine code back to your TypeScript. Off by default (it makes the build bigger). |\n| `optimizeLevel` | `number` | `3` | How hard the optimizer works on **speed**, from `0` (none) to `3` (most). `3` is a production release setting. |\n| `shrinkLevel` | `number` | `1` | How hard the optimizer works on **size**, from `0` to `2`. `1` trades a little speed for a smaller binary. |\n| `converge` | `boolean` | `true` | Re-run the optimizer until the output stops getting better. Squeezes out a bit more at the cost of a slower build. |\n| `noAssert` | `boolean` | `false` | Strip `assert(...)` checks from the output (replace them with just their value, no trap). `false` keeps the safety checks in. |\n| `enable` | `string[]` | (see below) | Which modern WebAssembly features the compiled module is allowed to use. |\n| `runtime` | `string` | `\"stub\"` | The memory-management runtime baked into the module. `\"stub\"` is a minimal runtime that never frees memory, which fits toiljs's model exactly: the edge runs one fresh instance per request and throws its whole memory away when the request ends, so there is nothing to garbage-collect. |\n| `lib` | `string[]` | `[\"node_modules/toiljs/server/globals\"]` | Extra library paths whose top-level exports become **ambient globals** (usable with no `import`). This is what makes toiljs's server globals (like `crypto` and the auth primitives) available everywhere in `server/`. |\n| `memoryBase` | `number` | `65536` | The byte offset where your server's static data starts. Requests that fit below this offset use the reserved prefix at offset 0. The current edge and dev host deliver larger requests above the heap; the toiljs entrypoint reserves the input before decoding. Raising this value enlarges the prefix fast path and costs initial memory; it is not a hard upload-body cap. |\n| `initialMemory` | `number` | `4` | How much linear memory the module starts with, in **pages**. One WebAssembly page is 64 KiB, so `4` is 256 KiB. It grows on demand past this. |\n| `debug` | `boolean` | `false` | Include debug information (names and the like) in the binary. Off for production. |\n| `trapMode` | `string` | `\"allow\"` | What happens on a trapping operation (like a bad float-to-int conversion). `\"allow\"` lets it trap (the default and correct choice); `\"clamp\"` replaces traps with clamping instead. |\n\nThe `enable` array turns on WebAssembly features that are off by default in the compiler. The scaffold enables the modern set the compiled server relies on: `sign-extension`, `mutable-globals`, `nontrapping-f2i` (non-trapping float-to-int conversions), `bulk-memory` (fast `memory.copy` / `memory.fill`), `simd` (vector operations), `reference-types`, and `multi-value` (functions that return more than one value). Leave this list as scaffolded unless you have a specific reason to change it; removing an entry can make the module fail to compile or run.\n\n### The `--rpcModule` build flag\n\nOne toilscript flag is worth knowing even though it lives in your npm scripts, not in `toilconfig.json`: `--rpcModule shared/server.ts`. It tells the compiler to also emit `shared/server.ts`, the fully typed client the browser imports to call your server (the `@data` codec plus the typed `Server` surface). toiljs adds this flag for you on the request build. If your `build:server` script is missing it (older projects predate it), `toiljs doctor --fix` injects it. See [the CLI reference](../cli/README.md#what---fix-repairs).\n\n## Gotchas\n\n- **Wrap the config in `defineConfig`.** Without it you lose autocomplete and type errors on typos.\n- **Secrets never go here.** API keys, session secrets, and passwords belong in the environment, not in `toil.config.ts` (which is committed to git). See [Environment and secrets](../services/environment.md).\n- **`server.srcDir` / `server.outDir` do not move your server.** The server source location is governed by `toilconfig.json` entries. Editing these two config fields has no effect today.\n- **`nodeMode` and `daemon` are dev/self-host only.** In production the edge assigns each server its role; these settings never override that.\n- **An invalid `nodeMode` does not crash the build.** It warns and falls back to `all`.\n\n## Related\n\n- [The CLI](../cli/README.md): the commands that read this config.\n- [Environment and secrets](../services/environment.md): runtime values and how they differ from build-time config.\n- [Auth configuration](../auth/configuration.md): the auth-related environment settings behind `server.auth`.\n- [Email and 2FA](../services/email.md): the full email setup behind `server.email`.\n- [Metadata and SEO](../frontend/metadata.md): the full `client.seo` field list.\n- [Daemons and scheduled jobs](../background/daemons.md): the background layer behind `server.daemon`.\n- [Compute tiers (L1 to L4)](../concepts/tiers.md): what `server.nodeMode` selects.\n- [Images](../frontend/images.md) and [Styling](../frontend/styling.md): the features `client.images` and `toiljs configure` control.\n",
|
|
@@ -26,7 +26,7 @@ export const TOIL_DOCS = {
|
|
|
26
26
|
"database/events.md": "# Events (append-only logs)\n\nAn `Events` collection is an **append-only log**: a growing list of immutable\nfacts under a key. Normal writes append events rather than editing history.\nUse explicit [redaction](#redacting-personal-data) when a stored payload must be erased.\n\n## What and why\n\nAn \"append-only log\" means exactly two things happen to it:\n\n1. You **append** a new event (it goes on the end).\n2. You **read** events back (newest first).\n\nEach event records something that happened: \"Ada signed the guestbook\", \"order\n1234 was refunded\", \"user logged in from a new device\".\n\nUse an `Events` log when you care about the **history**, not just the latest\nstate:\n\n- **Activity feeds:** \"what happened in this room, newest first\".\n- **Audit trails:** a tamper-resistant record of every important action.\n- **Event sourcing:** treat the log as the source of truth and compute other\n values (totals, summaries, leaderboards) from it.\n\nIf you only ever need the *current* value of something (a user's profile, a\nshopping cart), use [Documents](./documents.md) instead. If you only need a\nrunning total (likes, page views), use a [Counter](./counters.md). Events are\nfor keeping the full list of what happened.\n\n```mermaid\nflowchart LR\n A[\"append(user, event)\"] --> L[(\"Event log for user<br/>(oldest ... newest)\")]\n B[\"append(user, event)\"] --> L\n L --> R[\"latest(user, 20)<br/>returns newest 20\"]\n```\n\n## The type\n\nAn `Events` collection has two type parameters: the **key** (which log) and the\n**event** (what each entry stores).\n\n```ts\nEvents<K, V>\n```\n\n- `K` is the key type: it picks *which* log you are appending to. One key, one\n log. For a per-user activity feed, the key is the user id.\n- `V` is the event type: the shape of a single entry. Both `K` and `V` are\n [`@data`](../concepts/types.md) classes so they can be stored as bytes.\n\nYou declare it as a `@collection` field inside a `@database` class:\n\n```ts\n@data\nclass UserKey {\n id: string = '';\n constructor(id: string = '') { this.id = id; }\n}\n\n@data\nclass Activity {\n action: string = ''; // 'login', 'post.create', 'comment', ...\n detail: string = '';\n at: u64 = 0; // unix seconds\n}\n\n@database\nclass FeedDb {\n @collection static activity: Events<UserKey, Activity>;\n}\n```\n\nThe field name (`activity`) names the collection; the class name (`FeedDb`) names\nthe database. You reach the log through `FeedDb.activity`.\n\n## Operations\n\nThe [toilscript](../concepts/decorators.md) API exposes these operations:\n\n| Operation | Signature | What it does |\n| --- | --- | --- |\n| `append` | `append(key: K, event: V): void` | Add one event to the end of the log. |\n| `appendOnce` | `appendOnce(key: K, eventId: string, event: V): bool` | Add one event, but only if that `eventId` was never seen before. |\n| `get` | `get(key: K, eventId: string): V \\| null` | Read the event stored by `appendOnce` under this ID. Bounded request-path read. |\n| `last` | `last(key: K): V \\| null` | Read the newest appended event. Bounded request-path read. |\n| `latest` | `latest(key: K, limit: i32): V[]` | Read up to `limit` events, newest first. |\n| `since` | `since(key: K, limit: i32): V[]` | Read the NEXT batch of events past a `@derive`'s checkpoint, oldest first. For folding a growing log incrementally. |\n\n### `append`\n\nThe everyday operation. It adds one event and returns nothing.\n\n```ts\nFeedDb.activity.append(\n new UserKey('ada'),\n new Activity('login', 'from Paris', now),\n);\n```\n\n`append` is a **write**, so you may call it from an action handler (`@post`,\n`@put`, `@patch`, `@del`) but not from a read-only `@get`. See\n[decorators](../concepts/decorators.md) for what each handler kind may do.\n\n### `appendOnce` (idempotent append)\n\n`appendOnce` adds an event **at most once** for a given id. \"Idempotent\" means\nrunning it more than once has the same effect as running it once: the extra calls\nchange nothing.\n\nWhy you want this: networks retry. A client (or another service) may send the\nsame request twice because the first reply got lost. With plain `append`, that\ndouble-delivery writes the event twice. With `appendOnce`, you pass a stable\n`eventId` (a string you choose that is unique for that logical event), and the\nlog records it only once.\n\n```ts\n// `orderId` uniquely identifies this order event. If the request is retried,\n// the second call is a no-op and the log still has exactly one entry.\nconst first = FeedDb.activity.appendOnce(\n new UserKey('ada'),\n `order-refunded:${orderId}`, // the idempotency id\n new Activity('order.refund', orderId, now),\n);\n// first === true -> this call appended the event\n// first === false -> a previous call already appended it; nothing happened now\n```\n\nThe return value tells you which happened:\n\n- `true`: this call appended the event (it was the first with that id).\n- `false`: a call with the same id already appended it, so this was a no-op.\n\nPick an `eventId` that is truly unique per event: an order id, a payment id, a\nmessage uuid. Do not reuse one id for two different events, or the second will be\nsilently dropped as a \"duplicate\".\n\n### `get` and `last`\n\nRead one event directly from a route, query, action, derive, or job:\n\n```ts\nconst event = FeedDb.activity.get(new UserKey('ada'), 'order-refunded:123');\nconst newest = FeedDb.activity.last(new UserKey('ada'));\n```\n\n`get` uses the exact string ID supplied to `appendOnce`, scoped to that collection\nand stream key. Ordinary `append` does not expose an ID for `get`. `last` reads the\nnewest appended event from either write method; it does not interpret application\nrevision numbers or sort event IDs. A duplicate `appendOnce` neither replaces the\noriginal event nor moves it to the end of the stream.\n\nBoth operations return `null` when absent. As with Documents reads, inspect\n`Db.lastError()` immediately after a null result to distinguish absence from a\nretryable storage failure. Values decode using their stored schema version, so\n`@migrate` applies without rewriting the immutable event.\n\nThese are bounded point reads, permitted on the request path. `latest` and `since`\nremain background-only scans. Point reads do not add compare-and-swap semantics,\nor validate revision sequences. Production async reads fetch from the stream’s home\ncell; synchronous execution reports unavailable when the home is remote. Event IDs\nand stream keys are limited to 4096 encoded bytes. Deploy the updated backend and\nmesh peers before deploying modules that call these methods.\n\nDev snapshots created before event ID positions were recorded cannot reliably\nassociate old deduplication IDs with events. For such IDs, `get` reports\n`DbError.Unavailable`; `last`, `latest`, and `since` can still read the stored events.\nNew `appendOnce` entries retain their ID positions across dev-server restarts.\n\n### `latest`\n\nReads the newest events back, up to a limit, newest first.\n\n```ts\nconst recent = FeedDb.activity.latest(new UserKey('ada'), 20);\n// recent[0] is the most recent event, recent[1] the next, ...\n```\n\n`latest` is a **scan**: it can walk many rows. Scans are barred on the request\npath, so you **cannot** call `latest` from a `@get` or a `@post`. This is a\ndeliberate guardrail: a log can grow without bound, and a per-request scan would\nget slower and more expensive as the log grows.\n\nInstead you scan the log **off** the request path, in a\n[`@derive`](../background/derive.md), and publish a small precomputed\n[View](./views.md) that your routes read cheaply. The next section shows the full\npattern.\n\n### `since` (incremental read for a `@derive`)\n\n`latest` rescans the tail every time. For a log that grows without bound, that gets slower and slower. `since`\nreads only the events you have **not folded yet**: the next batch past a saved cursor, oldest first, up to a\nlimit. It is how a [`@derive`](../background/derive.md) folds a huge log incrementally instead of rescanning\nit from the start on every change.\n\nYou do not pass or manage the cursor. The host owns it: it seeds `since` from a durable checkpoint, advances\nit as it hands you events, and saves it after your fold's view publish lands. So you just loop until the\nbatch is empty:\n\n```ts\n@derive\nrollup(): void {\n const key = new StatsKey('global');\n const view = StatsDb.summary.get(key) ?? new Summary(); // the running view\n let batch = StatsDb.events.since(key, 500);\n while (batch.length > 0) { // drain in bounded batches\n for (let i = 0; i < batch.length; i++) view.apply(batch[i]);\n batch = StatsDb.events.since(key, 500);\n }\n StatsDb.summary.publish(key, view);\n}\n```\n\nTwo rules:\n\n- **`@derive` or `@job` only.** Like `latest`, `since` is a scan, so it is barred on the request path (`@get`\n / `@post`). Calling it there is rejected.\n- **Your fold must be idempotent per event.** A rare crash-recovery case (a \"healed\" event that arrives at an\n older position) makes the host re-read the log from the start that one run, so applying an event twice must\n not change the result. Use set-style updates (`view.byId[e.id] = e.value`), not blind accumulation\n (`view.count += 1`). If your fold cannot be idempotent, use `latest` and recompute instead.\n\n`since` is the efficient path; `latest` (recompute) is the simple path. See [`@derive`](../background/derive.md)\nfor the full picture.\n\n## Worked example: an activity feed\n\nHere is the whole loop: an action appends an event, a `@derive` folds the log\ninto a view, and a `@get` serves the view.\n\n```ts\nimport { Activity } from '../models/Activity';\nimport { UserKey } from '../models/UserKey';\nimport { FeedView } from '../models/FeedView';\nimport { NewActivity } from '../models/NewActivity';\n\n@database\nclass FeedDb {\n // The source of truth: every activity, appended forever.\n @collection static activity: Events<UserKey, Activity>;\n // The precomputed snapshot the GET serves: the newest 20 activities.\n @collection static feed: View<UserKey, FeedView>;\n\n // A @derive runs OFF the request path, so it is allowed to scan the log\n // (`latest`) and publish the view. The runtime runs it right after an append,\n // so the view reflects the new event on the next read.\n @derive\n rebuild(): void {\n const key = new UserKey('ada'); // (real code would loop per active user)\n const view = new FeedView();\n view.items = FeedDb.activity.latest(key, 20);\n FeedDb.feed.publish(key, view);\n }\n}\n\n@rest('feed')\nclass Feed {\n // GET reads the precomputed view: a single keyed read, NOT a scan.\n @get('/')\n public list(): FeedView {\n const view = FeedDb.feed.get(new UserKey('ada'));\n return view == null ? new FeedView() : view;\n }\n\n // POST appends one activity. The @derive republishes `feed` right after.\n @post('/')\n public record(input: NewActivity): FeedView {\n const at = <u64>(Date.now() / 1000);\n FeedDb.activity.append(new UserKey('ada'), new Activity(input.action, input.detail, at));\n // We do NOT scan here (that would be barred). We just acknowledge; the GET\n // serves the updated list from the view the derive rebuilds.\n return new FeedView();\n }\n}\n```\n\nThe two `@data` models:\n\n```ts\n@data\nexport class Activity {\n action: string = '';\n detail: string = '';\n at: u64 = 0;\n}\n\n@data\nexport class FeedView {\n items: Activity[] = [];\n}\n```\n\nThe key idea: the **log** is the durable history, and the **view** is a small,\nread-cheap snapshot derived from it. Reads hit the view; the log grows quietly in\nthe background.\n\n## Ordering and consistency\n\nToilDB is a **worldwide** database: your data lives in many regions at once. A\nfew honest details about what that means for events:\n\n- **Ordering within a log is stable.** Each key (each log) has one \"home\" location\n where appends are serialized and stamped with an increasing sequence number. So\n `latest` always returns a consistent newest-first order for that key.\n- **`latest` is newest first.** Index `0` is the most recent event.\n- **Reads can lag slightly across regions.** An append is applied at the log's\n home first, then copied out to other regions in the background (this is called\n *asynchronous replication*, and the result is *eventual consistency*: given a\n little time, every region converges to the same log). A read served from a far\n region may briefly miss the very newest event. A `@derive` runs right after the\n append that triggered it, so the view it publishes reflects that write.\n- **`appendOnce` dedups by your id.** The at-most-once guarantee is anchored to\n the `eventId` you pass, so retries are safe even across a flaky network.\n- **A log grows without bound.** Appending never shrinks it. Keep events small,\n and read them through a bounded `latest(key, N)` in a derive rather than trying\n to hold the whole log in memory.\n\n## Gotchas\n\n- **Routes can read individual events, but cannot scan the log.** Use `get` or\n `last` for a single event. `latest` and `since` require a `@derive` or `@job`;\n use a [View](./views.md) when a route needs a precomputed list.\n- **Normal changes append new facts.** Record an `order.refund` event rather\n than editing `order.create`. Explicit redaction is available for erasing\n personal payloads; it preserves event references and deduplication.\n- **Choose `eventId` carefully for `appendOnce`.** It must be unique per logical\n event. Reusing an id drops the new event as a duplicate.\n- **Do not treat a counter as a log.** A [Counter](./counters.md) gives you a\n running total but forgets the individual events. If you need the list, use\n `Events`.\n\n## Related\n\n- [Documents](./documents.md): mutable, one value per key (edit in place).\n- [Counters](./counters.md): a single running total per key.\n- [Views](./views.md): the precomputed snapshot your routes read.\n- [`@derive`](../background/derive.md): folds an event log into a View, off the\n request path.\n- [Data types (`@data`)](../concepts/types.md): how keys and events are stored.\n- [Decorators](../concepts/decorators.md): which handler kinds may append vs scan.\n\n## Redacting personal data\n\n`Events.redact(key, reference)` erases one event payload and returns `true` when\nits current recovery copies are scrubbed. Each native call cleans a bounded\npage of append-once markers. A `false` result means cleanup is still pending:\nretain the reference and call again in a later action or job. The entry is hidden\nfrom `get`, `last`, `latest`, `since`, and `entries` after the first successful\ncall. Queries and derives cannot redact.\n\nFor an `appendOnce` entry, obtain the reference with `reference(key, eventId)`.\nThis is a bounded read allowed in request handlers and still returns a reference\nwhile cleanup is pending. It returns `null` when the ID is absent; consult\n`Db.lastError()` if the read failed.\n\n```ts\nconst reference = Logs.changes.reference(key, 'change:42');\nif (reference != null) {\n const complete = Logs.changes.redact(key, reference);\n // Persist the reference in the cleanup job when complete is false.\n}\n```\n\nFor ordinary appends, a background function can enumerate references with\n`entries(key, limit = 128, after = 0)`. It returns an `EventPage<V>` containing\n`entries` (`EventEntry<V>.value` and `.reference`), `.cursor`, and `.hasMore`.\nPages are oldest first, limits are 1..1024, and the cursor is exclusive. Resume\nwith the returned cursor while `hasMore` is true, including after an empty page.\nA full final page can require one additional empty read. This scan is allowed\nonly in jobs and derives, independently of the host-managed `since` checkpoint.\n\nReferences include both the sequence and original HLC, so erasing an old\nreference does not erase a newer event at a reused position. Erasure progress\nis stored durably and replicated; retrying an append-once delivery, replaying an\nold envelope, or installing a late snapshot must not restore the erased entry.\nThe dev database retains references and erasure state across restarts. Its\nappend-once index stores positions rather than payload copies, so cleanup can\nfinish in one call there.\n\nRedaction marks the source as changed and forces affected derives to reread\nfrom the start. Rebuild projections without the erased facts and use\n[View.delete](./views.md) for projection keys that no longer exist. Application\nfolds that reuse previous view state must clear that state before a full rebuild.\n\nThis operation scrubs current event and append-once recovery payloads. Historical\nreplication envelopes, retained backups, and obsolete storage files follow their\nown retention and compaction policies; redaction does not rewrite those archives.\nKeep the erasure ledger when restoring or rebuilding a database.\n",
|
|
27
27
|
"database/membership.md": "# Membership (sets and relationships)\n\nA `Membership` collection stores **sets**: unordered groups of members under a\nkey. You add and remove members, ask whether a member is in the set, and list a\nset's members. It is the natural fit for many-to-many relationships like group\nmembers, followers, and tags.\n\n## What and why\n\nA \"set\" is a collection with no duplicates and no order: a member is either in it\nor not. `Membership` gives you one set per key. The key names the set; each member\nis one item in it.\n\nThis maps directly onto **many-to-many relationships**, where each thing on one\nside can relate to many things on the other:\n\n- **Group members:** the key is a group id, the members are user ids.\n- **Followers:** the key is a user id, the members are the ids of their followers.\n- **Tags:** the key is an article id, the members are tag names.\n- **Access control:** the key is a resource id, the members are the users allowed\n in.\n\nYou could try to store these as an array field inside a [Document](./documents.md)\n(for example `group.memberIds: string[]`). That works for **small, rarely-changed**\nsets, but it has real limits:\n\n| | Array in a Document | `Membership` set |\n| --- | --- | --- |\n| Add/remove one member | Read the whole array, edit it, write it all back | One direct `add`/`remove`, no read-modify-write |\n| Concurrent edits | Two writers can clobber each other's array | Each `add`/`remove` is its own operation |\n| Big sets | The whole array is loaded on every read/write | Members are stored separately; you read a bounded page |\n| \"Is X a member?\" | Load the array and search it | One direct `contains` check |\n\nRule of thumb: a **handful** of stable items (a user's two or three roles) is fine\nas an array in a Document. A set that **grows** or is **edited concurrently**\n(followers, group members, tags across many articles) should be a `Membership`.\n\n```mermaid\nflowchart LR\n K[\"Group key: 'eng'\"] --> S[(\"Membership set\")]\n S --> M1[\"ada\"]\n S --> M2[\"grace\"]\n S --> M3[\"linus\"]\n```\n\n## The type\n\n```ts\nMembership<K, M>\n```\n\n- `K` is the key type: it picks *which* set. For group membership, the key is the\n group id.\n- `M` is the member type: the shape of one member. For user membership, that is a\n user id. Both are [`@data`](../concepts/types.md) classes.\n\nDeclare it as a `@collection` field on a `@database` class:\n\n```ts\n@data\nclass GroupKey {\n group: string = '';\n constructor(group: string = '') { this.group = group; }\n}\n\n@data\nclass Member {\n userId: string = '';\n constructor(userId: string = '') { this.userId = userId; }\n}\n\n@database\nclass GroupsDb {\n @collection static members: Membership<GroupKey, Member>;\n}\n```\n\n## Operations\n\nFour operations, matching the toilscript API exactly. Exact signatures:\n\n| Operation | Signature | What it does |\n| --- | --- | --- |\n| `contains` | `contains(key: K, member: M): bool` | Is `member` in the set? |\n| `add` | `add(key: K, member: M): void` | Put `member` in the set (idempotent). |\n| `remove` | `remove(key: K, member: M): void` | Take `member` out of the set (idempotent). |\n| `list` | `list(key: K, limit: i32): M[]` | Up to `limit` members of the set. |\n| `listPage` | `listPage(key: K, limit: i32, cursor?: Uint8Array \\| null): MembershipPage<M>` | An ordered page and an opaque cursor for the next page. |\n\n### `contains`\n\nA direct membership check. It is a keyed read, so it is allowed from any handler,\nincluding a read-only `@get`.\n\n```ts\nif (GroupsDb.members.contains(new GroupKey('eng'), new Member('ada'))) {\n // ada is in the 'eng' group\n}\n```\n\n### `add` and `remove`\n\nBoth are **writes**, so you call them from an action handler (`@post`, `@put`,\n`@patch`, `@del`), not from a `@get`. Both are **idempotent**: \"idempotent\"\nmeans doing it again has no extra effect.\n\n- `add` on a member already in the set: no change, no error.\n- `remove` on a member not in the set: no change, no error.\n\n```ts\nGroupsDb.members.add(new GroupKey('eng'), new Member('ada')); // ada joins\nGroupsDb.members.add(new GroupKey('eng'), new Member('ada')); // no-op, still one 'ada'\nGroupsDb.members.remove(new GroupKey('eng'), new Member('linus')); // linus leaves (or no-op)\n```\n\nBecause they are idempotent, you do not need to check `contains` before calling\nthem. Just `add` to join and `remove` to leave.\n\n### `listPage`: walk a large set\n\nUse `listPage` in a job or derive to resume strictly after the last encoded member.\nPass `null` for the first page, then pass the returned `cursor` unchanged. Keep\nthat raw cursor: re-encoding a migrated member could produce different bytes.\nA page contains `members: M[]` and `cursor: Uint8Array | null`. A short page ends\nthe walk; a full final page may require one empty call to establish the end.\n\n```ts\nlet cursor: Uint8Array | null = null;\ndo {\n const page = GroupsDb.members.listPage(key, 100, cursor);\n // Process this bounded page; persist cursor when checkpointing a job.\n cursor = page.cursor;\n} while (cursor != null);\n```\n\nLimits must be positive and at most 1024, within the background scan budget.\nThe ordering is lexicographic over encoded member bytes, not alphabetical over\ndisplay names. Removing the cursor member does not prevent continuation. This is\nnot a snapshot: additions behind the cursor appear on a subsequent walk, and\nconcurrent removals can omit members. Reads remain scoped to one collection\nand set key. `listPage`, like `list`, is barred on Query and Action paths.\n\n### `list`\n\nReturns up to `limit` members of a set.\n\n```ts\nconst roster = GroupsDb.members.list(new GroupKey('eng'), 100);\n```\n\n`list` is a **scan** (it can walk many rows), so it is barred on the request\npath: you **cannot** call it from a `@get` or a `@post`. Like reading a whole\nevent log, listing a whole set belongs off the request path, in a\n[`@derive`](../background/derive.md) or a `@job`, which publishes a\n[View](./views.md) your routes read. The [worked example](#worked-example-group-membership)\nbelow shows this.\n\n`list` returns up to `limit` members. A set can be larger than one page, so treat\nthe result as \"a bounded page of members\", not necessarily \"every member\".\n\n## Worked example: group membership\n\nA user joins or leaves a group from a route; a derive lists the roster into a\nview; a route reads the view. Membership checks happen directly in the route.\n\n```ts\nimport { GroupKey } from '../models/GroupKey';\nimport { Member } from '../models/Member';\nimport { Roster } from '../models/Roster';\nimport { JoinRequest } from '../models/JoinRequest';\n\n@database\nclass GroupsDb {\n // The set: who is in each group.\n @collection static members: Membership<GroupKey, Member>;\n // A precomputed roster for the GET (listing is a scan, barred on routes).\n @collection static roster: View<GroupKey, Roster>;\n\n @derive\n rebuild(): void {\n const key = new GroupKey('eng');\n const r = new Roster();\n r.users = GroupsDb.members.list(key, 500); // scan, allowed in a derive\n GroupsDb.roster.publish(key, r);\n }\n}\n\n@rest('groups')\nclass Groups {\n // GET the roster from the view (a keyed read, not a scan).\n @get('/eng')\n public list(): Roster {\n const r = GroupsDb.roster.get(new GroupKey('eng'));\n return r == null ? new Roster() : r;\n }\n\n // Direct membership check: `contains` is a keyed read, legal in a GET.\n @get('/eng/is-member')\n public isMember(userId: string): bool {\n return GroupsDb.members.contains(new GroupKey('eng'), new Member(userId));\n }\n\n // Join: `add` is idempotent, so joining twice is harmless.\n @post('/eng/join')\n public join(input: JoinRequest): bool {\n GroupsDb.members.add(new GroupKey('eng'), new Member(input.userId));\n return true; // the @derive rebuilds the roster view right after\n }\n\n // Leave: `remove` is idempotent, so leaving when not a member is harmless.\n @post('/eng/leave')\n public leave(input: JoinRequest): bool {\n GroupsDb.members.remove(new GroupKey('eng'), new Member(input.userId));\n return true;\n }\n}\n```\n\nThe models:\n\n```ts\n@data\nexport class Member {\n userId: string = '';\n}\n\n@data\nexport class Roster {\n users: Member[] = [];\n}\n```\n\nThe split is the point: `contains`/`add`/`remove` act on a single member and are\nlegal on the request path; `list` scans the whole set and lives in a derive that\nfeeds a view.\n\n## Consistency\n\n- **Add and remove serialize at the set's home.** Each set key has one home\n location where its writes are applied in order, so an `add` followed by a\n `remove` of the same member lands in that order and the set ends up correct.\n- **Reads can lag slightly across regions.** ToilDB is worldwide. A change made\n at a set's home is copied to other regions in the background (asynchronous\n replication, giving eventual consistency: every region converges after a short\n delay). A `contains` or a `list` served from a far region may briefly miss a\n just-made change.\n- **`add`/`remove` are idempotent**, so retries are safe: re-adding an existing\n member or re-removing an absent one does nothing.\n- **A set can grow without bound.** Adding never shrinks it. Read it through a\n bounded `list(key, N)` in a derive, not all at once.\n\n## Gotchas\n\n- **You cannot `list` from a route.** `list` is a scan, legal only in a `@derive`\n or `@job`. To show a roster on a page, publish it to a [View](./views.md) from a\n derive and `get` that view in the route.\n- **`list` returns a bounded page.** A large set may have more members than your\n `limit`. Do not assume the returned array is the entire set.\n- **Sets have no application-level order.** `listPage` uses encoded-byte order for continuation. Do not treat it as an order by display name or creation time.\n- **Use `Membership` for growth or concurrency.** A small, stable set is fine as\n an array in a Document. Reach for `Membership` when the set grows, is edited\n concurrently, or you frequently ask \"is X a member?\".\n\n## Related\n\n- [Documents](./documents.md): when a small array field is enough (and when it is\n not).\n- [Views](./views.md): where a roster/list belongs so routes can read it cheaply.\n- [`@derive`](../background/derive.md): runs `list` off the request path.\n- [Data types (`@data`)](../concepts/types.md): how set keys and members are stored.\n- [Decorators](../concepts/decorators.md): which handler kinds may add/remove vs scan.\n",
|
|
28
28
|
"database/README.md": "# ToilDB\n\nToilDB is the database built into toiljs. It is **global**, it needs **no setup or connection string**, and your backend talks to it with plain typed method calls.\n\nIf you have used Postgres, MySQL, or MongoDB before, ToilDB will feel a little different, and this page explains why. If you have never used a database, that is fine too: read on and every term is defined as it appears.\n\nThe conditional-write, event-redaction, Membership cursor, and View-delete APIs\nrequire **toiljs 0.0.125 or later**, **toilscript 0.1.64 or later**, and a backend\nbuilt with their corresponding `data.*` imports. Rebuild the hot and cold\nartifacts together to include background materializers and the updated schema.\n\n## What \"a globally distributed edge database\" means\n\nLet us take that phrase one word at a time.\n\n- A **database** is a place your program stores data so it is still there on the next request. Your handler runs, finishes, and forgets everything in memory (a fresh WebAssembly instance runs each request), so anything you want to keep, a user account, a like count, a guestbook entry, has to go into a database.\n- The **edge** is a network of servers spread around the world, each one physically close to some of your users. Your compiled backend runs on the edge server nearest whoever is calling it. (For the full picture of where code runs, see [Tiers](../concepts/tiers.md).)\n- **Globally distributed** means the database is not one machine in one city. Its data lives on many machines in many regions at once. A user in Tokyo and a user in Paris each read from a copy near them, so reads are fast everywhere instead of fast in one place and slow everywhere else.\n\nPut together: ToilDB is storage that lives out on that same worldwide edge, next to your code, so a database read does not have to fly across an ocean and back.\n\n```mermaid\nflowchart LR\n subgraph edge[\"Dacely edge (worldwide)\"]\n direction LR\n H1[\"Your backend<br/>(Tokyo)\"] --> D1[(\"ToilDB<br/>copy near Tokyo\")]\n H2[\"Your backend<br/>(Paris)\"] --> D2[(\"ToilDB<br/>copy near Paris\")]\n end\n D1 <-->|\"replication<br/>(keeps copies in sync)\"| D2\n UA[\"User in Tokyo\"] --> H1\n UB[\"User in Paris\"] --> H2\n```\n\nYou never pick a region, open a connection, or run a migration script. You declare what you want to store (see [Setup](./setup.md)) and ToilDB is there.\n\n## Why seven families instead of one big \"table\"\n\nMost databases give you one general-purpose tool: a table (or a collection) that you read and write however you like. That is flexible, but on a system spread across the whole planet, the flexible tool is also the slow and error-prone tool. The classic example is a counter. If two servers on opposite sides of the world both try to do \"read the number, add one, write it back\" at the same moment, one of the two increments is silently lost, because each read the same starting value.\n\nToilDB solves this by giving you **seven specialized collection types**, called **families**. Each family is tuned for one job and exposes only the operations that are safe and fast for that job. A counter family, for instance, has no \"set to this value\" operation at all: you can only `add` a delta, and the database merges concurrent deltas from around the world without losing any. You cannot misuse it, because the unsafe operation does not exist.\n\nSo instead of one generic table you reach for, you pick the family that matches what you are doing. Picking the right one is the main skill, and the guide below walks you through it.\n\nThe seven families are:\n\n| Family | It stores | Read a page |\n| --- | --- | --- |\n| **Documents** | Records you look up by id (users, posts, orders). | [Documents](./documents.md) |\n| **Unique** | A claim on a value that must be one-of-a-kind (usernames, emails, slugs). | [Unique](./unique.md) |\n| **Counter** | A running total that many callers increment at once (likes, views). | [Counters](./counters.md) |\n| **Events** | An append-only log of things that happened (feeds, audit trails). | [Events](./events.md) |\n| **Capacity** | A limited quantity you hand out without overselling (tickets, seats). | [Capacity](./capacity.md) |\n| **Membership** | Sets of \"who belongs to what\" (followers, tags, room members). | [Membership](./membership.md) |\n| **View** | A precomputed, read-optimized snapshot (home pages, leaderboards). | [Views](./views.md) |\n\n## The one idea shared by all seven: key then value\n\nEvery family, underneath, works the same way: it maps a **key** to a **value**.\n\n- A **key** is how you find your data. Think of it as the label on a drawer: a user id, a username, a room name.\n- A **value** is what is in the drawer: the user record, the owner of a username, the members of a room.\n\nIn ToilDB, both the key and the value are ordinary TypeScript classes that you tag with `@data`. The `@data` tag tells toilscript (the compiler that turns your backend into WebAssembly) how to pack that class into bytes for storage and unpack it again. You write a normal class with normal fields, give each field a default, and the compiler does the rest.\n\n```ts\n// A key: how you address one record.\n@data\nclass UserId {\n id: string = '';\n constructor(id: string = '') { this.id = id; }\n}\n\n// A value: what you store under that key.\n@data\nclass User {\n id: string = '';\n name: string = '';\n score: u64 = 0;\n}\n```\n\nYou then declare a collection that maps that key type to that value type, for example `Documents<UserId, User>`. Every family is generic over its key and value types in exactly this way, so once you understand `@data` keys and values you understand all seven. The full `@data` reference is on the [data types page](../backend/data.md); how to declare collections is on [Setup](./setup.md).\n\n## Eventual consistency, in plain words\n\nBecause ToilDB keeps copies of your data in many regions, there is one honest trade-off you need to understand: **eventual consistency**.\n\nEvery key has one **home**: a single region that officially owns that key's data. All **writes** to a key travel to its home, where they are applied one at a time, in order. That is what makes writes safe: even if a thousand servers write the same key at once, the home lines them up so nothing is lost or corrupted.\n\n**Reads**, on the other hand, are served from the copy nearest the reader, which is fast but may be a beat behind. After a write lands at the home, it takes a brief moment (usually milliseconds) to fan out to the other regions' copies. During that moment, a reader in another region might still see the previous value. The copies always catch up; they are *eventually* consistent, not *instantly* consistent everywhere.\n\nWhat this means in practice:\n\n- **It is usually invisible.** The lag is tiny, and most apps never notice.\n- **Do not assume a write is visible everywhere the instant it returns.** For example, right after you create a record, a read from a far-away region might briefly not see it yet.\n- **Some families are stronger.** Because writes to a single key are serialized at its home, operations that must never race, claiming a unique username, reserving the last ticket, are decided at the home and are safe. The Unique and Capacity families lean on this so two callers can never both win. See [Unique](./unique.md) and [Capacity](./capacity.md).\n\nEach family page spells out its own consistency behavior, so you always know what to expect.\n\n## Choosing a family: the decision guide\n\nStart from what you are trying to do, not from a data structure. This table maps a real need to the family that was built for it.\n\n| I need to... | Use | Why |\n| --- | --- | --- |\n| Store and update a thing I look up by id (a user, a post, an order). | **Documents** | The general-purpose record store: create, read, update, delete by key. |\n| Guarantee a value is used by only one owner across the whole world (username, email, slug). | **Unique** | Claims are decided at the key's home, so two people cannot claim the same name. |\n| Count something that many people bump at the same time (likes, page views, stock tally). | **Counter** | Concurrent `add`s from anywhere merge without ever losing an increment. |\n| Keep a growing list of things that happened, in order (activity feed, audit log). | **Events** | Append-only: you add events and read the newest, but never edit history. |\n| Hand out a limited quantity and never sell more than exist (tickets, seats, inventory). | **Capacity** | Reserve/confirm/cancel holds at the home prevent overselling. |\n| Track which members belong to a set (followers, tags, room members, permissions). | **Membership** | Add/remove/check membership without loading a whole record to edit a list. |\n| Serve a precomputed, ready-to-render result quickly (leaderboard, home page snapshot). | **View** | A background job computes it once; requests read it with a single fast lookup. |\n\nIf two families seem to fit, this flowchart resolves it. Read it top to bottom.\n\n```mermaid\nflowchart TD\n START[\"What are you storing?\"] --> Q1{\"Is it a running<br/>total (a number many<br/>callers increase)?\"}\n Q1 -->|Yes| COUNTER[\"Counter\"]\n Q1 -->|No| Q2{\"Must a value be<br/>one-of-a-kind across<br/>everyone (username,<br/>email, slug)?\"}\n Q2 -->|Yes| UNIQUE[\"Unique\"]\n Q2 -->|No| Q3{\"Is it a limited stock<br/>you must not oversell<br/>(tickets, seats)?\"}\n Q3 -->|Yes| CAPACITY[\"Capacity\"]\n Q3 -->|No| Q4{\"Is it an append-only<br/>log of events that<br/>happened, kept in order?\"}\n Q4 -->|Yes| EVENTS[\"Events\"]\n Q4 -->|No| Q5{\"Is it a set of members<br/>you add/remove/check<br/>(followers, tags)?\"}\n Q5 -->|Yes| MEMBERSHIP[\"Membership\"]\n Q5 -->|No| Q6{\"Is it a precomputed<br/>read-only snapshot a<br/>background job builds?\"}\n Q6 -->|Yes| VIEW[\"View\"]\n Q6 -->|No| DOCUMENTS[\"Documents<br/>(the default record store)\"]\n```\n\nA quick sanity check when you land somewhere: **Documents is the default.** If you are storing \"a thing with fields that I update by id,\" it is almost always Documents. The other six exist for the specific jobs above where Documents would be slow, unsafe under concurrency, or awkward.\n\nReal apps mix families freely. The demo guestbook, for example, uses **Events** for the log of signatures, a **Counter** for the running total, and a **View** for the ready-to-serve snapshot, all in one small feature.\n\n## Where to go next\n\n- [Setup](./setup.md): declare a database, its collections, and reach them from a handler.\n- [Documents](./documents.md): the general-purpose record store (start here).\n- [Unique](./unique.md), [Counters](./counters.md), [Events](./events.md), [Views](./views.md), [Membership](./membership.md), [Capacity](./capacity.md): the specialized families.\n\n## Related\n\n- [Setup](./setup.md): how to declare `@database`, `@collection`, and `@data` types.\n- [Data types (`@data`)](../backend/data.md): the typed keys and values every family uses.\n- [Tiers](../concepts/tiers.md): where your backend and its data run.\n- [Decorators](../concepts/decorators.md): `@query`, `@action`, `@derive`, and friends.\n",
|
|
29
|
-
"database/setup.md": "# Declaring a database\n\nYou set up ToilDB by **declaring** it in code: a `@database` class listing your collections, plus the `@data` key and value types those collections use. There is no schema file, no migration command, and no connection string.\n\n## What and why\n\nA **database declaration** tells toiljs three things at compile time: which collections exist, which family each one is, and what key and value types it stores. The compiler bakes that list into your server's WebAssembly, and both the dev server and the production edge read it back to set up your collections automatically. You declare; the platform provisions.\n\nReach for this on day one: before you can read or write any data, you need a `@database` with at least one `@collection`.\n\nGroup related data into collections within a small number of databases. A new\nfeature usually needs a new collection, not another `@database` class. Database\nnamespaces do not replace application authorization: enforce account and\norganization access in server routes and scope collection keys appropriately.\n\n## How: the three pieces\n\nA working database is always these three pieces together.\n\n1. **`@data` key and value types.** Plain classes, tagged `@data`, that describe what you store.\n2. **A `@database` class** whose static fields are `@collection`s, each typed by a family.\n3. **A handler** (a `@rest` route or an RPC method) that reads and writes those collections.\n\nHere is a complete, minimal example: a store of users you look up by id.\n\n```ts\n// A @data key: how you address one user.\n@data\nclass UserId {\n id: string = '';\n constructor(id: string = '') { this.id = id; }\n}\n\n// A @data value: what you store for each user.\n@data\nclass User {\n id: string = '';\n name: string = '';\n score: u64 = 0;\n}\n\n// The database declaration. Each @collection is one collection,\n// typed by its family (here, Documents) with <Key, Value>.\n@database\nclass AppDb {\n @collection static users: Documents<UserId, User>;\n}\n\n// A route that reads and writes the collection.\n@rest('users')\nclass Users {\n @get('/:id')\n public getUser(ctx: RouteContext): User {\n const user = AppDb.users.get(new UserId(ctx.param('id')));\n return user == null ? new User() : user;\n }\n\n @post('/')\n public createUser(input: User): User {\n AppDb.users.create(new UserId(input.id), input);\n return input;\n }\n}\n```\n\nThat is the whole setup. Run `toiljs dev` and `AppDb.users` works immediately.\n\n### The `@data` types\n\nBoth the key and the value are `@data` classes. `@data` is what makes a class storable: the compiler synthesizes a binary codec (pack to bytes, unpack from bytes) so ToilDB can persist it. The rules that matter here:\n\n- **Give every field a default** (`= ''`, `= 0`, and so on). The decoder builds an empty instance and fills it, so it needs defaults.\n- **A value type must be default-constructible** (creatable with `new User()` and no arguments). Keys often add a convenience constructor, as `UserId` does above, so you can write `new UserId('abc')`.\n- Fields may be numbers (`u8`..`u256`, `i8`..`i256`, `f32`, `f64`), `bool`, `string`, another `@data` class, or an array of any of these.\n\nThe full reference, including how the same type becomes a typed client type, is on the [data types page](../backend/data.md).\n\n### The `@database` class and `@collection` fields\n\n```ts\n@database\nclass AppDb {\n @collection static users: Documents<UserId, User>;\n @collection static likes: Counter<UserId>;\n}\n```\n\n- `@database` marks the class as a database. You can have more than one `@database` class; each is a separate namespace of collections.\n- Each `@collection` is one collection. Declare it as a **`static`** field with **no initializer**: you write the type, and the compiler wires up the actual handle for you.\n- The field's **type is its family**: `Documents<K, V>`, `Counter<K>`, `Events<K, V>`, `Unique<K, V>`, `Membership<K, M>`, `Capacity<K>`, or `View<K, V>`. The compiler reads the family straight from this type, so getting it right here is how you pick a family (see [Choosing a family](./README.md#choosing-a-family-the-decision-guide)).\n\nYou reach a collection through the class, statically: `AppDb.users.get(...)`, `AppDb.likes.add(...)`. There is nothing to instantiate.\n\n### Reaching collections from a handler\n\nAny backend function can read and write collections by referencing them on the database class. What that function is allowed to do depends on its **kind**, covered next.\n\n## How access is gated: `@query`, `@action`, and friends\n\nToilDB will not let every function do everything. Each backend function runs as one **function kind**, and each kind is allowed a different slice of database operations. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path.\n\nYou rarely write these decorators by hand, because routes get a sensible kind automatically:\n\n- A **`@get`** route (a safe, read-only HTTP method) runs as a **Query**.\n- A **`@post`** route (a mutating method) runs as an **Action**.\n- A plain RPC **`@remote`** method defaults to a **Query** (read-only) because it has no HTTP method to infer from. Tag it `@action` if it writes.\n\nYou can override the default with `@query` or `@action` on the method when the automatic choice is wrong (for example, a `@get` that genuinely needs to write, though that is unusual).\n\nWhat each kind may do:\n\n| Kind | Set by | May do | May **not** do |\n| --- | --- | --- | --- |\n| **Query** | `@get`, plain `@remote`, or `@query` | Point reads: `get`, `getMany`, `exists`, `lookup`, `contains`, counter `get`, view `get`, Events `get`/`last`, capacity `available`. | Any write. Any scan. |\n| **Action** | `@post` or `@action` | Everything a Query can, plus bounded writes: `create`, `patch`, `delete`, `getDelete`, `enqueue`, `append`, `appendOnce`, counter `add`, membership `add`/`remove`, unique `claim`/`release`, capacity `reserve`/`confirm`/`cancel`. | Scans. Publishing a View. |\n| **Derive / Job** | `@derive` / `@job` (background work) | Reads including **scans** (`latest`, membership `list`), plus `publish` a View. | (Run off the request path; see below.) |\n\nTwo rules trip people up, so they are worth stating plainly:\n\n- **Scans are barred from request handlers.** Reading \"the newest N events\" (`events.latest`) or \"the members of this set\" (`membership.list`) can fan out across many rows, so a `@get` or `@post` cannot call them. Do the scan in a `@derive` (a small function that recomputes a snapshot off the request path) and have the request read the snapshot. See [Views](./views.md) and [@derive](../background/derive.md).\n- **Only a `@derive` or `@job` may `publish` a View.** Requests read views; background work writes them.\n\nBoth gates are enforced twice: the compiler rejects an illegal call at build time, and the edge rejects it again at runtime, so a hand-edited module cannot sneak past. For the decorator catalog, see [Decorators](../concepts/decorators.md).\n\n## No manual provisioning: how it actually gets set up\n\nYou never create a table or run a migration. Here is the machinery, so the \"it just works\" is not a mystery.\n\nWhen toilscript compiles your backend, it scans every `@database` class and writes a small catalog into the `.wasm` file: for each collection, its name, its family, its key and value type names, and the value's schema version. This catalog rides *inside* the compiled module.\n\nThen, wherever your backend runs, the host reads that catalog once at startup and builds your collections to match:\n\n- Under **`toiljs dev`**, the host is an in-process, in-memory emulator. It reads the catalog and stands up all seven families in memory. This is a development store: single process, single tenant, and cleared when you restart. It exists so you can build and test against real ToilDB behavior with no services to run.\n- On the **Dacely edge**, the host is the real ToilDB, backed by a globally distributed ScyllaDB cluster. It reads the *same* catalog and serves the *same* operations, now durable and worldwide.\n\nBecause both sides read the same catalog, **the same code runs unchanged in dev and in production.** There is no connection string to swap and no provisioning step to run.\n\n```mermaid\nflowchart TD\n SRC[\"@database AppDb<br/>@collection static users: Documents...<br/>@data UserId / User\"]\n SRC -->|toilscript compile| WASM[\"server.wasm<br/>(with an embedded<br/>toildb catalog section)\"]\n WASM -->|toiljs dev reads the catalog| DEV[\"In-memory dev store<br/>(one process, cleared on restart)\"]\n WASM -->|edge reads the same catalog| EDGE[(\"ToilDB on the Dacely edge<br/>(ScyllaDB, worldwide, durable)\")]\n```\n\n## Gotchas\n\n- **Declare collections as `static` fields with no initializer.** Do not try to `new` a collection or assign a handle yourself; the compiler owns that.\n- **Every `@data` field needs a default, and value types must be default-constructible.** A missing default fails to compile.\n- **Dev data is not durable.** The `toiljs dev` store lives in memory and resets on restart. Do not rely on it to persist across dev runs; that is what the edge is for.\n- **Changing a `@data` type is a format change.** Reordering fields or changing a field's type changes the stored layout. Add new fields at the end, and use a migration when you evolve a stored type. See [data types](../backend/data.md).\n- **Pick the family at the type.** The family is read from the collection's declared type, so `Counter<K>` versus `Documents<K, V>` is a real, load-bearing choice, not a hint. Revisit [Choosing a family](./README.md#choosing-a-family-the-decision-guide) if unsure.\n\n## Related\n\n- [ToilDB overview](./README.md): the seven families and how to choose.\n- [Documents](./documents.md): the general-purpose record family (a good first collection).\n- [Data types (`@data`)](../backend/data.md): keys, values, and the codec.\n- [Decorators](../concepts/decorators.md): `@database`, `@collection`, `@query`, `@action`, `@derive`.\n- [@derive](../background/derive.md): recompute snapshots and run scans off the request path.\n",
|
|
29
|
+
"database/setup.md": "# Declaring a database\n\nYou set up ToilDB by **declaring** it in code: a `@database` class listing your collections, plus the `@data` key and value types those collections use. There is no schema file, no migration command, and no connection string.\n\n## What and why\n\nA **database declaration** tells toiljs three things at compile time: which collections exist, which family each one is, and what key and value types it stores. The compiler bakes that list into your server's WebAssembly, and both the dev server and the production edge read it back to set up your collections automatically. You declare; the platform provisions.\n\nReach for this on day one: before you can read or write any data, you need a `@database` with at least one `@collection`.\n\nGroup related data into collections within a small number of databases. A new\nfeature usually needs a new collection, not another `@database` class. Database\nnamespaces do not replace application authorization: enforce account and\norganization access in server routes and scope collection keys appropriately.\n\n## How: the three pieces\n\nA working database is always these three pieces together.\n\n1. **`@data` key and value types.** Plain classes, tagged `@data`, that describe what you store.\n2. **A `@database` class** whose static fields are `@collection`s, each typed by a family.\n3. **A handler** (a `@rest` route or an RPC method) that reads and writes those collections.\n\nHere is a complete, minimal example: a store of users you look up by id.\n\n```ts\n// A @data key: how you address one user.\n@data\nclass UserId {\n id: string = '';\n constructor(id: string = '') { this.id = id; }\n}\n\n// A @data value: what you store for each user.\n@data\nclass User {\n id: string = '';\n name: string = '';\n score: u64 = 0;\n}\n\n// The database declaration. Each @collection is one collection,\n// typed by its family (here, Documents) with <Key, Value>.\n@database\nclass AppDb {\n @collection static users: Documents<UserId, User>;\n}\n\n// A route that reads and writes the collection.\n@rest('users')\nclass Users {\n @get('/:id')\n public getUser(ctx: RouteContext): User {\n const user = AppDb.users.get(new UserId(ctx.param('id')));\n return user == null ? new User() : user;\n }\n\n @post('/')\n public createUser(input: User): User {\n AppDb.users.create(new UserId(input.id), input);\n return input;\n }\n}\n```\n\nThat is the whole setup. Run `toiljs dev` and `AppDb.users` works immediately.\n\n### The `@data` types\n\nBoth the key and the value are `@data` classes. `@data` is what makes a class storable: the compiler synthesizes a binary codec (pack to bytes, unpack from bytes) so ToilDB can persist it. The rules that matter here:\n\n- **Give every field a default** (`= ''`, `= 0`, and so on). The decoder builds an empty instance and fills it, so it needs defaults.\n- **A value type must be default-constructible** (creatable with `new User()` and no arguments). Keys often add a convenience constructor, as `UserId` does above, so you can write `new UserId('abc')`.\n- Fields may be numbers (`u8`..`u256`, `i8`..`i256`, `f32`, `f64`), `bool`, `string`, another `@data` class, or an array of any of these.\n\nThe full reference, including how the same type becomes a typed client type, is on the [data types page](../backend/data.md).\n\n### The `@database` class and `@collection` fields\n\n```ts\n@database\nclass AppDb {\n @collection static users: Documents<UserId, User>;\n @collection static likes: Counter<UserId>;\n}\n```\n\n- `@database` marks the class as a database. You can have more than one `@database` class; each is a separate namespace of collections.\n- Each `@collection` is one collection. Declare it as a **`static`** field with **no initializer**: you write the type, and the compiler wires up the actual handle for you.\n- The field's **type is its family**: `Documents<K, V>`, `Counter<K>`, `Events<K, V>`, `Unique<K, V>`, `Membership<K, M>`, `Capacity<K>`, or `View<K, V>`. The compiler reads the family straight from this type, so getting it right here is how you pick a family (see [Choosing a family](./README.md#choosing-a-family-the-decision-guide)).\n\nYou reach a collection through the class, statically: `AppDb.users.get(...)`, `AppDb.likes.add(...)`. There is nothing to instantiate.\n\n### Reaching collections from a handler\n\nAny backend function can read and write collections by referencing them on the database class. What that function is allowed to do depends on its **kind**, covered next.\n\n## How access is gated: `@query`, `@action`, and friends\n\nToilDB will not let every function do everything. Each backend function runs as one **function kind**, and each kind is allowed a different slice of database operations. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path.\n\nYou rarely write these decorators by hand, because routes get a sensible kind automatically:\n\n- A **`@get`** route (a safe, read-only HTTP method) runs as a **Query**.\n- A **`@post`** route (a mutating method) runs as an **Action**.\n- A plain RPC **`@remote`** method defaults to a **Query** (read-only) because it has no HTTP method to infer from. Tag it `@action` if it writes.\n\nYou can override the default with `@query` or `@action` on the method when the automatic choice is wrong (for example, a `@get` that genuinely needs to write, though that is unusual).\n\nWhat each kind may do:\n\n| Kind | Set by | May do | May **not** do |\n| --- | --- | --- | --- |\n| **Query** | `@get`, plain `@remote`, or `@query` | Point reads: `get`, `getMany`, `exists`, `lookup`, `contains`, counter `get`, view `get`, Events `get`/`last`, capacity `available`. | Any write. Any scan. |\n| **Action** | `@post` or `@action` | Everything a Query can, plus bounded writes: `create`, `patch`, `delete`, `getDelete`, `enqueue`, `append`, `appendOnce`, counter `add`, membership `add`/`remove`, unique `claim`/`release`, capacity `reserve`/`confirm`/`cancel`. | Scans. Publishing a View. |\n| **Derive / Job** | Host dispatches Derive / Job; compiler checks `@derive` / `@job` | Reads including **scans** (`latest`, membership `list`), plus `publish` a View. | (Run off the request path; see below.) |\n\n**The host determines the execution kind.** With toilscript 0.1.65, toiljs 0.0.126, and a v2-capable edge host, a `@scheduled` method tagged `@job` runs as **Job**, including scans and `View.publish`. Daemon `onStart` and untagged ticks run as **Action**. Tagging a helper `@job` does not change its caller's runtime permissions. Job catalogs use v2; older edge hosts reject them rather than silently downgrade to Action. See [daemon permissions](../background/daemons.md#onstart-run-once-at-boot).\n\nTwo rules trip people up, so they are worth stating plainly:\n\n- **Scans are barred from request handlers.** Reading \"the newest N events\" (`events.latest`) or \"the members of this set\" (`membership.list`) can fan out across many rows, so a `@get` or `@post` cannot call them. Do the scan in a `@derive` (a small function that recomputes a snapshot off the request path) and have the request read the snapshot. See [Views](./views.md) and [@derive](../background/derive.md).\n- **Only a call executing as Derive or Job may `publish` a View.** A `@job` annotation on code called from an Action does not change the caller's execution kind.\n\nBoth gates are enforced twice: the compiler rejects an illegal call at build time, and the edge rejects it again at runtime, so a hand-edited module cannot sneak past. For the decorator catalog, see [Decorators](../concepts/decorators.md).\n\n## No manual provisioning: how it actually gets set up\n\nYou never create a table or run a migration. Here is the machinery, so the \"it just works\" is not a mystery.\n\nWhen toilscript compiles your backend, it scans every `@database` class and writes a small catalog into the `.wasm` file: for each collection, its name, its family, its key and value type names, and the value's schema version. This catalog rides *inside* the compiled module.\n\nThen, wherever your backend runs, the host reads that catalog once at startup and builds your collections to match:\n\n- Under **`toiljs dev`**, the host is an in-process, in-memory emulator. It reads the catalog and stands up all seven families in memory. This is a development store: single process, single tenant, and cleared when you restart. It exists so you can build and test against real ToilDB behavior with no services to run.\n- On the **Dacely edge**, the host is the real ToilDB, backed by a globally distributed ScyllaDB cluster. It reads the *same* catalog and serves the *same* operations, now durable and worldwide.\n\nBecause both sides read the same catalog, **the same code runs unchanged in dev and in production.** There is no connection string to swap and no provisioning step to run.\n\n```mermaid\nflowchart TD\n SRC[\"@database AppDb<br/>@collection static users: Documents...<br/>@data UserId / User\"]\n SRC -->|toilscript compile| WASM[\"server.wasm<br/>(with an embedded<br/>toildb catalog section)\"]\n WASM -->|toiljs dev reads the catalog| DEV[\"In-memory dev store<br/>(one process, cleared on restart)\"]\n WASM -->|edge reads the same catalog| EDGE[(\"ToilDB on the Dacely edge<br/>(ScyllaDB, worldwide, durable)\")]\n```\n\n## Gotchas\n\n- **Declare collections as `static` fields with no initializer.** Do not try to `new` a collection or assign a handle yourself; the compiler owns that.\n- **Every `@data` field needs a default, and value types must be default-constructible.** A missing default fails to compile.\n- **Dev data is not durable.** The `toiljs dev` store lives in memory and resets on restart. Do not rely on it to persist across dev runs; that is what the edge is for.\n- **Changing a `@data` type is a format change.** Reordering fields or changing a field's type changes the stored layout. Add new fields at the end, and use a migration when you evolve a stored type. See [data types](../backend/data.md).\n- **Pick the family at the type.** The family is read from the collection's declared type, so `Counter<K>` versus `Documents<K, V>` is a real, load-bearing choice, not a hint. Revisit [Choosing a family](./README.md#choosing-a-family-the-decision-guide) if unsure.\n\n## Related\n\n- [ToilDB overview](./README.md): the seven families and how to choose.\n- [Documents](./documents.md): the general-purpose record family (a good first collection).\n- [Data types (`@data`)](../backend/data.md): keys, values, and the codec.\n- [Decorators](../concepts/decorators.md): `@database`, `@collection`, `@query`, `@action`, `@derive`.\n- [@derive](../background/derive.md): recompute snapshots and run scans off the request path.\n",
|
|
30
30
|
"database/unique.md": "# Unique\n\nThe **Unique** family enforces that a value is claimed by only one owner across the entire world. It is how you make usernames, email addresses, and URL slugs one-of-a-kind, safely, even when two people try to grab the same one at the same instant.\n\n## What and why\n\nA **Unique collection** maps a `@data` **claim key** (the thing that must be unique, like a username) to a `@data` **owner value** (who or what claimed it, like a user id). At any moment a claim key is either **unclaimed** or **owned by exactly one owner**. The family lets you look up one or several owners, claim a key, and release it.\n\nReach for Unique whenever a value must be globally singular:\n\n- usernames or handles\n- email addresses\n- URL slugs or workspace names\n- any \"reserve this name for me\" scenario\n\nWhy not just check a Documents record first? Because a check-then-write has a race: two requests can both check \"is `alice` free?\", both see yes, and both create it. Unique closes that gap. The claim is decided at the key's single **home** (see [consistency](./README.md#eventual-consistency-in-plain-words)), where claims are processed one at a time, so exactly one of the two racers wins.\n\nDeclare one by typing a `@collection` as `Unique<ClaimKey, OwnerValue>`:\n\n```ts\n@data\nclass Username {\n name: string = '';\n constructor(name: string = '') { this.name = name; }\n}\n\n@data\nclass OwnerId {\n userId: string = '';\n constructor(userId: string = '') { this.userId = userId; }\n}\n\n@database\nclass AppDb {\n @collection static usernames: Unique<Username, OwnerId>;\n}\n```\n\n## The operations\n\n`K` is the claim-key type, `V` the owner value type.\n\n| Operation | Signature | Returns | Use it to |\n| --- | --- | --- | --- |\n| `lookup` | `lookup(key: K): V \\| null` | the current owner, or `null` if unclaimed | find out who owns a name |\n| `claim` | `claim(key: K, value: V): ClaimResult<V>` | a `ClaimResult` (see below) | try to take a name for an owner |\n| `release` | `release(key: K, value: V): void` | nothing; **traps** if you are not the owner | give a name back |\n\n`lookup` is a read, so it works in any function. `claim` and `release` are writes, so they need an **Action** (a `@post` route or an `@action`); see [Setup](./setup.md#how-access-is-gated-query-action-and-friends).\n\n### `ClaimResult`\n\n`claim` returns a small object that tells you what happened:\n\n```ts\nclass ClaimResult<V> {\n claimed: bool; // true if YOU own the key now\n owner: V | null; // when claimed is false, who owns it instead\n}\n```\n\n- **`claimed == true`**: you own the key. This covers both a fresh claim and an **idempotent re-claim**: if you claim a key you already own (same owner value), you still get `true`, so retrying a claim is safe. `owner` is `null` in this case.\n- **`claimed == false`**: someone else got there first. `owner` is their value, so you can tell the user \"that name is taken.\"\n\n### `lookup`\n\n`lookup` just reads the current owner without changing anything:\n\n```ts\nconst owner = AppDb.usernames.lookup(new Username('alice'));\nif (owner == null) {\n // 'alice' is free\n} else {\n // owner.userId currently holds 'alice'\n}\n```\n\nNote that `lookup` is subject to [eventual consistency](./README.md#eventual-consistency-in-plain-words): a claim made moments ago in another region may not show up in a far-away `lookup` yet. Do not use `lookup` as your uniqueness guard. `lookup` is for display (\"this name is taken by ...\"); the real guarantee comes from `claim`, which is decided at the home and cannot race.\n\n### `lookupMany`\n\n`lookupMany(keys)` returns owners in input order, with `null` for unclaimed keys.\nIt uses one database operation for the batch and the same tenant isolation and\nconsistency rules as `lookup`. The host enforces its batch-size limit (32 keys for queries, 64 for actions); split larger lists into bounded pages. The batch is not a transaction\nor a consistent snapshot across keys. It never replaces `claim` as a uniqueness\ncheck.\n\n```ts\nconst owners = AppDb.usernames.lookupMany([new Username('alice'), new Username('bob')]);\n```\n\n### `claim`\n\n`claim` is the operation that actually enforces uniqueness. You pass the key and the owner value:\n\n```ts\nconst result = AppDb.usernames.claim(new Username('alice'), new OwnerId('u_123'));\nif (result.claimed) {\n // 'alice' is now yours\n} else {\n // taken; result.owner is the current owner\n}\n```\n\nBecause claims are serialized at the key's home, this is race-safe. If two requests anywhere in the world call `claim('alice', ...)` at the same moment, the home applies them in order: the first gets `claimed: true`, the second gets `claimed: false` with the first as `owner`. There is no window where both win.\n\n### `release`\n\n`release` gives a claim back so the key becomes available again. Only the **current owner** may release: you pass both the key and the owner value, and if that value is not the current owner, `release` **traps** (aborts the request). This prevents one user from releasing another user's name.\n\n```ts\nAppDb.usernames.release(new Username('alice'), new OwnerId('u_123'));\n```\n\nRelease when a name is being changed or an account is deleted, so the name returns to the pool.\n\n## The claim / release lifecycle\n\nA claim is a small state machine: unclaimed, owned, and back to unclaimed.\n\n```mermaid\nstateDiagram-v2\n [*] --> Unclaimed\n Unclaimed --> Owned: claim(key, me) -> claimed: true\n Owned --> Owned: claim(key, me) again -> claimed: true (idempotent)\n Owned --> Owned: claim(key, someoneElse) -> claimed: false, owner: me\n Owned --> Unclaimed: release(key, me)\n```\n\nThe key insight: **claiming is the guard, releasing is cleanup.** You claim to reserve, you release to free. A claim that is never released stays owned forever (there is no automatic expiry; if you want time-limited holds, that is what the [Capacity](./capacity.md) family's TTL reservations are for).\n\n## Worked example: reserving a username on signup\n\nThe safe signup pattern is: claim the username first, then create the account, and if creating the account fails, release the claim so the name is not stranded.\n\n```ts\nimport { Response } from 'toiljs/server/runtime';\n\n@data\nclass Username {\n name: string = '';\n constructor(name: string = '') { this.name = name; }\n}\n\n@data\nclass OwnerId {\n userId: string = '';\n constructor(userId: string = '') { this.userId = userId; }\n}\n\n@data\nclass UserId {\n id: string = '';\n constructor(id: string = '') { this.id = id; }\n}\n\n@data\nclass User {\n id: string = '';\n username: string = '';\n}\n\n@data\nclass SignupInput {\n username: string = '';\n userId: string = '';\n}\n\n@database\nclass AppDb {\n @collection static usernames: Unique<Username, OwnerId>;\n @collection static users: Documents<UserId, User>;\n}\n\n@rest('signup')\nclass Signup {\n // POST /signup (Action: may claim and write)\n @post('/')\n public signup(input: SignupInput): Response {\n const owner = new OwnerId(input.userId);\n\n // 1. Claim the username. This is the uniqueness guard.\n const claim = AppDb.usernames.claim(new Username(input.username), owner);\n if (!claim.claimed) {\n return Response.text('username taken', 409);\n }\n\n // 2. Create the account. If this fails, undo the claim so the name is free.\n const user = new User();\n user.id = input.userId;\n user.username = input.username;\n if (!AppDb.users.create(new UserId(input.userId), user)) {\n AppDb.usernames.release(new Username(input.username), owner);\n return Response.text('could not create account', 409);\n }\n\n return Response.json(user.toJSON().toString());\n }\n}\n```\n\nTwo things make this correct:\n\n- **The claim happens before the account write**, so uniqueness is decided up front.\n- **The claim is released if the follow-up write fails**, so a failed signup does not permanently burn a username.\n\nBecause `claim` is idempotent for the same owner, a client that retries the whole request after a network hiccup does not get a false \"taken\" error: the second `claim('alice', u_123)` returns `claimed: true` again.\n\n## Consistency notes\n\n- **`claim` and `release` are strongly consistent at the key's home.** Two callers can never both own the same key; the home serializes claims. This is the whole point of the family.\n- **`lookup` is eventually consistent.** It reads a possibly-nearby copy, so a very recent claim from elsewhere may not appear yet. Never gate uniqueness on `lookup`; gate it on the result of `claim`.\n- **Claims do not expire.** A claim stays until someone releases it. Use [Capacity](./capacity.md) if you need holds that auto-release after a timeout.\n\n## Gotchas\n\n- **Do not use `lookup` as the uniqueness check.** Its answer can be stale. Call `claim` and trust its `claimed` flag.\n- **Release on failure and on account deletion.** A claim you forget to release stays owned forever, quietly blocking that name.\n- **`release` traps for a non-owner.** Pass the correct owner value, or you abort the request. If you are not sure you own it, `lookup` first (for display) but expect `release` to enforce ownership regardless.\n- **The owner value is data you choose.** Store enough in it (a user id, say) to know who holds the claim, so you can display and release it later.\n\n## Related\n\n- [ToilDB overview](./README.md): the seven families and how to choose.\n- [Setup](./setup.md): declaring the collection and which function kinds may claim.\n- [Documents](./documents.md): the account record you create alongside a claim.\n- [Capacity](./capacity.md): time-limited holds that auto-release (a different kind of \"reserve\").\n- [Data types (`@data`)](../backend/data.md): the claim key and owner value.\n",
|
|
31
31
|
"database/views.md": "# Views (materialized views)\n\nA `View` collection is a **precomputed, read-optimized result** that you publish\nonce and read many times. Reads are a single cheap keyed lookup, no scanning, no\nrecomputing.\n\n## What and why\n\nA \"materialized view\" is a fancy name for a simple idea: instead of computing an\nanswer every time someone asks, you compute it **once**, store the finished\nanswer, and just hand it out on each request. \"Materialized\" means the result is\nmade real and saved (as opposed to computed on the fly). \"View\" means it is a\nread-friendly projection of your underlying data.\n\nThink of a leaderboard. Computing \"the top 10 players by score\" means scanning\nevery player and sorting. You do not want to do that on every page load. So you\ncompute it in the background, save the finished top-10 list as a view, and every\npage load just reads that saved list. Fast, cheap, and it does not get slower as\nyou add players.\n\nUse a `View` when:\n\n- A page needs a result that is **expensive to compute** (a scan, a sort, a fold\n over many rows).\n- That result is **read far more often than it changes** (a home page, a feed, a\n leaderboard, a \"latest N\" list, a rendered summary).\n\nThe rule of thumb: if a route wants \"the newest N of something\" or \"the top N of\nsomething\", that is a scan, and scans are barred on the request path. A `View`\nis how you serve that result without scanning per request.\n\n```mermaid\nflowchart LR\n subgraph off[\"Off the request path (@derive)\"]\n S[(\"Source data<br/>events / counters / records\")] --> D[\"@derive: scan + compute\"]\n D --> P[\"publish(key, result)\"]\n end\n P --> V[(\"View\")]\n subgraph on[\"On the request path (@get)\"]\n G[\"view.get(key)\"] --> V\n end\n```\n\n## The type\n\nA `View` has two type parameters: the **key** (which view) and the **value**\n(the precomputed result).\n\n```ts\nView<K, V>\n```\n\n- `K` is the key type: it picks *which* view to read or publish. For a single\n global leaderboard, the key can be a fixed constant like `'main'`.\n- `V` is the value type: the finished result you serve. Both are\n [`@data`](../concepts/types.md) classes.\n\nDeclare it as a `@collection` field inside a `@database` class, alongside the\nsources it is built from:\n\n```ts\n@database\nclass BoardDb {\n @collection static scores: Events<GameKey, ScoreEvent>; // a source\n @collection static board: View<GameKey, Leaderboard>; // the view\n}\n```\n\n## Operations\n\nA `View` has exactly two operations you use directly (plus `require`, a\nconvenience wrapper). Exact signatures:\n\n| Operation | Signature | What it does |\n| --- | --- | --- |\n| `get` | `get(key: K): V \\| null` | Read the published view, or `null` if nothing has been published yet. |\n| `require` | `require(key: K): V` | Like `get`, but traps (aborts the request) if nothing is published. |\n| `publish` | `publish(key: K, value: V): void` | Overwrite the view for `key` with a fresh result. |\n| `delete` | `delete(key: K): void` | Remove a projection key, idempotently; allowed in Action, Derive, or Job. |\n\n### `delete`: remove a projection key\n\n`AppDb.pages.delete(key)` removes the current projection and invalidates the\nlocal read cache. The delete is tenant-scoped, owner-routed, and replicated with\nits own authenticated operation. It does not retrigger the view's materializer.\nRemove or redact the corresponding source data as part of the same cleanup;\na later source write can otherwise rebuild and republish the key. A failed\ndelete aborts with the host status rather than reporting successful erasure.\n\n### `get`\n\nA plain keyed read. It is cheap and is allowed from **any** handler, including a\nread-only `@get` route.\n\n```ts\nconst board = BoardDb.board.get(new GameKey('main'));\nif (board == null) {\n // Nothing published yet (e.g. brand-new game). Serve an empty default.\n return new Leaderboard();\n}\nreturn board;\n```\n\nAlways handle the `null` case: until a `@derive` (or `@job`) has published at\nleast once, `get` returns `null`.\n\n### `publish`\n\nOverwrites the stored view with a new value. This is how the view gets its\ncontent.\n\n```ts\nBoardDb.board.publish(new GameKey('main'), freshBoard);\n```\n\n`publish` is **restricted**: you may only call it from a\n[`@derive`](../background/derive.md) or a `@job` (a background task), never from a\nrequest handler (`@get`/`@post`/...). The compiler enforces this, and so does the\nedge at runtime. The reason: a view is meant to be maintained off the request\npath from the source of truth, not written ad hoc by whichever request happens to\nrun.\n\n### How `publish` differs from a Documents write\n\nA `View`'s `publish` looks like writing a value, but it is not the same as a\n[Documents](./documents.md) `patch`/`create`. The differences matter:\n\n| | `Documents` write (`create` / `patch`) | `View` `publish` |\n| --- | --- | --- |\n| Who may call it | Any action handler (`@post`, ...) | Only a `@derive` or `@job` |\n| Meaning | The record IS the source of truth | The view is a **copy** derived from a source |\n| Version control | You may version-check (optimistic concurrency) | The host assigns the version; a later publish always wins (last writer wins) |\n| Read path | Standard keyed read | A read-optimized fast path (views are heavily read) |\n| Who owns correctness | You (each writer edits the true value) | The derive (it recomputes from the source, so the view always converges) |\n\nIn short: a Document is the truth; a View is a saved snapshot of a computation\nover the truth. If the view is ever wrong or stale, the derive just recomputes and\nrepublishes it. You never hand-edit a view from a route.\n\n## Automatic maintenance with `@derive`\n\nYou rarely call `publish` by hand. The normal pattern is a\n[`@derive`](../background/derive.md): a method on your `@database` class that\nreads the sources, builds the value, and publishes it. The runtime runs it for\nyou:\n\n- **Right after a write to a source.** When a request writes one of the\n database's source collections (an `append`, a `counter.add`, a record write),\n the database's derives run right after the response is produced. So the view\n reflects the new data on the next read.\n- **On box load.** When the server starts or reloads, the views are rebuilt from\n their sources before the first read is served.\n\nYou never call a derive yourself, and a derive's own `publish` never re-triggers\nit. See [`@derive`](../background/derive.md) for the full rules.\n\n## Worked example: a leaderboard\n\nA game appends score events; a derive folds them into a top-10 list; a route\nserves that list with a single `get`.\n\n```ts\nimport { ScoreEvent } from '../models/ScoreEvent';\nimport { Leaderboard } from '../models/Leaderboard';\nimport { GameKey } from '../models/GameKey';\nimport { NewScore } from '../models/NewScore';\n\n@database\nclass BoardDb {\n // The source of truth: every score, appended as it happens.\n @collection static scores: Events<GameKey, ScoreEvent>;\n // The materialized view: the current top entries, ready to serve.\n @collection static board: View<GameKey, Leaderboard>;\n\n // Off the request path: scan the recent scores, sort, publish the top 10.\n @derive\n rebuild(): void {\n const key = new GameKey('main');\n const recent = BoardDb.scores.latest(key, 500); // a scan, allowed in a derive\n const board = new Leaderboard();\n board.entries = topTen(recent); // your own sort/aggregate\n BoardDb.board.publish(key, board);\n }\n}\n\n@rest('leaderboard')\nclass LeaderboardRoutes {\n // GET reads the precomputed view: one keyed read, no scan.\n @get('/')\n public top(): Leaderboard {\n const board = BoardDb.board.get(new GameKey('main'));\n return board == null ? new Leaderboard() : board;\n }\n\n // POST records a score. The @derive rebuilds `board` right after.\n @post('/')\n public submit(input: NewScore): Leaderboard {\n BoardDb.scores.append(new GameKey('main'), new ScoreEvent(input.player, input.score));\n return new Leaderboard(); // ack; the GET serves the updated board from the view\n }\n}\n```\n\nThe models:\n\n```ts\n@data\nexport class ScoreEvent {\n player: string = '';\n score: u64 = 0;\n}\n\n@data\nexport class Leaderboard {\n entries: ScoreEvent[] = [];\n}\n```\n\nA \"latest N\" view is the same shape: the derive calls `latest(key, N)` and\npublishes the list. See the [Events](./events.md) page for that variant end to\nend.\n\n## Consistency\n\n- **Last writer wins.** The host assigns each `publish` a version, and a later\n publish always supersedes an earlier one. Because a derive recomputes the whole\n view from the source of truth, the view converges to a correct snapshot; you do\n not need to coordinate concurrent publishes.\n- **A view can be briefly stale.** ToilDB is worldwide, and a view is published\n at its home then copied to other regions in the background (asynchronous\n replication). A read from a far region can lag the newest publish by a moment.\n This is usually fine for the things views hold (feeds, leaderboards, summaries):\n a leaderboard that is a second behind is still a good leaderboard.\n- **`get` returns `null` until the first publish.** Always default the empty case.\n\n## Gotchas\n\n- **`publish` is derive/job only.** You cannot publish from a route. If you need\n to update a view in response to a request, write the *source* (append an event,\n add to a counter) from the action and let the `@derive` republish.\n- **Handle `null` from `get`.** A never-published view reads as `null`, not as an\n empty value. Return a sensible default.\n- **A view is a copy, not the truth.** Never store data *only* in a view. Keep\n the real data in a source family (Documents, Events, Counters) and treat the\n view as a disposable, recomputable projection.\n- **Keep views bounded.** A view value is read whole on every request, so build\n it from a bounded read (top N, latest N, a total), not the entire history.\n\n## Related\n\n- [`@derive`](../background/derive.md): the normal way a view is maintained.\n- [Events](./events.md): the append-only log a view is often folded from.\n- [Counters](./counters.md): another common source for a view (totals).\n- [Documents](./documents.md): mutable source-of-truth records (and how a write\n differs from a publish).\n- [Data types (`@data`)](../concepts/types.md): how view keys and values are stored.\n",
|
|
32
32
|
"frontend/components.md": "# Components\n\nA toiljs frontend is a normal React app, so the components you write are just React components: functions that return JSX, hold state with hooks, and compose however you like. There is no toiljs-specific base class, no decorator, and no registration step. On top of your own components, toiljs ships a small set of ready-made ones on the `Toil` global (an image, a script loader, a form, and a few more) for the jobs a plain React app makes you wire up by hand. This page covers both: how your components fit in, and a reference for every `Toil.*` component.\n\n## Writing your own components\n\nAnything you already know about writing React components applies unchanged. You write ordinary functions, use `useState` / `useEffect` / `useMemo` and any hooks you like, and return JSX:\n\n```tsx\n// client/components/Counter.tsx\nimport { useState } from 'react';\n\nexport default function Counter({ start = 0 }: { start?: number }) {\n const [n, setN] = useState(start);\n return <button onClick={() => setN(n + 1)}>Clicked {n} times</button>;\n}\n```\n\nYour reusable components live in `client/components/`, and you import them the normal way from anywhere in `client/`:\n\n```tsx\n// client/routes/index.tsx\nimport Counter from '../components/Counter';\n\nexport default function Home() {\n return (\n <main>\n <h1>Welcome</h1>\n <Counter start={10} />\n </main>\n );\n}\n```\n\nA few things are worth spelling out, because they are the parts that are handled for you rather than by you:\n\n- **Route files must `export default` a component.** A file under `client/routes/` becomes a page only if it default-exports a component (see [Routing](./routing.md)). Files under `client/components/` have no such rule; export them however you please (default or named).\n- **The `Toil.*` globals need no import.** `Toil.Link`, `Toil.useParams()`, `Toil.Image`, and everything else on `Toil` are ambient. The compiler generates a `toil-env.d.ts` that types the global, so your editor autocompletes `Toil.` and type-checks it with no `import` line. The same goes for `Server` (the typed backend client) and the `FastMap` / `DataWriter` data utilities. See the [Frontend overview](./README.md) for the whole ambient surface.\n- **The JSX runtime is configured for you.** You do not write `import React from 'react'` at the top of every file. The build sets up the automatic JSX runtime, so JSX just works. Import named hooks and types from `react` when you need them (`import { useState, type ReactNode } from 'react'`), but the bare React import for JSX is unnecessary.\n\nIn other words, a component in a toiljs app is indistinguishable from a component in any React app. Two things are genuinely special, and both are opt-in.\n\n### Special case 1: components inside an SSR route\n\nIf a route opts into server rendering with `export const ssr = true`, its component tree is rendered once at build time into a template, then filled per request on the edge (see [Rendering and SSR](./rendering.md)). For that to work, any part of the JSX that changes per request (a value from the URL, a list from a loader, a block of user HTML) has to be wrapped in one of the [SSR marker primitives](#ssr-marker-primitives) below, or isolated in a `Toil.Island`. Anything you leave unwrapped gets frozen into the template at its build-time value.\n\nYou do not have to get this perfect: if an SSR route (or a layout above it) cannot render on the server, toiljs skips SSR for that route at build, prints a warning telling you what to move, and falls back to plain client rendering. So the route still works, it just loses its server first paint until you address the warning.\n\n### Special case 2: importing images and other assets\n\nImporting an image with the `?toil` suffix gives you an object carrying the resolved URL, the intrinsic width and height, and an auto-generated blur placeholder (a `blurDataURL`), ready to hand straight to `Toil.Image`:\n\n```tsx\nimport hero from './hero.webp?toil';\n// hero is { src, width, height, blurDataURL }\n\n<Toil.Image src={hero} alt=\"Our office\" />;\n```\n\nPassing the whole object lets `Toil.Image` reserve the correct aspect ratio and paint the blur while the real image loads, with no extra props. See [Images](./images.md) for the full treatment.\n\nPlain asset imports are typed for you as well. A bare `import logo from './logo.svg'` (or `.png`, `.webp`, and so on) resolves to the hashed URL string, and the vite-imagetools query forms (`?url`, `?as=srcset`, `?as=metadata`) are typed too, so you get autocomplete and type-checking on all of them without declaring any modules yourself.\n\n## The toiljs component primitives\n\nThese are the components toiljs provides on the `Toil` global. They cover the framework-level jobs a plain React app leaves to you. All are ambient (no import), and all are fully typed.\n\n| Component | What it is | Renders |\n| --- | --- | --- |\n| `Toil.Image` | Layout-shift-free `<img>` replacement with lazy-load and blur placeholder. | An `<img>` (optionally wrapped in a sizing box). |\n| `Toil.Script` | One-time external or inline `<script>` loader with a load strategy. | Nothing (`null`). |\n| `Toil.Form` | A `<form>` that runs an action on submit and revalidates loader data. | A `<form>`. |\n| `Toil.Slot` | Renders a named parallel-route slot for the current URL. | The slot's route tree, or a fallback. |\n| `Toil.Head` | Declarative `<head>` contribution (title, meta, link). | Nothing (`null`). |\n| `Toil.Metadata` | Declarative route-style metadata applied from any component. | Nothing (`null`). |\n\nThe [SSR marker primitives](#ssr-marker-primitives) (`Toil.Hole`, `Toil.Repeat`, `Toil.RawHtml`, `Toil.attr`, `Toil.Island`) are a separate group, covered in their own section below.\n\n### `Toil.Image`\n\nA drop-in `<img>` replacement that prevents layout shift and lazy-loads by default. You give it `width` and `height` (or `fill`) so it reserves space before the image arrives, it decodes asynchronously, it lazy-loads unless you mark it `priority`, and it can fade in from a blur placeholder. It accepts either a string URL or a `?toil` import object (which auto-fills the size and blur):\n\n```tsx\nimport hero from './hero.webp?toil';\n\nexport default function Home() {\n return (\n <>\n <Toil.Image src={hero} alt=\"Our office\" priority placeholder=\"blur\" />\n <Toil.Image src=\"/team.jpg\" alt=\"The team\" width={800} height={600} />\n </>\n );\n}\n```\n\nKey props are `src`, `alt` (required, pass `alt=\"\"` for a decorative image), `width` / `height` or `fill`, `priority` (for an above-the-fold hero), and `placeholder` (`'empty'` or `'blur'`). It also accepts any standard `<img>` attribute. This is kept brief on purpose: [Images](./images.md) covers sizing, `fill`, blur placeholders, and how it stops layout shift in full.\n\n### `Toil.Script`\n\nLoads an external or inline `<script>` exactly once for the whole life of the app (even across client-side navigations), and lets you choose when it runs with a `strategy` prop. Use it instead of a hand-written `<script>` tag for third-party snippets like analytics or chat widgets, which a plain `<script>` in a single-page app runs unreliably or twice:\n\n```tsx\n// client/layout.tsx\nexport default function Layout({ children }: { children?: React.ReactNode }) {\n return (\n <div className=\"app\">\n <Toil.Script src=\"https://cdn.example-analytics.com/analytics.js\" />\n {children}\n </div>\n );\n}\n```\n\nThe `strategy` is `afterInteractive` (default), `lazyOnload`, or `beforeInteractive`, and there are `onLoad` / `onReady` / `onError` callbacks. `Toil.Script` renders nothing. See [Scripts](./scripts.md) for the strategies, inline scripts, dedup rules, and the full prop table.\n\n### `Toil.Form`\n\nA `<form>` that submits without reloading the page. On submit it runs your `action` (which receives the form's `FormData`), tracks pending and error state, and on success revalidates the current route's loader data so the page reflects the write. It is the convenient front end of the loader/action data loop:\n\n```tsx\n// A guestbook that refreshes its list after a successful sign.\nexport default function Guestbook() {\n const entries = Toil.useLoaderData<typeof loader>();\n\n const sign = async (data: FormData) => {\n const author = String(data.get('author'));\n const message = String(data.get('message'));\n await Server.REST.guestbook.sign({ body: new NewMessage(author, message) });\n };\n\n return (\n <Toil.Form action={sign} resetOnSuccess>\n {({ pending, error }) => (\n <>\n <input name=\"author\" placeholder=\"Your name\" />\n <textarea name=\"message\" />\n <button disabled={pending}>{pending ? 'Signing...' : 'Sign'}</button>\n {error ? <p className=\"err\">Could not sign, try again.</p> : null}\n </>\n )}\n </Toil.Form>\n );\n}\n```\n\nPass a render function as the child to read the live submit state: it receives `{ pending, error, data }`, which is how you disable the button while pending or show an error. The props:\n\n| Prop | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `action` | `(data: FormData) => void \\| Promise<void>` | (required) | Runs on submit, receiving the form's `FormData`. May be async. |\n| `revalidate` | `RevalidateTarget` | `true` | Which loader data to refetch after a successful submit. `true` is the current route. |\n| `onSuccess` | `() => void` | (none) | Called after a successful submit. |\n| `onError` | `(error: unknown) => void` | (none) | Called when the action throws. |\n| `resetOnSuccess` | `boolean` | `false` | Reset the form fields after a successful submit. |\n| `className` | `string` | (none) | Class on the `<form>` element. |\n| `children` | `ReactNode` or `(state) => ReactNode` | (none) | Form contents. A function child receives `{ pending, error, data }`. |\n\nFor writes that are not form submits (a delete button, a like toggle), reach for the underlying `Toil.useAction` hook instead. Both are covered in [Fetching data](./data-fetching.md).\n\n### `Toil.Slot`\n\nRenders the named parallel-route slot for the current URL. A folder starting with `@` (like `@modal`) under `client/routes/` is a whole second route tree that matches the URL independently of the main page; placing `<Toil.Slot name=\"modal\" />` is where that tree renders. If no slot route matches the current URL, it renders the `fallback` (nothing by default):\n\n```tsx\n// client/routes/gallery/layout.tsx\nexport default function GalleryLayout({ children }: { children?: React.ReactNode }) {\n return (\n <div>\n {children}\n <Toil.Slot name=\"modal\" fallback={null} />\n </div>\n );\n}\n```\n\n| Prop | Type | Default | What it does |\n| --- | --- | --- | --- |\n| `name` | `string` | (required) | The slot name: the `@name` directory under `routes/`, without the `@`. |\n| `fallback` | `ReactNode` | `null` | Rendered when no slot route matches the current URL. |\n\nParallel slots and the intercepting routes that fill them (the \"click a photo, open it in a modal\" pattern) are explained in [Routing](./routing.md).\n\n### `Toil.Head`\n\nA declarative way for any component (a page, a layout, a deep child) to contribute to the document `<head>`: a title, `<meta>` tags, and `<link>` tags. It renders nothing; it just applies its head entries for the lifetime of the component and reverts them on unmount. Entries compose across the tree, with later or deeper ones winning per key:\n\n```tsx\nexport default function ArticlePage() {\n const post = Toil.useLoaderData<typeof loader>();\n return (\n <>\n <Toil.Head\n title={post.title}\n meta={[\n { name: 'description', content: post.summary },\n { property: 'og:title', content: post.title },\n ]}\n link={[{ rel: 'canonical', href: `https://example.com/blog/${post.id}` }]}\n />\n <article>{/* ... */}</article>\n </>\n );\n}\n```\n\n`<Toil.Head>` takes `title`, `meta` (an array of `{ name | property, content }` tags), and `link` (an array of `{ rel, href }` tags). The hook form `Toil.useHead(spec)` and the shorthand `Toil.useTitle(title)` do the same job imperatively. See [Metadata and SEO](./metadata.md).\n\n### `Toil.Metadata`\n\nThe declarative, convenience-shaped cousin of `Toil.Head`. Instead of raw meta and link arrays, you pass a route-style metadata object (the same shape a route file's `export const metadata` uses), and toiljs expands the convenience fields (`description`, `keywords`, `canonical`, `openGraph`, and so on) into the right tags. It renders nothing, and applies for the component's lifetime. Use it to set metadata from a component that is not itself a route file (a rendered article, a widget):\n\n```tsx\n<Toil.Metadata\n title=\"Our pricing\"\n description=\"Simple, flat pricing for teams of any size.\"\n openGraph={{ title: 'Pricing', image: 'https://example.com/og/pricing.png' }}\n/>\n```\n\nBecause a route's own `metadata` export is applied last (highest priority), `Toil.Metadata` fills in for routes that declare none and yields to a route that sets the same key. The full field list lives in [Metadata and SEO](./metadata.md).\n\n## SSR marker primitives\n\nThese five primitives matter only for routes with `export const ssr = true`. They are how you tell the build which parts of a server-rendered page are dynamic (filled per request) versus static (baked into the template once). On a client-only route they do nothing special, so you never need them unless you opt a route into SSR.\n\n**The mental model.** In the browser these markers are transparent: `<Toil.Hole>` renders its children, `<Toil.Repeat>` maps its rows, `<Toil.RawHtml>` renders a raw-HTML wrapper, and `Toil.attr(id, value)` returns the value unchanged. Your client-side app runs exactly as written. But under the build-time SSR extractor, each marker instead emits a sentinel token that marks an insertion point. The extractor strips those tokens and records their positions, producing a static template with numbered holes. Per request, your compiled backend fills only the hole values, and the edge splices them into the template. Because the static scaffold around each hole is React's own rendering output and the hole values are escaped exactly as React escapes them, the browser hydrates the result byte-for-byte, with no mismatch.\n\n| Marker | Use it for | Shape |\n| --- | --- | --- |\n| `Toil.Hole` | A single dynamic **text** value. | `<Toil.Hole id=\"...\">{value}</Toil.Hole>` |\n| `Toil.Repeat` | A **list**: one row template repeated over an `each` array. | `<Toil.Repeat id=\"...\" each={rows}>{(item, i) => ...}</Toil.Repeat>` |\n| `Toil.RawHtml` | A block of **trusted, pre-rendered HTML**. | `<Toil.RawHtml id=\"...\" html={s} as=\"section\" />` |\n| `Toil.attr` | A dynamic value in **attribute position** (an `href`, a `class`). | `href={Toil.attr('id', value)}` (a function call) |\n| `Toil.Island` | Content that must render **only in the browser** (escape hatch). | `<Toil.Island>{children}</Toil.Island>` |\n\nA few rules that keep the template valid:\n\n- **Every marker needs a stable `id`**, a short name unique within the page. The build maps each id to a numbered slot, so keep the ids constant across builds.\n- **`Toil.Repeat` needs at least one row at build time.** It captures that first row as the sub-template for every row, so the build render must see a sample with one or more items. An empty `each` gives it nothing to capture.\n- **`Toil.RawHtml` renders inside a wrapper element** (a `<div>` by default, `as=\"section\"` to change the tag), and you own sanitising that HTML, exactly like React's `dangerouslySetInnerHTML`.\n- **`Toil.attr` is a function, not an element.** An attribute is not a child node, so it cannot be a JSX element. You call `Toil.attr(id, value)` right where the attribute value goes, and it composes with literal text around it (`` className={`btn ${Toil.attr('kind', d.kind)}`} ``).\n- **`Toil.Island` renders nothing on the server and on the first (hydration) render**, then reveals its children after mount. So an island gets no server first paint and no SEO, by design. It is the place for anything that genuinely cannot run on the server (reads `window`, calls `Date.now()`, depends on the live URL).\n\nHere is a compact SSR route wiring several of them together. The title, tag list, and author link all come from the route's `loader`:\n\n```tsx\n// client/routes/blog/[id].tsx\nexport const ssr = true;\n\nexport const loader = async ({ params }: Toil.LoaderArgs) => {\n // Illustrative shape: { title, tags, authorUrl }.\n return Server.REST.blog.get({ params: { id: params.id } });\n};\n\nexport default function BlogPost() {\n const post = Toil.useLoaderData<typeof loader>();\n return (\n <article>\n <h1>\n <Toil.Hole id=\"title\">{post.title}</Toil.Hole>\n </h1>\n\n {/* A dynamic attribute: call attr() in attribute position. */}\n <a href={Toil.attr('authorUrl', post.authorUrl)}>By the author</a>\n\n {/* A list: one row template, stamped once per item on the server. */}\n <ul>\n <Toil.Repeat id=\"tags\" each={post.tags}>\n {(tag) => <li key={tag}>{tag}</li>}\n </Toil.Repeat>\n </ul>\n </article>\n );\n}\n```\n\nFor the whole SSR story (the template extraction flow, keeping a route SSR-safe, and the current SSR limitations), see [Rendering and SSR](./rendering.md).\n\n## Related\n\n- [Rendering and SSR](./rendering.md): how SSR routes render, hydrate, and use the marker primitives.\n- [Images](./images.md): the full `Toil.Image` reference, blur placeholders, and layout shift.\n- [Scripts](./scripts.md): `Toil.Script` strategies, inline scripts, and dedup.\n- [Fetching data](./data-fetching.md): `Toil.Form`, `useAction`, loaders, and the typed backend clients.\n- [Metadata and SEO](./metadata.md): `Toil.Head`, `Toil.Metadata`, and per-route metadata.\n- [Routing](./routing.md): pages, layouts, and the parallel slots `Toil.Slot` renders.\n",
|