toiljs 0.0.85 → 0.0.87

package/docs/background/README.md

@@ -0,0 +1,114 @@
+# Background work
+
+Background work is code that runs **without a user waiting for it**: on a timer, once globally, or automatically after your data changes. toiljs gives you two tools for it, `@daemon` and `@derive`, and this page helps you pick the right one.
+
+## Why background work
+
+Most of your server code runs **because a user asked**: a browser hits a [route](../backend/rest.md) or calls an [RPC](../backend/rpc.md), your handler runs, and a response goes back. But some work does not belong on that path:
+
+- It should happen **on a schedule** (every hour, every night at 2am), not when a request happens to arrive.
+- It would be **too slow** to do inside a request, so you want it precomputed and ready.
+- It must happen **exactly once across the whole world**, not once per user and not once per server.
+
+That is background work. In toiljs there are two kinds, and they solve two different problems.
+
+## The two tools
+
+```mermaid
+flowchart TD
+    Q{What do you need?}
+    Q -->|Run on a timer, or once globally| D["@daemon + @scheduled<br/>a single background worker"]
+    Q -->|Keep a fast read-view in sync with your data| V["@derive<br/>maintains a materialized View"]
+    D --> DD["e.g. nightly cleanup, hourly poll,<br/>periodic rollup"]
+    V --> VV["e.g. a leaderboard, a 'latest 10' feed,<br/>a home-page summary"]
+```
+
+### `@daemon`: a scheduled, global worker
+
+A [daemon](./daemons.md) is one long-lived background worker for your whole app. It runs on a **schedule** you set (an interval like every 5 minutes, or a cron time like "9:15 on weekdays"), and there is exactly **one** of it worldwide at any moment (a "singleton"). Use it for work that is driven by **time** or that must run **once globally**:
+
+- clean up stale rows every night;
+- poll an external API every few minutes;
+- send a daily digest email;
+- roll up yesterday's numbers into a summary.
+
+### `@derive`: keep a read-view up to date
+
+A [derive](./derive.md) is not on a timer. It runs **automatically whenever the data it depends on changes**, and its job is to keep a precomputed **View** (a read-optimized copy of your data) fresh. Use it when a page needs data that is **expensive to compute on every read**, like a leaderboard or a "latest 10 comments" list, so the read itself stays a single cheap lookup:
+
+- fold an [events](../database/events.md) log into a "latest N" list;
+- total up [counters](../database/counters.md) into a scoreboard;
+- assemble a home-page summary from several sources.
+
+## Which one do I reach for?
+
+| Question                                              | Use          |
+| ----------------------------------------------------- | ------------ |
+| "Run this every hour / at midnight."                  | `@daemon`    |
+| "Do this once for the whole app, not per server."     | `@daemon`    |
+| "Poll or call an outside service on a schedule."      | `@daemon`    |
+| "This page's data is too slow to compute per request."| `@derive`    |
+| "Keep a leaderboard / feed in sync as data changes."  | `@derive`    |
+
+A simple rule of thumb: if the trigger is **the clock**, use a `@daemon`. If the trigger is **a change to your data** and the goal is a fast read, use a `@derive`.
+
+They also combine well. A `@daemon` might do a heavy nightly aggregation and write a summary row, while a `@derive` keeps a small live view fresh on every write. They are different tiers of the edge and are covered on their own pages.
+
+## `@job`: the widest database surface for background work
+
+`@daemon` and `@derive` decide **where and when** background code runs. `@job` answers a different question: **what a function is allowed to do to the database.** It is one of the ToilDB **function kinds** (the same family as `@query`, `@action`, and `@derive`), and it grants the **widest** data surface of them all.
+
+A **function kind** is a label the compiler puts on a backend function to gate which database operations it may issue. This is a safety rail: a read-only endpoint physically cannot write, and an expensive scan cannot run on the hot request path. The kinds line up from narrowest to widest:
+
+| Kind       | Typical trigger           | Point reads | **Scan** (`latest`, membership `list`) | Writes            | `publish` a View |
+| ---------- | ------------------------- | ----------- | -------------------------------------- | ----------------- | ---------------- |
+| `@query`   | a `@get` / plain `@remote`| yes         | no                                     | no                | no               |
+| `@action`  | a `@post` / `@action`     | yes         | no                                     | yes (bounded)     | no               |
+| `@derive`  | your data changed         | yes         | yes                                    | append / counter add only | yes      |
+| `@job`     | you drive it (background) | yes         | yes                                    | yes (all)         | yes              |
+
+A **scan** is a read that can fan out across many rows (like "the newest 50 events" via `events.latest`, or "every member of this set" via `membership.list`). Scans are **barred from request handlers** because a request must stay fast and bounded. `@derive` and `@job` run **off the request path**, so they are the only kinds allowed to scan.
+
+### When to reach for `@job`
+
+Use `@job` when a piece of background work needs **more** database power than the default:
+
+- it must **scan** (fold `events.latest`, walk `membership.list`), and/or
+- it must **publish a View** *while also* doing arbitrary writes (`create`, `patch`, `delete`). A `@derive` can publish a View but cannot `patch` or `delete` a record; a `@job` can do everything.
+
+The compiler accepts `@job` on a method, including a `@daemon`'s `@scheduled` method. Tag a scheduled task `@job` when it needs that full surface (say, a nightly repair that scans a log, fixes rows, and republishes a summary View). A plain, **untagged** `@scheduled` method runs with the **`@action`** surface instead: point reads plus bounded writes, which is all most rollups and cleanups need.
+
+```ts
+@daemon
+class Jobs {
+    // A plain scheduled method: point reads + bounded writes (the @action surface).
+    @scheduled('1h')
+    rollup(): void { /* get a counter, patch a summary row */ }
+
+    // A scheduled method that also needs SCANS and to PUBLISH a View: tag it @job.
+    @scheduled('1d')
+    @job
+    nightlyRepair(): void {
+        // @job unlocks scan-class reads (events.latest / membership.list)
+        // and publishing a View, on top of ordinary reads and writes.
+    }
+}
+```
+
+### `@job` versus `@derive`
+
+They overlap (both run off the request path, both may scan, both may publish a View), but they are triggered differently and sized differently:
+
+- **`@derive`** is **change-triggered**: it re-runs automatically whenever its source data changes, and its narrow job is to keep one **View** in sync. It cannot `create` / `patch` / `delete` records (only append, counter-add, and publish). Reach for it when a read is too slow and you want it kept fresh on every write.
+- **`@job`** is **you-drive-it** background work (typically a `@scheduled` daemon method) with the **full** write surface. Reach for it when the work is clock-driven or one-off and needs to both scan and mutate freely.
+
+> **Rule of thumb.** Clock-driven and needs the full database surface: a `@job` (usually inside a `@daemon`). Change-driven and only maintains a View: a `@derive`. Clock-driven with modest reads and writes: a plain `@scheduled` method (no `@job` needed).
+
+For the complete permission grid see the [function-kind matrix](../database/setup.md#how-access-is-gated-query-action-and-friends), and for the decorator itself see [every decorator](../concepts/decorators.md#database-function-kinds-data-access-policy).
+
+## Related
+
+- [Daemons and scheduled jobs](./daemons.md): `@daemon`, `@scheduled`, interval vs cron, and how a single global worker fails over safely.
+- [Derived views (`@derive`)](./derive.md): keeping a materialized View in sync with its source data.
+- [Compute tiers (L1 to L4)](../concepts/tiers.md): where daemons and request handlers each run on the edge.
+- [Views](../database/views.md) and [Events](../database/events.md): the database families a `@derive` reads from and writes to.

package/docs/background/daemons.md

@@ -0,0 +1,230 @@
+# Daemons (`@daemon` / `@scheduled`)
+
+A `@daemon` is a single, long-lived background worker for your whole app. You mark a class `@daemon`, add `@scheduled` methods that fire on a timer, and the Dacely edge keeps exactly **one** copy of it running worldwide, restarting it elsewhere if the machine it is on fails.
+
+## What a daemon is
+
+The word "daemon" (say "DEE-mon") is an old computing term for a program that runs quietly in the background, not tied to any single user. That is exactly what this is.
+
+Compare the three ways your server code can run:
+
+| Kind                            | How many run                         | Lives for                     |
+| ------------------------------- | ------------------------------------ | ----------------------------- |
+| [Request handler](../backend/rest.md) (`@rest`) | a fresh one per request              | one request                   |
+| [Stream box](../realtime/streams.md) (`@stream`) | one per open connection              | one connection                |
+| **Daemon** (`@daemon`)          | **exactly one for the whole app**    | as long as it holds the lease |
+
+Because it is a single, resident instance, its fields persist across scheduled runs (a request handler forgets everything after each request; a daemon does not). It is the right home for work that must happen **once globally on a cadence**, not once per user and not once per server.
+
+```ts
+@daemon
+class Jobs {
+    @scheduled('1h')
+    hourly(): void {
+        // Runs once an hour, on the one elected worker. Put recurring background
+        // work here: rollups, cleanup, polling an upstream, and so on.
+    }
+}
+```
+
+## `@scheduled`: run on a cadence
+
+A `@scheduled` method fires on a schedule. The single string argument is the cadence, and it comes in two flavours.
+
+### Interval schedules
+
+An **interval** fires every fixed span of time. Write a number followed by a unit: `s` (seconds), `m` (minutes), `h` (hours), or `d` (days). The number must be at least 1, and the span may not exceed 7 days.
+
+```ts
+@scheduled('30s') everyHalfMinute(): void { /* ... */ }
+@scheduled('5m')  everyFiveMinutes(): void { /* ... */ }
+@scheduled('1h')  hourly(): void { /* ... */ }
+@scheduled('1d')  daily(): void { /* ... */ }
+```
+
+### Cron schedules
+
+A **cron** expression fires at wall-clock times ("every weekday at 9:15", "midnight on the first of the month"). Use it when you care about the actual time of day, not just a repeating gap. A cron spec is five fields separated by spaces, in this order:
+
+```
+minute  hour  day-of-month  month  day-of-week
+```
+
+toiljs recognises a cron spec by the spaces in it (an interval has none).
+
+```ts
+@scheduled('15 9 * * 1-5')   // 09:15, Monday to Friday
+weekdayMorning(): void { /* ... */ }
+
+@scheduled('0 0 1 * *')      // 00:00 on the 1st of every month
+monthlyReset(): void { /* ... */ }
+```
+
+Cron times are evaluated in **UTC** and are **minute-granular** (the smallest cron step is one minute). A `*` means "every value" for that field.
+
+### Rules
+
+- A `@scheduled` method takes **no arguments and returns `void`**.
+- A daemon class may have **several** `@scheduled` methods, each on its own cadence.
+- Because only the one elected worker fires them, a task runs **once per tick for the whole app**, never once per server.
+
+### `onStart`: run once at boot
+
+A daemon may also declare a plain `onStart(): void` method (not decorated). It runs **once**, when the daemon box first starts on the elected worker. Use it to set up state or kick off a long-running loop.
+
+```ts
+@daemon
+class Jobs {
+    onStart(): void {
+        // one-time setup when this daemon becomes active
+    }
+
+    @scheduled('1h')
+    hourly(): void { /* ... */ }
+}
+```
+
+## No backfill: missed runs are skipped, not replayed
+
+This is the single most important thing to understand about scheduling.
+
+If the daemon is down when a tick was due (say the leader failed and a standby is still taking over), or if the clock jumps forward, toiljs does **not** go back and run all the ticks you missed. It simply fires the **next** due run and moves on. This is called a **no-backfill** policy.
+
+Two practical consequences:
+
+1. **Design tasks to be safe to skip.** "Recompute the summary" is fine to miss (the next run fixes it). "Charge every user once" is not, unless you make it idempotent.
+2. **Make tasks idempotent where a missed run matters.** Idempotent means running it twice (or catching up later) has the same effect as running it once. For example, "set yesterday's total to X" is idempotent; "add 1 to a counter" is not.
+
+## One global worker, with safe failover
+
+There is exactly **one** daemon running for your app at any moment. A second machine keeps a **warm standby** ready but idle. If the active worker's hold on the job expires (it crashed, lost the network, or was shut down), the standby takes over and fires the following runs.
+
+The mechanism is a **lease**. Think of the lease as a "who is in charge" token that only one worker can hold at a time, and that has to be renewed to keep. Only the worker holding the lease (the **leader**) runs the schedule.
+
+```mermaid
+sequenceDiagram
+    participant A as Worker A (leader)
+    participant L as Lease
+    participant B as Worker B (standby)
+    A->>L: hold the lease, renew it
+    Note over A: A fires @scheduled ticks
+    Note over B: B stays idle, watching
+    A--xL: A crashes, stops renewing
+    Note over L: lease expires
+    B->>L: acquire the lease
+    Note over B: B is now leader, fires the next ticks
+```
+
+The important guarantee: **two workers never run the same tick at the same time.** This is called **at-most-once** scheduling. The trade-off is the no-backfill behaviour above: to be sure a tick is never run twice, the edge would rather skip the in-flight tick when a leader is lost than risk running it on two machines. You never start, stop, or place the daemon yourself; the edge elects the leader and drives it.
+
+## Leadership fencing: side effects only run on the leader
+
+A subtle risk with a warm standby is a "split brain": for a brief moment, two workers might both think they are the leader. To make that harmless, toiljs **fences** every side effect behind a leadership check. A **side effect** is anything that changes the outside world:
+
+- **Database writes** (creating, patching, deleting rows, adding to counters, appending events, publishing views).
+- **Outbound HTTP calls** (`http_call`, described below).
+
+These run **only** on the confirmed leader. If code that is not the leader tries one, the edge refuses it (a "not leader" error) rather than let it happen twice. Plain **reads** and computation are not fenced (they are safe to do anywhere). So even if two workers briefly overlap, only one of them can actually write or call out. You do not write the fencing yourself; it is automatic. The upshot: put your writes and outbound calls in `@scheduled` methods freely, and trust that they happen once.
+
+## `daemon.*` host calls
+
+A daemon has a small set of host abilities beyond ordinary computation:
+
+- **Database access.** A daemon reads and writes [ToilDB](../database/README.md) with the same collection handles you use in a route or a derive (`.get`, `.add`, `.append`, `.publish`, and so on). Writes are leader-fenced as described above.
+- **Outbound HTTP (`http_call`).** A daemon can call an external service (to poll an API, post to a webhook). This is the one place your server code reaches out to the internet, so it is deliberately restricted:
+  - It is **leader-only** (fenced, like any side effect).
+  - It is **SSRF-bounded**. SSRF (server-side request forgery) is an attack where code is tricked into calling internal addresses it should not reach. The edge resolves the target host and blocks private or internal addresses, so a daemon cannot use `http_call` to poke around inside the network.
+  - It is **metered**: making many calls or pulling huge responses costs budget, which caps abuse.
+- **Leadership info** (`is_leader`, `current_epoch`). A daemon can check whether it is currently the leader, which is useful for guarding a long `onStart` loop.
+
+> **Note:** In `toiljs dev` (the single-process local emulator), the daemon is always the leader (there is nothing to fail over to), and `http_call` is stubbed to return a "call failed" result rather than make real network requests. Everything else, including the schedule and your database writes, runs exactly as it does on the edge.
+
+### `http_call` is not callable from your code yet
+
+An honest caveat, because it is easy to assume otherwise: **there is no guest-callable `http_call` API in toiljs today.** The capability itself is fully built on the *host* side (the edge implements the leader-only, SSRF-bounded, metered outbound call as a low-level host import, `daemon.http_call(reqPtr, reqLen, outPtr, outCap) -> i64`, and the dev emulator reserves the same name), but toiljs does **not** ship a friendly TypeScript wrapper you can call from a `@scheduled` method. There is no `daemon.httpCall(...)` (or similarly named) function in the standard library, so your daemon code cannot make an outbound HTTP request at the moment.
+
+Treat outbound HTTP from a daemon as **planned, not yet available**. When the guest binding lands it will behave exactly as described above (leader-fenced, SSRF-bounded, and metered, and its usage will show up in the `Analytics` counters `daemonHttpCallAttempts` / `daemonHttpCallFailures`). Until then, if a daemon needs data from an outside service, have that service write into [ToilDB](../database/README.md) some other way, and let the daemon read it from there.
+
+## The `main.daemon.ts` file (a separate tier)
+
+Like streams, daemons live in their **own entry file**, `server/main.daemon.ts`, and compile into their **own artifact**, `build/server/release-cold.wasm`. Importing your `@daemon` module there pulls it into that artifact.
+
+```ts
+// server/main.daemon.ts
+import { revertOnError } from 'toiljs/server/runtime/abort/abort';
+
+import './daemon/Jobs'; // add each @daemon module here
+
+// NOTE: unlike main.ts / main.stream.ts, the daemon entry does NOT re-export the
+// request runtime. A daemon artifact exposes its schedule hooks, not the request handler.
+export function abort(message: string, fileName: string, line: u32, column: u32): void {
+    revertOnError(message, fileName, line, column);
+}
+```
+
+`toiljs build` produces `release-cold.wasm` automatically when your project declares a `@daemon` surface:
+
+```sh
+$ ls build/server/*.wasm
+build/server/release.wasm          # L1 request   (@rest / @service)
+build/server/release-stream.wasm   # L2/L3 stream  (@stream)
+build/server/release-cold.wasm     # L4 daemon     (@daemon)
+```
+
+There is **at most one** `@daemon` class per project, and it is a compile error to put a `@daemon` in the request build. See [Compute tiers](../concepts/tiers.md) for how one source tree becomes three artifacts.
+
+## Worked example: a periodic rollup
+
+A common daemon job: every hour, read a running total and store a small summary so pages can show it with one cheap read. This reads a [counter](../database/counters.md) and writes a summary [document](../database/documents.md); the write only takes effect on the leader.
+
+```ts
+// server/models/StatKey.ts and Summary.ts (@data types)
+@data
+class StatKey {
+    name: string = 'signups';
+    constructor(name: string = 'signups') { this.name = name; }
+}
+
+@data
+class Summary {
+    total: u64 = 0;
+    updatedAt: u64 = 0;
+}
+
+// server/data/StatsDb.ts
+@database
+class StatsDb {
+    @collection static signups: Counter<StatKey>;
+    @collection static summary: Documents<StatKey, Summary>;
+}
+
+// server/daemon/Jobs.ts
+@daemon
+class Jobs {
+    @scheduled('1h')
+    rollup(): void {
+        const key = new StatKey('signups');
+        const total = StatsDb.signups.get(key);   // a read (allowed anywhere)
+
+        const s = new Summary();
+        s.total = total;
+        s.updatedAt = <u64>(Date.now() / 1000);
+        StatsDb.summary.patch(key, s);            // a write: runs only on the leader
+    }
+}
+```
+
+Now any request handler can serve the summary with a single keyed read, and it is never more than an hour stale. Because the write is fenced to the leader and the rollup is idempotent (it *sets* the total rather than adding to it), a missed or failed-over tick is harmless: the next hourly run simply refreshes it.
+
+## When not to use a daemon
+
+- **When a user is waiting for the result.** Do that work in a [route](../backend/rest.md) or [RPC](../backend/rpc.md), on the request path.
+- **When you need a fast read-view kept in sync with data changes.** That is a [`@derive`](./derive.md), which runs on every write rather than on a timer.
+- **When you need exactly-once, never-skipped execution.** Scheduling is at-most-once with no backfill, so make tasks idempotent or safe to skip. A daemon is not a guaranteed job queue.
+
+## Related
+
+- [Background overview](./README.md): daemons versus `@derive`, and which to reach for.
+- [Derived views (`@derive`)](./derive.md): keep a read-view in sync on every write.
+- [Compute tiers (L1 to L4)](../concepts/tiers.md): the daemon runs on the L4 global tier.
+- [Counters](../database/counters.md), [Documents](../database/documents.md), [Events](../database/events.md): the data a daemon reads and writes.

package/docs/background/derive.md

@@ -0,0 +1,179 @@
+# Derived views (`@derive`)
+
+A `@derive` keeps a fast, precomputed **View** of your data in sync automatically. You write a method that reads your source data and publishes a summary; toiljs re-runs it whenever the source data changes, so your pages can read the summary with one cheap lookup instead of doing the work on every request.
+
+## The problem it solves
+
+Some reads are expensive. "Show the 10 newest comments" means scanning an [events](../database/events.md) log. "Show the leaderboard" means totalling up scores. A **scan** (walking many rows to build a result) can fan out across an unbounded amount of data, and that is too slow and too unpredictable to do while a user waits.
+
+So toiljs **bars scans on the request path**. A [route](../backend/rest.md) handler runs under a restricted mode:
+
+- A `@get` runs as a **query** (reads only, no scans).
+- A `@post` / `@put` / `@patch` / `@del` runs as an **action** (keyed reads and writes, still no scans).
+
+If you cannot scan in a route, how do you show "the latest 10"? You precompute it. A `@derive` does the scan **off** the request path, folds the result into a [View](../database/views.md) (a read-optimized snapshot stored by key), and your route reads that View with a single keyed lookup, which is not a scan and so is allowed.
+
+```mermaid
+flowchart LR
+    W["POST /guestbook<br/>(action: append + count)"] --> S[(Events + Counter<br/>source data)]
+    S -->|write triggers| D["@derive recompute()<br/>scans, builds, publishes"]
+    D --> V[(View<br/>precomputed snapshot)]
+    G["GET /guestbook<br/>(query: one keyed read)"] --> V
+```
+
+## A worked example: a guestbook
+
+Here is the whole pattern in one database class: an [events](../database/events.md) log of signatures, a [counter](../database/counters.md) of how many there are, and a [View](../database/views.md) that holds the ready-to-serve page.
+
+```ts
+@data
+class GuestKey {
+    room: string = 'main';
+    constructor(room: string = 'main') { this.room = room; }
+}
+
+@database
+class GuestbookDb {
+    @collection static entries: Events<GuestKey, GuestEntry>;  // a source: the log
+    @collection static totals: Counter<GuestKey>;              // a source: the count
+    @collection static book: View<GuestKey, GuestbookView>;    // the view we publish
+
+    // Recompute the view from the sources. It MAY scan and publish; a route may not.
+    @derive
+    recompute(): void {
+        const key = new GuestKey('main');
+        const view = new GuestbookView();
+        view.total = GuestbookDb.totals.get(key);          // a keyed read
+        view.entries = GuestbookDb.entries.latest(key, 10); // a scan: allowed here
+        GuestbookDb.book.publish(key, view);               // publish the snapshot
+    }
+}
+```
+
+The route then writes to the sources and reads the view:
+
+```ts
+@rest('guestbook')
+class Guestbook {
+    @get('/')
+    list(): GuestbookView {
+        const key = new GuestKey('main');
+        const view = GuestbookDb.book.get(key);   // one keyed read, not a scan
+        return view == null ? new GuestbookView() : view; // empty until first publish
+    }
+
+    @post('/')
+    sign(input: NewMessage): GuestbookView {
+        const key = new GuestKey('main');
+        GuestbookDb.entries.append(key, new GuestEntry(input.author, input.message, 0));
+        GuestbookDb.totals.add(key, 1);
+        // The @derive republishes `book` right after this action returns, so GET
+        // serves the new entry. The action just acks with the new total (a counter
+        // read is allowed here; scanning the entries list is not).
+        const view = new GuestbookView();
+        view.total = GuestbookDb.totals.get(key);
+        return view;
+    }
+}
+```
+
+Sign the guestbook twice and the total climbs across requests, because the data lives in the database (and its view), not in module memory.
+
+## What a `@derive` may do
+
+A `@derive` runs under a special **derive** mode that is more powerful than a route:
+
+| Ability                                      | Query (`@get`) | Action (`@post` ...) | Derive       |
+| -------------------------------------------- | :------------: | :------------------: | :----------: |
+| Keyed reads (`.get`, counter total)          | yes            | yes                  | yes          |
+| Writes (`create`, `patch`, `add`, `append`)  | no             | yes                  | yes          |
+| **Scans** (`events.latest`, membership list) | **no**         | **no**               | **yes**      |
+| `view.publish` / `view.append`               | no             | no                   | yes          |
+
+So a derive is exactly the place to do the reads and scans a route cannot, and to publish the result.
+
+## Declaring a derive
+
+A `@derive` is a method on your [`@database`](../database/setup.md) class, next to the collections it reads and the View it writes.
+
+```ts
+@database
+class MyDb {
+    @collection static events: Events<Key, Fact>;  // a source
+    @collection static home: View<Key, HomePage>;  // the materialized view
+
+    @derive
+    rebuild(): void {
+        // read sources, build the value, publish it
+    }
+}
+```
+
+Rules:
+
+- A `@derive` method takes **no arguments and returns `void`**.
+- A database may declare **multiple** `@derive` methods; each runs independently.
+- The View value and its key are ordinary [`@data`](../backend/data.md) types, so they round-trip through the codec like any other stored value.
+
+## When a derive runs
+
+You never call a derive yourself. The runtime runs it for you at two moments:
+
+1. **After a write to a source.** When a request writes one of the database's source collections (an `events.append`, a `counter.add`, a document `create` or `patch`), that database's derives run **right after the response is produced**, so the view reflects the new data on the next read. Many writes to one database in a single request are **coalesced** into one recompute (it does not run once per write).
+
+2. **On box load.** When a server box starts, hot-reloads, or notices the underlying data changed out of band, the views are rebuilt from their sources **before the first read is served**. This is also where a value type's [`@migrate`](../backend/data.md) runs against old stored events, as the derive re-reads and republishes them.
+
+A derive's own `view.publish` never re-triggers it, so there is no infinite loop.
+
+The same code runs under `toiljs dev` (the in-process emulator) and on the production edge, with no flags or wiring to change.
+
+## Folding a growing log incrementally
+
+The guestbook above uses `events.latest` and **recomputes** the view from scratch on every change. That is simple, correct, and the right default for a **bounded** read: the latest N, a counter total, a small set.
+
+Some views instead fold an **unbounded** log: a running total over every event ever, an activity rollup, an audit summary. Rescanning the whole log on every change gets slower as it grows. For those, read the source with [`events.since`](../database/events.md) instead of `latest`. `since` hands you only the events you have **not folded yet**, so the derive folds forward incrementally.
+
+```ts
+@derive
+rollup(): void {
+    const key = new StatsKey('all');
+    const view = StatsDb.summary.get(key) ?? new Summary();   // the running view so far
+    let batch = StatsDb.events.since(key, 500);
+    while (batch.length > 0) {                                 // drain the new events in bounded batches
+        for (let i = 0; i < batch.length; i++) view.apply(batch[i]);
+        batch = StatsDb.events.since(key, 500);
+    }
+    StatsDb.summary.publish(key, view);
+}
+```
+
+The host owns the cursor, you never manage it: it seeds `since` from a durable checkpoint, advances it as it hands you events, and saves it **only after** your `publish` lands (so a crash re-folds the batch instead of skipping it). On the next trigger the derive resumes exactly where it left off.
+
+**One rule: the fold must be idempotent per event.** A rare crash-recovery case (an event that "heals" at an older position after the derive already passed it) makes the host re-read the whole log that one run, so applying an event twice must not change the result. Use set-style updates keyed by the event's id, not blind `count += 1` accumulation. If your fold cannot be idempotent, stay on the simple `latest` recompute path.
+
+## Guarantees and limitations
+
+**Guarantees**
+
+- **It converges to a correct snapshot.** Publishes are last-writer-wins (the host versions each publish so a later one always supersedes an earlier one), and a derive recomputes from the source of truth, so the view always ends up matching its sources.
+- **Reads stay cheap.** A route serves the view with a single keyed lookup, never a scan.
+
+**Limitations (read these)**
+
+- **By default it recomputes from scratch each time.** A derive re-reads its sources and republishes on every trigger. That is the simple path and a great fit for a **bounded** read: the latest N, a counter total, a small set. To fold an **unbounded, ever-growing** log efficiently, read it with [`events.since`](../database/events.md) instead of `latest` (see [Folding a growing log incrementally](#folding-a-growing-log-incrementally) above); it folds only the new events since a checkpoint.
+- **It is eventually consistent, by a moment.** The view is republished right after the writing request finishes, so there is a tiny window where a reader could see the pre-write view. For most pages (a feed, a leaderboard) that is invisible and fine.
+- **`view.publish` is derive-only.** Routes cannot publish; they can only read the view. That is the whole point: the expensive build happens off the request path.
+
+## When not to use a derive
+
+- **When the read is already cheap.** If a route can answer with a plain keyed `.get`, you do not need a view at all.
+- **When you need it on a timer, not on a data change.** Recomputing "yesterday's report" at 2am is a [daemon](./daemons.md) job, not a derive (a derive is triggered by writes, not by the clock).
+- **When the source is an unbounded full-history fold.** See the limitation above; keep derives to bounded reads.
+
+## Related
+
+- [Views](../database/views.md): the `View<K, V>` family a derive publishes into, and how to read it.
+- [Events](../database/events.md): the append-only log a derive commonly folds into a view.
+- [Counters](../database/counters.md): running totals a derive can read.
+- [Background overview](./README.md): `@derive` versus `@daemon`, and which to reach for.
+- [Data types (`@data`)](../backend/data.md): the value and key types a view stores.