@bloklabs/server 1.15.1 → 1.16.0

package/README.md

@@ -72,10 +72,13 @@ public sealed class ArticleExport(IBlokDocumentConverter blok, ILogger<ArticleEx
 |--------|---------|
 | `ToMarkdownAsync` | the Markdown, plus every construct Markdown could not carry |
 | `ToHtmlAsync` | the document's HTML |
+| `ToHtmlAsync(doc, pages, pageHref)` / `ToMarkdownAsync(doc, pages, pageHref)` | the same, with page titles, icons and links from your `BlokPageInfo` map: a `null` value shows "Page not found", `NoAccess` shows "No access", an id left out shows "Page"; `pageHref` is called only for allowed pages |
 | `ToPlainTextAsync` | the document's readable text; `includeHiddenText: true` also emits an image's alt, a video/file url, an embed source, an audio title/artist/url and a bookmark description/url |
-| `FromMarkdownAsync` | the saved document, plus what Markdown could not carry into it |
-| `FromHtmlAsync` | the saved document parsed out of HTML, plus what the HTML could not carry into it |
+| `FromMarkdownAsync` | the saved document, plus what Markdown could not carry into it; rich text fields are segments |
+| `FromHtmlAsync` | the saved document parsed out of HTML, plus what the HTML could not carry into it; rich text fields are segments |
 | `ExtractTextsAsync` / `InjectTextsAsync` | the document's translatable strings, and the document with them put back |
+| `GetPageIndexAsync` | the pages a document owns, the pages it links to, and each block's searchable text: the same rows as `pageIndex` in Node; raise the timeout for very large documents |
+| `RemapPageDocumentAsync` | a copy of a page document under new block and page ids, to import next to the original; throws `ArgumentException` naming every block id your map misses |
 | `GetVersionAsync` | the `version` the editor stamps into a saved document |
 | `GetSchemaAsync` | the saved format as JSON Schema (draft 2020-12) |
 
@@ -84,8 +87,10 @@ public sealed class ArticleExport(IBlokDocumentConverter blok, ILogger<ArticleEx
 `FromHtmlAsync` is the inverse of `ToHtmlAsync`. It takes a fragment or a whole
 document and parses the structural subset a document body is made of: headings,
 paragraphs, lists (nested, ordered, checklists), tables (merged cells included),
-images, links and inline marks, code, blockquotes, toggles and dividers. Layout
-containers are unwrapped and their children converted in place.
+images, links and inline marks, code, blockquotes, toggles and dividers. Tabs
+written by `ToHtmlAsync` come back as tabs, with each tab's title, icon and
+content. Other layout containers are unwrapped and their children converted in
+place.
 
 ```csharp
 var import = await blok.FromHtmlAsync(html, ct);
@@ -212,6 +217,12 @@ docker run \
 
 `--collab` turns the sync routes on. `--doc-endpoint` names the routes in your own app that the service loads a document from and writes it back to. `BLOK_DOC_ENDPOINT_AUTH` holds the header value those routes expect, sent verbatim on every call. It has to be a single line: a value carrying a carriage return or newline refuses to start rather than losing the header on every call. `--collab-dir` (or `--collab-s3-prefix`) is where the working copy lives. It holds document content, so it must not be publicly readable, and it may not sit inside `--storage-dir`, where everything is served. In-process, the same switches are `options.CollabEnabled` and `options.DocEndpoint`, and the app must call `app.UseWebSockets()`.
 
+Documents written to your endpoint carry rich text fields as segments, the shape the editor's `save()` returns. A document your endpoint returns may hold HTML strings or segments. A custom block tool's rich fields go in `--rich-text-fields` (or `BLOK_RICH_TEXT_FIELDS`; the flag wins), as JSON keyed by block type: `--rich-text-fields '{"callout":["title"]}'`. In-process, the same list is `options.RichTextFields`. List exactly the fields the tool declares in `static richTextFields` on the client, or the two write the field in different shapes. The built-in paragraph, header, quote, toggle and list `text` fields need no entry. See [Rich text is segments](#rich-text-is-segments).
+
+Collaboration rooms are stored in format 2. Upgrade the service and the editor together. An older editor reads the format 2 room and ends its own session with `unsupported-format`. A room stored in format 1 is converted the first time it opens, and offline edits made in format 1 are not replayed.
+
+Add `--collab-journal` (or set `BLOK_COLLAB_JOURNAL=true`) to keep an operation journal under `--collab-dir`. This turns on the acknowledged profile described below. It needs `--collab`. It is refused with `--collab-s3-prefix`: the journal is on this machine's disk, so a second instance sharing the bucket would not see it. In-process, the switch is `options.CollabJournal`. A store registered with `UseCollabOperationStore<T>()` replaces it. The first open of each document adopts its working copy as the journal's starting point, then deletes it, so nothing the working copy held is lost. Turning it off again is not a restart: follow [Going back to the working-copy profile](#going-back-to-the-working-copy-profile) first, or documents edited under the journal reopen as they were on the day you switched.
+
 ## Point the editor at it
 
 ```ts
@@ -257,7 +268,19 @@ new Blok({
 
 The editor caches the pass and replaces it ahead of expiry, and uploads and link previews share the same one.
 
-A pass is a plain HS256 JWT carrying `user`, `doc`, `write` and `exp`, signed with the secret the service runs with (at least 32 characters). Any backend can mint one with its own JWT library; `blokTicket` exists so a JavaScript one does not have to. Routes running inside your own ASP.NET app need none of this: they already know who the caller is.
+A pass is a plain HS256 JWT carrying `user`, `doc`, `write` and `exp`, signed with the secret the service runs with (at least 32 characters). A .NET backend mints one with `BlokTicket.Create` from the `Blok.Server` package:
+
+```csharp
+using Blok.Server.Tickets;
+
+var ticket = BlokTicket.Create(secret, new BlokTicketClaims { User = userId, Doc = docId, Write = true });
+```
+
+`Blok.Server` also brings in Jint, AngleSharp and BouncyCastle, even when you only mint passes.
+
+Any other backend can sign a pass with its own JWT library. The header must be exactly `{"alg":"HS256","typ":"JWT"}`, keys in that order and nothing added, because the server compares it byte for byte.
+
+Passes are needed whenever the routes run with `Auth = "ticket"`, including routes mapped inside your own ASP.NET app. With `Auth` set to `none` or `proxy`, the routes never read a pass.
 
 ## Routes
 
@@ -270,20 +293,87 @@ A pass is a plain HS256 JWT carrying `user`, `doc`, `write` and `exp`, signed wi
 | `GET /sync/{doc}` | WebSocket; the editor's live collaboration connection to one document (with `--collab`) |
 | `POST /sync/{doc}/reset` | Drops the working copy, reloads the document from your endpoint and tells every open tab to pick it up |
 | `POST /sync/{doc}/edit` | Inserts, updates or removes blocks from outside; all-or-nothing, reaches every open tab, and requires an idempotency key |
+| `GET /sync/{doc}/state` | Returns the live document as JSON, with the journal head it reflects when there is a journal |
+
+On a journal-backed service, `POST /sync/{doc}/reset` first adopts a working copy the journal does not hold yet and writes it back to your endpoint, so the reset rebaselines from a record that includes it. It then retires that working copy before it resets. If the retire fails, the reset answers 503 and changes nothing; retry it.
 
 `POST /sync/{doc}/edit` needs one `Blok-Idempotency-Key` header with 1 to 128 printable ASCII characters. With an operation journal, retrying the same key returns the first result without applying it again; reusing it for different work receives 409. A 204 then means the edit is durable, and the response carries `Blok-Doc-Lineage` and `Blok-Doc-Sequence`. A working-copy-only service answers 204 without those headers and without that promise. If that journal cannot commit, the endpoint returns 503 without relaying the edit. A working-copy-only service does not deduplicate the key or make reuse a 409: requests have ordinary retry behavior, and its 204 starts the existing write-back retry path.
 
+A rich text field in an edit may be an HTML string or segments. The service reads the HTML into segments before it applies anything. A rich text field is the `text` of a paragraph, header, quote, toggle or list block, plus every field in `--rich-text-fields`.
+
+- If reading that HTML runs past the runtime's timeout, the edit answers 503 with `Retry-After` (2 seconds by default) and applies nothing. Retry with the same key.
+- If that HTML runs out of the runtime's memory budget, the edit answers 413 and applies nothing. The budget is per request, so the same body fails every time. Do not retry it; send less rich text.
+- HTML the reader refuses answers 422.
+
+With a journal, an edit that changes nothing also answers 204. That covers the same data sent again, and rich text spelled differently, such as `<b>` for `<strong>` or `&nbsp;` for a space.
+
+- Nothing is journalled. `Blok-Doc-Sequence` names the current head, not a new sequence.
+- Nothing is recorded under the key. A retry with that key runs the edit again. If someone edited in between, the retry applies to the new document. A different body under the same key is not a 409.
+- Send `If-Match` on every edit you may retry. Then a retry after the document moved answers 412 instead of applying.
+
+A working-copy-only service never deduplicates, so there every retry runs again, and `If-Match` answers 428.
+
+An edit may also send `If-Match: "<lineage>:<sequence>"`. It is one quoted tag, built from the `Blok-Doc-Lineage` and `Blok-Doc-Sequence` values exactly as the service prints them.
+
+- With a journal, the edit applies only if the document is still at that head. Otherwise it answers 412 and applies nothing. The 412 carries the current `Blok-Doc-Lineage` and `Blok-Doc-Sequence`.
+- A key that is already committed still returns its first result, even if its `If-Match` is now stale.
+- A working-copy-only service has no head to check, so any `If-Match` answers 428 and applies nothing.
+- A list, `*`, a weak tag, or any other shape answers 400, with or without a journal.
+- A 412 commits nothing, so you may retry the same key with a fresh tag.
+
+`GET /sync/{doc}/state` returns the live document as `application/json`, in the same shape your document endpoint receives, with rich text fields as segments. It includes edits made a moment ago. With a journal, it also sends `Blok-Doc-Lineage`, `Blok-Doc-Sequence` and `ETag: "<lineage>:<sequence>"`, naming the exact head the body reflects. Send that `ETag` back as `If-Match` to edit only if nothing changed in between. A working-copy-only service sends the body without those three headers. A purged document answers 403. A document that cannot be loaded, is held by another process, or is on a service that is shutting down answers 503. An export that ran past the runtime's timeout or allocation budget also answers 503, with `Retry-After` (2 seconds by default). A document the service cannot write as JSON answers 500.
+
+Both routes add `Blok-Doc-Lineage`, `Blok-Doc-Sequence` and `ETag` to `Access-Control-Expose-Headers` for an allowed origin, so a browser page can read them. Headers your app already exposes are kept.
+
 Upload routes exist only when local or S3-compatible storage is configured. Consumer-supplied URLs pass through one guarded outbound client that blocks private and cloud-metadata addresses. Send `POST /upload-by-url` a `{"url":"..."}` body with an `application/json` media type; parameters such as `charset=utf-8` are allowed, but JSON suffix types are not.
 
 A request that carries `Origin` must match an allowed origin in every auth mode. In `none` and `proxy`, a genuinely originless backend request remains allowed, but an originless browser request carrying `Sec-Fetch-Site: cross-site` is rejected. `ticket` always requires an allowed `Origin`.
 
-A ticket with `write: false` may call `GET /unfurl` and open `GET /sync/{doc}` read-only; both upload routes, `reset` and `edit` require `write: true`. The `doc` claim scopes the collaboration routes: `/sync/{doc}`, its `reset` and its `edit` are refused when the pass names no document or a different one. A collaboration pass must also name its `user`: `GET /sync/{doc}` closes one with an empty `user` as 4401 `pass names no user`, because the per-user connection cap and rate window key on that name. The upload and unfurl routes ignore it, so a pass minted for one page works for every upload and preview that page can make.
+A ticket with `write: false` may call `GET /unfurl`, `GET /sync/{doc}/state` and open `GET /sync/{doc}` read-only; both upload routes, `reset` and `edit` require `write: true`. The `doc` claim scopes the collaboration routes: `/sync/{doc}`, its `reset`, its `edit` and its `state` are refused when the pass names no document or a different one. `state` asks your `IBlokAuthorization` for read access only. A collaboration pass must also name its `user`: `GET /sync/{doc}` closes one with an empty `user` as 4401 `pass names no user`, because the per-user connection cap and rate window key on that name. The upload and unfurl routes ignore it, so a pass minted for one page works for every upload and preview that page can make.
+
+## Rich text is segments
+
+**Breaking.** Rich text fields leave the service as segments, not HTML. This covers the write-back `PUT` to your document endpoint and `GET /sync/{doc}/state`. Before, `data.text` was a string:
+
+```json
+{ "id": "p1", "type": "paragraph", "data": { "text": "a <b>bold</b> word" } }
+```
+
+Now it is a list of segments:
+
+```json
+{ "id": "p1", "type": "paragraph", "data": { "text": [{ "text": "a " }, { "text": "bold", "marks": { "bold": true } }, { "text": " word" }] } }
+```
+
+What to change in your app:
+
+- Your document endpoint must accept segments in rich text fields.
+- It may keep serving HTML. The service reads HTML on the way in, from your endpoint and from `POST /sync/{doc}/edit`, and writes segments back.
+- List your custom tools' rich text fields with `--rich-text-fields` or `options.RichTextFields`, the same list as each tool's `richTextFields`. A field missing from the list keeps the shape the service was handed: HTML sent to the service for it stays an HTML string.
+- Upgrade the service and the editor together. Rooms are now format 2. An editor from before this change sees the new service announce format 2 and ends its own session with `unsupported-format`. The new editor does the same with an old service.
+
+The first time the service opens a document stored in format 1, it moves it to format 2 once:
+
+- every rich text field becomes formatted text, with every character and mark kept;
+- the room gets a new lineage and `epoch + 1`;
+- edits a format-1 editor saved offline are quarantined on that editor, not replayed.
+
+If that move fails, the document stays in format 1. The service then holds the document off for a while (2 s at first, doubling up to 60 s). In that window, joins, edits and `/state` answer 503. The next open after the window tries again.
+
+### Your own sync server
+
+If you run your own server against Blok's sync protocol, it must speak format 2 too. See [blok-sync-v2.md](protocol/blok-sync-v2.md), type 100.
+
+- Announce `"format":2` in the type-100 control frame. The editor closes a session that names any other format.
+- Store each rich text field as a formatted `Y.XmlText`, one Yjs attribute per mark. A plain `Y.Text` in a rich field is read as format-1 HTML.
+- When you move a room from format 1, mint a new `lineage` and announce `epoch + 1`. Keeping the old lineage would let format-1 offline edits replay into the format-2 room.
+- `test/unit/server-conformance/blok-client-contract.test.ts` drives Blok's own collaboration provider against a server and expects format 2.
 
 ## Live collaboration profiles
 
 `--collab` (or `options.CollabEnabled`) gives you the working-copy profile: the service keeps a working copy of every open document and writes it back to your document endpoint. Nothing keeps a record of the individual changes that produced it, so `POST /sync/{doc}/edit` cannot tell a retry from new work, and a socket gets no per-change receipt.
 
-Registering an operation store turns on the acknowledged profile. The journal becomes the record. Every accepted change is appended to it before it is broadcast. The edit route deduplicates its `Blok-Idempotency-Key`, and answers 409 for a key reused for different work. A socket that negotiated `blok-sync.v2` receives one acknowledgement per operation, naming the sequence it committed at. The service ships no store you can switch on: there is no flag for one on the standalone host, and the working set under `--collab-dir` or `--collab-s3-prefix` is not a journal. An in-process app registers its own. The store's own bodies are elided below. Writing them is the work, and the laws further down are what they have to keep; the registration is complete as written:
+Registering an operation store, or `--collab-journal`, turns on the acknowledged profile. The journal becomes the record. Every accepted change is appended to it before it is broadcast. The edit route deduplicates its `Blok-Idempotency-Key`, and answers 409 for a key reused for different work. A socket that negotiated `blok-sync.v2` receives one acknowledgement per operation, naming the sequence it committed at. The standalone host's `--collab-journal` puts the built-in local journal under `--collab-dir`. It serves one instance only. The working set under `--collab-dir` or `--collab-s3-prefix` is not a journal. An app that runs more than one instance registers its own store. The store's own bodies are elided below. Writing them is the work, and the laws further down are what they have to keep; the registration is complete as written:
 
 ```csharp
 using Blok.Server.AspNetCore;
@@ -338,7 +428,7 @@ A backend that is not .NET implements the wire protocol instead of this interfac
 
 Stock `y-websocket` never offers `blok-sync.v2`, so it negotiates v1 and is compatible with the working-copy profile alone: ordinary y-protocol sync, no acknowledgement, no durability claim. On a journal-backed document a v1 write is still journaled before it is relayed; it earns no receipt. The same holds for any client that offers only v1.
 
-S3 stays v1-only. `--collab-s3-prefix` puts the working set in your bucket, and there is no S3 operation store, so an S3-configured service runs the working-copy profile unless it also registers one.
+S3 stays v1-only. `--collab-s3-prefix` puts the working set in your bucket, and there is no S3 operation store, so an S3-configured service runs the working-copy profile unless it also registers one. `--collab-journal` is refused next to `--collab-s3-prefix`.
 
 ### Going back to the working-copy profile
 
@@ -347,7 +437,8 @@ Registering an operation store is close to one-way per document. A journal-backe
 A build without your store does not read the journal. Unregistering the store, or rolling back to a binary that never had it, lands each document on whatever else it has:
 
 - **Journal-backed from the start.** There is no blob, so the room seeds from your document endpoint and comes back as the last projection that endpoint accepted. Every operation acknowledged since then is still in your journal and nothing serves it.
-- **Working set from before the switch.** A document that ran under `--collab-dir` or `--collab-s3-prefix` before you registered the store still holds the blob it had that day, and registering the store never touched it. A journal-backed room writes no blob, and it seeds its journal from your endpoint rather than from the blob. A blob with any frame in it is authoritative on open. The endpoint is never consulted, so that document comes back as it was on the day you switched. There is no error, and nothing in the log.
+- **Working set from before the switch.** The first open under the journal adopts a document's blob as the journal's baseline, keeping its lineage, and then retires it. Every later open retires any blob it finds beside the journal; a failed retire is logged and tried again on the next open. A journal-backed room never writes one. So a blob survives only for a document that has not been opened since you registered the store, or one journalled by an earlier Blok build and not reopened since.
+- **Unreadable working set.** Retiring removes only a blob that reads back. A damaged one is kept for repair: under `--collab-dir` it is moved aside as `<key>.unreadable-<time>`, and an S3 object is left where it is. The room then seeds from your endpoint. Only a purge deletes quarantined bytes. A blob with any frame in it is authoritative on open. The endpoint is never consulted, so that document comes back as it was on the day you switched. There is no error, and nothing in the log.
 
 Blok does not keep a second whole-document copy beside the journal to make the switch back instant. The journal is the record; the JSON is a projection of it. Buying instant rollback with a hidden dual write would mean two records that can disagree, and the second one carries no fence.
 
@@ -361,6 +452,35 @@ Run this drill before you roll back.
 
 Rolling forward again is not symmetric either. With the store registered, the journal wins the open, so whatever was typed while the old build was serving is not in it.
 
+## Pages
+
+A `page` block saves only its `pageId`. The page body is its own collaborative document, and its document id is the page id. So each page body loads and saves through your document endpoint, like any other document.
+
+| Call | What the service sends | What you answer |
+| --- | --- | --- |
+| `GET {DocEndpoint}/{docId}` | `Authorization`: your `DocEndpointAuth` value | `200` with the JSON literal `null`, or `{"data": null, "version": "0"}`, for a document you never saved; it opens empty. Otherwise `200` with `{"data": <document>, "version": "<v>"}`, or the bare document. |
+| `PUT {DocEndpoint}/{docId}` | The bare document. `Blok-Doc-Version`: the last version you answered, absent until you answer one. `Blok-Doc-Lineage` and `Blok-Doc-Sequence`: with a journal only. | Any `2xx`. A JSON body with `version` sets the next `Blok-Doc-Version`; an empty body keeps it. |
+
+- A first open fails on 404, 204, an empty 200 or any other non-2xx. The socket closes with 4503. Never answer those for a new page.
+- PUT is an upsert, and the version header is optional. With a journal, every reopen owes one PUT, even if nobody typed. It goes out at the room's next checkpoint, eviction or drain. For a page never saved, it carries `{"blocks":[]}` and no version, so it must create the row.
+- Keep the version on the service's own PUT. Bump it only for your own write, answer a stale PUT with 409, and call `POST /sync/{doc}/reset`.
+- A refused PUT is retried, and its room stays loaded until it lands. So delete a page in this order: commit a tombstone that your `IBlokAuthorization` and both routes refuse, purge it with `ICollabDocumentPurger`, then delete its rows. The guide below handles the purge's `UnauthorizedAccessException` and `DocumentOpenElsewhere`.
+
+Who may open which page goes through `IBlokAuthorization`. It runs before a room loads, on `/sync`, `/state`, `/edit` and `/reset`. It is registered as a singleton:
+
+```csharp
+builder.Services
+  .AddBlokServer(options =>
+  {
+    options.CollabEnabled = true;
+    options.DocEndpoint = "http://127.0.0.1:5080/internal/blok-docs";
+    options.DocEndpointAuth = docEndpointSecret;
+  })
+  .UseAuthorization<PageRules>();
+```
+
+The full walkthrough, with the page store, the document endpoint, delete, duplicate, export and drain: [Page blocks on an ASP.NET Core backend](https://github.com/JackUait/blok/blob/main/docs/maintainers/page-csharp-host.md).
+
 ## Who was in a document, and when
 
 The service can tell your app when a person was in a document, including people

package/dist/ticket.cjs

@@ -1,7 +1,7 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
 let node_crypto = require("node:crypto");
 //#region src/ticket.ts
-/** Must match the floor BlokServerOptions.Validate enforces, or the service refuses to start. */
+/** Must match BlokTicket.MinimumSecretLength (C#), the floor BlokServerOptions.Validate enforces, or the service refuses to start. */
 var MIN_SECRET_LENGTH = 32;
 var DEFAULT_TTL_SECONDS = 300;
 /**

package/dist/ticket.mjs

@@ -1,6 +1,6 @@
 import { createHmac } from "node:crypto";
 //#region src/ticket.ts
-/** Must match the floor BlokServerOptions.Validate enforces, or the service refuses to start. */
+/** Must match BlokTicket.MinimumSecretLength (C#), the floor BlokServerOptions.Validate enforces, or the service refuses to start. */
 var MIN_SECRET_LENGTH = 32;
 var DEFAULT_TTL_SECONDS = 300;
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bloklabs/server",
-  "version": "1.15.1",
+  "version": "1.16.0",
   "description": "Blok server delivery for ASP.NET Core, npm, and Docker — shared C# routes for file uploads and link previews.",
   "license": "Apache-2.0",
   "author": "JackUait",