@smthrs/std 0.0.0-stage → 1.0.0-rc.4

package/docs/guides/pin-a-checkpoint.md

@@ -0,0 +1,141 @@
+---
+title: "Pin a tree and read it later"
+description: "Record the working tree as a checkpoint, materialize it as a scratch directory for the length of one call, and relocate a read or a command onto it."
+sidebar:
+  order: 6
+---
+
+A checkpoint is a **value**. Minting one changes nothing about the workspace,
+and running a call against one leaves the live tree exactly as it stands. That
+is the whole of what it buys: before it existed, the only way to answer "did
+this fail before my change" was to undo the change, which means reverting the
+very work the reading was meant to prove.
+
+This module is the host half of [`@smthrs/harness`](/api/harness)'s
+`ctx.checkpoint()` and `ctx.call(flow, input, { at })`. The harness owns the
+surface, the identity, the bound, and the refusals; this owns the two host
+operations that need a real tree.
+
+## Bind the git store
+
+```ts
+import * as Checkpoints from "@smthrs/std/Checkpoints"
+
+const store = Checkpoints.layerGit({
+  root: "/workspace/repo", // the host path of the git repository
+  cwd: "/testbed", // where a container sees that same directory
+  baseRef: "refs/flows/capture-base" // optional
+})
+```
+
+`layerGit` requires a `ChildProcessSpawner`. A host that pins nothing binds
+`Checkpoints.layerNoop`, whose refusal says to take the reading on the live tree
+instead.
+
+## Capture and materialize
+
+```ts
+import * as Effect from "effect/Effect"
+
+const reading = Effect.gen(function*() {
+  const checkpoints = yield* Checkpoints.Checkpoints
+  const snapshot = yield* checkpoints.capture("before-fix")
+  return yield* checkpoints.materialize(snapshot.id, (tree) => Effect.succeed(tree.host))
+})
+```
+
+`capture(id)` records the working tree as it stands and returns a `Snapshot`
+with the `id` you gave and the store's own `ref` for it. `materialize(id, use)`
+hands the tree back as a directory for the length of the effect you pass, and
+removes it however that effect ends. It is scoped rather than returned, because
+a run killed at its wall-clock budget would otherwise leave a second checkout of
+the whole repository inside the tree whose diff is the run's answer.
+
+A `Materialized` carries four paths, because one directory can have two names:
+
+| Field       | Meaning                                               |
+| ----------- | ----------------------------------------------------- |
+| `host`      | Where a process on this machine finds the checkpoint. |
+| `guest`     | Where a container finds the same directory.           |
+| `root`      | The workspace itself, as this machine names it.       |
+| `guestRoot` | The workspace as a container names it.                |
+
+`Checkpoints.baseId` is `"base"`, the tree the run opened on. It is not minted:
+the store uses `baseRef` when one is declared. A declared ref that
+does not resolve is `not_found`. Otherwise it tries `TestRunner.captureBase`
+and then `HEAD`, matching `test`.
+
+## What the git binding does, and does not do
+
+`layerGit` records with `git stash create`, which is the one git command that
+records the working tree and changes nothing else: it does not move the
+worktree and does not touch the stash ref. It runs against a temporary shadow
+`GIT_DIR` that shares only the workspace's objects and reads a copy of the
+workspace index, so it never writes the agent's index and never runs a hook,
+`core.fsmonitor` command or filter driver the workspace's `.git` names. That matters
+because the agent runs `git` in this same workspace and its own `git diff` is
+the run's evidence. A capture that staged into the real index would be the
+harness editing the evidence while recording it. A tree with nothing to record
+prints nothing, and the checkpoint is then `HEAD`, which is exactly what the
+tree is.
+
+The commit is named in the repository's own git config, under
+`Checkpoints.configSection` (`flows-checkpoint`), and **not** by a ref. A ref is
+history: `git log --all` lists it, `git show` prints it, and `git log --all -S`
+searches it, so a checkpoint named by a ref would hand an agent a commit
+containing its own edit and let it read that back as if it were upstream work.
+Under config the commit object stays unreferenced, a checkout by id works
+perfectly well, and no command that walks refs can reach it.
+
+One command still sees it: `git fsck` reports it as a dangling commit, because
+that is what an unreferenced commit is. A checkout is a standalone repository
+rather than a worktree of the workspace, so `git log --all` in the workspace
+does not list it even while a call runs in it.
+
+Untracked files are not in the recorded tree, which matches how a patch is
+captured. Materialization is a detached checkout at
+`<root>/.flows-checkpoints/<id>-<lease>`, under `Checkpoints.scratchDirectory`.
+Its own `.git` names the commit in `HEAD` and reaches the workspace's objects
+through a relative `objects/info/alternates`, so git run in it through a
+container mount works.
+Each call gets a unique lease, so overlapping calls never remove each other's
+checkout. Existing checkouts are left alone; a `SIGKILL` can leave one behind.
+Inside the workspace is the only placement that works, for the same reason the
+`test` baseline is placed there: a container sees the workspace through a mount,
+and a scratch checkout anywhere else on the host is not visible to it.
+
+## Point a call at a checkpoint
+
+`relocate` rewrites one call's input so the call runs against a materialized
+checkpoint. It is a closed table, not a per-flow hook:
+
+```ts
+const relocation = Checkpoints.relocate("read", { path: "src/widen.ts" }, tree)
+
+if (relocation._tag === "Relocated") {
+  // relocation.input now reads .flows-checkpoints/<id>-<lease>/src/widen.ts
+}
+```
+
+| Flow           | The field it is relocated by                                           |
+| -------------- | ---------------------------------------------------------------------- |
+| `bash`         | `cwd`, into the checkpoint and into the same subdirectory it asked for |
+| `read`, `ls`   | `path`, prefixed with the checkpoint's workspace-relative directory    |
+| `grep`, `glob` | `root`, prefixed the same way                                          |
+
+`bash` is relocated from the guest side when the call names a container, because
+that is the path the container will be given. Keeping the subdirectory matters:
+a check declared in `tests/` that silently ran at the top would come back
+failing for a reason nobody chose.
+
+Three inputs are refused rather than rewritten, and the tag says which:
+
+- `UnsupportedFlow`, for any flow outside that table. `test` is deliberately
+  outside it: it answers this question already with `against: "base"`, and two
+  mechanisms pointed at one tree are two answers that can disagree.
+- `AbsolutePath`, because an absolute path in these runs is a container path and
+  the host cannot know which prefix of it names the tree.
+- `OutsideTree`, for a relative path that climbs past the checkpoint with `..`,
+  which names the live tree under another name. That is exactly the tree the
+  reading was taken to avoid, and a call key folds the checkpoint in, so a live
+  reading recorded under a checkpoint would replay as a pinned one forever.

package/docs/guides/query-a-language-server.md

@@ -0,0 +1,257 @@
+---
+title: "Query a language server"
+description: "Run the ten LSP operations through the lsp flow, spawn a server with NodeLanguageServer, and understand the 1-based coordinates and the pass-through result."
+sidebar:
+  order: 8
+---
+
+`lsp` is one flow with ten operations. It answers the questions a text search
+cannot: where a symbol is defined, who calls it, what type it has.
+
+## Spawn a server
+
+`NodeLanguageServer` speaks framed JSON-RPC over ordinary stdio pipes, spawned
+through [`@smthrs/kernel`](/api/kernel)'s `ChildProcessSpawner`. No terminal is
+involved.
+
+```ts
+import * as NodeLanguageServer from "@smthrs/std/NodeLanguageServer"
+
+const server = NodeLanguageServer.layer({
+  command: "typescript-language-server",
+  args: ["--stdio"],
+  cwd: "/workspace",
+  environment: { NODE_ENV: "development" },
+  timeoutMs: 30_000
+})
+```
+
+The layer sends `initialize` with `cwd` as the root URI and then `initialized`,
+so the service is ready when it resolves. `timeoutMs` defaults to 30 seconds and
+bounds every request as well as every write to the server's standard input.
+The child inherits only `PATH`, `HOME`, `USER`, `LANG`, `LC_*`, `TERM`, `TMPDIR`,
+and `SHELL`, with credential-shaped names withheld. Declare other names through
+`environment`; explicit values override the allowlist.
+
+### Keep the server's code out of the workspace
+
+The server runs on the host as the host user, so a program the workspace
+supplies would let files an agent wrote choose what the host executes. The
+layer fails with `permission_denied`, before spawning, when:
+
+- `command` resolves to a file under `cwd`, directly, through a `PATH` entry
+  such as `node_modules/.bin`, or through a symlink planted there;
+- an argument, or the value after `=` in one, names an existing file under
+  `cwd`;
+- `command` is a launcher whose arguments choose what runs: a shell
+  (`sh -c ...`), `env`, a package runner (`npx`, `pnpm exec`, `npm`, `yarn`,
+  `bunx`, `bun x`, `deno`), or `node` or `bun` given inline code or a preload
+  (`-e`, `-p`, `-r`, `--require`, `--import`, `--loader`). `npx` and
+  `pnpm exec` resolve the workspace's `node_modules/.bin` first, and inline
+  code resolves bare requires from `cwd`. Name the server's own host binary,
+  or an interpreter and a host script file, instead.
+
+The layer cannot see what the server loads after it starts. By default
+typescript-language-server runs the workspace's own `node_modules/typescript`;
+tsserver started with `--allowLocalPluginLoads` loads `tsconfig` plugins from
+the project; rust-analyzer runs build scripts and proc macros. Pin those to
+host copies, or run the server in a sandbox:
+
+```ts
+const server = NodeLanguageServer.layer({
+  command: "/opt/lsp/node_modules/.bin/typescript-language-server",
+  args: ["--stdio"],
+  cwd: "/workspace",
+  initializationOptions: { tsserver: { path: "/opt/lsp/node_modules/typescript/lib/tsserver.js" } }
+})
+```
+
+`initializationOptions` is sent on `initialize` unchanged. With `tsserver.path`
+outside the workspace and local plugin loads left off, tsserver probes for
+plugins only beside its own install.
+
+A host with no server binds `LanguageServer.layerNoop`, and every operation
+fails with `unsupported`.
+
+### One server per language
+
+Pass several configs and each file goes to the server whose `extensions`
+include its extension. A config without `extensions` takes every file no other
+server claims; a file nobody claims fails with `unsupported`.
+`workspaceSymbols` asks every server and concatenates the answers.
+
+`NodeLanguageServer.layerLazy` takes the same configs but starts each server on
+the first request for one of its files, so a host can bind servers a run may
+never need. The workspace checks above still run when the layer is built. A
+server starts with the services and permissions of the code that built the
+layer, not those of the request that first needs it. A server that fails to
+start fails that request and every later one for its files. `refresh` and
+`close` never start a server.
+
+```ts
+const servers = NodeLanguageServer.layer([
+  { command: "typescript-language-server", args: ["--stdio"], cwd: "/workspace", extensions: [".ts", ".tsx"] },
+  { command: "pyright-langserver", args: ["--stdio"], cwd: "/workspace", extensions: [".py"] }
+])
+```
+
+## Edits reach the server
+
+With a `LanguageServer` bound, `edit`, `write` and `apply_patch` send each file
+they write to its server after the entire operation succeeds: `textDocument/didOpen` the first time, then
+`textDocument/didChange` with the full text. A file `apply_patch` deletes or
+moves away gets `textDocument/didClose`. `edit` then returns `errors`, the
+error-severity diagnostics in the file after the edit (at most 20, 1-based
+positions). `errors` is absent when no server is bound or it did not answer;
+the edit itself never fails because of the server.
+
+`bash` does not know which files a command changed, so after a command on the
+host it calls `refresh`: the client re-reads every file it has open, sends the
+text of each that changed, and closes each that is gone. A command run in a
+container leaves the host's files alone and refreshes nothing.
+
+`diagnostics` pulls `textDocument/diagnostic`. A server that answers
+`MethodNotFound` is read from its `textDocument/publishDiagnostics`
+notifications instead: the client opens the file if it is not open yet and waits
+up to `settleMs` (5 seconds by default) for a publish for the latest synced
+text, then fails with `timeout`. After a publish it keeps waiting while later
+ones arrive less than `quietMs` (300 ms by default) apart, within `settleMs`, and
+answers with the last. Either way the answer is a report,
+`{ kind: "full", items }`.
+
+A publish that carries no version counts for the text synced before it arrived.
+Before each sync the client sends a `$/` request, which the specification has
+every server refuse at once, so a report the server wrote earlier (such as the
+empty one typescript-language-server publishes when a file closes) reaches the
+client first and never counts for the new text. A server that does not answer
+within a second is not asked again.
+
+### typescript-language-server
+
+typescript-language-server 6 has no pull diagnostics, publishes without
+versions, and publishes nothing for a change that leaves a file with no
+problems, so such a change always waits the full `settleMs` and fails with
+`timeout`. On the first open of a file it publishes syntax errors before
+semantic ones. Pick `settleMs` and `quietMs` from a measurement on your host:
+
+```sh
+node scripts/lsp-settle-bench.ts <typescript-language-server or gopls> <tsserver.js or -> <workspace> <iterations> <settleMs> <quietMs> <file>...
+```
+
+### Measured settle with real servers
+
+Repeat it with `sh scripts/lsp-real.sh <dir> [iterations] [settleMs] [quietMs]`.
+It installs typescript-language-server 6.0.1 with typescript 5.9.3 (typescript 7
+ships no `tsserver.js`) and, when Go is on `PATH`, gopls under `<dir>`, runs
+`test/NodeLanguageServerReal.test.ts` (a real server on `test/fixtures/lsp/ts`;
+skipped when no server is on `PATH` or named by `SMITHERS_LSP_TLS` and
+`SMITHERS_LSP_TSSERVER`), then writes `<dir>/out/ts.json` and `go.json`. Each
+file is opened with a type error, then cycles fix, comment-only change, restore,
+break, close and reopen, 5 times, at `settleMs` 3000 and `quietMs` 500.
+
+One run on a 14 CPU macOS host at load average 44 to 54, from other agents, on
+the two-file fixtures. Milliseconds from sync to the first publish (raw, p50 and
+max), and to the client's answer:
+
+| Server, step                 | First publish        | Client answer            | Correct                  |
+| ---------------------------- | -------------------- | ------------------------ | ------------------------ |
+| tls, fix                     | 377 to 623, max 1301 | 878 to 908, max 3010     | 9/10                     |
+| tls, break                   | 373 to 401, max 472  | 874 to 887, max 1202     | 10/10                    |
+| tls, reopen                  | 1 to 4, max 33       | 864 to 871, max 930      | 10/10                    |
+| tls, comment-only or restore | none                 | waits `settleMs` (3.0 s) | 1/20, no errors returned |
+| tls, cold open               | none in 4 s          | 3.0 s, `timeout`         | 0/2                      |
+| gopls, every step            | 2 to 5, max 24       | 1 to 3, max 6            | 50/50                    |
+| gopls, cold open             | 252 to 1541          | 415 to 553               | 2/2                      |
+
+`settleMs: 3000` and `quietMs: 500` hold for warm typescript-language-server
+edits. Limitations: one host, loaded, n of 5 per step and 2 files; a cold open
+under load exceeded 3 s for both files, and a change that leaves a file
+clean returns no `errors` after the full `settleMs`. gopls publishes within
+milliseconds, so it never needs more than the defaults. No Python server was
+measured. The raw JSON is written by the script and is not checked in.
+
+## Run a query
+
+```ts
+import * as Lsp from "@smthrs/std/Lsp"
+
+const definition = Lsp.run({
+  operation: "definition",
+  path: "/workspace/src/widen.ts",
+  line: 12,
+  character: 17
+})
+// definition.result is the server's own answer, passed through unchanged
+```
+
+| Operation               | What it takes               |
+| ----------------------- | --------------------------- |
+| `hover`                 | `path`, `line`, `character` |
+| `definition`            | `path`, `line`, `character` |
+| `references`            | `path`, `line`, `character` |
+| `implementation`        | `path`, `line`, `character` |
+| `prepareCallHierarchy`  | `path`, `line`, `character` |
+| `callHierarchyIncoming` | `path`, `line`, `character` |
+| `callHierarchyOutgoing` | `path`, `line`, `character` |
+| `documentSymbols`       | `path`                      |
+| `diagnostics`           | `path`                      |
+| `workspaceSymbols`      | `query`                     |
+
+`line` and `character` are **1-based**, which is how `read` and `grep` report
+them, so a hit from a search is a position you can pass straight in. The flow
+converts to the protocol's 0-based coordinates for you.
+
+`path` must be a normalized absolute path. A relative path, or a missing one for
+any operation except `workspaceSymbols`, fails with `invalid_input`, as does a
+position operation missing `line` or `character`. The server reads the file
+itself, so the flow first reads `path` through the bound `FileSystem`: a path the
+guarded filesystem denies fails with `permission_denied` before the server sees
+it.
+
+`references` includes the declaration. The two call-hierarchy directions run
+`prepareCallHierarchy` first and return an empty array when the server prepares
+no item, so a position that is not a callable is an empty answer rather than a
+failure.
+
+## The result is not a schema
+
+`Output` is `{ result: unknown }`. The server's answer is passed through in the
+server's own shape, because LSP responses vary by server and by version, and
+narrowing them here would be a second, staler schema. Decode it on the caller's
+side against what the server you bound actually returns.
+
+## Bounds a server cannot exceed
+
+The client refuses a malformed or oversized stream rather than growing without
+limit:
+
+| Bound                                     | Value                                               |
+| ----------------------------------------- | --------------------------------------------------- |
+| One JSON-RPC frame body                   | 8 MiB                                               |
+| One frame's headers                       | 8 KiB                                               |
+| `NodeLanguageServer.MAX_QUEUED_FRAMES`    | 256 frames buffered for the server's standard input |
+| `NodeLanguageServer.MAX_PENDING_REQUESTS` | 512 concurrent in-flight requests                   |
+
+Every queued write uses the request timeout, so a server that stops reading
+produces a typed `timeout` rather than an unbounded queue or a new hang. Process
+exit or a closed stdout fails every pending request after at most 100 ms for
+stderr to finish draining.
+
+Request errors include the protocol `method`. A server refusal keeps its numeric
+code, message, and optional data in `StdError.rpcError`; the response frame limit
+bounds that data. The main error message also includes the server message.
+`StdError.stderr`, when present, contains the latest stderr tail, captured from
+at most 64 KiB. Initialization failures and exits retain this diagnostic context.
+
+## Bring your own server
+
+`LanguageServer` is an ordinary service interface: ten query methods, each
+taking a `Position` (`path`, `line`, `character`) or a string, plus
+`sync(path, text)` and `close(path)`. A host with its own client, in-process
+index, or remote service implements those twelve methods and
+binds them with `LanguageServer.make`. Nothing above the service knows which one
+answered.
+
+One detail to implement against: a `Position` reaching the service is already
+**0-based**. The 1-based to 0-based conversion happens in the flow, so the
+service speaks the protocol's own coordinates.

package/docs/guides/reach-the-network.md

@@ -0,0 +1,141 @@
+---
+title: "Reach the network"
+description: "Get a URL with fetch, post a body with http-post, render a page with webfetch, and search the web through a provider bound to the WebSearch service."
+sidebar:
+  order: 7
+---
+
+Four flows leave the machine, and they differ in what they promise. All of them
+go through [`@smthrs/kernel`](/api/kernel)'s `HttpClient`, so the permission
+kernel sees the request.
+
+| Flow        | Tier           | Capability   | Use it for                                            |
+| ----------- | -------------- | ------------ | ----------------------------------------------------- |
+| `fetch`     | `sealed`       | `net:get:*`  | A raw GET whose body you want as text.                |
+| `webfetch`  | `sealed`       | `net:get:*`  | A page you want rendered as Markdown or plain text.   |
+| `http-post` | `irreversible` | `net:post:*` | Sending data. The remote side may already have acted. |
+| `websearch` | `sealed`       | `net:post:*` | A search through a configured provider.               |
+
+Provide the host's `GrantStore` alongside the pin-capable `HttpClient` from
+`@smthrs/platform-node/EgressHttpClient.layer(process.env)` (also used by the
+Bun host bundle). Unsupported transports fail closed. Every destination,
+including redirects, is checked before DNS or transport. Private destinations
+also need an exact `net:private` action grant for the normalized URL origin.
+See [HTTP destination permissions](../reference/flows.md#http-destination-permissions).
+
+## Get a URL
+
+```ts
+import * as Fetch from "@smthrs/std/Fetch"
+
+const response = Fetch.run({
+  url: "https://example.com/data.json",
+  headers: { accept: "application/json" }
+})
+// response.status, response.body, response.truncated, response.notice
+```
+
+An error status is an ordinary value: a 500 comes back as `status: 500` with its
+body. Only a transport failure, a permission refusal, or a body that exceeds the
+cap uses the error channel.
+
+The URL must be absolute `http` or `https` with no user information, or the call
+fails with `invalid_input`. A response larger than 5 MiB fails with
+`response_too_large` before decoding. The returned body is capped at 60,000
+bytes, head first, and says so through `truncated` and `notice`.
+
+## Post a body
+
+```ts
+import * as HttpPost from "@smthrs/std/HttpPost"
+
+const response = HttpPost.run({
+  url: "https://example.com/hooks/build",
+  body: JSON.stringify({ ref: "main" }),
+  contentType: "application/json"
+})
+```
+
+`contentType` defaults to `application/json`. The body is sent verbatim. The
+output shape and the limits match `fetch`.
+
+This is the one network flow declared `irreversible`, which is what makes a
+scheduler treat it as a real side effect rather than a retrievable read.
+
+## Render a page
+
+`webfetch` is for reading documentation rather than consuming an API:
+
+```ts
+import * as WebFetch from "@smthrs/std/WebFetch"
+
+const page = WebFetch.run({
+  url: "https://effect.website/docs",
+  format: "markdown",
+  timeout: 30
+})
+// page.url is the final URL after redirects
+// page.status, page.contentType, page.content, page.truncated, page.notice
+```
+
+`format` is `markdown` (the default), `text`, or `html`. HTML is converted only
+when the response is actually HTML; anything else is passed through as it
+arrived.
+
+Five behaviors are worth knowing:
+
+- Up to 10 redirects are followed, and `page.url` is where it ended. Crossing an
+  origin drops the `authorization` and `cookie` headers.
+- A content type that is not `text/*` and carries neither `json` nor `xml` fails
+  with `unsupported_content_type`. This flow renders text, not binaries.
+- `timeout` is in seconds, defaults to 30, and is capped at 120. It bounds the
+  request and the body read, and expiry is a `timeout` failure.
+- A response past 5 MiB is `response_too_large`, checked against
+  `content-length` first and then while reading.
+- The rendered `content` keeps its first 60,000 bytes. A longer page sets
+  `truncated: true` and a `notice` with the kept and total bytes.
+
+## Search the web
+
+`websearch` has no built-in provider. It calls the `WebSearch` service, so the
+host decides who answers:
+
+```ts
+import * as WebSearch from "@smthrs/std/WebSearch"
+
+const results = WebSearch.run({
+  query: "effect schema optional fields",
+  numResults: 5,
+  freshness: "month"
+})
+// results.results: [{ title, url, snippet, publishedAt? }]
+```
+
+`numResults` is 1 through 20 and defaults to 8. `freshness` is `day`, `week`,
+`month`, or `year`.
+
+The package ships one provider. `ExaWebSearch.layer` reads its API key from a
+named credential rather than from the environment:
+
+```ts
+import * as ExaWebSearch from "@smthrs/std/ExaWebSearch"
+
+const provider = ExaWebSearch.layer("exa-api-key")
+```
+
+It needs [`@smthrs/control`](/api/control)'s `Credential` service and
+[`@smthrs/kernel`](/api/kernel)'s `HttpClient`. It maps provider outcomes onto
+the standard codes: a 429 or a refusal carrying `Retry-After` becomes
+`rate_limited` with the advice in the message, a 401 or 403 becomes
+`provider_unavailable`, a 5xx becomes `provider_unavailable` naming the status,
+and any other non-2xx becomes `request_failed`. A request or response deadline
+becomes `timeout`. Snippets are clipped at 2,000 characters, the request is
+bounded at 30 seconds, and a response body over 5 MiB fails with
+`response_too_large` before it is decoded.
+
+A host with no provider binds `WebSearch.layerNoop`, and the call fails with
+`provider_unavailable`.
+
+`websearch` is not in `Manifest.readOnly`, because its provider contract
+requires `net:post` authority, which is mutating under the kernel capability
+taxonomy. A read-only tool surface gets `fetch` and `webfetch` instead.