@smthrs/std 0.0.0-stage → 1.0.0-rc.12

package/docs/guides/run-the-test-suite.md

@@ -0,0 +1,154 @@
+---
+title: "Run the test suite and attribute a failure"
+description: "Declare how the project under test runs its suite, select what to run, and use against:'base' to separate the failures your edit introduced from the ones that were already there."
+sidebar:
+  order: 5
+---
+
+The `test` flow answers three questions a shell command cannot: what the suite
+reported, which of the failures are yours, and whether the run failed about the
+code at all.
+
+## Declare the runner once
+
+A caller of `test` selects **which** tests, never **how** to run them. The
+invocation is a host declaration, bound through the `TestRunner` service:
+
+```ts
+import * as TestRunner from "@smthrs/std/TestRunner"
+
+const runner = TestRunner.layer({
+  command: "python -m pytest -q",
+  cwd: "/testbed",
+  root: "/workspace/repo",
+  container: "testbed",
+  env: { PYTHONDONTWRITEBYTECODE: "1" },
+  timeoutMs: 900_000
+})
+```
+
+| Field       | Meaning                                                              |
+| ----------- | -------------------------------------------------------------------- |
+| `command`   | The runner command line, with no test selection in it.               |
+| `cwd`       | Where the runner runs. The container's path when `container` is set. |
+| `root`      | The host path of the git repository, when it differs from `cwd`.     |
+| `container` | A container to route the run through, via the `Container` transport. |
+| `env`       | Environment the runner needs.                                        |
+| `baseRef`   | The ref whose commit is the pristine base.                           |
+| `timeoutMs` | Default wall-clock budget for one run.                               |
+
+`cwd` and `root` differ exactly when the runner runs in a container: the
+container sees the repository at `cwd`, the host sees it at `root`, and a
+baseline checkout created at `<root>/.flows-test-base/run-<lease>` is visible to the runner
+at `<cwd>/.flows-test-base/run-<lease>`, because it is one directory under two names.
+
+A host with no runner binds `TestRunner.layerNoop`. The flow then fails with
+`provider_unavailable` and tells the caller to use `bash` with the command the
+project's own documentation gives.
+
+## Run a selection
+
+```ts
+import * as TestRun from "@smthrs/std/TestRun"
+
+const result = TestRun.run({
+  selection: ["tests/test_admin.py::AdminViewBasicTest::test_change_list"]
+})
+```
+
+Selection entries reach the runner as arguments, never as text spliced into the
+command line. The invocation is `bash -lc '<command> "$@"' <command> id...`, so a
+test id holding `::`, `[`, `]`, spaces, or shell metacharacters stays data.
+Omitting `selection` runs everything.
+
+The result is a reading, not a log:
+
+| Field                               | Meaning                                                                    |
+| ----------------------------------- | -------------------------------------------------------------------------- |
+| `command`                           | The logical invocation, without container transport arguments.             |
+| `exitCode`                          | The runner's exit code.                                                    |
+| `passed`                            | Tests reported passing. Zero when `parsed` is false.                       |
+| `failed`                            | Ids of the tests reported failing or erroring.                             |
+| `parsed`                            | Whether the runner's complete report could be read.                        |
+| `tail`                              | The end of the runner's combined output.                                   |
+| `tailTruncated`, `tailDroppedBytes` | Whether `tail` is a fragment, and by how much.                             |
+| `invalidProbe`                      | Present when the exit code describes the command, not the code under test. |
+
+When `parsed` is false, `tail` is all there is, and base attribution is omitted
+rather than guessed at.
+
+Unittest skips and expected failures, and TAP skips and TODO failures, do not
+count as passing or failing tests. A successful TAP TODO counts as passing.
+Unittest unexpected successes count as failures; attribution requires their
+reported ids. Python 3.10 class-only headers and Python 3.11+ full test ids are
+both supported. Missing failure ids or unknown unittest summary fields leave
+`parsed` false.
+
+## Who the failure belongs to
+
+A non-zero exit is a runner's verdict about the code it ran, not about the
+command it was handed. A run that names a test, a file, a module, an
+environment, or a program that does not exist never reaches any code and still
+exits non-zero, so it reads the same before and after a correct fix.
+`invalidProbe` is present when this run was one of those, with a `reason` from
+that closed list and a `message` saying what the result does and does not
+prove.
+
+Jev makes that attribution. The flow asks the `probe/attribution` classifier
+through the `Evaluator` service of [`@smthrs/model`](/api/model), so a host
+binds an evaluator on a subscription seat. Only a zero exit is attributed without a judge. Exit codes 126 and 127
+also go through attribution: a runner can return them after executing tests.
+A judge that cannot
+be reached, refuses, times out, or answers something unusable fails the call
+with `provider_unavailable`, `timeout`, or `request_failed`, because a guess at
+whether a reproduction reproduced is worth less than no answer at all.
+
+## Attribute a failure to your own edit
+
+The native CLI supports workspace tests. Baseline comparison refuses before any
+command starts; isolated staging is tracked in
+[#3234](https://github.com/smithersai/smithers/issues/3234). A host declares this
+with `baseline: false`.
+
+`against: "base"` runs the same selection a second time against the pristine
+base commit, in a scratch checkout, and differences the two failure sets:
+
+```ts
+const result = TestRun.run({
+  selection: ["tests/test_admin.py"],
+  against: "base"
+})
+
+result.introduced // failing here and not on the base tree: yours
+result.preexisting // failing on both: not yours, and not worth investigating
+result.fixed // failing on the base tree and passing here
+result.base // the base run's own outcome, plus its ref and commit
+```
+
+That is the whole of attribution, in one call. Without it, the same question
+costs a revert of the very work it was meant to prove.
+
+The baseline commit is resolved from the runner's `baseRef`, then from
+`TestRunner.captureBase` (`refs/flows/capture-base`), then from `HEAD`. A
+`baseRef` that does not resolve is an error rather than a fallback, because a
+baseline against the wrong tree answers the attribution question wrong.
+
+The baseline is a detached checkout at `<root>/.flows-test-base/run-<lease>`, under
+`TestRun.scratchDirectory`. Each call gets a unique lease. Existing checkouts
+are left alone; a `SIGKILL` can leave one behind. Inside the repository is the only place that works:
+a runner reaching the repository through a container mount sees a scratch
+checkout anywhere else on the host as a path that does not exist. It is removed
+when the call ends, however it ends. It is written from a temporary shadow
+`GIT_DIR`, like a checkpoint, so no hook or filter driver in the workspace's
+`.git` runs on the host and the repository's format is never changed.
+
+`against: "base"` needs a repository directory. A runner declaring neither `root`
+nor `cwd` fails with `invalid_input`.
+
+## Bound the run
+
+`timeoutMs` on the call overrides the runner's own, which overrides
+`TestRun.DEFAULT_TIMEOUT_MS` (ten minutes). The run holds at most
+`TestRun.MAX_CAPTURE_BYTES` of output in memory, and `tail` carries the last
+30,000 bytes of it, so `tail` holds the part of a long run that says what
+failed. A timeout is a `timeout` failure, not a result.

package/docs/guides/search-a-workspace.md

@@ -0,0 +1,148 @@
+---
+title: "Search a workspace"
+description: "Find files with glob and search their contents with grep: roots, ordered include and exclude patterns, context lines, per-file caps, and the notices that keep an empty answer from lying."
+sidebar:
+  order: 3
+---
+
+Both flows go through the `Search` service, so bind either
+[`PortableSearch.layer` or `NativeSearch.layer`](./bind-the-standard-flows.md)
+first. The semantics are identical; see
+[The search contract](../concepts/search-contract.md) for why.
+
+## Choose the search root
+
+`root` defaults to `.`, resolved by the host filesystem. A guarded agent host
+resolves it inside its workspace. Pass a root to search another directory:
+
+```ts
+import * as Glob from "@smthrs/std/Glob"
+
+const files = Glob.run({ pattern: "**/*.ts", root: "/workspace" })
+// files.paths, files.total, files.truncated, files.notice
+```
+
+Every pattern is matched against a candidate's path **relative to that root**.
+An absolute pattern such as `/workspace/tests/**` is read as the root-relative
+path `workspace/tests/**`, which nothing under `/workspace` can match. When that
+happens, `notice` says so rather than returning a silent empty result.
+
+Three rules follow from root-relative matching:
+
+- No `/` in the pattern matches the basename at any depth: `*.ts` finds every
+  TypeScript file in the subtree.
+- A `/` in the pattern anchors it at the root: `src/*.ts` matches
+  `<root>/src/a.ts` and not `<root>/lib/src/a.ts`.
+- A leading `/` or `./` is the same anchor spelled out.
+
+`glob` returns at most 1,000 paths per call, and `limit` lowers that. Pass
+`hidden: true` to include dot files.
+
+Both flows apply `.gitignore` rules from the search root and nested directories,
+even outside Git repositories. `noIgnore` is an optional boolean defaulting to
+`false`; set `noIgnore: true` to include ignored paths. Rules above the root and
+global ignore files do not apply. Hidden-file and fixed directory skips still
+apply when `noIgnore` is true.
+
+## Search contents
+
+```ts
+import * as Grep from "@smthrs/std/Grep"
+
+const found = Grep.run({
+  pattern: "def widen",
+  root: "/workspace",
+  globs: ["**/*.py", "!**/test_*.py"],
+  context: 2,
+  maxCount: 3,
+  limit: 50
+})
+```
+
+| Input                                      | What it does                                                              |
+| ------------------------------------------ | ------------------------------------------------------------------------- |
+| `pattern`                                  | A Smithers Ripgrep ASCII v1 expression, or a literal with `fixedStrings`. |
+| `globs`                                    | Ordered include and exclude patterns, `!` for exclusion.                  |
+| `ignoreCase`, `smartCase`                  | Case handling. The two are mutually exclusive.                            |
+| `beforeContext`, `afterContext`, `context` | Context lines. `context` cannot be combined with the other two.           |
+| `maxCount`                                 | At most this many matches per file. At least 1.                           |
+| `filesWithMatches`                         | Report file names only.                                                   |
+| `hidden`                                   | Include dot files.                                                        |
+| `noIgnore`                                 | Include paths excluded by `.gitignore` when true. Defaults to false.      |
+| `symbols`                                  | Report the definition enclosing each hit. True by default.                |
+| `limit`                                    | Global match budget, capped at 200.                                       |
+
+The output is match-centric:
+
+| Output                | Meaning                                                                   |
+| --------------------- | ------------------------------------------------------------------------- |
+| `matches`             | Each hit with its own `before` and `after` context and optional `symbol`. |
+| `files`               | The file list, which is what `filesWithMatches` fills.                    |
+| `filesSearched`       | Every file the globs admitted, including binaries and unopenable files.   |
+| `skippedBinary`       | How many of those held a NUL byte.                                        |
+| `truncated`, `notice` | Whether the budget bit, and what was shown out of what there was.         |
+| `retriedAsLiteral`    | Present when these results came from re-running the pattern literally.    |
+
+`limit` counts matches, not rows, so context can never crowd out the hit it
+belongs to.
+
+## Read the definition a hit sits in
+
+Each hit carries the definition enclosing it, when the file's shape says so
+plainly. That is a read window, so the follow-up read is exact rather than a
+guess at `line - 10`:
+
+```ts
+import * as Read from "@smthrs/std/Read"
+
+const hit = found.matches[0]
+if (hit?.symbol !== undefined) {
+  const page = Read.run({
+    path: hit.file,
+    offset: hit.symbol.startLine,
+    limit: hit.symbol.endLine - hit.symbol.startLine + 1
+  })
+}
+```
+
+Set `symbols: false` to skip the annotation. When the shape is not plain, the
+heuristic answers nothing rather than guessing a window.
+
+## When a regular expression finds nothing
+
+A pattern full of metacharacters that finds nothing is the most common way a
+search lies. If a pattern containing metacharacters produces no matches and no
+files, the search runs again with `fixedStrings: true`, and if the literal
+reading finds something the result carries `retriedAsLiteral: true` and says so
+in `notice`. The retry costs one more pass only in the case that already found
+nothing.
+
+To search a literal on purpose, pass `fixedStrings: true` yourself and skip the
+regular expression grammar entirely.
+
+## What is refused
+
+These are `invalid_input` rather than a best guess, because two peer
+implementations must not each pick their own reading:
+
+- `ignoreCase` and `smartCase` both true.
+- `context` together with `beforeContext` or `afterContext`.
+- `maxCount` below 1.
+
+File-type registries are unsupported; there is no `types` input field.
+
+An unsupported pattern is `invalid_pattern`, with a message naming the
+construct. A `root` that does not exist is `not_found`.
+
+## What a walk never enters
+
+Twelve directory names are skipped during descent, including `.git`,
+`node_modules`, `__pycache__`, and `.venv`. Skipping applies to descent only, so
+naming one of them as the `root` searches it. An entry the process cannot read
+is skipped and the walk continues.
+
+The portable peer streams file contents when the host supports it. A guarded
+host may provide atomic file reads but refuse streaming handles. If a stream
+fails before delivering any bytes, search tries that same filesystem's atomic
+read. A stream that fails after delivering bytes is discarded without a second
+read, so two versions of a changing file cannot be combined into one result.

package/docs/installation.md

@@ -0,0 +1,115 @@
+---
+title: "Installation"
+description: "Install @smthrs/std, its runtime requirements and import forms, the browser-safe subpaths, and the packages a host adds to make the handlers runnable."
+sidebar:
+  order: 1
+---
+
+## Install the package
+
+```bash
+npm install @smthrs/std@next effect
+```
+
+The `next` tag carries the 1.0 release candidates.
+
+It requires
+Node.js 26.4.0 or later and ships as both ESM and CommonJS with TypeScript
+declarations. Its runtime dependencies install with it:
+[`effect`](https://effect.website), plus [`@smthrs/core`](/api/core) for the
+flow declaration type, [`@smthrs/kernel`](/api/kernel) for the permission-aware
+host services, [`@smthrs/capability`](/api/capability) for the capability
+vocabulary, and [`@smthrs/control`](/api/control) for the credential seam
+`ExaWebSearch` reads its API key through.
+
+## Import forms
+
+The root entry point re-exports every module as a namespace:
+
+```ts
+import { Grep, Manifest, Read, StdError } from "@smthrs/std"
+```
+
+Each module is also importable from its own subpath, which is the form this
+documentation uses and the form that keeps a bundle to what it needs:
+
+```ts
+import * as Read from "@smthrs/std/Read"
+import * as StdError from "@smthrs/std/StdError"
+```
+
+Two subpath forms are not public: `@smthrs/std/internal/*` and
+`@smthrs/std/<Module>/index`. Both are blocked in the export map and carry no
+promise. `@smthrs/std/package.json` is exported.
+
+## The browser-safe subset
+
+The root entry point is Node-only. It re-exports `NodeLanguageServer`, which
+imports `node:url`, so importing `@smthrs/std` in a browser bundle pulls a Node
+built-in in with it.
+
+Four subpaths are browser-safe, because nothing they import reaches a Node
+built-in:
+
+- `@smthrs/std/Grep`
+- `@smthrs/std/Glob`
+- `@smthrs/std/Search`
+- `@smthrs/std/PortableSearch`
+
+`PortableSearch` performs its walk through the injected `FileSystem` service, so
+a browser host that provides one gets working `grep` and `glob` with no external
+binary.
+
+## What a runnable host adds
+
+The declarations run nowhere on their own. Handlers ask for host services
+through Effect's context, so a composition adds a platform package that provides
+them:
+
+```bash
+pnpm add @effect/platform-node @effect/platform-node-shared
+```
+
+`NodeServices.layer` from `@effect/platform-node` provides `FileSystem`, `Path`,
+and `ChildProcessSpawner`, which is everything the filesystem, search, and shell
+handlers require. A host that wants the permission kernel's capability checks in
+front of those services builds them through
+[`@smthrs/kernel`](/api/kernel)'s `HostServices` instead, which is what the
+Smithers CLI does.
+
+| Handler                                     | Services it requires                                                           |
+| ------------------------------------------- | ------------------------------------------------------------------------------ |
+| `Read.run`, `Edit.run`                      | `FileSystem`                                                                   |
+| `Write.run`, `Ls.run`, `ApplyPatch.run`     | `FileSystem`, `Path`                                                           |
+| `Grep.run`, `Glob.run`                      | `Search`                                                                       |
+| `Bash.run`                                  | `ChildProcessSpawner`, `Path`, and `Container` when a call names one           |
+| `TestRun.run`                               | `ChildProcessSpawner`, `TestRunner`, and `Container` when the runner names one |
+| `ShellCommand.run`                          | `ChildProcessSpawner`                                                          |
+| `Fetch.run`, `HttpPost.run`, `WebFetch.run` | `HttpClient`, `GrantStore`                                                     |
+| `WebSearch.run`                             | `WebSearch`                                                                    |
+| `Lsp.run`                                   | `LanguageServer`                                                               |
+| `UpdatePlan.run`                            | none                                                                           |
+
+## The six injected services
+
+Six flows reach the host through a service rather than through a platform layer.
+Each one has a refusal layer for a host that cannot serve it, so a flow whose
+service is missing says so instead of succeeding quietly.
+
+| Service          | Refusal layer              | Working layer                                                                   |
+| ---------------- | -------------------------- | ------------------------------------------------------------------------------- |
+| `Search`         | `Search.layerNoop`         | `PortableSearch.layer` or `NativeSearch.layer`                                  |
+| `Container`      | `Container.layerNoop`      | `Container.layerCommand({ program: "docker" })`                                 |
+| `TestRunner`     | `TestRunner.layerNoop`     | `TestRunner.layer(runner)`                                                      |
+| `Checkpoints`    | `Checkpoints.layerNoop`    | `Checkpoints.layerGit({ root })` or `Checkpoints.layerContainer({ container })` |
+| `WebSearch`      | `WebSearch.layerNoop`      | `ExaWebSearch.layer(credentialId)`                                              |
+| `LanguageServer` | `LanguageServer.layerNoop` | `NodeLanguageServer.layer(config)`                                              |
+
+`NativeSearch.layer` drives the `rg` executable, so a host that binds it needs
+ripgrep on the path. `PortableSearch.layer` needs nothing beyond `FileSystem`
+and `Path`. `ExaWebSearch.layer` additionally requires
+[`@smthrs/control`](/api/control)'s `Credential` service and
+[`@smthrs/kernel`](/api/kernel)'s `HttpClient`.
+
+Composing these is one job, covered in
+[Bind the standard flows into a host](./guides/bind-the-standard-flows.md).

package/docs/quickstart.md

@@ -0,0 +1,125 @@
+---
+title: "Quickstart"
+description: "Search a real directory with grep, then read the exact definition the hit sits in, using the in-process search peer and no external binary."
+sidebar:
+  order: 2
+---
+
+This quickstart runs two standard flows end to end against a real directory.
+`grep` finds a line and reports the definition that encloses it; `read` turns
+that definition's line range into a page. By the end you will have taken the two
+steps an agent takes hundreds of times a run, and you will have taken them
+without an external binary and without an API key.
+
+## Prerequisites
+
+- Node.js 26.4.0 or later.
+- A project with the packages installed: `npm install @smthrs/std@next @effect/platform-node effect`.
+
+## Make a tree to search
+
+Two files, so the glob filter has something to exclude:
+
+```bash
+mkdir -p /tmp/std-quickstart/src
+printf 'export function widen(value: number): number {\n  return value * 2\n}\n' > /tmp/std-quickstart/src/widen.ts
+printf 'widen is defined in src/widen.ts\n' > /tmp/std-quickstart/README.md
+```
+
+## Compose the host
+
+`Grep.run` asks for the `Search` service; `Read.run` asks for `FileSystem`.
+`PortableSearch` is the in-process implementation of `Search`, so the whole
+composition is two layers:
+
+```ts
+import { NodeServices } from "@effect/platform-node"
+import * as PortableSearch from "@smthrs/std/PortableSearch"
+import * as Layer from "effect/Layer"
+
+/** FileSystem, Path, and ChildProcessSpawner, from the Node platform. */
+const platform = NodeServices.layer
+
+/** The in-process Search peer, built over that platform. */
+const search = PortableSearch.layer.pipe(Layer.provide(platform))
+
+const host = Layer.merge(platform, search)
+```
+
+Swapping `PortableSearch.layer` for `NativeSearch.layer` runs the same calls
+through the `rg` executable instead. Nothing else in this file changes, which is
+the point of the seam. See [The search contract](./concepts/search-contract.md).
+
+## Search, then read what you found
+
+Each returned hit carries the definition it sits inside, when the file's shape
+says so plainly. That is a line range, so it is a read window rather than a
+guess:
+
+```ts
+import * as Grep from "@smthrs/std/Grep"
+import * as Read from "@smthrs/std/Read"
+import * as Effect from "effect/Effect"
+
+const program = Effect.gen(function*() {
+  const found = yield* Grep.run({
+    pattern: "widen",
+    root: "/tmp/std-quickstart",
+    globs: ["*.ts"]
+  })
+  const hit = found.matches[0]
+  if (hit === undefined) return "no match"
+  console.log(`${hit.file}:${hit.line}  ${hit.text}`)
+  console.log(`searched ${found.filesSearched} file(s)`)
+
+  const symbol = hit.symbol
+  if (symbol === undefined) return "no enclosing definition"
+  const page = yield* Read.run({
+    path: hit.file,
+    offset: symbol.startLine,
+    limit: symbol.endLine - symbol.startLine + 1
+  })
+  return `${symbol.kind} ${symbol.name}\n${page.content}`
+})
+
+console.log(await Effect.runPromise(Effect.provide(program, host)))
+```
+
+Run the file with your TypeScript runner. The output is the hit, the file count,
+and the whole definition:
+
+```text
+/tmp/std-quickstart/src/widen.ts:1  export function widen(value: number): number {
+searched 1 file(s)
+function widen
+export function widen(value: number): number {
+  return value * 2
+}
+```
+
+`filesSearched` is 1 because `globs: ["*.ts"]` excluded `README.md`. A glob
+without a `/` matches a basename at any depth, so `*.ts` reaches
+`src/widen.ts` without naming the directory.
+
+## What just happened
+
+- `page.content` is raw file text. The line numbers came back as the sibling
+  fields `startLine`, `endLine`, and `totalLines`, never as a gutter inside the
+  text, so any line of `content` can be pasted straight into `edit` as an
+  anchor.
+- `hit.symbol` is a heuristic that reports only what a file's shape says
+  plainly. When it cannot read a definition, it answers `undefined` rather than
+  guessing a window.
+- Neither call took a limit from you, and both applied one: `grep` returns at
+  most 200 matches and `read` at most 2,000 lines. Whenever a limit bites, the
+  result says so in `truncated` and `notice`. See
+  [Limits are disclosed, never silent](./concepts/limits-and-disclosure.md).
+
+## Next steps
+
+- [Search a workspace](./guides/search-a-workspace.md) for globs, context lines,
+  case rules, and the literal retry.
+- [Read and edit a file](./guides/read-and-edit-a-file.md) for the two anchor
+  forms `edit` accepts and what a miss reports.
+- [Bind the standard flows into a host](./guides/bind-the-standard-flows.md) to
+  offer all 17 flows to an agent instead of calling two by hand.