@smthrs/std 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/search-a-workspace.md

@@ -0,0 +1,148 @@
+---
+title: "Search a workspace"
+description: "Find files with glob and search their contents with grep: roots, ordered include and exclude patterns, context lines, per-file caps, and the notices that keep an empty answer from lying."
+sidebar:
+  order: 3
+---
+
+Both flows go through the `Search` service, so bind either
+[`PortableSearch.layer` or `NativeSearch.layer`](./bind-the-standard-flows.md)
+first. The semantics are identical; see
+[The search contract](../concepts/search-contract.md) for why.
+
+## Choose the search root
+
+`root` defaults to `.`, resolved by the host filesystem. A guarded agent host
+resolves it inside its workspace. Pass a root to search another directory:
+
+```ts
+import * as Glob from "@smthrs/std/Glob"
+
+const files = Glob.run({ pattern: "**/*.ts", root: "/workspace" })
+// files.paths, files.total, files.truncated, files.notice
+```
+
+Every pattern is matched against a candidate's path **relative to that root**.
+An absolute pattern such as `/workspace/tests/**` is read as the root-relative
+path `workspace/tests/**`, which nothing under `/workspace` can match. When that
+happens, `notice` says so rather than returning a silent empty result.
+
+Three rules follow from root-relative matching:
+
+- No `/` in the pattern matches the basename at any depth: `*.ts` finds every
+  TypeScript file in the subtree.
+- A `/` in the pattern anchors it at the root: `src/*.ts` matches
+  `<root>/src/a.ts` and not `<root>/lib/src/a.ts`.
+- A leading `/` or `./` is the same anchor spelled out.
+
+`glob` returns at most 1,000 paths per call, and `limit` lowers that. Pass
+`hidden: true` to include dot files.
+
+Both flows apply `.gitignore` rules from the search root and nested directories,
+even outside Git repositories. `noIgnore` is an optional boolean defaulting to
+`false`; set `noIgnore: true` to include ignored paths. Rules above the root and
+global ignore files do not apply. Hidden-file and fixed directory skips still
+apply when `noIgnore` is true.
+
+## Search contents
+
+```ts
+import * as Grep from "@smthrs/std/Grep"
+
+const found = Grep.run({
+  pattern: "def widen",
+  root: "/workspace",
+  globs: ["**/*.py", "!**/test_*.py"],
+  context: 2,
+  maxCount: 3,
+  limit: 50
+})
+```
+
+| Input                                      | What it does                                                              |
+| ------------------------------------------ | ------------------------------------------------------------------------- |
+| `pattern`                                  | A Smithers Ripgrep ASCII v1 expression, or a literal with `fixedStrings`. |
+| `globs`                                    | Ordered include and exclude patterns, `!` for exclusion.                  |
+| `ignoreCase`, `smartCase`                  | Case handling. The two are mutually exclusive.                            |
+| `beforeContext`, `afterContext`, `context` | Context lines. `context` cannot be combined with the other two.           |
+| `maxCount`                                 | At most this many matches per file. At least 1.                           |
+| `filesWithMatches`                         | Report file names only.                                                   |
+| `hidden`                                   | Include dot files.                                                        |
+| `noIgnore`                                 | Include paths excluded by `.gitignore` when true. Defaults to false.      |
+| `symbols`                                  | Report the definition enclosing each hit. True by default.                |
+| `limit`                                    | Global match budget, capped at 200.                                       |
+
+The output is match-centric:
+
+| Output                | Meaning                                                                   |
+| --------------------- | ------------------------------------------------------------------------- |
+| `matches`             | Each hit with its own `before` and `after` context and optional `symbol`. |
+| `files`               | The file list, which is what `filesWithMatches` fills.                    |
+| `filesSearched`       | Every file the globs admitted, including binaries and unopenable files.   |
+| `skippedBinary`       | How many of those held a NUL byte.                                        |
+| `truncated`, `notice` | Whether the budget bit, and what was shown out of what there was.         |
+| `retriedAsLiteral`    | Present when these results came from re-running the pattern literally.    |
+
+`limit` counts matches, not rows, so context can never crowd out the hit it
+belongs to.
+
+## Read the definition a hit sits in
+
+Each hit carries the definition enclosing it, when the file's shape says so
+plainly. That is a read window, so the follow-up read is exact rather than a
+guess at `line - 10`:
+
+```ts
+import * as Read from "@smthrs/std/Read"
+
+const hit = found.matches[0]
+if (hit?.symbol !== undefined) {
+  const page = Read.run({
+    path: hit.file,
+    offset: hit.symbol.startLine,
+    limit: hit.symbol.endLine - hit.symbol.startLine + 1
+  })
+}
+```
+
+Set `symbols: false` to skip the annotation. When the shape is not plain, the
+heuristic answers nothing rather than guessing a window.
+
+## When a regular expression finds nothing
+
+A pattern full of metacharacters that finds nothing is the most common way a
+search lies. If a pattern containing metacharacters produces no matches and no
+files, the search runs again with `fixedStrings: true`, and if the literal
+reading finds something the result carries `retriedAsLiteral: true` and says so
+in `notice`. The retry costs one more pass only in the case that already found
+nothing.
+
+To search a literal on purpose, pass `fixedStrings: true` yourself and skip the
+regular expression grammar entirely.
+
+## What is refused
+
+These are `invalid_input` rather than a best guess, because two peer
+implementations must not each pick their own reading:
+
+- `ignoreCase` and `smartCase` both true.
+- `context` together with `beforeContext` or `afterContext`.
+- `maxCount` below 1.
+
+File-type registries are unsupported; there is no `types` input field.
+
+An unsupported pattern is `invalid_pattern`, with a message naming the
+construct. A `root` that does not exist is `not_found`.
+
+## What a walk never enters
+
+Twelve directory names are skipped during descent, including `.git`,
+`node_modules`, `__pycache__`, and `.venv`. Skipping applies to descent only, so
+naming one of them as the `root` searches it. An entry the process cannot read
+is skipped and the walk continues.
+
+The portable peer streams file contents when the host supports it. A guarded
+host may provide atomic file reads but refuse streaming handles. If a stream
+fails before delivering any bytes, search tries that same filesystem's atomic
+read. A stream that fails after delivering bytes is discarded without a second
+read, so two versions of a changing file cannot be combined into one result.

package/docs/installation.md

@@ -0,0 +1,111 @@
+---
+title: "Installation"
+description: "Install @smthrs/std, its runtime requirements and import forms, the browser-safe subpaths, and the packages a host adds to make the handlers runnable."
+sidebar:
+  order: 1
+---
+
+## Install the package
+
+Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+It requires
+Node.js 26.4.0 or later and ships as both ESM and CommonJS with TypeScript
+declarations. Its runtime dependencies install with it:
+[`effect`](https://effect.website), plus [`@smthrs/core`](/api/core) for the
+flow declaration type, [`@smthrs/kernel`](/api/kernel) for the permission-aware
+host services, [`@smthrs/capability`](/api/capability) for the capability
+vocabulary, and [`@smthrs/control`](/api/control) for the credential seam
+`ExaWebSearch` reads its API key through.
+
+## Import forms
+
+The root entry point re-exports every module as a namespace:
+
+```ts
+import { Grep, Manifest, Read, StdError } from "@smthrs/std"
+```
+
+Each module is also importable from its own subpath, which is the form this
+documentation uses and the form that keeps a bundle to what it needs:
+
+```ts
+import * as Read from "@smthrs/std/Read"
+import * as StdError from "@smthrs/std/StdError"
+```
+
+Two subpath forms are not public: `@smthrs/std/internal/*` and
+`@smthrs/std/<Module>/index`. Both are blocked in the export map and carry no
+promise. `@smthrs/std/package.json` is exported.
+
+## The browser-safe subset
+
+The root entry point is Node-only. It re-exports `NodeLanguageServer`, which
+imports `node:url`, so importing `@smthrs/std` in a browser bundle pulls a Node
+built-in in with it.
+
+Four subpaths are browser-safe, because nothing they import reaches a Node
+built-in:
+
+- `@smthrs/std/Grep`
+- `@smthrs/std/Glob`
+- `@smthrs/std/Search`
+- `@smthrs/std/PortableSearch`
+
+`PortableSearch` performs its walk through the injected `FileSystem` service, so
+a browser host that provides one gets working `grep` and `glob` with no external
+binary.
+
+## What a runnable host adds
+
+The declarations run nowhere on their own. Handlers ask for host services
+through Effect's context, so a composition adds a platform package that provides
+them:
+
+```bash
+pnpm add @effect/platform-node@4.0.0-rc.115 @effect/platform-node-shared@4.0.0-rc.115
+```
+
+`NodeServices.layer` from `@effect/platform-node` provides `FileSystem`, `Path`,
+and `ChildProcessSpawner`, which is everything the filesystem, search, and shell
+handlers require. A host that wants the permission kernel's capability checks in
+front of those services builds them through
+[`@smthrs/kernel`](/api/kernel)'s `HostServices` instead, which is what the
+Smithers CLI does.
+
+| Handler                                     | Services it requires                                                           |
+| ------------------------------------------- | ------------------------------------------------------------------------------ |
+| `Read.run`, `Edit.run`                      | `FileSystem`                                                                   |
+| `Write.run`, `Ls.run`, `ApplyPatch.run`     | `FileSystem`, `Path`                                                           |
+| `Grep.run`, `Glob.run`                      | `Search`                                                                       |
+| `Bash.run`                                  | `ChildProcessSpawner`, `Path`, and `Container` when a call names one           |
+| `TestRun.run`                               | `ChildProcessSpawner`, `TestRunner`, and `Container` when the runner names one |
+| `ShellCommand.run`                          | `ChildProcessSpawner`                                                          |
+| `Fetch.run`, `HttpPost.run`, `WebFetch.run` | `HttpClient`, `GrantStore`                                                     |
+| `WebSearch.run`                             | `WebSearch`                                                                    |
+| `Lsp.run`                                   | `LanguageServer`                                                               |
+| `UpdatePlan.run`                            | none                                                                           |
+
+## The six injected services
+
+Six flows reach the host through a service rather than through a platform layer.
+Each one has a refusal layer for a host that cannot serve it, so a flow whose
+service is missing says so instead of succeeding quietly.
+
+| Service          | Refusal layer              | Working layer                                   |
+| ---------------- | -------------------------- | ----------------------------------------------- |
+| `Search`         | `Search.layerNoop`         | `PortableSearch.layer` or `NativeSearch.layer`  |
+| `Container`      | `Container.layerNoop`      | `Container.layerCommand({ program: "docker" })` |
+| `TestRunner`     | `TestRunner.layerNoop`     | `TestRunner.layer(runner)`                      |
+| `Checkpoints`    | `Checkpoints.layerNoop`    | `Checkpoints.layerGit({ root })`                |
+| `WebSearch`      | `WebSearch.layerNoop`      | `ExaWebSearch.layer(credentialId)`              |
+| `LanguageServer` | `LanguageServer.layerNoop` | `NodeLanguageServer.layer(config)`              |
+
+`NativeSearch.layer` drives the `rg` executable, so a host that binds it needs
+ripgrep on the path. `PortableSearch.layer` needs nothing beyond `FileSystem`
+and `Path`. `ExaWebSearch.layer` additionally requires
+[`@smthrs/control`](/api/control)'s `Credential` service and
+[`@smthrs/kernel`](/api/kernel)'s `HttpClient`.
+
+Composing these is one job, covered in
+[Bind the standard flows into a host](./guides/bind-the-standard-flows.md).

package/docs/quickstart.md

@@ -0,0 +1,125 @@
+---
+title: "Quickstart"
+description: "Search a real directory with grep, then read the exact definition the hit sits in, using the in-process search peer and no external binary."
+sidebar:
+  order: 2
+---
+
+This quickstart runs two standard flows end to end against a real directory.
+`grep` finds a line and reports the definition that encloses it; `read` turns
+that definition's line range into a page. By the end you will have taken the two
+steps an agent takes hundreds of times a run, and you will have taken them
+without an external binary and without an API key.
+
+## Prerequisites
+
+- Node.js 26.4.0 or later.
+- A package with the dependencies installed. Not on npm yet; see [Installation](https://github.com/smithersai/smithers/blob/main/packages/smithers/flows/flow/docs/installation.md#use-the-libraries).
+
+## Make a tree to search
+
+Two files, so the glob filter has something to exclude:
+
+```bash
+mkdir -p /tmp/std-quickstart/src
+printf 'export function widen(value: number): number {\n  return value * 2\n}\n' > /tmp/std-quickstart/src/widen.ts
+printf 'widen is defined in src/widen.ts\n' > /tmp/std-quickstart/README.md
+```
+
+## Compose the host
+
+`Grep.run` asks for the `Search` service; `Read.run` asks for `FileSystem`.
+`PortableSearch` is the in-process implementation of `Search`, so the whole
+composition is two layers:
+
+```ts
+import { NodeServices } from "@effect/platform-node"
+import * as PortableSearch from "@smthrs/std/PortableSearch"
+import * as Layer from "effect/Layer"
+
+/** FileSystem, Path, and ChildProcessSpawner, from the Node platform. */
+const platform = NodeServices.layer
+
+/** The in-process Search peer, built over that platform. */
+const search = PortableSearch.layer.pipe(Layer.provide(platform))
+
+const host = Layer.merge(platform, search)
+```
+
+Swapping `PortableSearch.layer` for `NativeSearch.layer` runs the same calls
+through the `rg` executable instead. Nothing else in this file changes, which is
+the point of the seam. See [The search contract](./concepts/search-contract.md).
+
+## Search, then read what you found
+
+Each returned hit carries the definition it sits inside, when the file's shape
+says so plainly. That is a line range, so it is a read window rather than a
+guess:
+
+```ts
+import * as Grep from "@smthrs/std/Grep"
+import * as Read from "@smthrs/std/Read"
+import * as Effect from "effect/Effect"
+
+const program = Effect.gen(function*() {
+  const found = yield* Grep.run({
+    pattern: "widen",
+    root: "/tmp/std-quickstart",
+    globs: ["*.ts"]
+  })
+  const hit = found.matches[0]
+  if (hit === undefined) return "no match"
+  console.log(`${hit.file}:${hit.line}  ${hit.text}`)
+  console.log(`searched ${found.filesSearched} file(s)`)
+
+  const symbol = hit.symbol
+  if (symbol === undefined) return "no enclosing definition"
+  const page = yield* Read.run({
+    path: hit.file,
+    offset: symbol.startLine,
+    limit: symbol.endLine - symbol.startLine + 1
+  })
+  return `${symbol.kind} ${symbol.name}\n${page.content}`
+})
+
+console.log(await Effect.runPromise(Effect.provide(program, host)))
+```
+
+Run the file with your TypeScript runner. The output is the hit, the file count,
+and the whole definition:
+
+```text
+/tmp/std-quickstart/src/widen.ts:1  export function widen(value: number): number {
+searched 1 file(s)
+function widen
+export function widen(value: number): number {
+  return value * 2
+}
+```
+
+`filesSearched` is 1 because `globs: ["*.ts"]` excluded `README.md`. A glob
+without a `/` matches a basename at any depth, so `*.ts` reaches
+`src/widen.ts` without naming the directory.
+
+## What just happened
+
+- `page.content` is raw file text. The line numbers came back as the sibling
+  fields `startLine`, `endLine`, and `totalLines`, never as a gutter inside the
+  text, so any line of `content` can be pasted straight into `edit` as an
+  anchor.
+- `hit.symbol` is a heuristic that reports only what a file's shape says
+  plainly. When it cannot read a definition, it answers `undefined` rather than
+  guessing a window.
+- Neither call took a limit from you, and both applied one: `grep` returns at
+  most 200 matches and `read` at most 2,000 lines. Whenever a limit bites, the
+  result says so in `truncated` and `notice`. See
+  [Limits are disclosed, never silent](./concepts/limits-and-disclosure.md).
+
+## Next steps
+
+- [Search a workspace](./guides/search-a-workspace.md) for globs, context lines,
+  case rules, and the literal retry.
+- [Read and edit a file](./guides/read-and-edit-a-file.md) for the two anchor
+  forms `edit` accepts and what a miss reports.
+- [Bind the standard flows into a host](./guides/bind-the-standard-flows.md) to
+  offer all 17 flows to an agent instead of calling two by hand.