valise-sh 0.2.0 → 0.3.0

package/docs/contrib-modules.md

@@ -1,246 +1,200 @@
-# Contributing Modules
+# Writing modules
 
-This guide covers modules loaded by the valise-sh engine. A module is
-usually a metadata file and a shell body; consumer projects may add their
-own naming, review, and release rules.
+A module is a metadata file and, usually, a shell body. This guide covers
+writing one in a valise project, checking and testing it, and deciding
+whether it belongs in a shareable layer. [module-abi.md](module-abi.md) is
+the full metadata contract; [projects.md](projects.md) covers bundles,
+variants, and builds.
 
-## Module structure
-
-A file-backed module has a metadata file and a body. POSIX bodies usually
-end in `.sh`; Fish bodies end in `.fish`:
+## Create a module
 
+```sh
+valise new demo
+cd demo
+valise module new hello --exports=hello
 ```
-module.d/<name>.meta
-modules/<area>/<name>.sh
+
+`valise module new` writes `module.d/hello.meta` and `modules/hello.sh` and
+adds `hello` to the `requires=` of the bundle's POSIX root, so a build
+loads it. When the default bundle has several POSIX roots, name the ones to
+change with `--into=<module>[,<module>...]`, or pass `--no-wire` to wire
+the module yourself. Edit the body, then:
+
+```sh
+valise lint     # check metadata, bundles, variants, and syntax
+valise apply    # build, install, write the integration, verify
 ```
 
-The metadata file is read by the engine before any body is sourced. It uses
-one `key=value` field per line:
+A module's metadata looks like this:
 
 ```text
-name=example_tool
-requires=
+name=hello
+version=1.0.0
+side_effect=read-only
+requires=valise_path
 optional=
-shells=sh bash zsh ksh
+shells=posix
 platforms=any
-side_effect=read-only
-exports=example_tool_loaded
-updates=
-version=1.0.0
-load=modules/system/example_tool.sh
+exports=hello
+commands=hello
+tests=tests/hello.sh
+load=modules/hello.sh
 ```
 
-### Metadata fields
-
-| Field | Required | Description |
-|---|---|---|
-| `name` | Yes | Stable identifier matching `[a-z_][A-Za-z0-9_]*` |
-| `requires` | No | Space-separated hard dependencies (loaded first) |
-| `optional` | No | Space-separated best-effort dependencies |
-| `shells` | Yes | Supported shells: `sh`, `bash`, `zsh`, `ksh`, `fish`. `sh` means every POSIX shell (sh, bash, zsh, ksh, dash) |
-| `platforms` | Yes | `any`, `darwin`, `linux`, or BSD family values |
-| `side_effect` | Yes | One of: `read-only`, `path-env`, `generated-file`, `update-network`, `install-time`, `local-secret`, `local-state`, `work-only` |
-| `exports` | No | Public functions this module declares |
-| `updates` | No | Update registration identifiers mapped in the project update manifest |
-| `version` | No | Module version (digit-first, digits/letters/`.`/`_`/`-`); different versions of one name across roots fail closed with a skew diagnostic |
-| `load` | Conditionally | Repository-relative path to the module body |
-
-At least one of `load` or `requires` must be present. `load` and `requires`
-are not mutually exclusive — a module
-may declare both and will load its dependencies first, then its own body. A
-module with `requires` and no load target is a **meta module**: it names
-dependencies and has no body. See
-[module-abi.md](module-abi.md#meta-modules).
-
-### Naming
-
-Modules that live only in a layer root (see
-[module-abi.md](module-abi.md) multi-root search path) must carry the
-`contrib_` name prefix; the loader rejects unprefixed layer-only modules
-fail closed. Modules in the primary root have no prefix requirement.
-Same-name modules across roots must be byte-identical or the load fails
-closed naming both paths.
-
-## Engine capabilities
-
-Use `sp_cap_*` functions from `lib/capabilities.sh` instead of open-coding
-platform-specific commands. The POSIX entrypoint sources the capability
-library before the loader, so module bodies can call these directly.
-
-**The helpers stay available after the profile loads.** A compiled plan
-sources the capability prelude and leaves it in place, so a function your
-module defines can call `sp_cap_os` or any other helper long after the plan
-finished loading. Both the compiled plan and the loader keep the helpers
-available at runtime.
-
-Available capabilities include:
-
-- `sp_cap_canonical_path <path>` — resolve absolute path through symlinks
-- `sp_cap_path_under <base> <target>` — check if target is under base
-- `sp_cap_file_mtime <path>` — portable modification time (epoch seconds)
-- `sp_cap_mktemp_dir [prefix] [parent]` — secure temporary directory
-- `sp_cap_mktemp_file <parent> <prefix>` — secure temporary file
-- `sp_cap_homebrew_prefix` — locate Homebrew installation
-- `sp_cap_command_exists <name>` — portable command existence check
-- `sp_cap_command_path <name>` — resolve command to absolute path
-- `sp_cap_os` — print normalized OS name
-- `sp_cap_cpu_count` — print available CPU count
-
-See [module-abi.md](module-abi.md) for the full capability contract and
-[architecture.md](architecture.md) for scope. Shared modules — the ones
-other people consume — carry additional governance: namespace ownership,
-review, deprecation, and removal are policy questions, and the written
-policy plus the curated index lives in
-[shared-modules.md](shared-modules.md).
-
-## Authoring workflow
-
-### In a consuming project
-
-Inside a project that uses the CLI, the CLI creates the files for you:
+Choose `shells` by what the body needs:
+
+- `posix` for a body that runs in every POSIX shell: dash, BusyBox ash,
+  bash, zsh, and ksh.
+- `sh` for a body that runs only in the generic class (dash, ash, and other
+  plain `sh` implementations), listed together with the explicit shells it
+  also supports: `shells=sh bash ksh` loads in dash, bash, and ksh, and not
+  in zsh.
+- `bash`, `zsh`, or `ksh` for a body that uses that shell's features.
+- `fish` for a native Fish body (`.fish`). A Fish module never runs a POSIX
+  body, so a feature that needs both shells is two modules.
+
+Zsh runs module functions with the user's options. A body that relies on
+word splitting of unquoted expansions behaves differently in zsh, where it
+is off by default; quote expansions, split explicitly, or leave zsh out of
+`shells`.
+
+`side_effect` says what the body does when sourced. Startup modules use
+`read-only`, `path-env`, `local-state` (writes under the XDG state
+directory), `local-secret` (reads secrets that live on this machine), or
+`work-only`. A build refuses `generated-file`, `update-network`, and
+`install-time` in a startup closure.
+
+List every public function in `exports`. The plan refuses to load a module
+whose export already exists, checks that each export is defined after the
+body runs, and removes them again when the module fails. List exports that
+should also run from scripts and other programs in `commands`; see
+[module-abi.md](module-abi.md#published-commands).
+
+Use the `sp_cap_*` helpers from the capability prelude instead of
+platform-specific commands. They stay defined after the profile loads, so
+your functions can call them later:
+
+- `sp_cap_os`, `sp_cap_cpu_count`, `sp_cap_homebrew_prefix`
+- `sp_cap_command_exists <name>`, `sp_cap_command_path <name>`
+- `sp_cap_canonical_path <path>`, `sp_cap_path_under <base> <target>`
+- `sp_cap_file_mtime <path>`
+- `sp_cap_secure_local_bin_check <home> [reject|allow-owned-symlinks]`
+- `sp_cap_mktemp_dir [prefix] [parent]`, `sp_cap_mktemp_file <parent> <prefix>`
+
+`sp_cap_secure_local_bin_check` defaults to `reject`, which refuses every
+symlink in `~/.local/bin`. `allow-owned-symlinks` accepts a symlink whose
+link entry is owned by the current user (the pipx and uv shape) while still
+rejecting unowned entries and group- or other-writable regular files.
+
+The engine also ships modules a project can require: `valise_path`
+(`valise_path_prepend` and `valise_path_append`), `valise_editor`,
+`valise_history`, `valise_pager`, `valise_xdg`, and the `valise_platform_*`
+modules. [shared-modules.md](shared-modules.md) lists them.
+
+## Checking and testing modules
+
+`valise lint [--require-parsers] [module...]` checks the project and reports
+every problem it finds:
+
+- each metadata record, its name, its prefix, and the files it names;
+- each layer: `contrib_` names, `version=`, safe file modes, and
+  dependencies that stay inside the layer or the engine;
+- the `lint_layer_forbid` pattern from `valise.conf`, if set;
+- every bundle compiled for every variant and shell, which catches unknown
+  roots and exclusions, missing dependencies, and exclusions that break a
+  remaining module;
+- the syntax of each body and test, with the parser of every shell the module
+  declares (`dash` or a plain `sh`, `bash -n`, `zsh -n`, `ksh -n`, and
+  `fish --no-execute`). A missing parser is reported as a skip, or as a
+  failure with `--require-parsers`.
+
+`valise test [--shell=<class>] [--require-shells] [module...]` runs each
+module's `tests=` scripts. For every shell class the module declares whose
+interpreter is installed, it builds a release with the module as its only
+root, starts the shell with `env -i` and a fresh `HOME`, `TMPDIR`, and XDG
+tree, loads the module in strict mode, and runs the test. `PATH` is
+`${VALISE_TEST_PATH:-/usr/bin:/bin:/usr/sbin:/sbin}` plus the interpreter's
+directory. A test passes when it exits 0:
 
 ```sh
-valise new demo                 # scaffold a project
-cd demo
-valise module new hello --exports=hello
+# tests/hello.sh
+[ "$(hello world)" = 'hello world' ] || { echo "got: $(hello world)"; exit 1; }
 ```
 
-`valise module new` writes `shell/module.d/hello.meta` and
-`shell/modules/hello.sh`, and wires `hello` into the seed aggregator's
-`requires=` so the module is actually reachable. It refuses a duplicate
-export name. Edit the body, then:
+The output names each run (`ok hello [bash] tests/hello.sh`), shows a
+failing test's output, and ends with a count; any failure makes the command
+fail. A test that needs a file before the module loads (a secret, a config
+file) writes it and calls `valise_test_reload`, which loads the module again.
+A test that starts another shell loads the module there from
+`VALISE_TEST_PLAN`, the plan the test was loaded with, with
+`VALISE_MODULE_ROOT` set to `VALISE_TEST_RELEASE`:
 
 ```sh
-valise build                    # build to build/release
-valise apply                    # build, install, integrate, verify
+bash -c 'VALISE_MODULE_ROOT=$VALISE_TEST_RELEASE; . "$VALISE_TEST_PLAN"; hello world'
 ```
 
-You never hand-write the registry: `valise build` and `valise apply`
-regenerate `shell/release/module-registry.tsv` from the metadata.
-
-### In the engine repository
-
-1. Write the two files. Start from an existing pair — `module.d/contrib_pager.meta`
-   plus `modules/system/contrib_pager.sh` is the smallest real example.
-2. Validate the metadata as data:
-
-   ```sh
-   . ./lib/metadata.sh
-   sp_metadata_read module.d/example_tool.meta
-   ```
-
-   The reader prints the normalized record or a precise diagnostic
-   (`invalid side_effect`, `unknown metadata field`, and so on). It never
-   sources the body.
-3. Load it through the entrypoint and call the export:
-
-   ```sh
-   VALISE_MODULE_ROOT="$PWD" \
-   VALISE_MODULES='example_tool' \
-   VALISE_MODE=strict \
-   . ./entrypoints/profile.sh
-   example_tool_loaded
-   ```
-
-   Strict mode fails closed on any problem: missing dependency, a declared
-   export missing after the body runs, an export that already exists,
-   reserved-utility shadowing, shell/platform mismatch. The loader checks
-   what `exports=` declares; it does not reject a body that defines extra
-   private functions, so list everything you consider public. Under zsh
-   the entrypoint handles word-splitting itself; source it the same way.
-4. Check parser coverage. `manifests/parser-support.tsv` declares which
-   POSIX shells must parse each file class. Place portable bodies under a
-   path pattern the table already covers (for example `modules/system/*.sh`)
-   or add a row for your pattern, then run the gate:
-
-   ```sh
-   make test-parser-support
-   ```
-
-   This target belongs to the engine's source repository. The npm package
-   ships the runtime and the docs, not the Makefile or the parser table, so
-   a package consumer checks parser coverage by hand: run every POSIX shell
-   you claim to support over the body (`sh -n`, `bash -n`, `zsh -n`,
-   `ksh -n`) and confirm each one accepts it.
-
-5. Publish a release containing your module tree and prove the whole
-   path end to end (make-manifest, install, release verify):
-
-   ```sh
-   valise make-manifest ./mytree
-   cd mytree && valise install && valise release verify
-   ```
-
-   See [installation.md](installation.md) for release integrity and
-   provenance.
-
-## Shipped modules
-
-The reusable `contrib_` modules are listed in
-[shared-modules.md](shared-modules.md). For example, load PATH and editor
-defaults from a checkout with:
+Tests see `VALISE_TEST_ROOT` (the scratch directory, which holds `HOME`),
+`VALISE_TEST_MODULE`, `VALISE_TEST_MODULE_DIR`, `VALISE_TEST_SHELL`,
+`VALISE_TEST_RELEASE`, `VALISE_TEST_PLAN` (`runtime/fish/profile.fish` in
+Fish), and `VALISE_PROJECT_ROOT`. Fish tests (`.fish`) run only in Fish; other tests
+run only in POSIX shells.
 
-```sh
-VALISE_MODULE_ROOT="$PWD" \
-VALISE_MODULES='contrib_path contrib_editor' \
-VALISE_MODE=strict \
-. ./entrypoints/profile.sh
-```
-## Testing checklist
-
-For every new module, run each step and confirm the named behavior. The
-commands in steps 1 to 3 work from a checkout of this repository or from an
-npm-installed package; the `make` targets belong to the source repository
-only.
-
-1. `sp_metadata_read module.d/<name>.meta` prints the normalized record
-   (step 2 above) and the load path is covered by a
-   `manifests/parser-support.tsv` row (`make test-parser-support` in the
-   source repository).
-2. Source the module in a clean environment — the body prints nothing when
-   its prerequisites are absent and defines every name in `exports=`:
-
-   ```sh
-   env -i HOME="$PWD/.fresh-home" PATH=/usr/bin:/bin \
-     VALISE_MODULE_ROOT="$PWD" VALISE_MODULES='<name>' VALISE_MODE=strict \
-     /bin/sh -c '. ./entrypoints/profile.sh'
-   ```
-
-   (Create `./.fresh-home` first or point `HOME` at any empty directory;
-   never source against your live profile.)
-3. Source it twice in one strict session or call the entrypoint repeatedly:
-   the second load must fail closed with `declared export already exists`
-   (the loader forbids double-definition), and calling an idempotent
-   function twice must leave the same state.
-4. The body performs no writes at source time: `side_effect` is
-   `read-only`, `path-env`, or `local-state`, and `generated-file`/
-   `update-network`/`install-time` bodies are rejected during ordinary
-   source.
-5. Every public function appears in `exports=` — the loader verifies each
-   declared export exists after the body runs, and cleans them up on
-   failure.
-6. Tests run in quarantined homes: this repository's suites already use
-   fresh temporary `HOME`, controlled `TMPDIR` and `PATH`, and cleared
-   startup hooks (`make check-fast` to confirm the engine gate). Stub
-   network and secrets in your own tests; never read a live profile.
-
-Run the engine's own fast gate while authoring:
+`valise run [--shell=<class>] [--module=<m>]... [--bundle=<b>]
+[--variant=<v>] [--keep-env] -- <script> [args...]` loads modules into a
+shell and runs a script there, for quick checks:
 
 ```sh
-make check-fast
+valise run --shell=zsh --module=hello -- 'hello "$@"' a b
+valise run --bundle=work -- 'command -v my_work_function'
 ```
 
-and the full matrix (`make VALISE_REQUIRE_TOOLING=1 check`) before claiming
-release readiness. The extended-runtime lanes are strict in CI and can be
-required locally with the same override. Both targets live in the source
-repository; the npm package ships the runtime and docs, not the Makefile.
-
-`valise layer-lint <layer-name>` validates a contribution layer's
-structure, namespace compliance, file permissions, metadata integrity, and
-dependency closure. It needs no Python, but it does need a layer tree: a
-provider directory under a consuming project's `shell/layers/`, or a path to
-a layer tree containing `module.d/`. In a consuming project:
+Its environment is quarantined like `valise test` unless `--keep-env` is
+given, and its exit status is the script's.
+
+None of these commands installs anything or reads your startup files. To
+check a layer tree outside a project, run `valise layer-lint <path>`.
+
+## What belongs in a layer
+
+A layer under `layers/<provider>/` is a set of modules another person could
+vendor unchanged. Before moving a module into a layer, check:
+
+- **No employer material**: no account IDs, hostnames, cloud profiles,
+  internal project names, or anything else tied to one organization.
+- **No personal configuration**: no home-directory paths, chosen editor,
+  toolchain, model, or theme.
+- **No dependency outside the layer** other than engine modules. A
+  dependency on one of your primary modules cannot be satisfied in someone
+  else's project.
+- **Mechanism, not preset.** Keep a reusable engine (a git helper, a path
+  function) separate from the presets that configure it for one person. Ship
+  the engine in the layer and keep presets in the primary modules, or in a
+  separate layer module that requires the engine.
+- **One purpose per module.** A grab-bag module forces a consumer to take
+  everything or nothing.
+- **An owner segment**: name modules `contrib_<owner>_<thing>`, so two
+  providers do not collide on a name.
+- **A version.** Every layer module needs `version=`; bump it on an
+  incompatible change.
+
+Set `lint_layer_forbid` in `valise.conf` to a pattern for the identifiers
+that must never reach the layer, such as an employer name, and `valise lint`
+will fail when one appears. [shared-modules.md](shared-modules.md) covers
+ownership, review, and deprecation for layers other people consume.
+
+## Modules in the engine repository
+
+The engine's own modules live in this repository's `module.d/` and
+`modules/` and use the `valise_` prefix. `manifests/parser-support.tsv`
+declares which parsers must accept each file class; it ships in the package,
+and `make test-parser-support` checks it in the source repository. Load an
+engine module from a checkout without installing anything through the
+development entrypoint:
 
 ```sh
-valise layer-lint <provider-name>
+env -i HOME="$(mktemp -d)" PATH=/usr/bin:/bin /bin/sh -c '
+  VALISE_MODULE_ROOT="$PWD" VALISE_MODULES="valise_path" VALISE_MODE=strict
+  . ./entrypoints/profile.sh && command -v valise_path_prepend'
 ```
+
+Run `make check-fast` while working and `make check` before committing.