valise-sh 0.1.0 → 0.2.0

package/docs/architecture.md

@@ -1,8 +1,9 @@
 # Architecture and scope
 
-`valise-sh` contains a shell-profile loader, a small catalog of reusable
-modules, and a CLI for installing and verifying sealed profile releases. It
-can be used from a source checkout or vendored by a consuming profile project.
+`valise-sh` contains a shell-profile loader, a compiler that turns module
+metadata into a static startup plan, a small catalog of reusable modules,
+and a CLI for installing and verifying sealed profile releases. It can be
+used from a source checkout or vendored by a consuming profile project.
 
 ## Components
 
@@ -11,17 +12,57 @@ can be used from a source checkout or vendored by a consuming profile project.
 | `entrypoints/profile.sh` | Sourced POSIX entrypoint. Locates the engine and module roots, loads the libraries, then invokes the loader. |
 | `lib/metadata.sh` | Parses and validates `module.d/*.meta` as data; it does not execute metadata. |
 | `lib/loader.sh` | Resolves dependencies, selects modules by shell/platform/phase, sources bodies, checks declared exports, and cleans up failures. |
+| `lib/compiler.awk` | Compiles the preprocessed module graph into a static POSIX startup plan. |
 | `lib/capabilities.sh` | Portable OS, command, path, temporary-file, and system-information helpers for modules. |
 | `lib/fish/` and `entrypoints/fish/` | Fish-native metadata reader, loader, and adapter. Fish modules are not POSIX shell scripts. |
 | `module.d/`, `modules/` | Metadata and module bodies, including generic platform and `contrib_` utilities. |
-| `bin/valise` | POSIX shell CLI for integration snippets, sealed release installation and verification, and source-tree commands used by consumer projects. |
+| `bin/valise` | POSIX shell CLI for integration snippets, sealed release installation and verification, project scaffolding, and the source-tree commands used by consumer projects. |
 | `manifests/parser-support.tsv` | Declares parser requirements for shipped shell files. |
 | `tests/`, `examples/` | Quarantined tests and runnable examples. |
 
 See [module-abi.md](module-abi.md) for metadata and loader contracts,
-[installation.md](installation.md) for CLI and release behavior, and
+[installation.md](installation.md) for CLI and release behavior,
+[variants.md](variants.md) for the compile and build contract, and
 [contrib-modules.md](contrib-modules.md) for authoring modules.
 
+## Two ways to load a profile
+
+A profile can be loaded two ways, and both are supported.
+
+The **loader** reads the metadata at startup, resolves the dependency
+graph, and sources the module bodies. It is the general mechanism: it
+honours every metadata field, including optional dependencies, and it
+works for a source tree that is still being edited.
+
+The **compiled plan** is a single static POSIX script per shell,
+generated ahead of time from the same metadata. It is faster to source,
+because nothing is parsed or resolved at startup. It carries the
+contract that matters for a startup file: platform gating, export
+checks, per-module failure reporting, strict and permissive failure
+policy, and the capability helpers left available to the modules it
+loads.
+
+The two are deliberately close in behavior, so a profile can be compiled
+for speed without changing what it does. The compile contract, including
+the exact fields the plan honors, is in [variants.md](variants.md).
+
+## Two release shapes
+
+A release directory takes one of two shapes, and the startup snippet
+handles both.
+
+A **built release** is what `valise build` produces. It carries
+`generated/installed-modules.<shell>` compiled plans, the capability
+prelude under `vendor/shell-profile/lib/`, the module bodies, and no
+`entrypoints/`.
+
+An **engine release** carries `entrypoints/profile.sh` and `lib/`, and no
+`generated/`. It loads through the loader rather than a compiled plan.
+
+The POSIX integration snippet prefers the compiled plan for the running
+shell and falls back to the loader, so a startup file written by either
+shape loads the release it is pointed at.
+
 ## Loading modules
 
 The POSIX entrypoint is sourced, not executed. A caller can select roots,
@@ -56,10 +97,35 @@ specific integrity boundary.
 
 The engine does not define a consumer's module catalog, default startup graph,
 workstation policy, project wrappers, or project-specific release build.
-Those belong in the consuming repository. The `compile`, `explain-variant`,
-`explain-bundles`, and `layer-lint` CLI commands therefore need source files
-from a consuming project; the rest of the documented bare-checkout commands
-are usable without one.
+Those belong in the consuming repository. `valise new` scaffolds an empty
+project so a consumer never has to invent the directory layout by hand, but
+the catalog itself is still the consumer's to write.
+
+The CLI splits into two groups by what they need.
+
+Commands that work in a bare checkout: `init`, `install`, `release`,
+`doctor`, `make-manifest`, `version`, and `new`.
+
+Commands that need a consuming project, because they read or write the
+project's source tree: `registry`, `module`, `build`, `apply`, `status`,
+`compile`, `explain-variant`, `explain-bundles`, and
+`layer-lint`. A project is recognised by `shell/module.d/` or
+`shell/release/module-registry.tsv`; the project commands run from the
+project root and read their defaults from `shell/release/project.conf`.
+
+## The registry is generated, not authored
+
+`shell/release/module-registry.tsv` is a build artifact. `valise registry`
+reads every `.meta` file under `shell/module.d/` and
+`shell/layers/<provider>/module.d/` and writes the registry from them, in the
+C locale, so repeated generation is byte-identical. `valise registry --check`
+compares the file against the metadata without rewriting it, which is what
+`valise status` reports as registry freshness. `build` regenerates the
+registry before compiling, so a project that contains only
+metadata and bodies still builds.
+
+This makes the `.meta` files the single source of truth. A hand-edited
+registry is not supported: the next build overwrites it.
 
 The `updates=` metadata field records identifiers for a consumer's update
 manifest. It does not make this engine a network updater. Shared modules are
@@ -75,3 +141,6 @@ suite. Extended runtime lanes run when their tools are installed; missing tools
 are visible skips locally and are required in CI. Set
 `VALISE_REQUIRE_TOOLING=1` to require them on a developer machine. The GitHub
 Actions workflow installs the shell runtimes used by the release check.
+
+These targets live in the source repository. The npm package ships the
+runtime, the docs, and the examples, not the Makefile or the test suite.

package/docs/contrib-modules.md

@@ -37,14 +37,20 @@ load=modules/system/example_tool.sh
 | `name` | Yes | Stable identifier matching `[a-z_][A-Za-z0-9_]*` |
 | `requires` | No | Space-separated hard dependencies (loaded first) |
 | `optional` | No | Space-separated best-effort dependencies |
-| `shells` | Yes | Supported shells: `sh`, `bash`, `zsh`, `ksh`, `fish` |
+| `shells` | Yes | Supported shells: `sh`, `bash`, `zsh`, `ksh`, `fish`. `sh` means every POSIX shell (sh, bash, zsh, ksh, dash) |
 | `platforms` | Yes | `any`, `darwin`, `linux`, or BSD family values |
-| `side_effect` | Yes | One of: `read-only`, `path-env`, `generated-file`, `update-network`, `install-time`, `local-secret`, `work-only` |
+| `side_effect` | Yes | One of: `read-only`, `path-env`, `generated-file`, `update-network`, `install-time`, `local-secret`, `local-state`, `work-only` |
 | `exports` | No | Public functions this module declares |
 | `updates` | No | Update registration identifiers mapped in the project update manifest |
 | `version` | No | Module version (digit-first, digits/letters/`.`/`_`/`-`); different versions of one name across roots fail closed with a skew diagnostic |
-| `load` | Yes | Repository-relative path to the module body (mutually exclusive with `load_hook`; exactly one required) |
-| `load_hook` | No | Shell function hook for bridge-backed records (mutually exclusive with `load`) |
+| `load` | Conditionally | Repository-relative path to the module body |
+
+At least one of `load` or `requires` must be present. `load` and `requires`
+are not mutually exclusive — a module
+may declare both and will load its dependencies first, then its own body. A
+module with `requires` and no load target is a **meta module**: it names
+dependencies and has no body. See
+[module-abi.md](module-abi.md#meta-modules).
 
 ### Naming
 
@@ -60,6 +66,13 @@ closed naming both paths.
 Use `sp_cap_*` functions from `lib/capabilities.sh` instead of open-coding
 platform-specific commands. The POSIX entrypoint sources the capability
 library before the loader, so module bodies can call these directly.
+
+**The helpers stay available after the profile loads.** A compiled plan
+sources the capability prelude and leaves it in place, so a function your
+module defines can call `sp_cap_os` or any other helper long after the plan
+finished loading. Both the compiled plan and the loader keep the helpers
+available at runtime.
+
 Available capabilities include:
 
 - `sp_cap_canonical_path <path>` — resolve absolute path through symlinks
@@ -82,6 +95,31 @@ policy plus the curated index lives in
 
 ## Authoring workflow
 
+### In a consuming project
+
+Inside a project that uses the CLI, the CLI creates the files for you:
+
+```sh
+valise new demo                 # scaffold a project
+cd demo
+valise module new hello --exports=hello
+```
+
+`valise module new` writes `shell/module.d/hello.meta` and
+`shell/modules/hello.sh`, and wires `hello` into the seed aggregator's
+`requires=` so the module is actually reachable. It refuses a duplicate
+export name. Edit the body, then:
+
+```sh
+valise build                    # build to build/release
+valise apply                    # build, install, integrate, verify
+```
+
+You never hand-write the registry: `valise build` and `valise apply`
+regenerate `shell/release/module-registry.tsv` from the metadata.
+
+### In the engine repository
+
 1. Write the two files. Start from an existing pair — `module.d/contrib_pager.meta`
    plus `modules/system/contrib_pager.sh` is the smallest real example.
 2. Validate the metadata as data:
@@ -119,12 +157,18 @@ policy plus the curated index lives in
    make test-parser-support
    ```
 
+   This target belongs to the engine's source repository. The npm package
+   ships the runtime and the docs, not the Makefile or the parser table, so
+   a package consumer checks parser coverage by hand: run every POSIX shell
+   you claim to support over the body (`sh -n`, `bash -n`, `zsh -n`,
+   `ksh -n`) and confirm each one accepts it.
+
 5. Publish a release containing your module tree and prove the whole
-   path end to end (make-manifest, install, verify-active):
+   path end to end (make-manifest, install, release verify):
 
    ```sh
    valise make-manifest ./mytree
-   cd mytree && valise install && valise verify-active
+   cd mytree && valise install && valise release verify
    ```
 
    See [installation.md](installation.md) for release integrity and
@@ -145,11 +189,14 @@ VALISE_MODE=strict \
 ## Testing checklist
 
 For every new module, run each step and confirm the named behavior. The
-commands all exist in this repository.
+commands in steps 1 to 3 work from a checkout of this repository or from an
+npm-installed package; the `make` targets belong to the source repository
+only.
 
 1. `sp_metadata_read module.d/<name>.meta` prints the normalized record
    (step 2 above) and the load path is covered by a
-   `manifests/parser-support.tsv` row (`make test-parser-support`).
+   `manifests/parser-support.tsv` row (`make test-parser-support` in the
+   source repository).
 2. Source the module in a clean environment — the body prints nothing when
    its prerequisites are absent and defines every name in `exports=`:
 
@@ -166,8 +213,9 @@ commands all exist in this repository.
    (the loader forbids double-definition), and calling an idempotent
    function twice must leave the same state.
 4. The body performs no writes at source time: `side_effect` is
-   `read-only` or `path-env`, and `generated-file`/`update-network`/
-   `install-time` bodies are rejected during ordinary source.
+   `read-only`, `path-env`, or `local-state`, and `generated-file`/
+   `update-network`/`install-time` bodies are rejected during ordinary
+   source.
 5. Every public function appears in `exports=` — the loader verifies each
    declared export exists after the body runs, and cleans them up on
    failure.
@@ -184,10 +232,15 @@ make check-fast
 
 and the full matrix (`make VALISE_REQUIRE_TOOLING=1 check`) before claiming
 release readiness. The extended-runtime lanes are strict in CI and can be
-required locally with the same override.
+required locally with the same override. Both targets live in the source
+repository; the npm package ships the runtime and docs, not the Makefile.
 
 `valise layer-lint <layer-name>` validates a contribution layer's
-structure, namespace compliance, file permissions, and metadata integrity,
-but it requires a consuming project's `shell/layers/` tree and
-`hack/validate-layer.py`, so it does not run from a bare checkout and is
-not part of this workflow.
+structure, namespace compliance, file permissions, metadata integrity, and
+dependency closure. It needs no Python, but it does need a layer tree: a
+provider directory under a consuming project's `shell/layers/`, or a path to
+a layer tree containing `module.d/`. In a consuming project:
+
+```sh
+valise layer-lint <provider-name>
+```

package/docs/installation.md

@@ -2,8 +2,9 @@
 
 `bin/valise` is a `/bin/sh` CLI. Its normal install and verification commands
 need the platform utilities used by the release manifest and integrity checks
-(for example, a SHA-256 utility). The consuming-project commands `compile`,
-`explain-variant`, `explain-bundles`, and `layer-lint` also require Python 3.
+(for example, a SHA-256 utility). Every documented command runs without
+Python 3. `explain-bundles` reads the generated registry and the bundle
+roots; `layer-lint` reads layer metadata directly.
 
 ## Install the CLI
 
@@ -38,38 +39,96 @@ packed CLI without a registry, and loads a module from the extracted package.
 ## Shell integration
 
 `valise install` installs the release described by the manifest in the
-current project and deploys shell integration. It adds a POSIX snippet for
-bash, zsh, or ksh when that shell is named by `$SHELL`, and a Fish `conf.d`
-hook when it finds Fish in a trusted system path. On macOS the POSIX snippet
-goes to `.bash_profile`, `.zprofile`, or `.kshrc`; on other systems it goes to
-`.bashrc`, `.zshrc`, or `.kshrc`. The Fish hook is
-`${XDG_CONFIG_HOME:-$HOME/.config}/fish/conf.d/valise.fish`. Existing POSIX
-profile content is preserved; a marker makes repeat installs idempotent.
-Symlinked POSIX profile files are left alone.
-
-To print a POSIX integration snippet without editing a startup file:
+current directory and writes shell integration by default. It always writes
+`~/.profile`, which login `sh`, `ksh`, and bash sessions read when no other
+file shadows it. It then adds the files each shell reads instead:
+
+- `~/.bash_profile` or `~/.bash_login` when one exists, because login bash
+  reads the first of those instead of `~/.profile`. valise never creates
+  either file.
+- `~/.bashrc` when bash is `$SHELL` or the file already exists.
+- `~/.zprofile` when zsh is on `PATH` or the file already exists. It is the
+  login file zsh reads instead of `~/.profile`.
+- `~/.zshrc` when zsh is `$SHELL` or the file already exists.
+- `${XDG_CONFIG_HOME:-$HOME/.config}/fish/conf.d/valise.fish` when valise
+  finds a Fish executable in a trusted location.
+
+`--no-integration` writes no startup file and no Fish hook. `valise apply`
+builds, installs, and then uses the same writer.
+
+Each POSIX file receives a managed block between `# BEGIN valise managed
+block` and `# END valise managed block`. Existing content is preserved, and
+a repeat install replaces the block in place. An unpaired marker is refused
+and stops the install; a file already processed keeps its updated block. A
+symlinked POSIX startup file is reported and left untouched. If the Fish
+hook path is a symlink or a directory, install fails rather than write
+through it. Install prints every path it wrote and how to remove the
+integration.
+
+The POSIX snippet adapts to the release shape:
+
+- A project release built by `valise build` carries
+  `generated/installed-modules.<shell>` for each shell its bundle seeds
+  (`bash`, `zsh`, and `ksh` for the scaffold and the example). The snippet
+  sources the plan for the running shell.
+- An engine release carries `entrypoints/profile.sh` and the loader. The
+  snippet sources that entrypoint.
+- If the release has neither a plan for the running shell nor the loader,
+  the snippet leaves the session alone. This is the result for a login
+  `dash` session with a built release: `dash` has no plan, and a built
+  release has no loader.
+
+`valise install` writes `~/.profile`, which ksh reads at login. A non-login
+interactive ksh reads the file named by `$ENV` when that variable is set.
+valise does not choose or edit that file; to cover non-login ksh, put the
+POSIX block in the file `$ENV` names.
+
+Zsh does not read `~/.profile` on its own: zsh login shells read
+`.zprofile`, not `~/.profile`. Install writes the same managed block to
+`~/.zprofile`, so zsh needs no separate pointer.
+
+A built project release has no Fish surface: no Fish entrypoint, generated
+Fish plan, or native Fish loader. The Fish hook verifies the active release,
+then loads nothing and prints nothing. Only an engine release with
+`entrypoints/fish/profile.fish` uses the native Fish loader.
+
+The Fish hook verifies the active release before loading it. The check
+validates the active pointer, the release manifest, and every listed file.
+If verification fails, the hook loads no release code. Fish continues
+starting even if a `conf.d` hook fails, so this does not abort the
+interactive shell.
+
+To write integration by hand, print it once and paste or save it:
 
 ```sh
-eval "$(valise init)"
+valise init            # POSIX block, or the Fish hook when $SHELL is fish
+valise init posix      # POSIX managed block
+valise init fish       # Fish conf.d hook; save it as fish/conf.d/valise.fish
 ```
 
-For Fish, source the generated integration in `~/.config/fish/config.fish`:
+Evaluating `valise init` output directly at every prompt —
+`eval "$(valise init)"` — re-runs CLI startup code in every shell and is
+not the routine path; prefer `valise install` or `valise apply`, which
+write the files once.
 
-```fish
-valise init fish | source
-```
+`valise doctor` verifies the active release against its own manifest, checks
+the startup files the running shell actually reads for valise integration,
+including the `.bash_profile` or `.bash_login` that bash reads instead of
+`~/.profile`. It exits nonzero when the active release is missing or fails
+verification, and when no startup file carries the integration.
+
+Two options adjust its scope:
 
-The Fish integration verifies the active release before loading it. The
-check validates the active pointer, the release manifest, and every listed
-file. If verification fails, the hook loads no release code. Fish continues
-starting even if a `conf.d` hook fails, so this does not abort the interactive
-shell.
+- `--pre-install` skips the active-release and startup-file checks, so doctor
+  can run before the first install.
+- `--no-integration` skips the startup-file check and reports only the
+  release.
 
-`valise verify-active` runs the release verification without sourcing shell
+`valise release verify` runs the release verification without sourcing shell
 code:
 
 ```sh
-valise verify-active
+valise release verify
 ```
 
 The development-only `valise init fish --source-tree` mode skips installed
@@ -99,18 +158,21 @@ Install and verify from the tree root:
 ```sh
 cd mytree
 valise install
-valise verify-active
+valise release verify
 ```
 
 Install stages a sealed, read-only copy under the XDG data directory and
-updates the active symlink. `verify-active` checks the installed bytes against
+updates the active symlink. `release verify` checks the installed bytes against
 their manifest. Fish repeats that check at startup; the POSIX startup snippets
-do not, so run `verify-active` separately if you need a check before sourcing
+do not, so run `release verify` separately if you need a check before sourcing
 in a POSIX shell.
 
-The engine does not build a consuming project's module catalog or release
-tree. The included [e2e example](../examples/e2e/README.md) demonstrates a
-small complete profile.
+`valise build` builds a consuming project's release tree: the module
+registry, the per-shell plans, every module body, and the manifest inputs.
+The engine itself ships as an engine release rather than a built project
+release; it carries `entrypoints/profile.sh` and `lib/`, not `generated/`.
+The included [e2e example](../examples/e2e/README.md) demonstrates a small
+complete profile.
 
 ## Integrity and provenance
 
@@ -135,6 +197,30 @@ controls.
 
 The active path is a symlink to the selected release.
 
+## Managing installed releases
+
+`valise release list` shows the installed releases and marks the active one.
+`valise release use <release-id>` points the active symlink at an existing
+release.
+
+`valise release prune --keep=<n>` removes old releases. It always keeps the
+active release and the `n` newest others, and it never removes a release that
+fails verification against its own manifest — a release that cannot be
+verified is reported and kept, so a damaged tree is not silently discarded.
+The default is `--keep=3`.
+
+## Command usage
+
+Every command prints its own contract:
+
+```sh
+valise compile --help
+valise build --help
+valise install --help
+```
+
+`valise help` prints the global command list.
+
 ## Updating and removing
 
 Update a source checkout with Git. To update an npm installation, repack and
@@ -142,9 +228,10 @@ reinstall the tarball. `valise install` updates the active profile release;
 it does not update the CLI itself.
 
 There is no `valise uninstall` command. Before removing the CLI, remove the
-managed integration from your shell startup file: in POSIX files, delete only
-the block beginning `# valise shell integration — managed by valise install`;
-for Fish, remove
+managed integration from every startup file valise wrote: delete only the
+block between `# BEGIN valise managed block` and `# END valise managed
+block` in `~/.profile` and in any of `~/.bash_profile`, `~/.bash_login`,
+`~/.bashrc`, `~/.zshrc`, and `~/.zprofile`. For Fish, remove
 `${XDG_CONFIG_HOME:-$HOME/.config}/fish/conf.d/valise.fish` if it is still
 the valise-managed hook. Then uninstall the CLI and remove the valise XDG
 state. Installed release files are read-only, so make them writable first: