structured-context 0.11.0 → 0.13.0

package/docs/config.md

@@ -1,4 +1,8 @@
-# structured-context Configuration Reference
+---
+title: "Configuration Reference"
+---
+
+# Configuration Reference
 
 ## Config file location
 
@@ -26,6 +30,12 @@ A space is a named directory or single file registered in the config. Example:
 }
 ```
 
+**`schema`** — the schema used to validate nodes in the space. Can be:
+- A bundled schema name (e.g. `strategy_general.json`, `strict_ost.json`)
+- A local path relative to the config file (e.g. `./schemas/custom.json`) or absolute path
+- A plugin schema prefixed with the plugin name (e.g. `sctx-wardley-mapping/wardley_map.json` or `wardley-mapping/wardley_map.json`). Specifying a plugin-prefixed schema automatically loads that plugin for the space.
+- A simple filename (e.g. `wardley_map.json`) provided by a plugin explicitly declared under `plugins`.
+
 **`includeSpacesFrom`** — import space definitions from other config files. Useful for aggregating spaces from multiple projects into a central config. Duplicate space names are not allowed.
 
 A config must define at least one of `spaces` or `includeSpacesFrom`. When using `includeSpacesFrom` alone, `spaces` can be omitted entirely.
@@ -53,6 +63,73 @@ Use `plugins` to load parse plugins that read spaces from non-markdown sources.
 
 All plugin names must start with `sctx-` (the prefix is optional in config and normalised on load). The special name `markdown` refers to the built-in markdown plugin. External plugins are resolved in order: config-adjacent (`{configDir}/plugins/{name}`), then npm. Each plugin must export a `configSchema` JSON Schema; config is validated against it on load. Fields annotated `format: 'path'` in a plugin's `configSchema` are resolved relative to the config file directory.
 
+Plugins can also contribute schemas and partials via a `<pluginRoot>/schemas/` directory (or `"sctx": { "schemas": "..." }` in `package.json`). If a space specifies a plugin-prefixed schema (e.g. `"sctx-wardley-mapping/wardley_map.json"`), schemas are resolved statically without executing plugin code, and the plugin's code hooks are loaded when parsing or rendering.
+
+## Model providers and semantic rules
+
+structured-context evaluates [semantic rules](semantic-rules.md) with a **model provider**: a named definition that says which wire API to speak, which model to use and where. A provider definition combines an `api`, explicit values or environment-variable names, and the key source.
+
+**Bundled providers** work without any configuration: a fixed model per vendor chosen by this project, plus `-env` variants that also read the vendor's usual model and endpoint variables (`OPENAI_MODEL`, `OPENAI_BASE_URL`, `ANTHROPIC_MODEL`), falling back to the bundle's defaults where a variable is unset. Bundles without `-env` read no environment beyond the key: they behave the same on every machine.
+
+Run `sctx providers` for the authoritative list of bundled and configured providers with their effective API, endpoint, model and key variable. Keys are never shown. `bun run test:models` checks that each bundled model resolves on its endpoint.
+
+### Wire APIs and bundled-provider conventions
+
+A provider's `api` names the **request shape**, potentially supported by multiple vendors.
+
+| API | Shape | Bundled providers |
+|---|---|---|
+| `openrouter-decisions` | Decisions API, closed-set `choice` questions, answers with confidence and probabilities | `openrouter-jev`, `openrouter-openai-decisions` |
+| `typesafe-systemone` | Same decisions shape (currently) | `typesafe-jev` |
+| `openai-decisions` | OpenAI's Decisions API: same concept, its own request/response shape | `openai-decisions` |
+| `anthropic-messages` | Chat: structured JSON verdict | `anthropic`, `anthropic-haiku-5-5`, `anthropic-env` |
+| `openai-completions` | Chat: OpenAI-compatible; works with Ollama, vLLM and OpenRouter | `openai`, `openai-env`, `openrouter-openai` |
+
+Bundled provider names follow the pattern:
+
+- `<vendor>` — this project's recommended model for that vendor; can change between releases.
+- `<vendor>-<model>` (e.g. `anthropic-haiku-5-5`) — pins a model id, so it does not drift when the `<vendor>` bundle's recommendation moves.
+- `<vendor>-env` — hydrates model, endpoint etc from the vendor's usual environment variables.
+- `openrouter-<vendor>-<purpose>` (e.g. `openrouter-openai-decisions`) — aggregating vendor's endpoint variant
+
+**Configured providers** live in `modelProviders` at the top level, and replace a bundled provider of the same name. Fields a configured provider leaves out still take the API defaults:
+
+```json
+{
+  "modelProviders": {
+    "mine": { "api": "openrouter-decisions", "apiKeyEnv": "MY_JEV_KEY" },
+    "local": { "api": "openai-completions", "endpoint": "http://localhost:11434/v1" }
+  }
+}
+```
+
+| Field | Description |
+|---|---|
+| `api` | **Required.** The wire API (see the table above and [Choosing a model provider](semantic-rules.md#choosing-a-model-provider)), or `fake` (for tests). |
+| `model` | A fixed model. Its default depends on the API. |
+| `modelEnv` | Environment variable holding a model override. An explicit `model` wins; a variable set but empty is an error. |
+| `endpoint` | A fixed API URL. |
+| `endpointEnv` | Environment variable holding an endpoint override. Same precedence as `modelEnv`. |
+| `apiKeyEnv` | The environment variable that holds the key. Defaults to the API's usual variable (see the table above). A local `openai-completions` endpoint can need no key. |
+| `apiKey` | The key itself. Use `apiKeyEnv` instead, so that keys stay out of config files. |
+
+Resolution per field: the explicit value wins, then the named environment variable, then the API default. Provider implementations read no environment; all hydration happens before one runs.
+
+The environment holds only keys and the variables a `-env` bundle names. No environment variable selects a provider.
+
+The top-level `semantic.provider` names the model provider that evaluates semantic rules: a bundled provider or an entry in `modelProviders`. A space can set its own `semantic`, which replaces the top-level one for that space, for example to keep a private space on a local model:
+
+```json
+{
+  "semantic": { "provider": "openrouter-jev" },
+  "spaces": [
+    { "name": "private-notes", "path": "/path/to/notes", "semantic": { "provider": "local" } }
+  ]
+}
+```
+
+Config does not turn semantic evaluation on. Each run asks for it with `--semantic`. `--semantic <provider>` also selects a different provider for that run.
+
 ## Markdown plugin config
 
 Set under `plugins.markdown` per space.

package/docs/rules.md

@@ -1,6 +1,15 @@
+---
+title: "Executable Rules"
+---
+
 # Executable Rules
 
-Rules are JSONata expressions in schema metadata (`$metadata.rules`). They run after structural JSON Schema validation and let you enforce cross-node checks and workflow constraints.
+Rules check content after JSON Schema validation, for example cross-node checks and workflow constraints. A schema declares rules in `$metadata.rules`. A rule has an `expression`, a `semantic` part, or both:
+
+- An **`expression`** is a JSONata test that must return `true`.
+- A **`semantic`** part asks a model to evaluate a node. See [Semantic rules](semantic-rules.md).
+
+A rule with neither is a schema error.
 
 For metadata structure and composition behavior, see [docs/schemas.md](schemas.md).
 
@@ -14,40 +23,76 @@ Rules are a flat array.
     "id": "active-outcome-count",
     "category": "workflow",
     "description": "Only one outcome should be active at a time",
-    "scope": "global",
-    "check": "$count(nodes[resolvedType='outcome' and status='active']) <= 1"
+    "global": true,
+    "expression": "$count(nodes[resolvedType='outcome' and status='active']) <= 1"
   }
 ]
 ```
 
-| Field | Required | Description |
+Every rule has a category, which describes its intent and supplies a default severity.
+
+| Category | Typical use | Default severity |
 |---|---|---|
-| `id` | yes | Unique rule identifier |
-| `category` | yes | `validation` \| `coherence` \| `workflow` \| `best-practice` |
-| `description` | yes | Human-readable rule intent |
-| `check` | yes | JSONata expression; must evaluate to `true` |
-| `type` | no | Restrict rule to nodes of this `resolvedType` |
-| `scope` | no | Use `"global"` to evaluate once for the whole space |
-| `override` | no | Only for merge conflicts: allows later duplicate `id` to replace earlier |
+| `validation` | Hard correctness constraints | `error` |
+| `coherence` | Cross-node consistency checks | `warning` |
+| `workflow` | Process/operating-discipline checks | `warning` |
+| `best-practice` | Advisory quality checks | `info` |
 
-## Categories
+Severity can also be set per rule. It applies to a failed `expression` and to the default `fail` verdict of a [semantic rule](semantic-rules.md#severity).
 
-Categories label violations for reporting. They do not change expression execution semantics.
+| Severity | Meaning | Fails validation |
+|---|---|---|
+| `error` | Content is wrong | yes |
+| `warning` | Content is likely inconsistent or incomplete | no |
+| `info` | Content could be better | no |
 
-| Category | Typical use |
-|---|---|
-| `validation` | Hard correctness constraints |
-| `coherence` | Cross-node consistency checks |
-| `workflow` | Process/operating-discipline checks |
-| `best-practice` | Advisory quality checks |
+Only `error` violations fail validation: `sctx validate` and `sctx validate-file` exit non-zero, and JSON output reports `valid: false`. Warnings and info violations are reported but don't fail, so a work-in-progress space can still pass while advisory checks remain visible.
+
+Prefer choosing the right category and relying on its default. Set `severity` on an individual rule only when that rule genuinely differs from the rest of its category:
+
+```json5
+{
+  "id": "goal-not-blocked",
+  "category": "workflow",
+  "severity": "error",
+  "description": "Goals must not be left blocked",
+  "type": "goal",
+  "expression": "current.status != 'blocked'"
+}
+```
+
+`sctx schemas show` lists each category with its default severity and marks rules that override it.
+
+## Selecting nodes
+
+- A rule with `global: true` runs once for the space. It needs an `expression`, and it cannot have `when`, `type` or `semantic`.
+- Other rules run once for each node.
+- `type` limits a rule to nodes with that `resolvedType`.
+- `when` limits a rule to nodes where the applicability condition returns `true`. See [Conditional rules](#conditional-rules-when).
+
+### Conditional rules (`when`)
 
-## Evaluation model
+`when` is the rule's applicability condition: a JSONata test that determines whether a rule applies to a node. It has the same context as `expression` and runs before it. Use `when` when `type` cannot select the nodes a rule applies to:
 
-- Rules with `scope: "global"` run once.
-- Other rules run per applicable node.
-- If `type` is set, only nodes with matching `resolvedType` are evaluated.
+```json
+{ "id": "ready-has-owner", "category": "workflow", "type": "task",
+  "description": "Ready tasks have an owner",
+  "when": "current.status = 'ready'", "expression": "$exists(current.owner)" }
+```
+
+`when` costs almost nothing, so use it to stop an expensive rule from running where it does not apply. For a semantic rule, this saves a model call on each node that `when` excludes.
+
+### Order within a rule
+
+A rule runs its parts in this order: `when`, then `expression`, then the `semantic` part. If the `expression` fails, the `semantic` part does not run on that node.
+
+An error in a `when` or `expression` is reported as a non-gating `error` violation.
 
-## Expression context
+## Writing expressions
+
+An `expression` (and `when`) is JSONata. This section covers what an expression can see and how to write expressions.
+
+### Expression context
 
 Each evaluation receives:
 
@@ -65,7 +110,7 @@ Useful resolved fields on nodes:
 
 Prefer `resolvedType` over raw `type` so aliases are handled correctly.
 
-## Predicate scoping (`$$`)
+### Predicate scoping (`$$`)
 
 Inside `nodes[...]`, bare names refer to each candidate node. Use `$$` to reference outer variables:
 
@@ -73,29 +118,39 @@ Inside `nodes[...]`, bare names refer to each candidate node. Use `$$` to refere
 $count(nodes[resolvedParentTitle=$$.current.title and resolvedType='solution'])
 ```
 
-## Rule imports (`$ref`)
+### Common patterns
 
-`$metadata.rules` can include `$ref` entries that import:
-- one rule
-- a rule-set object with `rules: []`
+```jsonata
+$exists(current.metric) = true
+$count(current.sources) >= 1
+$count(nodes[resolvedType='outcome' and status='active']) <= 1
+current.status != 'active' or $exists(parent) = false or parent.status = 'active'
+```
 
-Example:
+#### Dates and times
 
-```json5
-"rules": [
-  { "$ref": "sctx://rule-pack#/$defs/workflowRule" },
-  { "$ref": "sctx://rule-pack#/$defs/coreRuleSet" }
-]
+Dates and datetimes reach rules as ISO 8601 strings. Convert with `$toMillis()` to compare them, including date against datetime, and use `$millis()` for the current time:
+
+```jsonata
+$not($exists(current.due)) or $toMillis(current.due) > $millis()
+$toMillis(current.stale_after) > $toMillis(current.generated.at)
 ```
 
-Imported entries are normalized into the same flat runtime list.
+A datetime without an offset (`2026-07-01T00:00:00`) is read in the machine's local time zone, so require an offset with `format: "date-time"` when results must not depend on where validation runs. `$toMillis()` also needs seconds: it cannot parse a minute-precision datetime such as `2026-07-01T09:15`, which `format: "date-or-date-time"` accepts. A value `$toMillis()` cannot parse makes the rule fail.
+
+The same applies to [filter expressions](concepts.md#filter-expressions): compare dates with `$toMillis()`, not as strings.
+
+## Sharing rules
 
-## Merge and conflict behavior
+To share rules, put them in a schema's `$metadata.rules` and list that schema's `$id` in `$metadata.imports`. An import brings the whole schema's metadata, including its own imports. Merged rules combine by `id`; a duplicate `id` is an error, and a later rule replaces an earlier one only with `override: true`.
 
-When metadata is composed across `$ref`:
-- Rules are merged by `id`.
-- Different payloads for the same `id` are an error by default.
-- A later rule may replace an earlier one only with `override: true`.
+### Overriding rules
+
+```json5
+"$metadata": {
+  "imports": ["sctx://_rule-pack"]
+}
+```
 
 Example override:
 
@@ -105,16 +160,79 @@ Example override:
   "override": true,
   "category": "workflow",
   "description": "Require exactly one active outcome",
-  "scope": "global",
-  "check": "$count(nodes[resolvedType='outcome' and status='active']) = 1"
+  "global": true,
+  "expression": "$count(nodes[resolvedType='outcome' and status='active']) = 1"
 }
 ```
 
-## Common patterns
+## Cascade gating
+
+When a schema uses `$metadata.hierarchy`, rule evaluation cascades top-down through the
+hierarchy tree (breadth-first from roots and orphans). A **gate** occurs when a node fails
+in a way that makes its descendants' rules untrustworthy or wasteful to run:
+
+- **Schema validation failure** on a node — always gates. The node's own rules are
+  suppressed too, since its data is structurally invalid.
+- **Hierarchy structure violation** on a node (e.g. an invalid parent) — always gates the
+  node's subtree, though the node's own rules still run.
+- **A rule failure with `gatesDescendants: true`** — gates the node's subtree, though the
+  failing node's own other rules still run.
+
+Gating exists so that expensive or semantic downstream rules (see [Semantic rules](semantic-rules.md)) aren't wasted
+evaluating nodes whose ancestors are already known-bad, and so noise doesn't drown out the
+actionable failure. It is per-subtree: once a node is gated, its descendants inherit the same
+gate (the original blocker), rather than each generating a new one. Cascade gating only applies to nodes within the
+hierarchy DAG.
+
+A rule has one `gatesDescendants` setting. It applies to whichever part failed, the `expression` or the `semantic` part. Its default comes from the severity of that failure: `true` for `error`, and `false` for `warning` and `info`. Set it explicitly when that default doesn't fit.
+
+### Coverage output
+
+`sctx validate` reports gating as **coverage gates**: which node's failure (schema,
+hierarchy, or a specific `ruleId`) suppressed rule evaluation, and the list of files gated
+as a result. This makes the validation "edge" visible — e.g. "this blocked goal means all
+opportunities beneath it were not evaluated" — for both human reading and agent consumption
+(JSON output via `--json`, under `coverageGates`).
+
+Use `--include-gated` to evaluate rules on gated nodes anyway (still reported as gated) —
+useful for surfacing cheap, bottom-up checks (e.g. orphan detection, content quality) that
+don't depend on an ancestor being valid.
+
+### Single-file validation
+
+`sctx validate-file <path>` applies the same gating as `sctx validate`, so a file's rule results match its entry in a full run.
+
+- Expressions are evaluated for the whole space; the output is restricted to the file.
+- Semantic parts are evaluated only for the file and its gating ancestors (see [Semantic rules](semantic-rules.md#how-semantic-evaluation-works)).
+- If the file is gated, its rules are skipped and a `gated:<file>` entry under `warnings` names the blocker, so a gated file doesn't read as clean. `--include-gated` evaluates the file's rules anyway.
+
+## Controlling validation runs
+
+These options restrict which rules run, without changing rule definitions.
+
+### Filtering rules
+
+`--rule-filter <dimension>=<value>[,<value>]` runs only the rules that match. It works on `sctx validate` and `sctx validate-file`.
+
+- Dimensions: `category`, `severity`, `type` and `id`.
+- Values in one dimension combine with OR. Repeat the option to add a dimension; dimensions combine with AND.
+- A rule matches each severity that it can report, including the severities in a semantic verdict map.
+- Gating comes only from the rules that ran.
+
+For example, `--rule-filter category=workflow,coherence --rule-filter severity=error` runs only workflow and coherence rules that can report an error.
+
+### Depth-limited evaluation
+
+`--depth <n>` restricts validation to hierarchy nodes within `n` breadth-first levels from
+the top (roots and orphans are depth 1). This is a manual, coarser control than cascade
+gating — intended for staged workflows (e.g. an agent fixes the top level, then requests
+the next). Excluded nodes are reported under `skippedByDepth`, separate from gated nodes.
+
+## Output
+
+- **`sctx validate`** — the summary counts rule violations per severity, and details are grouped by severity with each line showing the rule's category and id.
+- **`sctx validate --json`** — `errors` includes only `error` severity rule violations (keyed `rule:<id>`, with a `severity` field). A `ruleViolations` array lists every violation with `file`, `ruleId`, `category`, `severity` and `description` (`file` is empty for global rules). A violation from a failed `expression` has the evidence `expression failed: <expression>` when the rule also has a `semantic` part, and no verdict.
+- **`sctx validate-file`** — `error` severity violations appear under `errors`; `warning` and `info` violations appear under `warnings`. Rule entries carry a `severity` field.
+
+Gated files: see [Single-file validation](#single-file-validation).
 
-```jsonata
-$exists(current.metric) = true
-$count(current.sources) >= 1
-$count(nodes[resolvedType='outcome' and status='active']) <= 1
-current.status != 'active' or $exists(parent) = false or parent.status = 'active'
-```