@kici-dev/compiler 0.1.26 → 0.2.0

package/dist/llm-context/llms-architecture.txt

@@ -19,7 +19,7 @@ Per-orchestrator settings loaded from a YAML file. These are instance-specific a
 ```typescript
 interface LocalConfig {
   database: { url: string };
-  instance?: { id?: string; mode?: 'platform' | 'hybrid' | 'independent' };
+  instance?: { id?: string; mode?: 'platform' | 'hybrid' | 'independent' | 'observed' };
   server?: { port?: number; basePath?: string; logLevel?: string };
   scaler?: { configPath?: string; configDir?: string };
 }
@@ -56,7 +56,7 @@ The merged result type used throughout the codebase. Combines `LocalConfig` + `S
 ```typescript
 interface AppConfig {
   instanceId: string; // From local config or auto-generated
-  mode: 'platform' | 'hybrid' | 'independent';
+  mode: 'platform' | 'hybrid' | 'independent' | 'observed';
   databaseUrl: string; // Flattened from database.url
   port: number; // Flattened from server.port
   basePath: string;
@@ -376,7 +376,7 @@ Source: https://docs.kici.dev/architecture/data-flows/
 
 This document describes the key data flows through the KiCI architecture: webhook delivery, job execution, developer-initiated remote runs, dependency caching, re-run and cancel, trace ID propagation, internal event routing, and generic webhook ingestion.
 
-> **Lock file schema version:** The lock file uses schema version 29. The orchestrator rejects any fetched lock whose `schemaVersion` does not exactly match the engine version it was built against, so a stale lock must be recompiled with `kici compile` and pushed again after any SDK upgrade that bumps the schema.
+> **Lock file schema version:** The orchestrator accepts a compatibility window of lock schema versions rather than an exact match. A lock is accepted when its `schemaVersion` is at or above the orchestrator's oldest supported version (additive bumps add fields older readers ignore) and the orchestrator's own schema is at or above the lock's `minReaderVersion` (the newest breaking version at compile time). A lock below the floor must be recompiled with `kici compile` and pushed; a lock requiring a newer reader means the orchestrator must be upgraded. Both out-of-window cases are rejected with an actionable error rather than a silent mis-route. See [lock file and drift](https://docs.kici.dev/user/lock-file-and-drift/#schema-compatibility-window).
 
 ## Webhook delivery flow
 
@@ -479,7 +479,7 @@ The working-tree overlay tarball uploads **directly** from the developer machine
 
 The run is dispatched to the developer's active organization (selected with `kici org use`, or overridden per-run). The orchestrator anchors its bound organization with a system-managed **remote source** (routing key `remote:<orgId>`) that it auto-provisions — no manual webhook source is required, so a zero-source org is immediately routable for remote runs. The Platform forces the run's routing key to `remote:<orgId>` server-side; the developer never sets a routing key. When an org has more than one connected orchestrator cluster, the CLI selects the target cluster explicitly (or relies on the per-org default), and a single connected cluster is auto-selected.
 
-Remote runs are offered by the Platform; an orchestrator with no Platform connection cannot serve them. Executing workflow steps on the developer machine with no orchestrator is the separate `kici run local` path.
+Remote runs are offered by the Platform; an orchestrator with no Platform connection cannot serve them. Executing workflow steps on the developer machine with no orchestrator is the separate `kici run <event> --local` path.
 
 ## Source and dependency caching flow
 
@@ -582,7 +582,7 @@ Dep cache misses alone do **not** trigger a build job. Deps are platform-specifi
 
 ### Cross-source / no-contentHash workflows
 
-- **Lock files without `contentHash`** (schema v1) skip the source cache entirely; agents compile from source. Regenerate lock files with `kici compile` to enable caching. The current lock file schema version is 29.
+- **Lock files without `contentHash`** (schema v1) skip the source cache entirely; agents compile from source. Regenerate lock files with `kici compile` to enable caching. The current lock file schema version is 31.
 - **Cross-source / global-workflow dispatch** (a workflow registered against source A fired by a webhook on source B) bypasses both caches. The registration's lock file entry still carries `contentHash`, but the cross-source path always clone-and-installs — the eval temp dir doesn't ship `@kici-dev/sdk`. The execution agent still verifies `contentHash` against the cloned source for drift detection.
 
 ### Build deduplication
@@ -670,7 +670,7 @@ Agents receive pre-signed S3 GET URLs (15-minute expiry) directly in `job.dispat
 
 The source/dep cache above is internal: the orchestrator owns its keys and decides when to hit or build. The **user-facing cache** is driven by the workflow author — the declarative `cache: { key, paths, restoreKeys? }` on a job/step, or the imperative `ctx.cache.restore()` / `ctx.cache.save()` API (see [SDK caching reference](https://docs.kici.dev/user/sdk/caching/)). It reuses the same object-storage backend and the same direct-to-storage presigned-URL transport, but the agent — not the orchestrator — initiates each restore and save over WebSocket.
 
-The agent's cache module archives `paths` into a gzipped tarball (computing a SHA-256 over the bytes) and streams downloads back through a checksum-verified extract pipeline. The orchestrator's `UserCache` owns the `cache/<orgId>/<repoId>/<scope>/<key>` namespacing, the immutable first-save check, the `restoreKeys` prefix scan, the two-phase atomic save, and per-org quota/TTL eviction.
+The agent's cache module archives `paths` into a gzipped tarball (computing a SHA-256 over the bytes) and streams downloads back through a checksum-verified extract pipeline. The orchestrator's `UserCache` owns the `cache/<orgId>/<repoId>/<scope>/<key>-<discriminator>` namespacing (the discriminator is a hash of the exact cache key, so two keys differing only by case stay two objects on a case-insensitive store), the immutable first-save check, the `restoreKeys` prefix scan, the two-phase atomic save, and per-org quota/TTL eviction.
 
 ### Restore flow
 
@@ -951,7 +951,7 @@ POST /webhook/:orgId/generic/:sourceId
 
 ## Database topology
 
-The orchestrator owns its own PostgreSQL database, with the authoritative `execution_runs`, `execution_jobs`, `execution_steps`, `dispatch_queue`, `dedup_cache`, `workflow_registrations`, `environments` / `scoped_secrets` / `environment_bindings`, `agent_tokens`, `cluster_meta`, and related tables. Each orchestrator deployment uses its own `KICI_DATABASE_URL`; database users are scoped per service.
+The orchestrator owns its own PostgreSQL database, with the authoritative `execution_runs`, `execution_jobs`, `execution_steps`, `dispatch_queue`, `dedup_cache`, `workflow_registrations`, `contexts` / `scoped_secrets` / `context_bindings`, `agent_tokens`, `cluster_meta`, and related tables. Each orchestrator deployment uses its own `KICI_DATABASE_URL`; database users are scoped per service.
 
 ## Execution reporting flow
 
@@ -1135,7 +1135,7 @@ Output chaining allows steps to consume outputs from preceding steps (within a j
 When workflow code runs at definition time (`step()`, `job()` calls):
 
 - `step()` creates an `OutputProxy<T>` via `createStepOutputProxy(stepName)` and attaches it as `.result`
-- `job()` creates an `OutputProxy<any>` via `createJobOutputProxy(jobName)` and attaches it as `.result`
+- `job()` creates an `OutputProxy<TOutputs>` (the job's inferred output shape — nested by step name for a multi-step job, flat for the `run:` shorthand) via `createJobOutputProxy(jobName)` and attaches it as `.result`, so cross-job reads type-check
 - The proxy is an ES6 `Proxy` object that defers all property access to a module-global `OutputsMap`
 - No outputs exist yet -- accessing `.result.field` before execution throws "has not produced outputs yet"
 
@@ -1150,7 +1150,7 @@ The compiler processes the workflow definition:
 
 ### Execution time (local test runner)
 
-When `kici run local` runs a workflow:
+When `kici run <event> --local` runs a workflow:
 
 1. **SDK module resolution:** The runner resolves `setStepOutputsMap` / `setJobOutputsMap` from the same `@kici-dev/sdk` module instance that the workflow uses (ensures the proxy reads from the same map)
 2. **Map injection:** Fresh `OutputsMap` and `StepRefMap` are created and injected via `setStepOutputsMap()` / `setStepRefMap()` before each job
@@ -1219,7 +1219,7 @@ The Platform tier exposes a `/ws/browser` WebSocket endpoint for dashboard clien
 - [Architecture overview](https://docs.kici.dev/architecture/overview/) -- three-tier model and component responsibilities
 - [Protocol messages](https://docs.kici.dev/architecture/protocol-messages/) -- WebSocket message schemas
 - [Event system internals](https://docs.kici.dev/architecture/webhooks/event-system/) -- event router, registration model, cron scheduler
-- [State machine](https://docs.kici.dev/architecture/execution/state-machine/) -- job execution state transitions
+- [Execution lifecycle](https://docs.kici.dev/architecture/execution/state-machine/) -- run, job, and step status vocabularies and terminal states
 - [Webhook delivery](https://docs.kici.dev/architecture/webhooks/webhook-delivery/) -- detailed webhook processing pipeline
 - [Operator: dependency caching](https://docs.kici.dev/operator/dependency-caching/) -- configuration guide
 - [Operator: monitoring & tracing](https://docs.kici.dev/operator/observability/monitoring/) -- trace fields and Loki queries
@@ -1248,6 +1248,7 @@ flowchart LR
     AGENT_B["Agent\n(arm64)"]
 
     GH -- "HTTP\n(webhooks)" --> PLATFORM
+    GH -. "HTTP direct webhook\n(hybrid / observed / independent)" .-> ORCH_A
     PLATFORM <-- "WebSocket\n(relay + telemetry)" --> ORCH_A
     PLATFORM <-- "WebSocket\n(relay + telemetry)" --> ORCH_B
     ORCH_A <-- "WebSocket P2P\n(reroute + progress\n+ Raft)" --> ORCH_B
@@ -1260,7 +1261,19 @@ flowchart LR
 
 **Why three tiers?** Trust boundaries. The Platform relay never sees customer code -- it only verifies webhook signatures and forwards payloads. The orchestrator matches triggers against the lock file without cloning repositories. Only the agent, running on customer infrastructure, clones code and executes steps.
 
-This model also supports an independent deployment without the hosted Platform: the orchestrator and agent run on customer infrastructure, with the orchestrator receiving webhooks directly from GitHub instead of through the Platform relay.
+This model also supports pointing webhooks **directly** at the orchestrator,
+bypassing the Platform relay. In **hybrid** mode the orchestrator keeps its
+Platform connection for the dashboard and telemetry while GitHub delivers events
+straight to the orchestrator's ingress — so a Platform outage never drops a build
+trigger. **Observed** mode drops the relay leg entirely: webhooks reach only the
+orchestrator's own ingress (no payload ever transits KiCI) while the Platform
+connection stays up for the hosted dashboard, and the orchestrator's sources
+register as observe-only — recorded and dashboard-visible, but excluded from
+every relay-candidate lookup. In a fully **independent** deployment the
+orchestrator and agent run on customer infrastructure with no Platform at all,
+receiving webhooks directly. For
+exactly which capabilities the hosted Platform provides in each case, see
+[What requires the hosted Platform](https://docs.kici.dev/operator/orchestrator/platform-capabilities/).
 
 ## Component responsibilities
 
@@ -1275,7 +1288,7 @@ The Platform never processes, stores, or executes customer code, and never sees
 The orchestrator is the execution brain. It decides what to run and dispatches work to agents.
 
 - **Trigger matching** -- Evaluates lock file triggers against webhook payloads to determine which jobs to run. Uses branch, path, and event matching via picomatch.
-- **Lock file caching** -- Fetches `kici.lock.json` from the configured provider's API (GitHub, generic webhook, universal-git, or internal). An LRU cache wraps the per-provider fetcher, keyed by `{provider}:{repo}:{ref}` so cross-provider fallback resolutions stay isolated.
+- **Lock file caching** -- Fetches `kici.lock.json` via the configured source's fetcher (GitHub API, universal-git clone for generic webhook sources backed by a git URL, or the local filesystem for `file://` sources). An LRU cache wraps the per-provider fetcher, keyed by `{provider}:{repo}:{ref}` so cross-provider fallback resolutions stay isolated.
 - **Agent registry** -- Tracks connected agents with label-based routing for job dispatch.
 - **Job queue** -- PostgreSQL-backed FIFO queue for reliable dispatch.
 - **Webhook pipeline** -- Dedup, event mapping, lock file fetch, trigger matching, and job dispatch in a single pipeline.
@@ -1290,12 +1303,13 @@ The orchestrator is the execution brain. It decides what to run and dispatches w
 The agent is the execution worker. It runs on customer infrastructure and has full access to customer code.
 
 - **Repository cloning** -- Clones the target repo with token-based auth (token in HTTP headers, not URLs, to prevent leakage).
-- **Step execution** -- Runs steps sequentially with full `StepContext` (zx shell, logger, environment, workflow/job metadata).
-- **Docker support** -- Container-based step execution via `docker exec` for isolated environments.
+- **Step execution** -- Runs steps in declaration order with full `StepContext` (zx shell, logger, environment, workflow/job metadata). Steps wrapped in a `parallel()` group run concurrently behind a `maxParallel` window, and each child reports as its own observable step with its own logs, status, timing, and retry.
+- **Execution sandboxes** -- Runs the workflow runner as a separate child process with a sanitized environment, in one of three sandboxes: bare metal (process fork, with optional bubblewrap namespace isolation), a container runtime (the whole job lifecycle runs inside a disposable container), or inside a Firecracker microVM. Agent-internal credentials never reach customer workflow code.
 - **Log streaming** -- Chunked log streaming back to the orchestrator with configurable size limits.
+- **Dependency caching** -- Packs, uploads, and restores installed workflow dependencies so repeat runs skip the install step.
 - **Graceful shutdown** -- SIGTERM with 10s grace period, SIGUSR1 for drain mode.
 
-> Source: `packages/agent/src/execution/job-runner.ts` (job lifecycle), `packages/agent/src/server.ts` (entry point)
+> Source: `packages/agent/src/execution/job-runner.ts` (job lifecycle), `packages/agent/src/execution/sandbox/` (execution sandboxes and the parallel step scheduler), `packages/agent/src/server.ts` (entry point)
 
 ## Supporting packages
 
@@ -1306,14 +1320,22 @@ Shared business logic used by all three tiers. Single source of truth for cross-
 - Protocol message schemas (Zod-based, direction-specific unions including dashboard REST-over-WS, browser live streaming, the test-relay control plane, log pull, run events, peer-to-peer, cluster join, and source registration)
 - Provider interfaces (WebhookNormalizer, LockFileFetcher, ChangedFilesFetcher, CloneTokenProvider, RepoUrlBuilder, ContributorResolver, CheckStatusPoster)
 - Trigger matching engine (branch, path, event evaluation)
-- Execution state machine (11 states, 16 events, pure functions)
+- Dispatch inputs (input descriptors, extraction from the trigger event, and coercion to typed values)
+- Matrix expansion and fanout (combination expansion with include/exclude, job-name suffix formatting, and materialization of one matrix or multi-host job into N dispatchable children)
+- Execution status vocabulary (run/job/step status enums + terminal-state sets; lifecycle owned by the orchestrator's execution tracker)
+- Check mode (the idempotent run modes `apply` / `check` / `check-fail-on-drift` and the per-step outcome vocabulary)
 - Webhook signature verification (HMAC-SHA256, timing-safe)
 - WebSocket close codes (unified across all tiers)
 - WebSocket rate limiting (WsRateLimiter)
 - Environment allowlist (safe env var filtering)
 - Secrets management (secret context resolution)
-- Environment model (scoped secrets, env merge, protection gates)
+- Context model (scoped secrets, ordered context merge, protection gates)
+- Approval requirements (normalized approver clauses shared by the orchestrator gate, the resolver, the held-run store, and the agent step round-trip)
+- Build provenance (in-toto statement schema, DSSE envelope, attestation bundle, verification)
+- Artifact name contract (the shared filesystem/URL-safe name schema the orchestrator, agent, and SDK all validate against)
+- Developer MCP tool schemas (argument schemas for the AI-agent tool surface)
 - Label utilities (platform label derivation, runsOn normalization, `kici:*` set-only reserved namespace, role labels)
+- Host inventory (the canonical queryable host-roster schema shared by the orchestrator's roster store, the agent-facing inventory API, and the SDK's `ctx.kici.inventory`)
 - Audit policy and retention (per-action access-log sampling, warm-retention windows for cold-store eligibility, federated activity row schema)
 - Scaler backend type enum (`container`, `bare-metal`, `firecracker`, `kubernetes`)
 - Registration trigger type enum (registerable trigger discriminator)
@@ -1331,7 +1353,9 @@ User-facing SDK for defining workflows in TypeScript. Provides factory functions
 
 CLI tooling for workflow authors. Compiles `.kici/workflows/*.ts` to `.kici/kici.lock.json`, provides watch mode, local test execution, project initialization, and pre-commit hook integration.
 
-> Source: `packages/compiler/src/`
+It also runs the **local dev plane** -- an on-demand, fully local execution stack (embedded PostgreSQL, an orchestrator process, and a bare-metal-scaled agent) that lets an author run a workflow end-to-end on their own machine. That is why the compiler depends on `@kici-dev/orchestrator` and `@kici-dev/agent`: it resolves and spawns their built entry points rather than reimplementing them. See [Local dev plane](https://docs.kici.dev/operator/orchestrator/local-dev-plane/).
+
+> Source: `packages/compiler/src/` (`local-plane/` for the local dev plane)
 
 ### `@kici-dev/core`
 
@@ -1357,7 +1381,7 @@ Unscoped wrapper package that provides the `kici` CLI command. Re-exports `@kici
 
 ### `kici-admin` (admin CLI wrapper)
 
-Unscoped wrapper package that provides the `kici-admin` CLI command. Re-exports `@kici-dev/orchestrator/cli` for orchestrator administration tasks.
+Unscoped wrapper package that ships two binaries: `kici-admin`, which re-exports `@kici-dev/orchestrator/cli` for orchestrator administration tasks, and `kici-agent`, which re-exports `@kici-dev/agent/server` to run an agent. It therefore depends on both `@kici-dev/orchestrator` and `@kici-dev/agent` (the `KICIADMIN → AGENT` edge in the graph below).
 
 > Source: `packages/kici-admin/`
 
@@ -1380,10 +1404,13 @@ flowchart TD
     DASH --> ENGINE
     DASH -.->|dev| PLATFORM
     SHARED --> CORE
+    SHARED --> ENGINE
     SDK --> ENGINE
     SDK --> CORE
     COMPILER --> ENGINE
     COMPILER --> CORE
+    COMPILER --> ORCH
+    COMPILER --> AGENT
     COMPILER -.->|peer| SDK
     PLATFORM --> ENGINE
     PLATFORM --> SHARED
@@ -1402,10 +1429,12 @@ flowchart TD
     KICIADMIN --> AGENT
 ```
 
-**Leaf packages** (no `@kici` dependencies): `@kici-dev/core` and `@kici-dev/engine`. These can be tested and built independently. `@kici-dev/shared` builds on `@kici-dev/core` and re-exports it. The dashboard depends on `@kici-dev/engine` for shared types (protocol schemas, state machine) and imports the Platform's API type definitions as a dev dependency, but communicates with backend services at runtime via HTTP/WebSocket, not at compile time.
+**Leaf packages** (no `@kici` dependencies): `@kici-dev/core` and `@kici-dev/engine`. These can be tested and built independently. `@kici-dev/shared` builds on `@kici-dev/core` (which it re-exports) and on `@kici-dev/engine` for shared vocabularies. The dashboard depends on `@kici-dev/engine` for shared types (protocol schemas, execution status enums) and imports the Platform's API type definitions as a dev dependency, but communicates with backend services at runtime via HTTP/WebSocket, not at compile time.
 
 **Runtime tiers** (Platform, orchestrator, agent) all depend on `@kici-dev/engine` for shared business logic and `@kici-dev/shared` for utilities. Only the agent depends on `@kici-dev/sdk` (it loads workflow definitions at runtime).
 
+The `COMPILER → ORCH` and `COMPILER → AGENT` edges exist solely for the local dev plane: the compiler spawns a local orchestrator and agent so an author can execute a workflow end-to-end without any deployed infrastructure. Nothing in the compile path itself reaches into either tier.
+
 ## Connection overview
 
 KiCI uses three WebSocket layers for real-time communication.
@@ -1433,7 +1462,7 @@ KiCI uses application-level tenant isolation. The Platform dashboard API accepts
 ## See also
 
 - [Multi-Orchestrator Architecture](https://docs.kici.dev/architecture/clustering/multi-orchestrator/) -- P2P clustering, Raft consensus, job rerouting
-- [State Machine](https://docs.kici.dev/architecture/execution/state-machine/) -- execution lifecycle tracking across all tiers
+- [Execution lifecycle](https://docs.kici.dev/architecture/execution/state-machine/) -- run, job, and step status vocabularies and the tracker that owns lifecycle state
 - [Protocol Messages](https://docs.kici.dev/architecture/protocol-messages/) -- WebSocket message schemas for all three layers
 - [Webhook Delivery](https://docs.kici.dev/architecture/webhooks/webhook-delivery/) -- end-to-end trace of a webhook through all three tiers