@kici-dev/compiler 0.6.0 → 0.7.0

package/dist/llm-context/llms-architecture.txt

@@ -29,7 +29,7 @@ interface LocalConfig {
 
 ### SharedConfig
 
-Shared settings stored in the PostgreSQL `config_versions` table. Defined once, shared across all instances:
+Settings stored in the PostgreSQL `config_versions` table, written and read by the `/admin/config` routes and their `kici-admin config` commands:
 
 ```typescript
 interface SharedConfig {
@@ -73,11 +73,13 @@ interface AppConfig {
 
 ### How they merge
 
+`resolveFullConfig()` takes a `LocalConfig` and an optional `SharedConfig` and merges them:
+
 ```
 defaults (getDefaults())
     |
     v
-SharedConfig (from DB)        ──deepMerge──>  merged layer 1+2
+SharedConfig (argument)        ──deepMerge──>  merged layer 1+2
     |
     v
 LocalConfig (from YAML)       ──deepMerge──>  merged layer 1+2+3
@@ -94,29 +96,48 @@ appConfigSchema.safeParse()    ──validate──>   typed AppConfig
 
 The `deepMerge` function merges objects recursively, replaces arrays (does not merge item-by-item), and skips `undefined`/`null` source values (they do not override existing values).
 
+**The `SharedConfig` argument is `null` in the shipped wiring.** `ConfigReloader` is the only non-test caller of `resolveFullConfig()`, and it is constructed with `sharedStore: null`. So the DB layer is skipped and the effective chain is defaults → YAML → env. The `config_versions` table is read by the `/admin/config` write and inspection routes, by `kici-admin rotate-key`, and by the cluster join flow — never by a running orchestrator's own config.
+
 ## Resolution chain
 
-### Two-phase design
+### Startup
+
+`server.ts` and `standalone.ts` both call `loadConfig()`, which parses `KICI_*` environment variables against the flat schema in `config.ts`. No YAML file and no database row participates:
 
 ```
-Phase 1 (local-only):
-  YAML file + KICI_ env vars
-      |
-      v
-  resolveLocalConfig() -> { databaseUrl, instanceId, port, mode }
-      |
-      v
-  Connect to PostgreSQL
-      |
-      v
-Phase 2 (full merge):
-  defaults -> DB -> YAML -> env
-      |
-      v
-  resolveFullConfig() -> AppConfig
+Process start
+    |
+    v
+loadConfig() -> envDef.parse(process.env) -> AppConfig
+    |
+    v
+Connect to PostgreSQL, run migrations
+    |
+    v
+Start server (HTTP, WS, scaler, cluster)
 ```
 
-**Why two phases?** The database URL must come from local config (YAML or env var) because we need it to connect to PostgreSQL. But the shared config is stored in PostgreSQL. This circular dependency is broken by resolving local config first (Phase 1), connecting to the DB, then doing the full merge (Phase 2).
+The database URL therefore has to be an environment variable: the orchestrator needs it to reach PostgreSQL, and the shared config lives in PostgreSQL.
+
+### Reload
+
+`resolveLocalConfig()` and `resolveFullConfig()` run on the reload path, not at startup:
+
+```
+SIGHUP / POST /admin/config/reload / kici-admin config reload
+    |
+    v
+resolveLocalConfig()  -> YAML file + KICI_ env overlay
+    |
+    v
+resolveFullConfig(local, null)  -> defaults -> YAML -> env -> AppConfig
+    |
+    v
+Hold databaseUrl, port, instanceId and storage at their startup values
+    |
+    v
+Atomic swap into ConfigReloader.currentConfig
+```
 
 ### Env var processing
 
@@ -128,43 +149,6 @@ Environment variables are processed in two stages:
 
 Type coercion is applied based on known field types: numeric fields are parsed as numbers, boolean fields are compared against `"true"`, all others remain strings.
 
-## Two-phase bootstrap
-
-```
-┌─────────────────┐
-│  Process Start   │
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│  Load YAML +    │  resolveLocalConfig()
-│  Env Overrides  │  -> databaseUrl, instanceId, port, mode
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│  Connect to     │  PostgreSQL
-│  Database       │  Run migrations
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│  Load Shared    │  SharedConfigStore.getLatest()
-│  Config from DB │  -> decrypt -> SharedConfig
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│  Full Merge     │  resolveFullConfig(local, db, env)
-│  + Validate     │  -> AppConfig
-└────────┬────────┘
-         │
-         ▼
-┌─────────────────┐
-│  Start Server   │  HTTP, WS, scaler, cluster
-└─────────────────┘
-```
-
 ## DB schema
 
 ### config_versions table
@@ -264,7 +248,7 @@ flowchart TD
     trigger --> execute["executeReload()<br/>boolean mutex"]
 
     execute --> resolveLocal["resolveLocalConfig()"]
-    execute --> getLatest["getLatest (DB)"]
+    execute --> getLatest["getLatest (DB)<br/>skipped: sharedStore is null"]
     execute --> resolveFull["resolveFullConfig()<br/>(merge + validate)"]
 
     resolveLocal --> check["Check restart-required fields"]
@@ -283,26 +267,26 @@ flowchart TD
 - **Mutex:** Boolean flag prevents concurrent reloads. Second reload returns `{ success: false, errors: ["Reload already in progress"] }`.
 - **Debounce:** Rapid triggers (e.g., multiple SIGHUP signals) are collapsed into a single reload with a 500ms window.
 - **Validation before swap:** The new config must pass full schema validation. On failure, the old config is preserved and an error is logged.
-- **Restart-required detection:** Fields like `databaseUrl`, `port`, `instanceId` are compared. If changed, the old values are preserved in the applied config and a warning is logged.
+- **Restart-required detection:** `databaseUrl`, `port`, `instanceId` and `storage` are compared. If changed, the old values are preserved in the applied config and a warning is logged.
 - **No crash on failure:** The orchestrator always keeps running with the old config if anything goes wrong during reload.
 
 ### Subsystem callbacks
 
 The `ConfigReloader` uses a dependency injection pattern with callbacks for subsystem re-initialization:
 
-| Callback              | When Called                   | Purpose                                                                                      |
-| --------------------- | ----------------------------- | -------------------------------------------------------------------------------------------- |
-| `onProviderChange`    | Provider config changed       | Reserved callback (providers are now DB-managed via sources table; currently always a no-op) |
-| `onScalerReload`      | Always on successful reload   | Reload scaler YAML config                                                                    |
-| `onPlatformReconnect` | Platform URL or token changed | Reconnect WS to Platform relay                                                               |
-| `onConfigApplied`     | Always on successful reload   | Atomic config reference swap, increment local config version                                 |
+| Callback              | When Called                   | Purpose                                                                                                                        |
+| --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| `onProviderChange`    | Provider config changed       | Reserved callback. Providers are DB-managed via the sources table, so the change detector always reports no change             |
+| `onScalerReload`      | Always on successful reload   | Reload scaler YAML config, from the path the process started with                                                              |
+| `onPlatformReconnect` | Platform URL or token changed | Logs that the Platform connection settings changed. The connection is not re-established; `standalone.ts` registers no handler |
+| `onConfigApplied`     | Always on successful reload   | Atomic config reference swap, increment local config version                                                                   |
 
 ### Prometheus metrics
 
-| Metric                          | Type    | Labels                                                                  | Description                           |
-| ------------------------------- | ------- | ----------------------------------------------------------------------- | ------------------------------------- |
-| `kici_orch_config_reload_total` | Counter | `result` (attempted/success/failed), `source` (sighup/http/cluster/cli) | Config reload attempts and outcomes   |
-| `kici_orch_config_version`      | Gauge   | --                                                                      | Current shared config version from DB |
+| Metric                          | Type    | Labels                                                                  | Description                                                                                                    |
+| ------------------------------- | ------- | ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
+| `kici_orch_config_reload_total` | Counter | `result` (attempted/success/failed), `source` (sighup/http/cluster/cli) | Config reload attempts and outcomes                                                                            |
+| `kici_orch_config_version`      | Gauge   | --                                                                      | Shared config version from the DB. Set only when the reload path reads a version, so it carries no value today |
 
 ## Multi-Provider
 
@@ -334,40 +318,42 @@ Each source record contains its own `appId` and `privateKey` (stored as scoped s
 
 ### Heartbeat config version
 
-In clustered deployments, each orchestrator includes its current config version in Raft heartbeat metadata via the `configVersion` optional field on the `peerHeartbeatSchema`.
+In clustered deployments, each orchestrator includes its config version in Raft heartbeat metadata via the `configVersion` optional field on the `peerHeartbeatSchema`.
+
+**That number is a local reload counter, not a shared config version.** `onConfigApplied` increments it on every successful reload and publishes the new value to the peer registry. It counts how many times this instance has reloaded.
 
 When the `PeerRegistry` processes a heartbeat:
 
 1. Compare `localConfigVersion` with `peer.configVersion`
 2. If `peer.configVersion > localConfigVersion` AND both are > 0:
    - Invoke the `onConfigVersionBehind` callback
-   - This triggers a config reload from the database
+   - This triggers a config reload, which re-reads the environment and the local YAML file
 
 ### Auto-remediation flow
 
 ```
-Orchestrator A (version 5)          Orchestrator B (version 3)
+Orchestrator A (reloaded 5x)        Orchestrator B (reloaded 3x)
          │                                     │
          │──── heartbeat(configVersion=5) ────>│
          │                                     │
          │                           compare: 5 > 3
-         │                           trigger reload from DB
+         │                           trigger reload
          │                                     │
-         │                           resolveFullConfig()
-         │                           -> version 5
+         │                           resolveFullConfig(local, null)
+         │                           -> counter becomes 4
          │                                     │
-         │<── heartbeat(configVersion=5) ──────│
+         │<── heartbeat(configVersion=4) ──────│
          │                                     │
-         │   both at version 5 ✓               │
 ```
 
+B converges on A's count only after it has reloaded as many times as A has. Each instance reads its own environment and its own YAML file, so the two agree on content only when those inputs agree.
+
 ### Guard conditions
 
 - Version comparison only triggers when **both** local and peer versions are > 0
 - This prevents false triggers from:
-  - Legacy orchestrators that do not report `configVersion` (field is optional, defaults to 0)
-  - Newly started orchestrators before their first config load
-- The `localConfigVersion` is a monotonically incrementing local counter (incremented on each successful reload)
+  - Orchestrators that do not report `configVersion` (field is optional, defaults to 0)
+  - Newly started orchestrators before their first reload
 
 ## See also
 
@@ -398,7 +384,7 @@ GitHub  -->  Platform Relay  -->  Orchestrator  -->  Agent
 
 1. **Provider sends webhook** to the Platform relay endpoint.
 2. **Platform routes the webhook** to the right orchestrator over WebSocket and forwards the body bytes verbatim. Platform never sees customer HMAC secrets — signature verification happens entirely on the orchestrator after reassembly.
-3. **Orchestrator verifies signature** (HMAC-SHA256 against per-source webhook secret, with dual-secret rotation support).
+3. **Orchestrator admits the delivery**, then **verifies the signature** (HMAC-SHA256 against per-source webhook secret, with dual-secret rotation support). Admission runs first, on the routing key alone: when the ingest admission controller sheds, the orchestrator records an `event_log` breadcrumb with status `shed` and ACKs `shed_retry_later`, which the Platform answers as **429** with `Retry-After`. See [ingest admission shed](https://docs.kici.dev/architecture/webhooks/webhook-delivery/#ingest-admission-shed-step-3).
 4. **Orchestrator dedup check** against dual-layer `DedupCache` (in-memory set + `dedup_cache` DB table).
 5. **Orchestrator resolves provider** by looking up the provider bundle from the `ProviderRegistry` using `getByRoutingKey()` (exact match first, falls back to provider type prefix for backward compatibility). Skips processing if the provider is unknown.
 6. **Orchestrator normalizes** the webhook via the provider's `WebhookNormalizer` (extracts branch, event type, action, sender).
@@ -525,7 +511,7 @@ Build Job Dispatch --> Build Agent (kici:role:builder + matching kici:os:/kici:a
   |                      |-- npm ci in .kici/
   |                      |-- Pack .kici/ source (portable tar.gz, excludes node_modules)
   |                      |-- Pack .kici/node_modules (portable tar.gz)
-  |                      |-- Upload source tarball to cache (source/{contentHash}.tar.gz)
+  |                      |-- Upload source tarball to cache (source/v2/{orgId}/{sourceTarDigest}.tar.gz)
   |                      |-- Upload deps tarball to cache (deps/{plat}-{arch}/{depsHash}.tar.gz)
   |                      |-- Upload deps companion .hash file
   |                      |-- Report success (cache.upload.complete × 2)
@@ -600,7 +586,7 @@ Dep cache misses alone do **not** trigger a build job. Deps are platform-specifi
 
 ### Cross-source / no-contentHash workflows
 
-- **Lock files without `contentHash`** (schema v1) skip the source cache entirely; agents compile from source. Regenerate lock files with `kici compile` to enable caching. The current lock file schema version is 39.
+- **Lock files without `contentHash`** (schema v1) skip the source cache entirely; agents compile from source. Regenerate lock files with `kici compile` to enable caching. The current lock file schema version is 41.
 - **Cross-source / global-workflow dispatch** (a workflow registered against source A fired by a webhook on source B) bypasses both caches. The registration's lock file entry still carries `contentHash`, but the cross-source path always clone-and-installs — the eval temp dir doesn't ship `@kici-dev/sdk`. The execution agent still verifies `contentHash` against the cloned source for drift detection.
 
 ### Build deduplication
@@ -639,7 +625,7 @@ Both source and dep caches use `S3CacheStorage` as the sole backend. The `CacheS
 
 Cache keys reflect that source tarballs and deps have different platform characteristics:
 
-- **Source:** `source/{contentHash}.tar.gz` — platform-agnostic. Raw TypeScript source is identical regardless of CPU architecture, so one entry is shared across all platforms. `contentHash` is the per-workflow hash from the lock file (`SHA-256(COMPILE_SCHEMA_VERSION + ":" + rawSource [+ "\0" + assetDigest])`, where `COMPILE_SCHEMA_VERSION = 5` and line endings are normalized to LF so the hash agrees across platforms).
+- **Source:** `source/v2/{orgId}/{sourceTarDigest}.tar.gz`, with a `source/v2/{orgId}/{contentHash}.hash` pointer — platform-agnostic, and scoped to the owning organization so two repositories with matching `.kici/` trees never share one object. Raw TypeScript source is identical regardless of CPU architecture, so one entry is shared across all platforms. `contentHash` is the per-workflow hash from the lock file (`SHA-256(COMPILE_SCHEMA_VERSION + ":" + rawSource [+ "\0" + assetDigest])`, where `COMPILE_SCHEMA_VERSION = 7` and line endings are normalized to LF so the hash agrees across platforms).
 - **Deps:** `deps/{platform}-{arch}/{depsHash}.tar.gz`, with a
   `deps/{platform}-{arch}/{lockfileHash}.hash` pointer holding that hash — the
   tarball is addressed by its own content, so two builds sharing a lock file
@@ -1370,14 +1356,14 @@ Shared business logic used by all three tiers. Single source of truth for cross-
 - Label utilities (platform label derivation, runsOn normalization, `kici:*` set-only reserved namespace, role labels)
 - Host inventory (the canonical queryable host-roster schema shared by the orchestrator's roster store, the agent-facing inventory API, and the SDK's `ctx.kici.inventory`)
 - Audit policy and retention (per-action access-log sampling, warm-retention windows for cold-store eligibility, federated activity row schema)
-- Scaler backend type enum (`container`, `bare-metal`, `firecracker`, `kubernetes`, `event`) and the reserved `kici.` event-name prefix that keeps a user step from forging a system event
+- Scaler backend type enum (`container`, `bare-metal`, `firecracker`, `kubernetes`, `event`; the orchestrator config rejects `kubernetes`) and the reserved `kici.` event-name prefix that keeps a user step from forging a system event
 - Job resource vocabulary (the requests/limits shape the SDK accepts, the compiler validates and emits, the orchestrator uses for capacity math and kernel-side enforcement, and the dashboard displays)
 - Registration trigger type enum (registerable trigger discriminator)
 - Sandbox capability set (the Linux capability names a container sandbox may add or drop, shared by the SDK validator, the compiler, and the dispatch resolver)
 - Plan tier vocabulary (the hosted plan tiers and the purchasable subset, shared by the Platform and the browser dashboard)
 - Infrastructure alert vocabulary (the diagnostics alert types and severities the Platform mints and the dashboard and `kici` CLI render)
 - Metric catalog (the generated Prometheus metric inventory, its naming policy, and metric-kind compatibility checks)
-- Bundler config (shared bundler configuration consumed by `e2e/helpers/service-deploy.ts`; the agent runtime uses the `@kici-dev/core/ts-loader-hook` to transform TypeScript on import, with no runtime bundler step)
+- Bundler config (the shared workflow-bundle configuration factory on the barrel; the agent runtime uses the `@kici-dev/core/ts-loader-hook` to transform TypeScript on import, so no runtime path bundles a workflow)
 
 > Source: `packages/engine/src/`
 
@@ -1397,7 +1383,7 @@ It also runs the **local dev plane** -- an on-demand, fully local execution stac
 
 ### `@kici-dev/core`
 
-Light shared utilities with no server-side dependencies. It provides JSON-structured logging, error helpers, async-local-storage request context, and human-readable formatting (`formatBytes`/`formatDuration`/`formatUptime`). It also provides cryptographic helpers (`sha256`/`sha256File`/`deriveSharedSecret` plus symmetric encrypt/decrypt), retry-backoff computation, and the shared diagnostics-result contract. The rest of its surface ships as subpath entry points: the temp-directory allocator and its garbage collector, package-manager detection, CI-environment detection, and the idempotent-step runner (the check / confirm / apply primitive behind idempotent steps). Finally it supplies zx initialization (`initZx()`) and the TypeScript loader hook that transforms TypeScript on import. It is the dependency-light core that the SDK, compiler, and `kici` CLI consume directly so they stay free of heavier server-only dependencies. `@kici-dev/shared` re-exports it, so existing `@kici-dev/shared` import paths keep working.
+Light shared utilities with no server-side dependencies. It provides JSON-structured logging, error helpers, async-local-storage request context, and human-readable formatting (`formatBytes`/`formatDuration`/`formatUptime`). It also provides cryptographic helpers (`sha256`/`sha256File`/`deriveSharedSecret` plus symmetric encrypt/decrypt), retry-backoff computation, and the shared diagnostics-result contract. The rest of its surface ships as subpath entry points: the temp-directory allocator and its garbage collector, package-manager detection, CI-environment detection, and the idempotent-step runner (the check / confirm / apply primitive behind idempotent steps). One further subpath holds the `.kici/` source digest: the single content-hash definition the compiler writes into the lock file and the agent recomputes as its drift gate. Finally it supplies zx initialization (`initZx()`) and the TypeScript loader hook that transforms TypeScript on import. It is the dependency-light core that the SDK, compiler, and `kici` CLI consume directly so they stay free of heavier server-only dependencies. `@kici-dev/shared` re-exports it, so existing `@kici-dev/shared` import paths keep working.
 
 > Source: `packages/core/src/`