@norskvideo/norsk-auto-manager 0.1.0

package/README.md

@@ -0,0 +1,344 @@
+# `@norskvideo/norsk-auto-manager`
+
+The opinionated orchestration layer above Norsk Manager. Manager exposes a
+mechanical API — start node, terminate node, register job, report events —
+without policy. AutoManager owns policy: where jobs run, how nodes are sized
+and packed, how failures recover, when hot spares are provisioned.
+
+This README is the conceptual guide. The TSDoc comments on each public
+symbol are the authoritative API reference; run `npx tsc --declaration` or
+read `lib/src/automanager.d.ts` for the full signatures.
+
+For the design rationale, see `.ai/steve/tasks/37-auto-manager-design.md`
+in the repo root.
+
+---
+
+## Quick start
+
+```ts
+import { AutoManager, PlacementPool } from "@norskvideo/norsk-auto-manager";
+
+const awsEast: PlacementPool = {
+  name: "aws-east",
+  kind: "aws",
+  region: "eu-west-1",
+  packingStrategy: "binpack",
+  scaleOut: "elastic",
+  candidateInstanceTypes: [
+    { instanceType: "m5.xlarge", totalCapacity: 100, totalCores: 16, capabilities: [] },
+  ],
+};
+
+const [autoManager, runPromise] = await AutoManager.run({
+  url: "manager.example.com:6789",
+  pendingWindow: 60,
+  removeStoppedNodesAfter: 3600,
+  placementPools: [awsEast],
+
+  // Recommended callbacks for ops tooling:
+  onPlacementDecision: (info) => console.log("placement", info),
+  onBundleBroken: (id, reason) => console.warn("bundle broken", id, reason),
+  onConnectionStateChange: (state) => console.log("manager", state),
+});
+
+// AutoManager runs in the background. `runPromise` resolves when the
+// event-stream loop exits (e.g. after `autoManager.close()`).
+```
+
+The `AutoManager.run` call validates the settings up front (throws
+`AutoSettingsValidationError` on shape/typo problems) and connects to
+Manager. Once the event stream is established, AutoManager mirrors
+Manager's state (jobs, nodes, bundles, inventories) and starts running
+placement decisions.
+
+---
+
+## Bundles
+
+A **bundle** is the unit of work you submit. It has one or more **replicas**
+(1 = no backup, 2 = primary + backup, N>2 = N-way redundancy). Each replica
+contains one or more **jobs** — the atomic placeable units.
+
+```ts
+import { Bundle } from "@norskvideo/norsk-auto-manager";
+
+const bundle: Bundle = {
+  bundleId: "my-channel-1",
+  replicas: 2,                      // primary + backup
+  pools: ["aws-east", "aws-west"],  // preference order; spills if first is full
+  jobs: [
+    {
+      jobName: "encode",
+      requirements: {
+        requiredCapacity: 8,        // benchmarked reference-workload units
+        requiredCores: 4,           // explicit cpuset pinning
+        requiredCapabilities: [],   // hardware/software requirements
+        priorityBand: "gold",       // see "Priority bands" below
+      },
+    },
+  ],
+};
+
+await autoManager.createBundle(bundle);
+```
+
+Submit a bundle by calling `autoManager.createBundle(bundle)`. Manager
+expands it into one runtime Job per replica, AutoManager runs placement
+for each, and the workflow begins.
+
+### Placement preferences
+
+`pools` is an ordered preference list. Placement tries the first pool; if
+it can't satisfy the job there, it falls through to the next. Each replica
+can override the bundle default via `replicaOverrides`:
+
+```ts
+replicaOverrides: { 1: { pools: ["oci-fra", "aws-east"] } }
+```
+
+This is the canonical "primary in AWS, backup in OCI, fall back to the
+other on either side" cross-cloud DR pattern.
+
+### Intra/inter-replica rules
+
+Within a replica, jobs can be co-located or kept apart on different
+nodes (`coLocateWith`, `separateFrom` on `BundleJobSpec`). Between
+replicas, the defaults are "different node" (hard) and "same AZ" (soft),
+configurable via `intraReplicaPlacement` and `interReplicaPlacement`.
+
+---
+
+## Priority bands
+
+Every job declares a `priorityBand: "gold" | "silver" | "bronze"`. The band
+governs failure behaviour:
+
+| Band   | On failure       | Restart placement      |
+|--------|------------------|------------------------|
+| gold   | restart          | **prefers hot spare**  |
+| silver | restart          | any available          |
+| bronze | drop (no restart)| n/a                    |
+
+Restart bounds (`maxRestartsPerHour`, default 6) apply to gold and silver:
+if a replica's restart rate exceeds the ceiling, AutoManager marks the
+bundle `broken`, fires `onBundleBroken`, and stops trying. A successful
+restart clears the count for that replica.
+
+Bronze jobs that hit a recovery path fire `onJobDropped` and exit
+silently.
+
+You can configure each band's behaviour explicitly:
+
+```ts
+bands: {
+  gold:   { onFailure: "restart", restartPlacement: "preferHotSpare", maxRestartsPerHour: 6 },
+  silver: { onFailure: "restart", restartPlacement: "any",            maxRestartsPerHour: 6 },
+  bronze: { onFailure: "drop" },
+}
+```
+
+---
+
+## Hot spares
+
+A **hot spare** is a warm-running node that holds no jobs but is ready to
+take one immediately when a recovery placement picks it. Configure per pool
+(and optionally per band):
+
+```ts
+hotSpares: [
+  {
+    pool: "aws-east",
+    targetCount: 2,
+    forBand: "gold",      // optional: dedicate spares to a band
+    spec: { instanceType: "m5.xlarge", region: "eu-west-1" },
+  },
+],
+```
+
+The **reconciler** runs continually:
+
+- Under target → provisions more spares (calls `createAwsNode`/`createOciNode` with the `spare=true` tag).
+- Over target → terminates the oldest excess (supports rotation for security patching).
+- A spare that picks up a job is automatically counted as "no longer a spare"; underflow fires immediately and a replacement is provisioned.
+
+`onSpareUnderflow(pool, current, target)` fires whenever the live count
+drops below the target.
+
+For **cluster pools** (`scaleOut: "fixed"`), the reconciler can't
+auto-scale — it fires `onSpareUnderflow` but leaves the actual hardware
+to the operator's IT process.
+
+---
+
+## Drain / cordon
+
+Take a node out of service gracefully:
+
+```ts
+autoManager.cordonNode(nodeId);    // no new placements land here
+autoManager.uncordonNode(nodeId);  // reverse
+
+autoManager.drainNode(nodeId);     // cordon + terminate (existing jobs
+                                   // recover per band rules)
+
+autoManager.isCordoned(nodeId);    // → boolean
+```
+
+In v1, `drainNode` is the heavy-handed variant: cordon then terminate.
+Manager emits `nodeStopping` for every job hosted on the node, and the
+standard band-aware recovery path re-places them onto other nodes. Gold
+jobs prefer spares; bronze jobs are dropped.
+
+Phase E migration (next section) provides the drop-free variant.
+
+---
+
+## Migration
+
+Move a running job from one node to another with cooperation from the job
+code:
+
+```ts
+const migrationId = await autoManager.migrateJob(jobId, targetNodeId);
+
+// Track progress via callbacks:
+//   onMigrationPhase(state)         — fires on every phase transition
+//   onMigrationCompleted(id)        — handshake completed successfully
+//   onMigrationAborted(id, reason)  — workflow / operator aborted
+
+// Or poll:
+autoManager.migrationById(migrationId);
+autoManager.migrationForJob(jobId);
+autoManager.migrations();
+
+// Operator escape hatch:
+await autoManager.abortMigration(migrationId, "operator changed mind");
+```
+
+The protocol is **cooperative** — the workflow itself must implement the
+cutover hooks (workflow ↔ worker channel; not modelled here). For
+workflows that don't support migration, fall back to `drainNode` (which
+is a frame drop).
+
+No concurrent migrations per source jobId. The target instance wears
+role `"migrationTarget"`; the source keeps its original role.
+
+---
+
+## Metrics
+
+```ts
+const m = autoManager.metrics();
+// {
+//   startedAt: Date,
+//   uptimeMs: number,
+//   placementsAttempted / Succeeded / Failed,
+//   jobsRejected,
+//   bundlesBroken,
+//   jobsDropped,
+//   pools: [{ poolName, nodeCount, liveSpares, spareTarget, usedCapacityFraction }],
+//   brokenBundleIds: BundleId[],
+// }
+```
+
+Counters are cumulative since `startedAt`; per-pool gauges are computed
+at call time. Poll on your own cadence; AutoManager doesn't push.
+
+---
+
+## Validation
+
+Two checks worth running before submitting bundles:
+
+```ts
+// Static shape: replicas count, pool refs, capability satisfiability.
+const errors = autoManager.validateBundle(bundle);
+
+// Static + placement viability: "would each replica place right now?"
+const result = autoManager.validateBundleAdmission(bundle);
+//   → { kind: "ok" }
+//     | { kind: "validationErrors"; errors: ValidationError[] }
+//     | { kind: "noCapacity"; unplaceable: { replicaIndex, jobName, reason }[] }
+```
+
+`AutoSettings` itself is validated by `AutoManager.run` — bad settings
+throw `AutoSettingsValidationError` carrying the structured error list.
+
+---
+
+## Callbacks reference
+
+All optional. None block AutoManager's event loop on error — exceptions
+are caught and forwarded to `onError`.
+
+### Connection / lifecycle
+
+- `onConnectionStateChange(state)` — `"idle" | "connecting" | "ready" | "transientFailure" | "shutdown"`.
+- `onError(err)` — catch-all; back-compat surface.
+
+### Job lifecycle
+
+- `onJobUpdated(jobWithHistory)` — Manager pushed a job change.
+- `onJobDeleted(jobId)` — job removed from the cache.
+- `onJobRejected(info)` — worker rejected a placement (typed).
+- `onJobConfigUpdated(jobId, config)` — runtime config refresh.
+- `onJobDropped(jobId)` — bronze-band recovery dropped the job.
+
+### Node lifecycle
+
+- `onNodeStarting / onNodeStarted / onNodeStopped / onNodeTerminated`.
+- `onNodeUnhealthy(nodeId)` — rejection rate trip.
+
+### Bundle lifecycle
+
+- `onBundleUpdated(bundle)` / `onBundleDeleted(bundleId)`.
+- `onBundleBroken(bundleId, reason)` — restart-rate ceiling tripped.
+
+### Placement
+
+- `onPlacementDecision(info)` — fires every time placement runs, with full trace.
+- `onPlacementFailed(info)` — typed failure callback (also surfaces via `onError`).
+- `onSpareUnderflow(pool, current, target)` — hot-spare reconciler is short.
+
+### Migration
+
+- `onMigrationPhase / onMigrationCompleted / onMigrationAborted`.
+
+---
+
+## Operator-side config overrides (Manager)
+
+Independent of this package but worth mentioning: Manager supports an
+external JSON overrides file at `/var/data/manager-overrides.json` (or
+`NORSK_MANAGER_OVERRIDES_FILE`) for the cloud / region / AZ / instance-
+type fields of provider configs. Lets operators tune those without
+rebuilding the Manager container. See
+`.ai/steve/tasks/37-auto-manager-design.md` §D2 for the shape.
+
+---
+
+## Lifecycle
+
+```ts
+autoManager.close();   // cancels intervals, closes the gRPC channel,
+                       // resolves runPromise
+autoManager.dispose(); // alias for close()
+```
+
+Idempotent. After `close()`, `runPromise` settles.
+
+---
+
+## Worker-side dependencies
+
+Several capabilities depend on worker-side work that lives outside this
+package (in norsk-ctl, by deployment convention):
+
+- **Hardware inventory & capacity scores** — workers advertise their cores, GPUs, NICs, and capabilities. Without this, placement runs against an empty `inventoryMap` and falls back to elastic-pool provisioning.
+- **Worker-side enforcement** — workers refuse incoming starts that exceed their advertised capacity. Without this, AutoManager's optimistic placement isn't backstopped.
+- **Migration handshake** — workers translate the four `NodeControllerMigration*` messages into workflow-level signals, and translate workflow responses back. Without this, `migrateJob` runs the protocol but the workflow never actually performs cutover.
+
+See the `.ai/steve/tasks/37-auto-manager-design.md` §"Worker-side work"
+section for the full norsk-ctl contract.