@happyvertical/smrt-jobs 0.30.0

package/AGENTS.md

@@ -0,0 +1,71 @@
+# @happyvertical/smrt-jobs
+
+Background job execution with persistent queue, scheduling, and fluent builder API.
+
+## Architecture
+
+```
+SmrtObject.bg('method') → SmrtJob (in _smrt_jobs) → TaskRunner picks up → executes via ObjectRegistry
+AgentSchedule (cron) → ScheduleRunner creates SmrtJob → TaskRunner executes → ScheduleRunner updates
+TaskRunner.start() → SmrtWorker lease (in _smrt_workers) → recovery keys on worker liveness, not heartbeat
+```
+
+## SmrtJob
+
+Persistent in `_smrt_jobs`. Fields: `queue` (default), `objectType`, `objectId`, `method`, `args`, `runAt`, `priority` (higher=sooner), `status`, `attempts`/`maxAttempts`, `timeout` (default 5min), `retryStrategy`, `workerId` (the owning runner's incarnation key), `workerHeartbeat` (telemetry only — no longer gates recovery).
+
+Status: `pending → running → completed/failed/cancelled`.
+
+## TaskRunner
+
+Polling-based execution engine. Config: `concurrency` (5), `pollInterval` (1s), `heartbeatInterval` (30s, telemetry only), `leaseTtlMs` (30s), `leaseTickMs` (10s), `shutdownTimeout` (30s).
+
+1. `start()` calls `assertReady()` (fail fast if `_smrt_workers` unmigrated), registers a seeded `SmrtWorker` lease, and adds its worker key to the process-global live set — all **before** polling
+2. Polls `claimReady()` to atomically claim pending jobs (`runAt <= NOW`, ordered by `priority DESC, runAt ASC, created_at ASC, id ASC`)
+3. Claim sets `status='running'`, `workerId=<incarnation key>`, heartbeat/start timestamps, and increments `attempts`
+4. Resolves class via `ObjectRegistry.getClass(objectType)`, creates instance, calls method
+5. **Internal args**: `_agentConfig` and `_scheduleId` stripped from args before calling method
+6. Terminal/retry writes are **conditional** (`WHERE worker_id=? AND status='running'`) so a recovered row is never stomped
+7. Retry: uses strategy from `@happyvertical/jobs`, schedules future `runAt` on failure
+8. Events: `job:started`, `job:completed`, `job:failed`, `job:retrying`, `runner:started/stopped`
+
+## Worker liveness & recovery (#1474)
+
+Recovery keys on **worker-process liveness**, never per-job heartbeat freshness (a CPU-bound synchronous handler used to starve the heartbeat and false-recover its own running jobs).
+
+- **`SmrtWorker` / `_smrt_workers`**: one lease row per runner *incarnation* (`workerId` is per-incarnation unique via `createWorkerKey`, so a restart never looks like it still owns the previous crash's jobs). `leaseExpiresAt` is a `datetime` (an integer epoch-ms column overflows `int4`/`INT32` on Postgres/DuckDB). Stage 1 writes/compares it against the host clock — the same approach the old heartbeat recovery used, so it's no more skew-sensitive than the code it replaced.
+- **Process-global live set** (`worker-liveness.ts`, `globalThis.__smrtLiveWorkers`): checked synchronously, so it can't be starved by a blocked loop. Covers all same-process topologies.
+- **Off-loop ticker** (`worker-liveness-ticker.ts` + `worker-liveness-thread.ts`): for engines a second connection can reach (Postgres, file-backed SQLite — `offLoopEligible()`), `start()` spawns a `node:worker_threads` ticker that renews the lease on its own thread, so a CPU-bound synchronous handler on the main loop can't starve it. In-memory SQLite / DuckDB, a thread-spawn failure, a start-handshake timeout, or the thread dying mid-run all fall back to main-loop renewal; the in-process live set keeps same-process correct regardless. The worker entry is a separate build entry resolved via `import.meta.resolve('@happyvertical/smrt-jobs/worker-liveness-thread')`.
+- **Recovery rule** (both runners): a `running` job is orphaned iff its worker is *not alive* = not in the live set **and** no fresh `_smrt_workers` lease. The live set takes precedence over a stale lease. TaskRunner also never recovers a job in its own `activeJobs`. If `_smrt_workers` is absent, recovery skips lease checks (never mass-recovers). Recovery is swept at most once per lease tick, and terminal/recovery writes use `RETURNING id` (not `rowCount`, which DuckDB/JSON adapters always report as ≥1).
+- **Lease clock**: the lease is compared against the host clock (same as the old heartbeat; fine with NTP + a 30s TTL). A dead process stops renewing and the lease expires within its TTL — that is how recovery detects death. (Instant cross-process detection via Postgres session advisory locks was prototyped on `@happyvertical/sql`'s `acquireSession()` but deferred — treating a free lock as proof-of-death false-recovers any worker legitimately in main-loop fallback mode.)
+
+## ScheduleRunner
+
+Polls `_smrt_agent_schedules` every 60s for due entries. Creates SmrtJob with `queue='agents'`, `priority=75`. Wires to TaskRunner events for completion/failure tracking. Slot reconciliation keys on worker liveness (it has no in-process active-job set, so the lease/live-set is its whole mechanism).
+
+Custom cron parser: 5-field (minute hour dom month dow). `*`, ranges, lists, steps supported. **Not timezone-aware** (UTC).
+
+## JobBuilder — Fluent API
+
+```typescript
+const handle = await doc.background('analyze', { detailed: true })
+  .delay('5m').priority('high').retries(5).queue('analysis').timeout(600000).enqueue();
+
+await handle.wait({ timeout: 60000, pollInterval: 100 }); // polling-based
+```
+
+`bg()` is shorthand: `await doc.bg('analyze', args)` → enqueues immediately, returns JobHandle.
+
+## withBackgroundJobs(Class)
+
+Mixin that adds `bg()` and `background()` to any SmrtObject. Uses WeakMap for collection caching per DB instance.
+
+## Gotchas
+
+- **Cron not timezone-aware**: all times treated as UTC
+- **No dead letter queue**: failed jobs stay in DB with `status='failed'` — manual intervention
+- **Result storage**: `resultPointer` is just a string — app must implement result backend
+- **Lazy builder**: `background()` returns builder — nothing happens until `enqueue()`
+- **wait() is polling**: JobHandle.wait() polls DB every 100ms (configurable)
+- **Migrate before start()**: `TaskRunner.start()` throws if `_smrt_workers` is missing — run `smrt db:migrate` after upgrading. Tables are never created at runtime.
+- **Recovery is liveness-based**: don't reintroduce heartbeat-threshold recovery; a blocked event loop must not look dead (see Worker liveness section, #1474)

package/CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

package/LICENSE

@@ -0,0 +1,7 @@
+Copyright <2025> <Happy Vertical Corporation>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -0,0 +1,151 @@
+# @happyvertical/smrt-jobs
+
+Background job execution for SMRT objects. Provides persistent queue storage, retry strategies, cron-based scheduling, and a fluent `JobBuilder` API via the `withBackgroundJobs()` mixin.
+
+## Installation
+
+```bash
+pnpm add @happyvertical/smrt-jobs
+```
+
+## Usage
+
+### Add background capabilities to a SmrtObject
+
+```typescript
+import { withBackgroundJobs, TaskRunner } from '@happyvertical/smrt-jobs';
+import { Document } from './document.js';
+
+// Mixin adds .bg() and .background() to any SmrtObject class
+const BackgroundDocument = withBackgroundJobs(Document);
+const doc = new BackgroundDocument({ db });
+await doc.initialize();
+
+// Quick enqueue — runs immediately when a TaskRunner picks it up
+const handle = await doc.bg('generateSummary', { format: 'md' });
+
+// Fluent builder for advanced options
+const handle2 = await doc.background('generateSummary', { format: 'md' })
+  .delay('5m')
+  .priority('high')
+  .retries(5)
+  .queue('analysis')
+  .timeout(600000)
+  .enqueue();
+
+// Wait for result (polling-based)
+const result = await handle2.wait({ timeout: 60000, pollInterval: 100 });
+```
+
+### Run a TaskRunner to process jobs
+
+```typescript
+import { TaskRunner } from '@happyvertical/smrt-jobs';
+
+const runner = new TaskRunner({
+  concurrency: 5,
+  pollInterval: 1000,
+  queues: ['default', 'analysis'],
+});
+await runner.initialize(db);
+await runner.start();
+
+// Listen for events
+runner.on('job:completed', (job, result) => { /* ... */ });
+runner.on('job:failed', (job, error) => { /* ... */ });
+
+// Graceful shutdown
+process.on('SIGTERM', () => runner.stop());
+```
+
+### Heartbeat-safe job execution
+
+`TaskRunner` keeps jobs alive with a heartbeat timer. If your job blocks the
+Node.js event loop for longer than the effective stale-job threshold, the runner
+will recover that work as stale and mark it failed.
+
+Common causes:
+
+- `execSync`, `spawnSync`, or other synchronous subprocess APIs
+- long CPU-bound loops in the job method itself
+- large synchronous filesystem work
+
+Prefer async subprocess APIs (`spawn`, `execFile`, streamed stdio) for long
+exports/builds/uploads, or move CPU-heavy work into a separate process or
+worker thread. If a job is intentionally long-running, tune
+`heartbeatInterval` and `staleJobThresholdMs` together so the stale threshold
+comfortably exceeds the longest gap between heartbeats.
+
+`ScheduleRunner` uses the same stale-heartbeat recovery rules when reconciling
+scheduled jobs.
+
+### Schedule recurring jobs with ScheduleRunner
+
+The `ScheduleRunner` polls the `_smrt_agent_schedules` table for due cron entries and creates `SmrtJob` records for the `TaskRunner` to execute. Wire them together via events:
+
+```typescript
+import { ScheduleRunner } from '@happyvertical/smrt-jobs';
+
+const scheduleRunner = new ScheduleRunner({ pollInterval: 30000 });
+await scheduleRunner.initialize(db);
+await scheduleRunner.start();
+
+// Connect TaskRunner events to update schedule state
+taskRunner.on('job:completed', (job) => {
+  const scheduleId = job.args?._scheduleId;
+  if (scheduleId) scheduleRunner.handleJobCompletion(scheduleId, true);
+});
+taskRunner.on('job:failed', (job, error) => {
+  const scheduleId = job.args?._scheduleId;
+  if (scheduleId) scheduleRunner.handleJobCompletion(scheduleId, false, error.message);
+});
+```
+
+### System Tables
+
+| Table | Purpose |
+|-------|---------|
+| `_smrt_jobs` | Persistent job queue (SmrtJob records) |
+| `_smrt_agent_schedules` | Cron schedule entries polled by ScheduleRunner |
+
+## API
+
+### Classes
+
+| Export | Description |
+|--------|------------|
+| `SmrtJob` | Persistent job record stored in `_smrt_jobs` |
+| `SmrtJobCollection` | Collection with `claimReady()`, `listReady()`, `listByStatus()`, `stats()`, `cleanup()` |
+| `JobBuilder` | Fluent API: `.delay()`, `.priority()`, `.retries()`, `.queue()`, `.timeout()`, `.enqueue()` |
+| `JobHandle` | Track, wait, cancel, or retry an enqueued job |
+| `JobContextLogger` | Logger that auto-injects job context (jobId, attempt, queue) |
+| `TaskRunner` | Polling-based execution engine with concurrency control and heartbeats |
+| `ScheduleRunner` | Polls for due cron schedules and creates SmrtJob entries |
+
+`TaskRunner` uses `SmrtJobCollection.claimReady()` so multiple workers can poll
+the same queue without duplicate-claiming a pending row.
+
+### Functions
+
+| Export | Description |
+|--------|------------|
+| `createTaskRunner(config?)` | Factory for creating a configured TaskRunner |
+| `createScheduleRunner(config?)` | Factory for creating a configured ScheduleRunner |
+| `withBackgroundJobs(Class)` | Mixin that adds `.bg()` and `.background()` to any SmrtObject class |
+| `parseDelay(delay)` | Parse human-readable delay strings (`'5m'`, `'1h'`, `'30s'`) to milliseconds |
+| `priorityToNumber(priority)` | Convert priority label (`'critical'`/`'high'`/`'normal'`/`'low'`) to number |
+
+### Key Types
+
+`Priority`, `JobStatus`, `JobResult`, `WaitOptions`, `BgOptions`, `BackgroundCapable`, `TaskRunnerConfig`, `TaskRunnerEvents`, `ScheduleRunnerConfig`, `ScheduleRunnerEvents`, `ScheduleInfo`, `JobContext`, `TimeoutBehavior`, `SmrtJobData`, `ListReadyOptions`
+
+## Dependencies
+
+- `@happyvertical/smrt-core` -- ORM and code generation
+- `@happyvertical/smrt-config` -- configuration loading
+- `@happyvertical/smrt-types` -- shared type definitions
+- `@happyvertical/jobs` -- retry strategies
+- `@happyvertical/sql` -- database interface
+- `@happyvertical/logger` -- structured logging
+- `@happyvertical/utils` -- ID generation utilities
+- Peer (optional): `@happyvertical/smrt-svelte`, `svelte`

package/dist/__smrt-register__.d.ts

@@ -0,0 +1,2 @@
+export {};
+//# sourceMappingURL=__smrt-register__.d.ts.map

package/dist/__smrt-register__.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"__smrt-register__.d.ts","sourceRoot":"","sources":["../src/__smrt-register__.ts"],"names":[],"mappings":""}

package/dist/background-policy.d.ts

@@ -0,0 +1,121 @@
+/**
+ * Opt-in policy controls for background job creation and dispatch.
+ *
+ * @remarks
+ * These guards harden the background-jobs surface flagged by the S5 audit
+ * (#1402):
+ *
+ * - {@link MAX_JOB_RETRIES} caps the retry count a caller can request so a
+ *   misconfigured `.retries(n)` cannot pin a worker on a poison job forever.
+ * - {@link assertWithinTenantCreationCap} bounds how many jobs a single tenant
+ *   may hold in the queue at once, so one tenant cannot exhaust the shared
+ *   worker pool (a cross-tenant denial of service).
+ * - {@link isBackgroundEligibleMethod} / {@link backgroundEligible} provide an
+ *   opt-in allowlist of methods that may be invoked by the runner. The runner's
+ *   dispatch is already bounded to existing prototype methods (no eval / dynamic
+ *   import), but a class can further restrict which of its methods are reachable
+ *   from a persisted job row. This same marker is intended to be consumed by the
+ *   agents package, which dispatches methods through an equivalent path.
+ */
+/**
+ * Hard ceiling on retry attempts a caller may request via `.retries(n)` /
+ * `bg(..., { retries })`. Requests above this are clamped (not rejected) so
+ * existing callers keep working while the worst case stays bounded.
+ */
+export declare const MAX_JOB_RETRIES = 25;
+/**
+ * Default maximum number of non-terminal (pending/running) jobs a single
+ * tenant may hold in the queue at once. Configurable per call; `0` / negative
+ * disables the cap.
+ */
+export declare const DEFAULT_TENANT_JOB_CAP = 10000;
+/**
+ * Clamp a requested retry count to {@link MAX_JOB_RETRIES}.
+ *
+ * @param requested - The retry count supplied by the caller.
+ * @returns A non-negative integer no greater than {@link MAX_JOB_RETRIES}.
+ */
+export declare function clampRetries(requested: number): number;
+/**
+ * Error thrown when a tenant exceeds its allowed in-flight job count.
+ */
+export declare class TenantJobCapExceededError extends Error {
+    readonly tenantId: string;
+    readonly cap: number;
+    readonly current: number;
+    constructor(tenantId: string, cap: number, current: number);
+}
+/**
+ * Throw {@link TenantJobCapExceededError} when a tenant is at or above its cap.
+ *
+ * @param tenantId - Tenant the new job would belong to (`null` = global; not
+ *   subject to the per-tenant cap).
+ * @param current - Current count of non-terminal jobs for the tenant.
+ * @param cap - Maximum allowed; `<= 0` disables the check.
+ */
+export declare function assertWithinTenantCreationCap(tenantId: string | null | undefined, current: number, cap: number): void;
+/**
+ * Class shape that opts into a background-method allowlist by declaring a
+ * static set/array of method names.
+ */
+export interface BackgroundEligibleClass {
+    /**
+     * Method names that may be invoked by the job/agent runner. When present
+     * (even if empty), it is treated as an exhaustive allowlist. When absent,
+     * the runner falls back to its default behaviour (any existing method).
+     */
+    backgroundEligibleMethods?: ReadonlyArray<string> | ReadonlySet<string>;
+}
+/**
+ * Add method names to a class's background-eligible allowlist.
+ *
+ * Installs/extends a static `backgroundEligibleMethods` set on the constructor.
+ * Once any method is marked, the runner refuses to dispatch a job whose
+ * `method` is not in the set — turning the dispatch surface from "any prototype
+ * method" into an explicit contract. Use this when applying the
+ * {@link backgroundEligible} decorator is inconvenient (or in non-decorator
+ * code, including the agents package).
+ *
+ * @param ctor - The class constructor to annotate.
+ * @param methods - Method names to allow.
+ */
+export declare function markBackgroundEligible(ctor: object, ...methods: string[]): void;
+/**
+ * Decorator: mark a method as background-eligible.
+ *
+ * This is a legacy (`experimentalDecorators`) method decorator — the mode the
+ * SMRT monorepo compiles with. Applying it (one or more times) builds up the
+ * static `backgroundEligibleMethods` allowlist on the owning class. Once any
+ * method is marked, the runner will refuse to dispatch a job whose `method` is
+ * not in the set.
+ *
+ * @example
+ * ```ts
+ * class Report extends SmrtObject {
+ *   \@backgroundEligible()
+ *   async regenerate() {}   // reachable from a job
+ *
+ *   async deleteEverything() {} // NOT reachable from a job
+ * }
+ * ```
+ */
+export declare function backgroundEligible(): (target: object, propertyKey: string | symbol, descriptor?: PropertyDescriptor) => PropertyDescriptor | undefined;
+/**
+ * Resolve the declared allowlist for a class, if any.
+ *
+ * @param ctor - The target object's constructor.
+ * @returns A `Set` of allowed method names, or `null` when the class did not
+ *   opt in (runner should fall back to default behaviour).
+ */
+export declare function getBackgroundEligibleMethods(ctor: unknown): ReadonlySet<string> | null;
+/**
+ * Whether a method may be invoked by the runner for a given target class.
+ *
+ * @param ctor - Constructor of the resolved target class.
+ * @param method - Method name from the persisted job row.
+ * @returns `true` when the class declared no allowlist (default) or when the
+ *   method is on the allowlist; `false` when an allowlist exists and excludes
+ *   the method.
+ */
+export declare function isBackgroundEligibleMethod(ctor: unknown, method: string): boolean;
+//# sourceMappingURL=background-policy.d.ts.map

package/dist/background-policy.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"background-policy.d.ts","sourceRoot":"","sources":["../src/background-policy.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;GAkBG;AAEH;;;;GAIG;AACH,eAAO,MAAM,eAAe,KAAK,CAAC;AAElC;;;;GAIG;AACH,eAAO,MAAM,sBAAsB,QAAS,CAAC;AAE7C;;;;;GAKG;AACH,wBAAgB,YAAY,CAAC,SAAS,EAAE,MAAM,GAAG,MAAM,CAQtD;AAED;;GAEG;AACH,qBAAa,yBAA0B,SAAQ,KAAK;aAEhC,QAAQ,EAAE,MAAM;aAChB,GAAG,EAAE,MAAM;aACX,OAAO,EAAE,MAAM;gBAFf,QAAQ,EAAE,MAAM,EAChB,GAAG,EAAE,MAAM,EACX,OAAO,EAAE,MAAM;CAQlC;AAED;;;;;;;GAOG;AACH,wBAAgB,6BAA6B,CAC3C,QAAQ,EAAE,MAAM,GAAG,IAAI,GAAG,SAAS,EACnC,OAAO,EAAE,MAAM,EACf,GAAG,EAAE,MAAM,GACV,IAAI,CAKN;AAED;;;GAGG;AACH,MAAM,WAAW,uBAAuB;IACtC;;;;OAIG;IACH,yBAAyB,CAAC,EAAE,aAAa,CAAC,MAAM,CAAC,GAAG,WAAW,CAAC,MAAM,CAAC,CAAC;CACzE;AAED;;;;;;;;;;;;GAYG;AACH,wBAAgB,sBAAsB,CACpC,IAAI,EAAE,MAAM,EACZ,GAAG,OAAO,EAAE,MAAM,EAAE,GACnB,IAAI,CASN;AAED;;;;;;;;;;;;;;;;;;GAkBG;AACH,wBAAgB,kBAAkB,KAE9B,QAAQ,MAAM,EACd,aAAa,MAAM,GAAG,MAAM,EAC5B,aAAa,kBAAkB,KAC9B,kBAAkB,GAAG,SAAS,CAOlC;AAED;;;;;;GAMG;AACH,wBAAgB,4BAA4B,CAC1C,IAAI,EAAE,OAAO,GACZ,WAAW,CAAC,MAAM,CAAC,GAAG,IAAI,CAK5B;AAED;;;;;;;;GAQG;AACH,wBAAgB,0BAA0B,CACxC,IAAI,EAAE,OAAO,EACb,MAAM,EAAE,MAAM,GACb,OAAO,CAIT"}