@semiont/jobs 0.5.29 → 0.5.31

package/README.md

@@ -98,10 +98,12 @@ interface JobMetadata {
   created: string;
   retryCount: number;
   maxRetries: number;
+  completedUnits?: string[];  // Checkpoint: work units (entity types) already
+                              // persisted by a failed attempt — the retry skips them
 }
 ```
 
-The `userName`, `userEmail`, and `userDomain` fields are an audit-only snapshot of the requesting user, persisted in the on-disk job file. Workers derive annotation `creator` attribution from `userId` via `didToAgent()`.
+The `userName`, `userEmail`, and `userDomain` fields are an audit-only snapshot of the requesting user, persisted in the on-disk job file. Workers derive annotation `creator` attribution from `userId` via `didToAgent()`. `completedUnits` is written only by `failJob`, unioned across attempts, and carried on the `job:fail` event — see Failure discipline below.
 
 ## Annotation Workers
 
@@ -118,7 +120,19 @@ The worker process (`worker-main.ts` → `startWorkerProcess` in `worker-process
 
 Detection logic lives in the `AnnotationDetection` class (`src/workers/annotation-detection.ts`); generation synthesis in `generateResourceFromTopic()` (`src/workers/generation/resource-generation.ts`). Processors never fetch content themselves — the worker process fetches it via `session.client.browse.resourceContent(resourceId)` and passes it in.
 
-Workers emit bus events via `session.client.transport.emit('mark:create' | 'job:start' | 'job:report-progress' | 'job:complete' | 'job:fail', payload)` — the Stower actor in @semiont/make-meaning handles persistence to the event log, and the job command handlers mirror the same events into the queue files (completion, retry-on-failure with `maxRetries`, progress-as-heartbeat).
+Workers emit lifecycle events via `session.client.transport.emit('job:start' | 'job:report-progress' | 'job:checkpoint' | 'job:complete' | 'job:fail', payload)` and persist annotations through the **awaited `mark:commit` operation** — a batch per unit of work that resolves only once the Stower actor in @semiont/make-meaning has appended every annotation to the event log. Unit completion and `job:complete` gate on that acknowledgement, never on emission, so a down persistence sink is a retryable failure instead of silent loss. The job command handlers mirror the lifecycle events into the queue files (completion, retry-on-failure with `maxRetries`, progress-as-heartbeat). `job:fail` carries the fields the worker computes: `completedUnits` (the checkpoint), `failureClass`, and `willRetry` — see Failure discipline.
+
+## Failure discipline
+
+Long inference work fails in bounded, classified, resumable ways — every piece below was built against a measured production failure, not a hypothetical:
+
+- **Every inference call is bounded and truly cancelled.** A call gets 10 minutes (`INFERENCE_TIMEOUT_MS`); at the bound the worker aborts it at the transport (`AbortSignal` through the provider SDK to the socket — milliseconds to rejection, no zombie billing on) and fails the job with a typed `InferenceTimeoutError`. An in-flight heartbeat reports elapsed-time liveness every 15 s during long calls.
+- **Budgets are derived, never tuned** (`workers/detection/detection-chunking.ts`). Input:output allocation is 1:2 per entity type asked for (`input ≤ outputBudget / (2 × typesPerCall)`), and **every** provider gets a duration cap: per-call output is bounded at what the provider's worst-case rate finishes in HALF the bound — the published rate when there is one, a conservative assumed floor (`ASSUMED_OUTPUT_TOKENS_PER_HOUR`, 30 tok/s) for rate-silent providers like Ollama, where an unbounded budget turned model repetition loops into hour-long transient burns.
+- **Size-shaped failures subdivide in place** (`callChunkSubdividing`). Four failure families descend, each with its own floor: a **truncation** descends by size and gets one same-size re-roll at the floor; a **timeout** descends two levels then propagates; an **`'unknown'`-stop unreadable response** (garbage output with no stop reason — measured size-correlated on real documents) descends by size and propagates at the floor; and a **flagged under-report** (below) descends by size, and at the floor its salvage — everything it did find, every span write-time-verified — is **accepted loudly** rather than discarded. A piece that cannot actually shrink is at its floor regardless of arithmetic: at temperature 0, an identical re-run returns the identical failure. Sub-piece overlap duplicates fall to the existing span-keyed dedupe.
+- **Successful-looking extractions are verified** (`assertYieldNotCollapsed`). A local model can return a clean, schema-conforming response carrying a fraction of the entities present — deterministic and otherwise invisible. When the provider declares `verifyDetectionYield` (all real providers do), each chunk's item count is checked against a cheap parallel count call; an extraction under half the count is flagged and subdivided. Every anchoring outcome and every call — including flagged and failed ones — is recorded to `semiont.detection.*` metrics (`@semiont/observability`).
+- **Entity types run concurrently up to the provider's declared capacity** (`client.maxConcurrency`): a hosted API with rate headroom runs several types at once; a local single-model server runs them sequentially, because concurrent requests only split one GPU. Jobs never switches on provider identity — both behaviors are capabilities declared on the `InferenceClient`.
+- **Failures are classified at the worker, where errors are still typed** (`failure-class.ts`). Only KNOWN-deterministic failures — truncation at the subdivision floor, unsupported media, non-throttle 4xx — skip the retry budget; everything unrecognized stays retryable. The class rides `job:fail` as `failureClass`.
+- **Retries resume from the checkpoint.** Reference-annotation persists each entity type's annotations as that unit completes; completed units ride `job:fail` into `metadata.completedUnits`, and the retry processes only what's left.
 
 ## Adding a Job Type
 

package/dist/index.d.ts

@@ -37,6 +37,16 @@ interface JobMetadata {
     created: string;
     retryCount: number;
     maxRetries: number;
+    /**
+     * Checkpointed resume (ABANDONED-INFERENCE P2, HD1): the entity-type
+     * units whose annotations were fully emitted by earlier failed
+     * attempts. Written only by `failJob`, unioned across attempts — and
+     * because `failJob` rebuilds the retried record by spreading metadata,
+     * the checkpoint survives every subsequent rebuild for free. A retried
+     * claim skips these units, so completed work is neither redone nor
+     * duplicated.
+     */
+    completedUnits?: string[];
 }
 /**
  * Locale conventions for detection/generation params.
@@ -316,9 +326,24 @@ interface JobQueue {
     /**
      * Move a running job back to `pending` (retry, re-announced) while
      * `retryCount < maxRetries`, else to `failed`. Returns what happened,
-     * or null if the job isn't running.
+     * or null if the job isn't running. `completedUnits` — the units the
+     * failing attempt fully emitted — are unioned into the record's
+     * checkpoint (ABANDONED-INFERENCE P2) so a retry skips them. A
+     * `failureClass` of 'deterministic' goes straight to `failed` with any
+     * budget remaining — a second identical attempt cannot succeed (P3).
      */
-    failJob(jobId: JobId, error: string): Promise<'retried' | 'failed' | null>;
+    failJob(jobId: JobId, error: string, completedUnits?: string[], failureClass?: 'transient' | 'deterministic'): Promise<'retried' | 'failed' | null>;
+    /**
+     * Persist a running job's completed-unit checkpoint AT unit completion —
+     * not only when a job fails (JOB-RESTART-SAFETY P2). `failJob` carries the
+     * checkpoint on a clean failure, but a worker that DIES (crash/OOM/kill)
+     * never emits `job:fail`, so its finished units would be lost and the
+     * janitor's recovery would redo them. This writes them into the running
+     * file's metadata as each unit lands, unioned with any existing
+     * checkpoint, so recovery resumes rather than restarts. Unthrottled (a
+     * unit completion must never be dropped); a no-op for non-running jobs.
+     */
+    checkpointUnits(jobId: JobId, completedUnits: string[]): Promise<void>;
     /** Write progress into a running job's file (throttled, best-effort). */
     recordProgress(jobId: JobId, progress: Record<string, unknown>): Promise<void>;
     /**
@@ -402,7 +427,20 @@ declare class FsJobQueue implements JobQueue {
      * re-announced); after that it lands in `failed` with the error.
      * Returns null (and changes nothing) if the job isn't running.
      */
-    failJob(jobId: JobId, error: string): Promise<'retried' | 'failed' | null>;
+    failJob(jobId: JobId, error: string, completedUnits?: string[], failureClass?: 'transient' | 'deterministic'): Promise<'retried' | 'failed' | null>;
+    /**
+     * Persist a running job's completed-unit checkpoint at unit completion
+     * (JOB-RESTART-SAFETY P2). `failJob` writes this checkpoint on a clean
+     * failure, but a worker that dies without emitting `job:fail` would lose
+     * its finished units — so the janitor's recovery (which re-queues with
+     * whatever `metadata.completedUnits` the running file holds) would redo
+     * them. Writing it HERE, as each unit lands, makes a crash lose at most
+     * the in-flight unit. Unioned with any existing checkpoint, unthrottled
+     * (a unit completion is rare and must never be dropped), a no-op for
+     * non-running jobs. Written directly, like `recordProgress`, so it also
+     * refreshes the mtime heartbeat.
+     */
+    checkpointUnits(jobId: JobId, completedUnits: string[]): Promise<void>;
     /**
      * Write progress into a running job's file. Throttled per job, and
      * a no-op for jobs that aren't running. Beyond surfacing live
@@ -524,7 +562,18 @@ interface ProcessorResult<R> {
 declare function processHighlightJob(content: string, inferenceClient: InferenceClient, params: HighlightDetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress): Promise<ProcessorResult<HighlightDetectionResult>>;
 declare function processCommentJob(content: string, inferenceClient: InferenceClient, params: CommentDetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress): Promise<ProcessorResult<CommentDetectionResult>>;
 declare function processAssessmentJob(content: string, inferenceClient: InferenceClient, params: AssessmentDetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress): Promise<ProcessorResult<AssessmentDetectionResult>>;
-declare function processReferenceJob(content: string, inferenceClient: InferenceClient, params: DetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress, logger: Logger): Promise<ProcessorResult<DetectionResult>>;
+/**
+ * Reference detection commits per UNIT — one entity type — through
+ * `onUnitComplete` (ABANDONED-INFERENCE P2, checkpointed resume): the
+ * callback receives the unit's deduped annotations, and only after it
+ * resolves does the unit count as complete. Emission belongs to the
+ * callback alone; the processor returns only the result — returning the
+ * annotations as well would recreate the post-run batch that N2 showed
+ * discards completed work wholesale.
+ */
+declare function processReferenceJob(content: string, inferenceClient: InferenceClient, params: DetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress, logger: Logger, onUnitComplete: (entityType: string, annotations: Annotation[]) => Promise<void>, signal?: AbortSignal): Promise<{
+    result: DetectionResult;
+}>;
 declare function processTagJob(content: string, inferenceClient: InferenceClient, params: TagDetectionParams, buildAnnotation: BuildAnnotation, onProgress: OnProgress): Promise<ProcessorResult<TagDetectionResult>>;
 declare function processGenerationJob(inferenceClient: InferenceClient, params: GenerationJobParams, onProgress: OnProgress, logger: Logger): Promise<{
     content: Uint8Array;
@@ -695,8 +744,21 @@ declare function generateResourceFromTopic(topic: string, entityTypes: string[],
  *
  * Thresholds are fixed by design (no env knobs) and deliberately
  * layered: inference timeout (10 min, P2) fires first; this watchdog
- * (15 min) catches the failure modes nobody predicted; the gateway's
- * dead-worker janitor (30 min) re-queues the job regardless.
+ * (15 min) catches wedges where the loop still turns but activity has
+ * stopped; the gateway's dead-worker janitor (30 min) re-queues the job
+ * regardless.
+ *
+ * The layering matters because this watchdog has a hard limit: it is an
+ * IN-PROCESS timer, so it cannot fire while the event loop itself is
+ * blocked — the exact condition a blocked loop creates
+ * (JOB-RESTART-SAFETY P7, the 2026-09-03 finalization hang: 18 min silent,
+ * this watchdog never fired, an empty /health confirming the loop was
+ * wedged). The unbounded emit that caused that specific hang is now bounded
+ * at the transport (`EMIT_TIMEOUT_MS`), so the loop errors instead of
+ * blocking; but for any future blocked-loop bug the ONLY backstop is the
+ * out-of-process one — the gateway's janitor sweeping this worker's job
+ * files by mtime (`fs-job-queue.ts` `recoverStaleRunningJobs`). A liveness
+ * guarantee a blocked loop defeats is not one; the janitor is the guarantee.
  */
 declare const STALL_THRESHOLD_MS: number;