@semiont/jobs 0.5.5 → 0.5.7

package/README.md

@@ -10,7 +10,7 @@ Job queue, worker infrastructure, and annotation workers for [Semiont](https://g
 
 ## Architecture Context
 
-Workers run in a separate separate process and connect to the Knowledge System (KS) over HTTP/SSE using `WorkerStateUnit` from `@semiont/api-client`. Workers receive job assignments via SSE push, claim jobs atomically, and emit domain events back to the KS via HTTP. The KS ingests these events onto its EventBus for SSE delivery to the frontend.
+Workers run in a separate process and connect to the Knowledge System (KS) over HTTP/SSE using a `SemiontSession` (from `@semiont/sdk`) driven by a `JobClaimAdapter`. Workers receive job assignments via an SSE `job:queued` subscription, claim jobs atomically, and emit domain events back to the KS via `session.client.transport.emit(...)`. The KS ingests these events onto its EventBus for SSE delivery to the frontend.
 
 ## Installation
 
@@ -19,20 +19,25 @@ npm install @semiont/jobs
 ```
 
 **Dependencies:**
-- `@semiont/core` — Core types, EventBus
-- `@semiont/api-client` — OpenAPI types
+- `@semiont/core` — Core types, `SemiontProject`, EventBus
+- `@semiont/sdk` — `SemiontSession`, `WorkerBus` (worker process)
+- `@semiont/http-transport` — HTTP transport, OpenAPI types
 - `@semiont/inference` — InferenceClient for AI operations
+- `@semiont/content` — Content storage URI derivation
+- `@semiont/event-sourcing` — Annotation id generation
+- `@semiont/observability` — Spans and job-outcome metrics
 
 ## Quick Start
 
 ```typescript
-import { JobQueue, type PendingJob, type GenerationParams } from '@semiont/jobs';
-import { EventBus, userId, resourceId, annotationId } from '@semiont/core';
-import { jobId } from '@semiont/api-client';
+import { FsJobQueue, type PendingJob, type GenerationParams } from '@semiont/jobs';
+import { EventBus, userId, resourceId, annotationId, jobId } from '@semiont/core';
+import { SemiontProject } from '@semiont/core/node';
 
-// Initialize
+// Initialize — jobs are stored under project.jobsDir
 const eventBus = new EventBus();
-const jobQueue = new JobQueue({ dataDir: './data' }, logger, eventBus);
+const project = new SemiontProject('/path/to/project');
+const jobQueue = new FsJobQueue(project, logger, eventBus);
 await jobQueue.initialize();
 
 // Create a job
@@ -84,58 +89,43 @@ interface JobMetadata {
   id: JobId;
   type: JobType;
   userId: UserId;
-  userName: string;       // For building W3C Agent creator
-  userEmail: string;      // For building W3C Agent creator
-  userDomain: string;     // For building W3C Agent creator
+  userName: string;       // Audit-only snapshot of the requesting user
+  userEmail: string;      // Audit-only snapshot of the requesting user
+  userDomain: string;     // Audit-only snapshot of the requesting user
   created: string;
   retryCount: number;
   maxRetries: number;
 }
 ```
 
-The `userName`, `userEmail`, and `userDomain` fields are used by workers to build the W3C `Agent` for annotation `creator` attribution via `userToAgent()`.
+The `userName`, `userEmail`, and `userDomain` fields are an audit-only snapshot of the requesting user, persisted in the on-disk job file. Workers derive annotation `creator` attribution from `userId` via `didToAgent()`.
 
 ## Annotation Workers
 
-Six workers process different annotation types:
+The worker process (`worker-main.ts` → `startWorkerProcess` in `worker-process.ts`) claims jobs over the bus via a `JobClaimAdapter` and dispatches by `jobType` to a processor function. There are no per-type worker classes; each job type maps to one `process*Job` function:
 
-| Worker | Job Type | Constructor |
-|--------|----------|------------|
-| `ReferenceAnnotationWorker` | `reference-annotation` | `(jobQueue, config, inferenceClient, eventBus, contentFetcher, logger)` |
-| `GenerationWorker` | `generation` | `(jobQueue, config, inferenceClient, eventBus, logger)` |
-| `HighlightAnnotationWorker` | `highlight-annotation` | `(jobQueue, config, inferenceClient, eventBus, contentFetcher, logger)` |
-| `AssessmentAnnotationWorker` | `assessment-annotation` | `(jobQueue, config, inferenceClient, eventBus, contentFetcher, logger)` |
-| `CommentAnnotationWorker` | `comment-annotation` | `(jobQueue, config, inferenceClient, eventBus, contentFetcher, logger)` |
-| `TagAnnotationWorker` | `tag-annotation` | `(jobQueue, config, inferenceClient, eventBus, contentFetcher, logger)` |
+| Job Type | Processor |
+|----------|-----------|
+| `reference-annotation` | `processReferenceJob` |
+| `generation` | `processGenerationJob` |
+| `highlight-annotation` | `processHighlightJob` |
+| `assessment-annotation` | `processAssessmentJob` |
+| `comment-annotation` | `processCommentJob` |
+| `tag-annotation` | `processTagJob` |
 
-Workers emit EventBus commands (`mark:create`, `job:start`, `job:complete`, etc.) — the Stower actor in @semiont/make-meaning handles persistence.
+Detection logic lives in the `AnnotationDetection` class (`src/workers/annotation-detection.ts`); generation synthesis in `generateResourceFromTopic()` (`src/workers/generation/resource-generation.ts`). Processors never fetch content themselves — the worker process fetches it via `session.client.browse.resourceContent(resourceId)` and passes it in.
 
-## Custom Workers
+Workers emit bus events via `session.client.transport.emit('mark:create' | 'job:start' | 'job:report-progress' | 'job:complete' | 'job:fail', payload)` — the Stower actor in @semiont/make-meaning handles persistence to the event log, and the job command handlers mirror the same events into the queue files (completion, retry-on-failure with `maxRetries`, progress-as-heartbeat).
 
-```typescript
-import { JobWorker, type AnyJob } from '@semiont/jobs';
-import type { Logger } from '@semiont/core';
-
-class MyWorker extends JobWorker {
-  constructor(jobQueue: JobQueue, logger: Logger) {
-    super(jobQueue, 1000, 5000, logger);
-    //              ^^^^  ^^^^
-    //              poll   error backoff
-  }
+## Adding a Job Type
 
-  protected getWorkerName(): string {
-    return 'MyWorker';
-  }
+Workers are not subclassed. To add a job type:
 
-  protected canProcessJob(job: AnyJob): boolean {
-    return job.metadata.type === 'generation';
-  }
+1. Add the new `JobType` and its params/result/progress types in `src/types.ts`.
+2. Add a `process*Job` function in `src/processors.ts` that runs the inference and returns the annotations/result.
+3. Dispatch the new `jobType` to that processor in `handleJobInner()` in `src/worker-process.ts`.
 
-  protected async executeJob(job: AnyJob): Promise<any> {
-    // Your processing logic — return result object
-  }
-}
-```
+Processors are transport-agnostic: they take content, an `InferenceClient`, the job params, the user id, the `generator` (W3C SoftwareAgent), and an `onProgress` callback, and return annotations plus a result. The worker process handles claiming, content fetching, and lifecycle event emission.
 
 ## Discriminated Unions
 
@@ -144,12 +134,12 @@ Jobs use TypeScript discriminated unions for type safety:
 ```typescript
 function handleJob(job: AnyJob) {
   if (job.status === 'running') {
-    console.log(job:progress);    // Available
+    console.log(job.progress);    // Available
     // console.log(job.result);   // Compile error
   }
   if (job.status === 'complete') {
     console.log(job.result);      // Available
-    // console.log(job:progress); // Compile error
+    // console.log(job.progress); // Compile error
   }
 }
 ```
@@ -159,7 +149,7 @@ function handleJob(job: AnyJob) {
 Jobs are stored as individual JSON files organized by status:
 
 ```
-data/jobs/
+{project.jobsDir}/
   pending/job-abc123.json
   running/job-def456.json
   complete/job-ghi789.json
@@ -182,7 +172,8 @@ Apache-2.0
 
 ## Related Packages
 
-- [`@semiont/core`](../core/) — Domain types, EventBus
-- [`@semiont/api-client`](../api-client/) — OpenAPI types
+- [`@semiont/core`](../core/) — Domain types, `SemiontProject`, EventBus
+- [`@semiont/sdk`](../sdk/) — `SemiontSession`, `WorkerBus`
+- [`@semiont/http-transport`](../http-transport/) — HTTP transport, OpenAPI types
 - [`@semiont/inference`](../inference/) — AI inference client
 - [`@semiont/make-meaning`](../make-meaning/) — Actor model, Knowledge Base, service orchestration

package/dist/index.d.ts

@@ -1,5 +1,4 @@
-import { Readable } from 'stream';
-import { ResourceId, JobId, UserId, EntityType, AnnotationId, Annotation, GatheredContext, TagSchema, Logger, EventBus, components } from '@semiont/core';
+import { JobId, UserId, ResourceId, EntityType, AnnotationId, Annotation, GatheredContext, TagSchema, Logger, EventBus, components, SupportedMediaType } from '@semiont/core';
 import { SemiontProject } from '@semiont/core/node';
 import { InferenceClient } from '@semiont/inference';
 
@@ -16,12 +15,6 @@ import { InferenceClient } from '@semiont/inference';
  * - State machine is explicit and type-safe
  */
 
-/**
- * Content fetcher - turns a ResourceId into a readable stream.
- * Workers use this to access resource content on demand.
- * The implementation is provided by the backend at startup.
- */
-type ContentFetcher = (resourceId: ResourceId) => Promise<Readable | null>;
 type JobType = 'reference-annotation' | 'generation' | 'highlight-annotation' | 'assessment-annotation' | 'comment-annotation' | 'tag-annotation';
 type JobStatus = 'pending' | 'running' | 'complete' | 'failed' | 'cancelled';
 /**
@@ -31,6 +24,13 @@ interface JobMetadata {
     id: JobId;
     type: JobType;
     userId: UserId;
+    /**
+     * Audit-only snapshot of the requesting user (with `userEmail` and
+     * `userDomain` below), stamped at job creation and persisted in the
+     * on-disk job file. No code path reads these back — annotation
+     * `creator` attribution is derived from `userId` via `didToAgent()`.
+     * Kept intentionally so job files are self-describing to a human.
+     */
     userName: string;
     userEmail: string;
     userDomain: string;
@@ -327,7 +327,22 @@ interface JobQueue {
     createJob(job: AnyJob): Promise<void>;
     getJob(jobId: JobId): Promise<AnyJob | null>;
     updateJob(job: AnyJob, oldStatus?: JobStatus): Promise<void>;
-    pollNextPendingJob(predicate?: (job: AnyJob) => boolean): Promise<AnyJob | null>;
+    /** Move a running job to `complete`. Returns false if the job isn't running. */
+    completeJob(jobId: JobId, result: Record<string, unknown>): Promise<boolean>;
+    /**
+     * Move a running job back to `pending` (retry, re-announced) while
+     * `retryCount < maxRetries`, else to `failed`. Returns what happened,
+     * or null if the job isn't running.
+     */
+    failJob(jobId: JobId, error: string): Promise<'retried' | 'failed' | null>;
+    /** Write progress into a running job's file (throttled, best-effort). */
+    recordProgress(jobId: JobId, progress: Record<string, unknown>): Promise<void>;
+    /**
+     * Cancel all pending jobs in a category — 'generation' is the
+     * `generation` type; 'annotation' is every `*-annotation` type.
+     * Running jobs are left to finish. Returns the number cancelled.
+     */
+    cancelPendingJobs(category: 'annotation' | 'generation'): Promise<number>;
     cancelJob(jobId: JobId): Promise<boolean>;
     getStats(): Promise<{
         pending: number;
@@ -342,33 +357,40 @@ interface JobQueue {
  * Job Queue Manager
  *
  * Filesystem-based job queue with atomic operations.
- * Jobs are stored in directories by status for easy polling.
+ * Jobs are stored in directories by status; status transitions are
+ * atomic delete + write across directories.
  */
 
 declare class FsJobQueue implements JobQueue {
     private eventBus?;
     private jobsDir;
     private logger;
-    private pendingQueue;
-    private watcher;
-    private loadDebounceTimer;
+    private reannounceTimer;
+    private cleanupTimer;
+    /** Per-job timestamp of the last progress write, for throttling. */
+    private lastProgressWrite;
     constructor(project: SemiontProject, logger: Logger, eventBus?: EventBus | undefined);
     /**
-     * Initialize job queue directories, load pending jobs, and start fs.watch
+     * Initialize job queue directories, announce any pending backlog,
+     * and start the re-announce interval. Idempotent.
      */
     initialize(): Promise<void>;
     /**
-     * Clean up watcher
+     * Stop the re-announce and retention intervals
      */
     destroy(): void;
     /**
-     * Load pending jobs from disk into in-memory queue
+     * Emit `job:queued` for a pending job, if an EventBus is wired and
+     * the job carries a `resourceId` (every current job type does).
      */
-    private loadPendingJobs;
+    private announce;
     /**
-     * Debounced version of loadPendingJobs — fs.watch can fire rapidly
+     * Announce every job currently in `pending/`. Files that vanish or
+     * fail to parse mid-scan (claimed, cancelled, partially written)
+     * are skipped — they're either gone for a good reason or picked up
+     * on the next tick.
      */
-    private debouncedLoadPendingJobs;
+    private announcePendingJobs;
     /**
      * Create a new job
      */
@@ -382,10 +404,25 @@ declare class FsJobQueue implements JobQueue {
      */
     updateJob(job: AnyJob, oldStatus?: JobStatus): Promise<void>;
     /**
-     * Poll for next pending job (FIFO) from in-memory queue.
-     * If a predicate is provided, returns the first matching job (skipping non-matching ones).
+     * Move a running job to `complete`. Returns false (and changes
+     * nothing) if the job is missing or not running — which also makes
+     * duplicate `job:complete` events harmless.
+     */
+    completeJob(jobId: JobId, result: Record<string, unknown>): Promise<boolean>;
+    /**
+     * Retry-or-fail a running job. While `retryCount < maxRetries` the
+     * job goes back to `pending` with the count bumped (and is
+     * re-announced); after that it lands in `failed` with the error.
+     * Returns null (and changes nothing) if the job isn't running.
+     */
+    failJob(jobId: JobId, error: string): Promise<'retried' | 'failed' | null>;
+    /**
+     * Write progress into a running job's file. Throttled per job, and
+     * a no-op for jobs that aren't running. Beyond surfacing live
+     * progress to `job:status-requested`, each write refreshes the
+     * file's mtime — the heartbeat `recoverStaleRunningJobs` watches.
      */
-    pollNextPendingJob(predicate?: (job: AnyJob) => boolean): Promise<AnyJob | null>;
+    recordProgress(jobId: JobId, progress: Record<string, unknown>): Promise<void>;
     /**
      * List jobs with filters
      */
@@ -394,6 +431,21 @@ declare class FsJobQueue implements JobQueue {
      * Cancel a job
      */
     cancelJob(jobId: JobId): Promise<boolean>;
+    /**
+     * Cancel all pending jobs in a category — the granularity of the
+     * `job:cancel-requested` UI signal. Running jobs are left to finish:
+     * interrupting a worker mid-inference would need a worker-side kill
+     * channel that doesn't exist.
+     */
+    cancelPendingJobs(category: 'annotation' | 'generation'): Promise<number>;
+    /**
+     * Recover running jobs orphaned by a dead worker: any `running/`
+     * file whose mtime is older than the stale window is fed through
+     * the same retry-or-fail path as `job:fail`. Progress writes
+     * refresh the mtime, so a live worker is never recovered out from
+     * under itself as long as it reports within the window.
+     */
+    recoverStaleRunningJobs(): Promise<number>;
     /**
      * Clean up old completed/failed jobs (older than retention period)
      */
@@ -415,80 +467,14 @@ declare class FsJobQueue implements JobQueue {
 }
 
 /**
- * Job Worker Base Class
- *
- * Abstract worker that polls the job queue and processes jobs.
- * Subclasses implement specific job processing logic.
- */
-
-declare abstract class JobWorker {
-    private running;
-    private currentJob;
-    private pollIntervalMs;
-    private errorBackoffMs;
-    protected jobQueue: JobQueue;
-    protected logger: Logger;
-    constructor(jobQueue: JobQueue, pollIntervalMs: number | undefined, errorBackoffMs: number | undefined, logger: Logger);
-    /**
-     * Start the worker (polls queue in loop)
-     */
-    start(): Promise<void>;
-    /**
-     * Stop the worker (graceful shutdown)
-     */
-    stop(): Promise<void>;
-    /**
-     * Poll for next job to process
-     */
-    private pollNextJob;
-    /**
-     * Process a job (handles state transitions and error handling)
-     */
-    private processJob;
-    /**
-     * Handle job failure (retry or move to failed)
-     */
-    protected handleJobFailure(job: AnyJob, error: any): Promise<void>;
-    /**
-     * Update job progress (best-effort, doesn't throw)
-     */
-    protected updateJobProgress(job: AnyJob): Promise<void>;
-    /**
-     * Sleep utility
-     */
-    protected sleep(ms: number): Promise<void>;
-    /**
-     * Emit completion event (optional hook for subclasses)
-     * Override this to emit job-specific completion events (e.g., job.completed)
-     */
-    protected emitCompletionEvent(_job: RunningJob<any, any>, _result: any): Promise<void>;
-    /**
-     * Get worker name (for logging)
-     */
-    protected abstract getWorkerName(): string;
-    /**
-     * Check if this worker can process the given job
-     */
-    protected abstract canProcessJob(job: AnyJob): boolean;
-    /**
-     * Execute the job (job-specific logic)
-     * This is where the actual work happens
-     * Return the result object (or void for jobs without results)
-     * Throw an error to trigger retry logic
-     */
-    protected abstract executeJob(job: AnyJob): Promise<any>;
-}
-
-/**
- * Job Processors — extracted from JobWorker subclasses
+ * Job Processors
  *
  * Pure functions that take content + inference client + params,
  * report progress via callback, and return annotations + results.
  *
  * No EventBus, no JobQueue, no side effects except calling inference.
- * Two callers:
- *   1. In-process JobWorker subclasses (existing path)
- *   2. Remote WorkerStateUnit via worker-process.ts (new path)
+ * Driven by the remote worker process (worker-process.ts), which claims
+ * jobs over SSE and dispatches by jobType to these functions.
  */
 
 type Agent = components['schemas']['Agent'];
@@ -513,7 +499,7 @@ declare function processTagJob(content: string, inferenceClient: InferenceClient
 declare function processGenerationJob(inferenceClient: InferenceClient, params: GenerationParams, onProgress: OnProgress, logger: Logger): Promise<{
     content: string;
     title: string;
-    format: string;
+    format: SupportedMediaType;
     result: GenerationResult;
 }>;
 
@@ -569,16 +555,11 @@ interface TagMatch {
  * 2. Call AI inference
  * 3. Parse and validate results using MotivationParsers
  *
- * All methods take content as a string parameter.
- * Workers are responsible for fetching content via ContentFetcher.
+ * All methods take content as a string parameter — the worker process
+ * fetches it and hands it in.
  */
 
 declare class AnnotationDetection {
-    /**
-     * Fetch content from a ContentFetcher and read the stream to a string.
-     * Shared helper for all workers.
-     */
-    static fetchContent(contentFetcher: ContentFetcher, resourceId: ResourceId): Promise<string>;
     /**
      * Detect comments in content.
      *
@@ -641,5 +622,5 @@ declare function generateResourceFromTopic(topic: string, entityTypes: string[],
     content: string;
 }>;
 
-export { AnnotationDetection, FsJobQueue, JobWorker, generateResourceFromTopic, isCancelledJob, isCompleteJob, isFailedJob, isPendingJob, isRunningJob, processAssessmentJob, processCommentJob, processGenerationJob, processHighlightJob, processReferenceJob, processTagJob };
-export type { AnyJob, AssessmentDetectionJob, AssessmentDetectionParams, AssessmentDetectionProgress, AssessmentDetectionResult, CancelledJob, CommentDetectionJob, CommentDetectionParams, CommentDetectionProgress, CommentDetectionResult, CompleteJob, ContentFetcher, DetectionJob, DetectionParams, DetectionProgress, DetectionResult, FailedJob, GenerationJob, GenerationParams, GenerationResult, HighlightDetectionJob, HighlightDetectionParams, HighlightDetectionProgress, HighlightDetectionResult, JobMetadata, JobQueryFilters, JobQueue, JobStatus, JobType, OnProgress, PendingJob, ProcessorResult, RunningJob, TagDetectionJob, TagDetectionParams, TagDetectionProgress, TagDetectionResult, YieldProgress };
+export { AnnotationDetection, FsJobQueue, generateResourceFromTopic, isCancelledJob, isCompleteJob, isFailedJob, isPendingJob, isRunningJob, processAssessmentJob, processCommentJob, processGenerationJob, processHighlightJob, processReferenceJob, processTagJob };
+export type { AnyJob, AssessmentDetectionJob, AssessmentDetectionParams, AssessmentDetectionProgress, AssessmentDetectionResult, CancelledJob, CommentDetectionJob, CommentDetectionParams, CommentDetectionProgress, CommentDetectionResult, CompleteJob, DetectionJob, DetectionParams, DetectionProgress, DetectionResult, FailedJob, GenerationJob, GenerationParams, GenerationResult, HighlightDetectionJob, HighlightDetectionParams, HighlightDetectionProgress, HighlightDetectionResult, JobMetadata, JobQueryFilters, JobQueue, JobStatus, JobType, OnProgress, PendingJob, ProcessorResult, RunningJob, TagDetectionJob, TagDetectionParams, TagDetectionProgress, TagDetectionResult, YieldProgress };