@colony2/jobdb 0.0.7 → 0.0.10

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +78 -658
  2. package/package.json +13 -13
package/README.md CHANGED
@@ -1,721 +1,141 @@
1
1
  # jobdb
2
2
 
3
- A durable workflow library for Go that provides reliable, long-running workflow orchestration with built-in retry logic, timeout handling, and persistent state management.
3
+ `jobdb` is a runtime server for durable jobs. The installed `jobdb` command
4
+ serves the JobDB runtime REST API over HTTP using one of the available storage
5
+ backends.
4
6
 
5
- ## What is jobdb?
6
-
7
- jobdb is a workflow orchestration library that helps you build reliable, distributed workflows. It handles the complexity of managing workflow state, retries, timeouts, and task coordination so you can focus on your business logic.
8
-
9
- **Key Features:**
10
- - **Durable Workflows**: Workflow state persists across failures and restarts
11
- - **Task Orchestration**: Break workflows into reusable task units
12
- - **Automatic Retries**: Configurable retry policies with exponential backoff
13
- - **Timeout Management**: Set invocation and total timeout limits
14
- - **Async Child Workflows**: Spawn and await child workflows
15
- - **Artifact Support**: Handle large files and binary data efficiently
16
- - **Multi-Tenant**: Built-in tenant isolation
17
- - **Job Querying**: List and filter jobs with flexible criteria
18
- - **Schedules**: First-class recurring jobs with pause/resume/archive support
19
- - **Embedded SQLite Runtime**: Durable local execution without external services
20
- - **Remote Runtime Protocol**: REST runtime adapter with tokenized lease operations
7
+ Use the server when you want a standalone runtime process that workers and other
8
+ clients can talk to over the remote runtime protocol.
21
9
 
22
10
  ## Installation
23
11
 
24
- ```bash
25
- go get github.com/colony-2/jobdb
26
- ```
27
-
28
- ## Local Runtime CLI
29
-
30
- The repo includes a Cobra-based local runtime server at `cmd/jobdb`.
31
-
32
- Run the default SQLite-backed embedded runtime:
12
+ Install the CLI with npm:
33
13
 
34
14
  ```bash
35
- go run ./cmd/jobdb --listen 127.0.0.1:9047 --db jobdb.db
15
+ npm install -g @colony2/jobdb
36
16
  ```
37
17
 
38
- The SQLite runtime stores jobs, chapters, artifacts, leases, and schedules in a
39
- local SQLite database plus a blob directory.
40
-
41
- Run the in-memory toy runtime explicitly:
18
+ Verify the command is available:
42
19
 
43
20
  ```bash
44
- go run ./cmd/jobdb toy --listen 127.0.0.1:9047
21
+ jobdb --help
45
22
  ```
46
23
 
47
- For Go module migration details, including moving embedded direct-runtime users
48
- to SQLite, see
49
- [`docs/MIGRATION-SQLITE-EMBEDDED-RUNTIME.md`](docs/MIGRATION-SQLITE-EMBEDDED-RUNTIME.md).
50
-
51
24
  ## Quick Start
52
25
 
53
- Here's a simple workflow that processes data through multiple tasks:
54
-
55
- ```go
56
- package main
57
-
58
- import (
59
- "context"
60
- "log"
61
-
62
- "github.com/colony-2/jobdb/pkg/jobdb"
63
- toyruntime "github.com/colony-2/jobdb/pkg/jobdb/runtime/toy"
64
- )
65
-
66
- // Define a job worker (orchestrates tasks)
67
- type DataProcessingJob struct{}
68
-
69
- func (j DataProcessingJob) Name() string { return "data_processing" }
70
-
71
- func (j DataProcessingJob) Run(ctx jobdb.JobContext, input jobdb.JobData) (jobdb.JobData, error) {
72
- // Execute tasks in sequence
73
- result, err := ctx.DoTask(jobdb.DefaultRunPolicy(), "validate", input)
74
- if err != nil {
75
- return nil, err
76
- }
77
-
78
- result, err = ctx.DoTask(jobdb.DefaultRunPolicy(), "transform", result)
79
- if err != nil {
80
- return nil, err
81
- }
82
-
83
- return result, nil
84
- }
85
-
86
- // Define task workers
87
- type ValidateTask struct{}
88
-
89
- func (t ValidateTask) Name() string { return "validate" }
90
-
91
- func (t ValidateTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
92
- // Your validation logic here
93
- return input, nil
94
- }
95
-
96
- type TransformTask struct{}
97
-
98
- func (t TransformTask) Name() string { return "transform" }
99
-
100
- func (t TransformTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
101
- // Your transformation logic here
102
- return input, nil
103
- }
104
-
105
- func main() {
106
- ctx := context.Background()
107
- runtime := toyruntime.New()
108
-
109
- // Build the engine
110
- engine, err := jobdb.NewEngineBuilder().
111
- WithRuntime(runtime).
112
- WithWorkerTenantId("my-tenant").
113
- PlusWorkers(DataProcessingJob{}, ValidateTask{}, TransformTask{}).
114
- BuildEngine()
115
- if err != nil {
116
- log.Fatal(err)
117
- }
118
-
119
- // Start the engine worker loop
120
- go engine.Run(ctx)
121
-
122
- // Start a job
123
- input := jobdb.NewTaskDataOrPanic(map[string]interface{}{"value": 42})
124
- jobKey, err := engine.SubmitJob(ctx, jobdb.SubmitJob{
125
- TenantId: "my-tenant",
126
- JobType: "data_processing",
127
- Data: input,
128
- })
129
- if err != nil {
130
- log.Fatal(err)
131
- }
132
-
133
- log.Printf("Started job: %s", jobKey)
134
- }
135
- ```
136
-
137
- ## Core Concepts
138
-
139
- ### Jobs vs Tasks
140
-
141
- - **Jobs** are the top-level workflows that orchestrate tasks. A job worker defines the workflow logic.
142
- - **Tasks** are individual units of work within a job. Task workers implement specific operations.
143
-
144
- Jobs use `JobContext` to execute tasks, wait, and spawn child workflows. Tasks receive `TaskContext` for execution context.
145
-
146
- ### JobWorker Interface
147
-
148
- ```go
149
- type JobWorker interface {
150
- Name() string
151
- Run(JobContext, JobData) (JobData, error)
152
- }
153
- ```
154
-
155
- Your job worker orchestrates the workflow:
156
-
157
- ```go
158
- func (j MyJob) Run(ctx jobdb.JobContext, input jobdb.JobData) (jobdb.JobData, error) {
159
- // Execute tasks
160
- result, err := ctx.DoTask(policy, "task-name", taskInput)
161
- if err != nil {
162
- return nil, err
163
- }
164
-
165
- // Wait/sleep
166
- if err := ctx.AwaitDuration(jobdb.Duration(5 * time.Minute)); err != nil {
167
- return nil, err
168
- }
169
-
170
- // Wait for another job in the same tenant
171
- if err := ctx.AwaitJobs(childJobID); err != nil {
172
- return nil, err
173
- }
174
-
175
- return result, nil
176
- }
177
- ```
178
-
179
- ### TaskWorker Interface
180
-
181
- ```go
182
- type TaskWorker interface {
183
- Name() string
184
- Run(TaskContext, TaskData) (TaskData, error)
185
- }
186
- ```
187
-
188
- Your task worker implements a specific operation:
189
-
190
- ```go
191
- func (t MyTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
192
- // Access job context
193
- ctx.Logger.Info("processing task", "job", ctx.JobKey, "step", ctx.Step)
194
-
195
- // Wait if needed
196
- ctx.AwaitDuration(jobdb.Duration(30 * time.Second))
197
-
198
- // Return result
199
- return jobdb.NewTaskData(result)
200
- }
201
- ```
202
-
203
- ## Working with Data
204
-
205
- ### Creating TaskData
206
-
207
- ```go
208
- // From a struct or map
209
- data, err := jobdb.NewTaskData(map[string]interface{}{
210
- "userId": 123,
211
- "action": "process",
212
- })
213
-
214
- // Panic version for tests/simple cases
215
- data := jobdb.NewTaskDataOrPanic(myStruct)
216
-
217
- // With artifacts
218
- data, err := jobdb.NewTaskData(payload, artifact1, artifact2)
219
- ```
220
-
221
- ### Reading TaskData
222
-
223
- ```go
224
- func (t MyTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
225
- // Get raw JSON data
226
- rawData, err := input.GetData()
227
- if err != nil {
228
- return nil, err
229
- }
230
-
231
- // Unmarshal into your struct
232
- var payload MyPayload
233
- if err := json.Unmarshal(rawData, &payload); err != nil {
234
- return nil, err
235
- }
236
-
237
- // Access artifacts
238
- artifacts, err := input.GetArtifacts()
239
- if err != nil {
240
- return nil, err
241
- }
242
-
243
- return jobdb.NewTaskData(result)
244
- }
245
- ```
246
-
247
- ## Working with Artifacts
248
-
249
- Artifacts represent file-like data that flows through workflows. They support lazy loading and automatic cleanup.
250
-
251
- ### Creating Artifacts
252
-
253
- ```go
254
- // From bytes (in-memory)
255
- artifact := jobdb.NewArtifactFromBytes("config.json", jsonBytes)
256
-
257
- // From a reader
258
- artifact := jobdb.NewArtifactFromReader("output.txt", reader, size)
259
-
260
- // From a file (auto-cleanup enabled)
261
- artifact, err := jobdb.NewArtifactFromFile("build.tar.gz", "/tmp/build.tar.gz")
262
-
263
- // From a file (no cleanup)
264
- artifact, err := jobdb.NewArtifactFromFileNoCleanup("data.csv", "/data/input.csv")
265
-
266
- // Custom artifact with full control
267
- artifact := jobdb.NewArtifact("custom.dat",
268
- func() (io.ReadCloser, int64, error) {
269
- // Your opener logic
270
- f, _ := os.Open(path)
271
- info, _ := f.Stat()
272
- return f, info.Size(), nil
273
- },
274
- func() error {
275
- // Your cleanup logic
276
- return os.Remove(path)
277
- },
278
- )
279
- ```
280
-
281
- ### Using Artifacts
282
-
283
- ```go
284
- // Get artifact metadata
285
- name := artifact.Name() // "output.tar.gz"
286
- size := artifact.Size() // size in bytes
287
-
288
- // Stream artifact contents
289
- rc, err := artifact.Open()
290
- if err != nil {
291
- return err
292
- }
293
- defer rc.Close()
294
- // ... read from rc
295
-
296
- // Write to a file
297
- err = artifact.SaveToFile(ctx, "/output/file.tar.gz")
298
-
299
- // Get full contents (use carefully for large files)
300
- data, err := artifact.Bytes(ctx)
301
-
302
- // Compute SHA256 hash
303
- hash, err := artifact.Sha256(ctx)
304
- ```
305
-
306
- ### Artifacts in Tasks
307
-
308
- ```go
309
- func (t ProcessFileTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
310
- artifacts, err := input.GetArtifacts()
311
- if err != nil {
312
- return nil, err
313
- }
314
-
315
- // Process the first artifact
316
- if len(artifacts) > 0 {
317
- inputFile := artifacts[0]
318
-
319
- // Save to local file for processing
320
- tmpPath := "/tmp/input.dat"
321
- if err := inputFile.SaveToFile(context.Background(), tmpPath); err != nil {
322
- return nil, err
323
- }
324
-
325
- // Process the file...
326
- processFile(tmpPath)
327
-
328
- // Create output artifact
329
- outputArtifact, err := jobdb.NewArtifactFromFile("output.dat", "/tmp/output.dat")
330
- if err != nil {
331
- return nil, err
332
- }
333
-
334
- return jobdb.NewTaskData(result, outputArtifact)
335
- }
336
-
337
- return input, nil
338
- }
339
- ```
340
-
341
- ## Retry and Timeout Policies
342
-
343
- ### RunPolicy Configuration
26
+ Run the default SQLite-backed server:
344
27
 
345
- ```go
346
- policy := jobdb.RunPolicy{
347
- Retry: jobdb.RetryPolicy{
348
- InitialInterval: jobdb.Duration(100 * time.Millisecond),
349
- BackoffCoefficient: 2.0,
350
- MaximumInterval: jobdb.Duration(30 * time.Second),
351
- MaximumAttempts: 5,
352
- NonRetryableErrorTypes: []string{"ValidationError"},
353
- },
354
- InvocationTimeout: jobdb.AsDuration(30 * time.Second), // Per attempt
355
- TotalTimeout: jobdb.AsDuration(10 * time.Minute), // Overall
356
- }
357
-
358
- result, err := ctx.DoTask(policy, "my-task", input)
359
- ```
360
-
361
- ### Default Policy
362
-
363
- ```go
364
- // Use the default policy
365
- result, err := ctx.DoTask(jobdb.DefaultRunPolicy(), "my-task", input)
366
-
367
- // Default values:
368
- // - InvocationTimeout: 30 seconds
369
- // - TotalTimeout: 30 minutes
370
- // - InitialInterval: 100ms
371
- // - BackoffCoefficient: 2.0
372
- // - MaximumInterval: 30 seconds
373
- // - MaximumAttempts: 3
374
- ```
375
-
376
- ## Error Handling
377
-
378
- ### Application Errors
379
-
380
- Regular errors returned from your workers are treated as application errors and will trigger retries according to the retry policy:
381
-
382
- ```go
383
- func (t MyTask) Run(ctx jobdb.TaskContext, input jobdb.TaskData) (jobdb.TaskData, error) {
384
- if err := validateInput(input); err != nil {
385
- return nil, fmt.Errorf("validation failed: %w", err)
386
- }
387
- return result, nil
388
- }
389
- ```
390
-
391
- ### System Errors
392
-
393
- System errors represent infrastructure failures:
394
-
395
- ```go
396
- if err := connectToDatabase(); err != nil {
397
- return nil, jobdb.NewSystemError(jobdb.SystemErrorPayload{
398
- Message: "database connection failed",
399
- Component: "database",
400
- Code: "connection_error",
401
- Retryable: true,
402
- })
403
- }
28
+ ```bash
29
+ jobdb --listen 127.0.0.1:9047 --db jobdb.db
404
30
  ```
405
31
 
406
- ### Non-Retryable Errors
32
+ This starts the runtime API at `http://127.0.0.1:9047`. SQLite is the default
33
+ backend and persists runtime state in `jobdb.db`; large artifacts are stored in a
34
+ blob directory that defaults to `<db>.blobs`.
407
35
 
408
- Mark errors as non-retryable to stop retry attempts immediately:
36
+ The explicit SQLite subcommand is equivalent:
409
37
 
410
- ```go
411
- type ValidationError struct {
412
- error
413
- }
414
-
415
- func (e ValidationError) NonRetryable() bool {
416
- return true
417
- }
418
-
419
- // Usage
420
- if !isValid(input) {
421
- return nil, ValidationError{errors.New("invalid input")}
422
- }
38
+ ```bash
39
+ jobdb sqlite --listen 127.0.0.1:9047 --db jobdb.db
423
40
  ```
424
41
 
425
- ### Checking Error Types
426
-
427
- ```go
428
- if jobdb.IsAppError(err) {
429
- // Handle application error
430
- }
431
-
432
- if jobdb.IsSystemError(err) {
433
- // Handle system error
434
- }
435
- ```
42
+ Stop the server with `Ctrl-C` or `SIGTERM`; the command shuts the HTTP server
43
+ down before closing backend resources.
436
44
 
437
- ## Advanced Features
45
+ ## Backend Options
438
46
 
439
- ### Awaiting Jobs
47
+ ### SQLite
440
48
 
441
- Wait for previously submitted jobs before continuing:
49
+ SQLite is the default embedded durable backend.
442
50
 
443
- ```go
444
- func (j ParentJob) Run(ctx jobdb.JobContext, input jobdb.JobData) (jobdb.JobData, error) {
445
- if err := ctx.AwaitJobs(childJobID); err != nil {
446
- return nil, err
447
- }
448
-
449
- return input, nil
450
- }
51
+ ```bash
52
+ jobdb sqlite \
53
+ --listen 127.0.0.1:9047 \
54
+ --db ./jobdb.db \
55
+ --blob-dir ./jobdb.blobs
451
56
  ```
452
57
 
453
- ### Job Restart
58
+ Flags:
454
59
 
455
- Restart a failed job from a specific step:
60
+ - `--db`: SQLite database path. Defaults to `jobdb.db`.
61
+ - `--blob-dir`: directory for large artifacts. Defaults to `<db>.blobs`.
62
+ - `--sqlite-dsn`: SQLite DSN. Overrides `--db` and `JOBDB_SQLITE_DSN`.
63
+ - `--listen`: HTTP listen address. Defaults to `127.0.0.1:9047`.
456
64
 
457
- ```go
458
- newJobKey, err := engine.SubmitRestartJob(ctx, jobdb.SubmitRestartJob{
459
- PriorJobKey: failedJobKey,
460
- LastStepToKeep: 5, // Replay from step 6 onwards
461
- ExtraTaskInput: newInput,
462
- ExtraTaskOutput: newOutput,
463
- })
464
- ```
65
+ Environment:
465
66
 
466
- ### Job Cancellation
67
+ - `JOBDB_SQLITE_DSN`: SQLite DSN used when `--sqlite-dsn` is not set.
467
68
 
468
- ```go
469
- err := engine.CancelJob(ctx, jobdb.CancelJob{
470
- JobKey: jobKey,
471
- Reason: "user requested cancellation",
472
- })
473
- ```
69
+ ### Toy
474
70
 
475
- ### Checking Job Status
71
+ The toy backend is in-memory. It is useful for local experiments and tests, not
72
+ for durable execution.
476
73
 
477
- ```go
478
- status, err := engine.CheckJobStatus(ctx, jobKey)
479
-
480
- switch status {
481
- case jobdb.JobStatusCompleted:
482
- // Job finished successfully
483
- case jobdb.JobStatusActive:
484
- // Job is running
485
- case jobdb.JobStatusCancelled:
486
- // Job was cancelled
487
- case jobdb.JobStatusReady:
488
- // Job is ready to run
489
- }
74
+ ```bash
75
+ jobdb toy --listen 127.0.0.1:9047
490
76
  ```
491
77
 
492
- ### Getting Job Results
493
-
494
- ```go
495
- result, err := engine.GetJobResult(ctx, jobKey)
496
- if err == jobdb.ErrJobNotComplete {
497
- // Job hasn't completed yet
498
- return
499
- }
500
-
501
- // Use result
502
- data, _ := result.GetData()
503
- ```
78
+ ### Direct
504
79
 
505
- ### Listing Jobs
80
+ The direct backend uses Postgres for job and chapter records, and a blobstore
81
+ URI for large artifact bytes. It installs or verifies the `pgwf` schema on
82
+ startup.
506
83
 
507
- ```go
508
- resp, err := engine.ListJobs(ctx, jobdb.ListJobsRequest{
509
- TenantIds: []string{"my-tenant"},
510
- Statuses: []jobdb.JobStatus{jobdb.JobStatusActive, jobdb.JobStatusCompleted},
511
- JobTypes: []string{"data-processing"},
512
- PageSize: 50,
513
- PageToken: "", // empty for first page
514
- })
515
-
516
- for _, job := range resp.Jobs {
517
- log.Printf("Job %s: %s", job.JobKey, job.Status)
518
- }
519
-
520
- // Get next page
521
- if resp.NextPageToken != "" {
522
- nextResp, err := engine.ListJobs(ctx, jobdb.ListJobsRequest{
523
- PageToken: resp.NextPageToken,
524
- // ... other filters
525
- })
526
- }
84
+ ```bash
85
+ JOBDB_POSTGRES_DSN='postgres://user:pass@localhost:5432/jobdb?sslmode=disable' \
86
+ jobdb direct --blob-store-uri 'blobfs:///var/lib/jobdb/blobs' --listen 127.0.0.1:9047
527
87
  ```
528
88
 
529
- ### Schedules
530
-
531
- Schedules are runtime-owned recurring job definitions. A schedule target is the
532
- same shape as a job start: job type, input `TaskData`, run policy, and app
533
- metadata. The runtime stores the target, including an artifact snapshot, and
534
- materializes each occurrence as a normal app job.
535
-
536
- ```go
537
- start := time.Now().UTC()
538
-
539
- info, err := engine.UpsertSchedule(ctx, jobdb.UpsertScheduleRequest{
540
- TenantId: "my-tenant",
541
- ScheduleId: "daily-cleanup",
542
- Trigger: jobdb.ScheduleTrigger{
543
- Kind: jobdb.ScheduleTriggerInterval,
544
- Interval: 24 * time.Hour,
545
- StartAt: &start,
546
- },
547
- Target: jobdb.ScheduleTarget{
548
- JobType: "data-processing",
549
- Data: jobdb.JobData(jobdb.NewTaskDataOrPanic(map[string]any{"bucket": "reports"})),
550
- Metadata: json.RawMessage(`{"owner":"analytics"}`),
551
- },
552
- OverlapPolicy: jobdb.ScheduleOverlapSerial,
553
- })
554
- if err != nil {
555
- return err
556
- }
557
-
558
- log.Printf("next scheduled job: %s", info.NextJobKey)
559
- ```
89
+ Flags:
560
90
 
561
- The schedule API includes `GetSchedule`, `ListSchedules`, `PauseSchedule`,
562
- `ResumeSchedule`, `ArchiveSchedule`, `TriggerSchedule`, and
563
- `ListScheduleRuns`. With serial overlap policy, the runtime submits the next
564
- occurrence before app execution starts, but makes it wait for the previous
565
- occurrence to complete before it can be leased.
91
+ - `--postgres-dsn`: Postgres DSN for `pgwf` state.
92
+ - `--blob-store-uri`: Blobstore URI for large artifacts. Defaults to local blobfs.
93
+ - `--listen`: HTTP listen address. Defaults to `127.0.0.1:9047`.
566
94
 
567
- ### External Task Completion
95
+ Environment:
568
96
 
569
- For tasks that require external input (e.g., human approval), you can complete them externally:
97
+ - `JOBDB_POSTGRES_DSN`: Postgres DSN used when `--postgres-dsn` is not set.
570
98
 
571
- ```go
572
- // Find tasks waiting for capability
573
- handles, err := engine.FindTasksWaitingForCapability(ctx,
574
- "approval-job", // job type
575
- "human-approval", // task type
576
- []string{"tenant-1"}, // tenants (nil for all)
577
- )
578
-
579
- for _, handle := range handles {
580
- // Get task input
581
- input, err := handle.Data()
582
-
583
- // ... process externally ...
584
-
585
- // Complete the task
586
- output := jobdb.NewTaskDataOrPanic(approvalResult)
587
- err = handle.Finish(ctx, output)
588
- }
589
- ```
99
+ ## Runtime API
590
100
 
591
- ## Engine Configuration
101
+ The server exposes the JobDB runtime REST API. The wire contract is documented
102
+ in [openapi/jobdb-runtime.yaml](openapi/jobdb-runtime.yaml).
592
103
 
593
- ### Builder Options
104
+ Go clients normally use the remote runtime adapter:
594
105
 
595
106
  ```go
596
- runtime := toyruntime.New()
597
-
598
- engine, err := jobdb.NewEngineBuilder().
599
- WithRuntime(runtime). // Required
600
- WithWorkerTenantId("tenant-1"). // Required when running workers
601
- WithMaxActive(10). // Concurrent task limit
602
- WithLogger(logger). // Custom logger
603
- WithAwaitRecycleThreshold(5 * time.Minute). // Await recycle threshold
604
- PlusWorkers(job1, task1, task2). // Register workers
605
- PlusWorkers(job2, task3). // Add more workers
606
- BuildEngine()
107
+ runtime, err := remoteruntime.New("http://127.0.0.1:9047", nil)
607
108
  ```
608
109
 
609
- ### Registering Workers
110
+ See [pkg/jobdb/README.md](pkg/jobdb/README.md) for the Go runtime API, data
111
+ types, and runtime package reference.
610
112
 
611
- #### At Engine Build Time
113
+ ## Go Workflow Workers
612
114
 
613
- Workers can be registered during engine construction:
115
+ Workflow workers are intentionally documented separately from the server. If you
116
+ are writing job workers, task workers, or a process that runs worker loops, use
117
+ the `pkg/workflow` package.
614
118
 
615
- ```go
616
- builder := jobdb.NewEngineBuilder().
617
- WithRuntime(runtime).
618
- WithWorkerTenantId("tenant-1")
619
-
620
- // Register a job with its tasks
621
- builder.PlusWorkers(
622
- MyJobWorker{},
623
- Task1{},
624
- Task2{},
625
- )
626
-
627
- // Register another job
628
- builder.PlusWorkers(
629
- AnotherJobWorker{},
630
- Task3{},
631
- )
632
-
633
- engine, err := builder.BuildEngine()
634
- ```
119
+ See [pkg/workflow/README.md](pkg/workflow/README.md).
635
120
 
636
- #### After Engine Start (Dynamic Registration)
121
+ ## Development
637
122
 
638
- Workers can also be registered after the engine has started:
123
+ The CLI source lives in `cmd/jobdb`. To run it directly from a checkout:
639
124
 
640
- ```go
641
- // Engine was built with WithWorkerTenantId("my-tenant") and is already running.
642
- go engine.Run(ctx)
643
-
644
- // Create a workset
645
- workset, err := jobdb.AsWorkSet(
646
- NewJobWorker{},
647
- NewTask1{},
648
- NewTask2{},
649
- )
650
- if err != nil {
651
- log.Fatal(err)
652
- }
653
-
654
- // Register dynamically
655
- err = engine.RegisterWorkers(workset)
656
- if err != nil {
657
- log.Fatal(err)
658
- }
659
-
660
- // The engine can now process jobs of type NewJobWorker.Name()
125
+ ```bash
126
+ go run ./cmd/jobdb --listen 127.0.0.1:9047 --db jobdb.db
661
127
  ```
662
128
 
663
- This is useful for:
664
- - Plugin systems where workers are loaded dynamically
665
- - Multi-tenant systems where different tenants have different workflows
666
- - Hot-reloading worker implementations without restarting the engine
667
-
668
- ### Running the Engine
669
-
670
- ```go
671
- ctx := context.Background()
672
-
673
- // Run the engine worker loop (blocks)
674
- engine.Run(ctx)
675
-
676
- // Or run in background
677
- go engine.Run(ctx)
129
+ Run the full test suite:
678
130
 
679
- // Cancel when done
680
- ctx, cancel := context.WithCancel(context.Background())
681
- defer cancel()
682
-
683
- go engine.Run(ctx)
684
- // ... do work ...
685
- cancel() // Gracefully stop the engine
131
+ ```bash
132
+ go test ./...
686
133
  ```
687
134
 
688
- Engines with registered workers must be built with `WithWorkerTenantId`.
689
- The worker loop polls only that tenant; run a separate engine for another
690
- tenant.
691
-
692
- ## Best Practices
693
-
694
- 1. **Keep Tasks Idempotent**: Tasks may be retried, so ensure they can safely run multiple times
695
- 2. **Use Appropriate Timeouts**: Set realistic invocation and total timeouts based on expected task duration
696
- 3. **Handle Large Data with Artifacts**: Use artifacts for files and binary data instead of embedding in TaskData
697
- 4. **Log Generously**: Use `ctx.Logger` to log progress and debug issues
698
- 5. **Design for Failure**: Workflows should gracefully handle task failures and retries
699
- 6. **Clean Up Resources**: Implement proper cleanup in artifact handlers
700
- 7. **Use Singleton Keys**: For jobs that should only run once (e.g., daily reports)
701
- 8. **Monitor Job Status**: Use ListJobs and CheckJobStatus to monitor workflow health
702
-
703
- ## Architecture Notes
704
-
705
- jobdb can run against several runtime backends:
706
-
707
- - **SQLite runtime**: Stores workflow state, leases, schedules, Strata row data,
708
- and blobfs artifacts locally. This is the default embedded runtime and the
709
- default `jobdb` mode.
710
- - **Postgres/Strata direct runtime**: Stores workflow state and coordinates
711
- distributed execution through pgwf, with workflow data and artifacts in
712
- Strata.
713
- - **Remote runtime**: Uses the same `WorkflowRuntime` API over REST. The server
714
- owns lease tokens and schedule preflight; clients and workers stay generic.
715
-
716
- Multiple engine instances can run concurrently when the selected runtime backend
717
- supports shared coordination.
718
-
719
- ## License
135
+ Useful references:
720
136
 
721
- See LICENSE file for details.
137
+ - [pkg/jobdb/README.md](pkg/jobdb/README.md): runtime API, data types, and backend packages.
138
+ - [pkg/workflow/README.md](pkg/workflow/README.md): workflow SDK, workers, and engines.
139
+ - [docs/MIGRATION-SWF-GO-TO-JOBDB.md](docs/MIGRATION-SWF-GO-TO-JOBDB.md): concise import migration from `swf-go`.
140
+ - [docs/API-SURFACE.md](docs/API-SURFACE.md): supported public packages.
141
+ - [docs/SPEC-OpenAPI-Runtime-Contract.md](docs/SPEC-OpenAPI-Runtime-Contract.md): runtime REST contract notes.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@colony2/jobdb",
3
3
  "type": "module",
4
- "version": "0.0.7",
4
+ "version": "0.0.10",
5
5
  "description": "jobdb job system",
6
6
  "scripts": {
7
7
  "postinstall": "node install.js",
@@ -30,51 +30,51 @@
30
30
  },
31
31
  "archives": {
32
32
  "darwin-arm64": {
33
- "name": "jobdb_0.0.7_Darwin_arm64.tar.gz",
34
- "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.7/jobdb_0.0.7_Darwin_arm64.tar.gz",
33
+ "name": "jobdb_0.0.10_Darwin_arm64.tar.gz",
34
+ "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.10/jobdb_0.0.10_Darwin_arm64.tar.gz",
35
35
  "bins": [
36
36
  "jobdb"
37
37
  ],
38
38
  "format": "tar.gz",
39
39
  "checksum": {
40
40
  "algorithm": "sha256",
41
- "digest": "462f01b3754e85129d7564cf5e7dddc03e68539e535aff01de60db26277c78e2"
41
+ "digest": "c6604dbd46347dce23c75df4092bc1e3803e3c1142269609f96b6eb010fc8355"
42
42
  }
43
43
  },
44
44
  "darwin-x64": {
45
- "name": "jobdb_0.0.7_Darwin_x86_64.tar.gz",
46
- "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.7/jobdb_0.0.7_Darwin_x86_64.tar.gz",
45
+ "name": "jobdb_0.0.10_Darwin_x86_64.tar.gz",
46
+ "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.10/jobdb_0.0.10_Darwin_x86_64.tar.gz",
47
47
  "bins": [
48
48
  "jobdb"
49
49
  ],
50
50
  "format": "tar.gz",
51
51
  "checksum": {
52
52
  "algorithm": "sha256",
53
- "digest": "0982242b1a0aa33553c966e823c81bfd6161558d7500def7e5ba624cf93ce8c9"
53
+ "digest": "2d9abbcf2cb67d4123fef17dbb14fe2c909a5b4a6a7e6812934ce4ab274fdc44"
54
54
  }
55
55
  },
56
56
  "linux-arm64": {
57
- "name": "jobdb_0.0.7_Linux_arm64.tar.gz",
58
- "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.7/jobdb_0.0.7_Linux_arm64.tar.gz",
57
+ "name": "jobdb_0.0.10_Linux_arm64.tar.gz",
58
+ "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.10/jobdb_0.0.10_Linux_arm64.tar.gz",
59
59
  "bins": [
60
60
  "jobdb"
61
61
  ],
62
62
  "format": "tar.gz",
63
63
  "checksum": {
64
64
  "algorithm": "sha256",
65
- "digest": "d4086e3fd83f8e3eb9227a70fbd14954950600a16918042626c68a1e976a0545"
65
+ "digest": "e1cb989ae67774d5b26c86dca07385f26b59ccd37b00af9b8965294edf6c48ea"
66
66
  }
67
67
  },
68
68
  "linux-x64": {
69
- "name": "jobdb_0.0.7_Linux_x86_64.tar.gz",
70
- "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.7/jobdb_0.0.7_Linux_x86_64.tar.gz",
69
+ "name": "jobdb_0.0.10_Linux_x86_64.tar.gz",
70
+ "url": "https://github.com/colony-2/jobdb/releases/download/v0.0.10/jobdb_0.0.10_Linux_x86_64.tar.gz",
71
71
  "bins": [
72
72
  "jobdb"
73
73
  ],
74
74
  "format": "tar.gz",
75
75
  "checksum": {
76
76
  "algorithm": "sha256",
77
- "digest": "d6eb564436f0a654f8355569db66588d60e21fcedd790601b437ca0712475495"
77
+ "digest": "6dc5af9b1aec10c0a9307a646012e3ceafb991952b77d282820f0a25e96acf70"
78
78
  }
79
79
  }
80
80
  },