mongo-job-scheduler 0.1.6 → 0.1.8

package/README.md

@@ -1,52 +1,40 @@
 # Mongo Job Scheduler
 
-A production-grade MongoDB-backed job scheduler for Node.js.
+A production-grade MongoDB-backed job scheduler for Node.js with distributed locking, retries, cron scheduling, and crash recovery.
 
-Designed for distributed systems that need:
-
-- reliable background jobs
-- retries with backoff
-- cron & interval scheduling
-- crash recovery
-- MongoDB sharding safety
+[![npm version](https://img.shields.io/npm/v/mongo-job-scheduler.svg)](https://www.npmjs.com/package/mongo-job-scheduler)
 
 ---
 
 ## Features
 
-- **Distributed locking** using MongoDB
-- **Multiple workers** support
-- **Retry with backoff**
-- **Cron jobs** (timezone-aware, non-drifting)
-- **Interval jobs**
-- **Resume on restart**
-- **Stale lock recovery**
-- **Automatic Lock Renewal** (Heartbeats): Long-running jobs automatically extend their lock.
-- **Sharding-safe design**
-
-## Distributed Systems
-
-This library is designed for distributed environments. You can run **multiple scheduler instances** (on different servers, pods, or processes) connected to the same MongoDB.
-
-- **Atomic Locking**: Uses `findOneAndUpdate` to safe-guard against race conditions.
-- **Concurrency**: Only one worker will execute a given job instance.
-- **Scalable**: Horizontal scaling is supported via MongoDB sharding.
+- ✅ **Distributed locking** — safe for multiple instances
+- ✅ **Atomic job execution** — no double processing
+- ✅ **Automatic retries** — with configurable backoff
+- ✅ **Cron scheduling** — timezone-aware, non-drifting
+- ✅ **Interval jobs** — repeated execution
+- ✅ **Crash recovery** — resume on restart
+- ✅ **Heartbeats** — automatic lock renewal for long jobs
+- ✅ **Query API** — filter, sort, paginate jobs
+- ✅ **Auto-indexing** — performance optimized out of the box
+- ✅ **Sharding-safe** — designed for MongoDB sharding
 
 ---
 
-## Install
+## Quick Start
+
+### Installation
 
 ```bash
 npm install mongo-job-scheduler
 ```
 
-## Basic Usage
+### Basic Usage
 
 ```typescript
 import { Scheduler, MongoJobStore } from "mongo-job-scheduler";
 import { MongoClient } from "mongodb";
 
-// ... connect to mongo ...
 const client = new MongoClient("mongodb://localhost:27017");
 await client.connect();
 const db = client.db("my-app");
@@ -62,133 +50,204 @@ const scheduler = new Scheduler({
 await scheduler.start();
 ```
 
-## Cron with Timezone
+---
+
+## Scheduling Jobs
+
+### One-Time Job
+
+```typescript
+await scheduler.schedule({
+  name: "send-email",
+  data: { userId: 123 },
+  runAt: new Date(Date.now() + 60000), // run in 1 minute
+});
+```
+
+### Cron Jobs (Timezone-Aware)
 
 ```typescript
 await scheduler.schedule({
   name: "daily-report",
-  // data: { type: "report" }, // optional payload
   repeat: {
-    cron: "0 9 * * *",
+    cron: "0 9 * * *", // every day at 9 AM
     timezone: "Asia/Kolkata", // default is UTC
   },
 });
 ```
 
-## Interval Scheduling
-
-Run a job repeatedly with a fixed delay between executions (e.g., every 5 minutes).
+### Interval Jobs
 
 ```typescript
 await scheduler.schedule({
   name: "cleanup-logs",
   data: {},
   repeat: {
-    every: 5 * 60 * 1000, // 5 minutes in milliseconds
+    every: 5 * 60 * 1000, // every 5 minutes (in milliseconds)
   },
 });
 ```
 
-## Job Deduplication
+### Bulk Scheduling
 
-Prevent duplicate jobs using `dedupeKey`. If a job with the same key exists, it returns the existing job instead of creating a new one.
+For high-performance ingestion:
 
 ```typescript
-await scheduler.schedule({
-  name: "email",
-  data: { userId: 123 },
-  dedupeKey: "email:user:123",
-});
+const jobs = await scheduler.scheduleBulk([
+  { name: "email", data: { userId: 1 } },
+  { name: "email", data: { userId: 2 } },
+  { name: "email", data: { userId: 3 } },
+]);
 ```
 
-## Job Cancellation
+---
+
+## Job Management
+
+### Get Job by ID
 
 ```typescript
-// Cancel a pending or running job
-await scheduler.cancel(jobId);
+const job = await scheduler.getJob(jobId);
 ```
 
-## Job Querying
+### Query Jobs
+
+List jobs with filtering, sorting, and pagination:
 
 ```typescript
-const job = await scheduler.getJob(jobId);
+const jobs = await scheduler.getJobs({
+  name: "daily-report",
+  status: "failed", // or ["failed", "pending"]
+  sort: { field: "updatedAt", order: "desc" },
+  limit: 10,
+  skip: 0,
+});
 ```
 
-## Job Persistence & Updates
+### Update Job
 
-Update job `data`, reschedule, or modify configuration safely.
+Update job data, reschedule, or modify configuration:
 
 ```typescript
 await scheduler.updateJob(jobId, {
   data: { page: 2 },
   nextRunAt: new Date(Date.now() + 60000), // delay by 1 min
-  repeat: { every: 60000 }, // Change to run every minute
+  repeat: { every: 60000 }, // change to run every minute
 });
 ```
 
-## Bulk Scheduling
+### Cancel Job
 
-For high-performance ingestion, use `scheduleBulk` to insert multiple jobs in a single database operation:
+```typescript
+await scheduler.cancel(jobId);
+```
+
+---
+
+## Advanced Features
+
+### Retries with Backoff
 
 ```typescript
-const jobs = await scheduler.scheduleBulk([
-  { name: "email", data: { userId: 1 } },
-  { name: "email", data: { userId: 2 } },
-]);
+// Simple: 3 attempts with instant retry
+await scheduler.schedule({
+  name: "webhook",
+  retry: 3,
+});
+
+// Advanced: custom delay and backoff
+await scheduler.schedule({
+  name: "api-call",
+  retry: {
+    maxAttempts: 5,
+    delay: 1000, // 1 second fixed delay
+    // or: delay: (attempt) => attempt * 1000 // dynamic backoff
+  },
+});
 ```
 
-## Events
+### Job Deduplication
 
-The scheduler emits typed events for lifecycle monitoring.
+Prevent duplicate jobs using idempotency keys:
 
 ```typescript
-// Scheduler events
-scheduler.on("scheduler:start", () => console.log("Scheduler started"));
-scheduler.on("scheduler:stop", () => console.log("Scheduler stopped"));
-scheduler.on("scheduler:error", (err) =>
-  console.error("Scheduler error:", err)
-);
+await scheduler.schedule({
+  name: "email",
+  data: { userId: 123 },
+  dedupeKey: "email:user:123", // only one job with this key
+});
+```
 
-// Worker events
-scheduler.on("worker:start", (workerId) =>
-  console.log("Worker started:", workerId)
-);
-scheduler.on("worker:stop", (workerId) =>
-  console.log("Worker stopped:", workerId)
-);
+### Event Monitoring
 
-// Job events
-scheduler.on("job:created", (job) => console.log("Job created:", job._id));
-scheduler.on("job:start", (job) => console.log("Job processing:", job._id));
+```typescript
 scheduler.on("job:success", (job) => console.log("Job done:", job._id));
 scheduler.on("job:fail", ({ job, error }) =>
   console.error("Job failed:", job._id, error)
 );
 scheduler.on("job:retry", (job) =>
-  console.warn("Job retrying:", job._id, job.attempts)
+  console.warn("Retrying:", job._id, "attempt", job.attempts)
 );
-scheduler.on("job:cancel", (job) => console.log("Job cancelled:", job._id));
+
+// More events: scheduler:start, scheduler:stop, worker:start,
+// worker:stop, job:created, job:start, job:cancel
 ```
 
-## Documentation
+### Graceful Shutdown
 
-See `ARCHITECTURE.md` for:
+Wait for in-flight jobs to complete:
 
-- job lifecycle
-- retry & repeat semantics
-- MongoDB indexes
-- sharding strategy
-- production checklist
+```typescript
+await scheduler.stop({
+  graceful: true,
+  timeoutMs: 30000,
+});
+```
 
-## Graceful Shutdown
+---
 
-Stop the scheduler and wait for in-flight jobs to complete:
+## Performance & Scaling
 
-```typescript
-await scheduler.stop({ graceful: true, timeoutMs: 30000 });
-```
+### Automatic Indexing
+
+**MongoDB indexes are created automatically** when you initialize `MongoJobStore`. No manual setup required.
+
+The library creates three indexes in background mode:
+
+- `{ status: 1, nextRunAt: 1 }` — for job polling (critical)
+- `{ dedupeKey: 1 }` — for deduplication (unique)
+- `{ lockedAt: 1 }` — for stale lock recovery
+
+These indexes prevent query time from degrading from O(log n) to O(n) at scale.
+
+### Distributed Systems
+
+Run **multiple scheduler instances** (different servers, pods, or processes) connected to the same MongoDB:
+
+- **Atomic Locking** — uses `findOneAndUpdate` to prevent race conditions
+- **Concurrency Control** — only one worker executes a job instance
+- **Horizontally Scalable** — supports MongoDB sharding
+
+See `architecture.md` for sharding strategy and production guidelines.
+
+---
+
+## Documentation
+
+- **`architecture.md`** — Internal design, MongoDB schema, sharding strategy, production checklist
+- **Job lifecycle** — pending → running → completed/failed
+- **Retry & repeat semantics** — at-most-once guarantees
+- **Correctness guarantees** — what we ensure and what we don't
+
+---
 
 ## Status
 
-**Early-stage but production-tested.**
+**Early-stage but production-tested.**  
 API may evolve before 1.0.0.
+
+---
+
+## License
+
+MIT

package/dist/core/scheduler.d.ts

@@ -2,6 +2,7 @@ import { SchedulerEventMap } from "../types/events";
 import { JobStore, JobUpdates } from "../store";
 import { Job } from "../types/job";
 import { ScheduleOptions } from "../types/schedule";
+import { JobQuery } from "../types/query";
 export interface SchedulerOptions {
     id?: string;
     store?: JobStore;
@@ -33,6 +34,10 @@ export declare class Scheduler {
      * Get a job by ID
      */
     getJob(jobId: unknown): Promise<Job | null>;
+    /**
+     * Query jobs
+     */
+    getJobs(query: JobQuery): Promise<Job[]>;
     /**
      * Update job data or schedule
      */

package/dist/core/scheduler.js

@@ -13,7 +13,7 @@ class Scheduler {
         this.handler = options.handler;
         this.workerCount = options.workers ?? 1;
         this.pollInterval = options.pollIntervalMs ?? 500;
-        this.lockTimeout = options.lockTimeoutMs ?? 30000;
+        this.lockTimeout = options.lockTimeoutMs ?? 10 * 60 * 1000; // default 10 minutes
         this.defaultTimezone = options.defaultTimezone;
     }
     on(event, listener) {
@@ -94,6 +94,15 @@ class Scheduler {
         }
         return this.store.findById(jobId);
     }
+    /**
+     * Query jobs
+     */
+    async getJobs(query) {
+        if (!this.store) {
+            throw new Error("Scheduler has no JobStore configured");
+        }
+        return this.store.findAll(query);
+    }
     /**
      * Update job data or schedule
      */

package/dist/store/in-memory-job-store.d.ts

@@ -1,5 +1,6 @@
 import { Job } from "../types/job";
 import { JobStore, JobUpdates } from "./job-store";
+import { JobQuery } from "../types/query";
 export declare class InMemoryJobStore implements JobStore {
     private jobs;
     private mutex;
@@ -24,4 +25,5 @@ export declare class InMemoryJobStore implements JobStore {
     findById(jobId: unknown): Promise<Job | null>;
     renewLock(jobId: unknown, workerId: string): Promise<void>;
     update(jobId: unknown, updates: JobUpdates): Promise<void>;
+    findAll(query: JobQuery): Promise<Job[]>;
 }

package/dist/store/in-memory-job-store.js

@@ -149,5 +149,31 @@ class InMemoryJobStore {
         }
         job.updatedAt = new Date();
     }
+    async findAll(query) {
+        let jobs = Array.from(this.jobs.values());
+        // Filter
+        if (query.name) {
+            jobs = jobs.filter((j) => j.name === query.name);
+        }
+        if (query.status) {
+            const statuses = Array.isArray(query.status)
+                ? query.status
+                : [query.status];
+            jobs = jobs.filter((j) => statuses.includes(j.status));
+        }
+        // Sort
+        if (query.sort) {
+            const { field, order } = query.sort;
+            jobs.sort((a, b) => {
+                const valA = a[field].getTime();
+                const valB = b[field].getTime();
+                return order === "asc" ? valA - valB : valB - valA;
+            });
+        }
+        // Skip/Limit
+        const start = query.skip ?? 0;
+        const end = query.limit ? start + query.limit : undefined;
+        return jobs.slice(start, end);
+    }
 }
 exports.InMemoryJobStore = InMemoryJobStore;

package/dist/store/job-store.d.ts

@@ -55,9 +55,14 @@ export interface JobStore {
      * Update job properties (data persistence)
      */
     update(jobId: unknown, updates: JobUpdates): Promise<void>;
+    /**
+     * Find all jobs matching query
+     */
+    findAll(query: JobQuery): Promise<Job[]>;
 }
 import { RetryOptions } from "../types/retry";
 import { RepeatOptions } from "../types/repeat";
+import { JobQuery } from "../types/query";
 export interface JobUpdates {
     data?: unknown;
     nextRunAt?: Date;

package/dist/store/mongo/mongo-job-store.d.ts

@@ -1,6 +1,7 @@
 import { Db, ObjectId } from "mongodb";
 import { JobStore, JobUpdates } from "../job-store";
 import { Job } from "../../types/job";
+import { JobQuery } from "../../types/query";
 export interface MongoJobStoreOptions {
     collectionName?: string;
     lockTimeoutMs?: number;
@@ -9,6 +10,10 @@ export declare class MongoJobStore implements JobStore {
     private readonly collection;
     private readonly defaultLockTimeoutMs;
     constructor(db: Db, options?: MongoJobStoreOptions);
+    /**
+     * Create necessary indexes for optimal query performance
+     */
+    private ensureIndexes;
     create(job: Job): Promise<Job>;
     createBulk(jobs: Job[]): Promise<Job[]>;
     findAndLockNext(options: {
@@ -30,4 +35,5 @@ export declare class MongoJobStore implements JobStore {
     }): Promise<number>;
     renewLock(id: ObjectId, workerId: string): Promise<void>;
     update(id: ObjectId, updates: JobUpdates): Promise<void>;
+    findAll(query: JobQuery): Promise<Job[]>;
 }

package/dist/store/mongo/mongo-job-store.js

@@ -5,6 +5,23 @@ class MongoJobStore {
     constructor(db, options = {}) {
         this.collection = db.collection(options.collectionName ?? "scheduler_jobs");
         this.defaultLockTimeoutMs = options.lockTimeoutMs ?? 30000;
+        // Auto-create indexes for performance
+        this.ensureIndexes().catch((err) => {
+            console.error("Failed to create indexes:", err);
+        });
+    }
+    /**
+     * Create necessary indexes for optimal query performance
+     */
+    async ensureIndexes() {
+        await Promise.all([
+            // Primary index for job polling (findAndLockNext)
+            this.collection.createIndex({ status: 1, nextRunAt: 1 }, { background: true }),
+            // Index for deduplication
+            this.collection.createIndex({ dedupeKey: 1 }, { unique: true, sparse: true, background: true }),
+            // Index for stale lock recovery
+            this.collection.createIndex({ lockedAt: 1 }, { sparse: true, background: true }),
+        ]);
     }
     // --------------------------------------------------
     // CREATE
@@ -195,5 +212,30 @@ class MongoJobStore {
             $set.repeat = updates.repeat;
         await this.collection.updateOne({ _id: id }, { $set });
     }
+    async findAll(query) {
+        const filter = {};
+        if (query.name) {
+            filter.name = query.name;
+        }
+        if (query.status) {
+            filter.status = Array.isArray(query.status)
+                ? { $in: query.status }
+                : query.status;
+        }
+        let cursor = this.collection.find(filter);
+        if (query.sort) {
+            cursor = cursor.sort({
+                [query.sort.field]: query.sort.order === "asc" ? 1 : -1,
+            });
+        }
+        if (query.skip) {
+            cursor = cursor.skip(query.skip);
+        }
+        if (query.limit) {
+            cursor = cursor.limit(query.limit);
+        }
+        const docs = await cursor.toArray();
+        return docs;
+    }
 }
 exports.MongoJobStore = MongoJobStore;

package/dist/types/job.d.ts

@@ -13,7 +13,7 @@ export interface Job<Data = unknown> {
     lockedBy?: string;
     attempts: number;
     lastError?: string;
-    retry?: RetryOptions;
+    retry?: RetryOptions | number;
     repeat?: RepeatOptions;
     dedupeKey?: string;
     createdAt: Date;

package/dist/types/query.d.ts

@@ -0,0 +1,11 @@
+import { JobStatus } from "./lifecycle";
+export interface JobQuery {
+    name?: string;
+    status?: JobStatus | JobStatus[];
+    limit?: number;
+    skip?: number;
+    sort?: {
+        field: "nextRunAt" | "createdAt" | "updatedAt";
+        order: "asc" | "desc";
+    };
+}

package/dist/types/query.js

@@ -0,0 +1,2 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });

package/dist/types/schedule.d.ts

@@ -11,7 +11,7 @@ export interface ScheduleOptions<T = unknown> {
     /**
      * Retry configuration
      */
-    retry?: RetryOptions;
+    retry?: RetryOptions | number;
     /**
      * Repeat configuration (cron or every)
      */

package/dist/worker/worker.js

@@ -10,7 +10,7 @@ class Worker {
         this.handler = handler;
         this.running = false;
         this.pollInterval = options.pollIntervalMs ?? 500;
-        this.lockTimeout = options.lockTimeoutMs ?? 30000;
+        this.lockTimeout = options.lockTimeoutMs ?? 10 * 60 * 1000; // default 10 minutes
         this.workerId =
             options.workerId ?? `worker-${Math.random().toString(36).slice(2)}`;
         this.defaultTimezone = options.defaultTimezone;
@@ -128,7 +128,10 @@ class Worker {
         catch (err) {
             const error = err instanceof Error ? err : new Error(String(err));
             const attempts = (job.attempts ?? 0) + 1;
-            const retry = job.retry;
+            let retry = job.retry;
+            if (typeof retry === "number") {
+                retry = { maxAttempts: retry, delay: 0 };
+            }
             if (retry && attempts < retry.maxAttempts) {
                 const nextRunAt = new Date(Date.now() + (0, retry_1.getRetryDelay)(retry, attempts));
                 await this.store.reschedule(job._id, nextRunAt, { attempts });

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "mongo-job-scheduler",
-  "version": "0.1.6",
+  "version": "0.1.8",
   "description": "Production-grade MongoDB-backed job scheduler with retries, cron, timezone support, and crash recovery",
   "license": "MIT",
   "author": "Darshan Bhut",