agenda 2.1.0 → 3.1.0

package/History.md

@@ -1,12 +1,62 @@
 Next
 ====
 
-  * https://github.com/agenda/agenda/pulls
+* https://github.com/agenda/agenda/pulls
+
+3.1.0 / 2020-04-07
+==================
+
+_Stay safe!_
+
+* Fix for skipImmediate resetting nextRunAt to current date ([#860](https://github.com/agenda/agenda/pull/860)) (Thanks @AshlinDuncan!)
+* Fix deprecated reconnect options ([#948](https://github.com/agenda/agenda/pull/948)) (Thanks @ekegodigital!)
+* Add ability to set a skip when querying jobs. ([#898](https://github.com/agenda/agenda/pull/898)) (Thanks @cjolif!)
+
+Internal:
+* Fixed deprecated MongoDB functions in tests ([#928](https://github.com/agenda/agenda/pull/928)) (Thanks @MichielDeMey!)
+* Updated devDependencies
+
+Thank you @koresar, @sampathBlam, and @MichielDeMey helping to review PRs for this release! 👏
+
+
+3.0.0 / 2020-02-13
+==================
+
+* Support MongoDB's Unified Topology Design ([#921](https://github.com/agenda/agenda/pull/921)) (Thanks @viktorzavadil!)
+* Fix: check that the new nextRunAt is different that the previous nextRunAt ([#863](https://github.com/agenda/agenda/pull/863)) (Thanks @RaphaelRheault!)
+* Update dependencies. Most notably MongoDB driver 3.4 → 3.5 ([#899](https://github.com/agenda/agenda/pull/899), [#900](https://github.com/agenda/agenda/pull/900), [#903](https://github.com/agenda/agenda/pull/903), [#906](https://github.com/agenda/agenda/pull/906), [#908](https://github.com/agenda/agenda/pull/908), [#910](https://github.com/agenda/agenda/pull/910), [#912](https://github.com/agenda/agenda/pull/912), [#913](https://github.com/agenda/agenda/pull/913), [#920](https://github.com/agenda/agenda/pull/920), [#922](https://github.com/agenda/agenda/pull/922))
+* Documentation updates, thanks @MichielDeMey and @Sunghee2. ([#923](https://github.com/agenda/agenda/pull/923) & [#907](https://github.com/agenda/agenda/pull/907))
+
+BREAKING
+--------
+
+* Stop testing for Node.js 8. This might still work but we're no longer actively testing for it. ([#925](https://github.com/agenda/agenda/pull/925))
+
+2.3.0 / 2019-12-16
+==================
+
+* Improved performance in situations when there are many "expired" jobs in the  database ([#869](https://github.com/agenda/agenda/pull/869)) (Thanks @mfred488!)
+* Fix periodic node.js process unhandledRejection ([#887](https://github.com/agenda/agenda/pull/887)) (Thanks @koresar and @Scorpil)
+* Update dependencies
+
+2.2.0 / 2019-11-24
+==================
+
+  * Fix `skipImmediate` option in `.every` ([#861](https://github.com/agenda/agenda/pull/861)) (Thanks @erics2783!)
+  * Add try/catch block to agenda#now method ([#876](https://github.com/agenda/agenda/pull/876)) (Thanks @sampathBlam!)
+  * Refactor job queuing mechanism. Agenda n ow guarantees priority when executing jobs scheduled the same datetime. Fixes also some tests. ([#852](https://github.com/agenda/agenda/pull/852)) (Thank you @dmbarreiro!)
+  * Update dependencies (Kudos @simison!)
+    Most notably `mongodb` ~3.2.7 -> ~3.3.0 ([changelog](https://github.com/mongodb/node-mongodb-native/tree/v3.3.0)) — highlights:
+    - Mongo DB Server Version 4.2 feature support
+    - Merged `mongodb-core` into `node-mongodb-native`
+    - Beta support for MongoDB Client-Side Encryption
+    - SRV Polling for Sharded Clusters
+  * Updates to documentation (Thank you @lautarobock, @sampathBlam, @indatawetrust)
 
 2.1.0 / 2019-09-09
 ==================
   * Support async functions in job processing ([#653](https://github.com/agenda/agenda/pull/653)) (thanks @princjef!)
-  * Allow sorting and limiting jobs when searching ((#665)[https://github.com/agenda/agenda/pull/665]) (thank you @edwin-jones)
+  * Allow sorting and limiting jobs when searching ([#665](https://github.com/agenda/agenda/pull/665)) (thank you @edwin-jones)
   * Update MongoClient connection settings with `useNewUrlParser: true` to remove the deprecation warning. ([#806](https://github.com/agenda/agenda/pull/806)) (thanks @dpawson905!)
   * Allow valid date strings when scheduling ([#808](https://github.com/agenda/agenda/pull/808)) (Thanks @wingsbob!)
   * Update dependencies ([#820](https://github.com/agenda/agenda/pull/820))

package/README.md

@@ -1,5 +1,5 @@
 <p align="center">
-  <img src="https://cdn.rawgit.com/agenda/agenda/master/agenda.svg" alt="Agenda" width="100" height="100">
+  <img src="https://cdn.jsdelivr.net/gh/agenda/agenda@master/agenda.svg" alt="Agenda" width="100" height="100">
 </p>
 <p align="center">
   A light-weight job scheduling library for Node.js
@@ -25,6 +25,7 @@
 - Event backed job queue that you can hook into.
 - [Agendash](https://github.com/agenda/agendash): optional standalone web-interface.
 - [Agenda-rest](https://github.com/agenda/agenda-rest): optional standalone REST API.
+- [inversify-agenda](https://github.com/lautarobock/inversify-agenda) - Some utilities for the development of agenda workers with Inversify
 
 ### Feature Comparison
 
@@ -48,7 +49,7 @@ Agenda is great if you need something that is simple and backed by MongoDB.
 | Persistence     | ✓             |  ✓  |   ✓    |
 | UI              | ✓             |     |   ✓    |
 | REST API        |               |     |   ✓    |
-| Optimized for   | Jobs / Messages | Jobs | Messages | Jobs |
+| Optimized for   | Jobs / Messages | Messages | Jobs |
 
 _Kudos for making the comparison chart goes to [Bull](https://www.npmjs.com/package/bull#feature-comparison) maintainers._
 
@@ -416,10 +417,6 @@ under `job.attrs.data`.
 `options` is an optional argument that will be passed to [`job.repeatEvery`](#repeateveryinterval-options).
 In order to use this argument, `data` must also be specified.
 
-`cb` is an optional callback function which will be called when the job has been
-persisted in the database.
-
-
 Returns the `job`.
 
 ```js
@@ -449,9 +446,6 @@ Schedules a job to run `name` once at a given time. `when` can be a `Date` or a
 `data` is an optional argument that will be passed to the processing function
 under `job.attrs.data`.
 
-`cb` is an optional callback function which will be called when the job has been
-persisted in the database.
-
 Returns the `job`.
 
 ```js
@@ -493,37 +487,33 @@ console.log('Job successfully saved');
 ## Managing Jobs
 
 
-### jobs(mongodb-native query, mongodb-native sort, mongodb-native limit)
+### jobs(mongodb-native query, mongodb-native sort, mongodb-native limit, mongodb-native skip)
 
-Lets you query (then sort and limit the result) all of the jobs in the agenda job's database. These are full [mongodb-native](https://github.com/mongodb/node-mongodb-native) `find`, `sort` and `limit` commands. See mongodb-native's documentation for details.
+Lets you query (then sort, limit and skip the result) all of the jobs in the agenda job's database. These are full [mongodb-native](https://github.com/mongodb/node-mongodb-native) `find`, `sort`, `limit` and `skip` commands. See mongodb-native's documentation for details.
 
 ```js
-const jobs = await agenda.jobs({name: 'printAnalyticsReport'}, {data:-1}, 3);
+const jobs = await agenda.jobs({name: 'printAnalyticsReport'}, {data:-1}, 3, 1);
 // Work with jobs (see below)
 ```
 
-### cancel(mongodb-native query, cb)
+### cancel(mongodb-native query)
 
-Cancels any jobs matching the passed mongodb-native query, and removes them from the database.
+Cancels any jobs matching the passed mongodb-native query, and removes them from the database. Returns a Promise resolving to the number of cancelled jobs, or rejecting on error.
 
 ```js
-agenda.cancel({name: 'printAnalyticsReport'}, (err, numRemoved) => {
-  // ...
-});
+const numRemoved = await agenda.cancel({name: 'printAnalyticsReport'});
 ```
 
 This functionality can also be achieved by first retrieving all the jobs from the database using `agenda.jobs()`, looping through the resulting array and calling `job.remove()` on each. It is however preferable to use `agenda.cancel()` for this use case, as this ensures the operation is atomic.
 
-### purge(cb)
+### purge()
 
-Removes all jobs in the database without defined behaviors. Useful if you change a definition name and want to remove old jobs.
+Removes all jobs in the database without defined behaviors. Useful if you change a definition name and want to remove old jobs. Returns a Promise resolving to the number of removed jobs, or rejecting on error.
 
 *IMPORTANT:* Do not run this before you finish defining all of your jobs. If you do, you will nuke your database of jobs.
 
 ```js
-agenda.purge((err, numRemoved) => {
-  // ...
-});
+const numRemoved = await agenda.purge();
 ```
 
 ## Starting the job processor
@@ -710,7 +700,7 @@ try {
   console.log('Successfully removed job from collection');
 } catch (e) {
   console.error('Error removing job from collection');
-});
+}
 ```
 
 ### disable()
@@ -960,7 +950,7 @@ const agenda = new Agenda(connectionOpts);
 const jobTypes = process.env.JOB_TYPES ? process.env.JOB_TYPES.split(',') : [];
 
 jobTypes.forEach(type => {
-  require('./lib/jobs/' + type)(agenda);
+  require('./jobs/' + type)(agenda);
 });
 
 if (jobTypes.length) {

package/lib/agenda/cancel.js

@@ -15,8 +15,8 @@ module.exports = async function(query) {
     const {result} = await this._collection.deleteMany(query);
     debug('%s jobs cancelled', result.n);
     return result.n;
-  } catch (err) {
+  } catch (error) {
     debug('error trying to delete jobs from MongoDB');
-    throw err;
+    throw error;
   }
 };

package/lib/agenda/database.js

@@ -25,8 +25,13 @@ module.exports = function(url, collection, options, cb) {
     url = 'mongodb://' + url;
   }
 
+  let reconnectOptions = {autoReconnect: true, reconnectTries: Number.MAX_SAFE_INTEGER, reconnectInterval: this._processEvery};
+  if (options && options.useUnifiedTopology && options.useUnifiedTopology === true) {
+    reconnectOptions = {};
+  }
+
   collection = collection || 'agendaJobs';
-  options = Object.assign({autoReconnect: true, reconnectTries: Number.MAX_SAFE_INTEGER, reconnectInterval: this._processEvery, useNewUrlParser: true}, options);
+  options = {...reconnectOptions, ...options};
   MongoClient.connect(url, options, (error, client) => {
     if (error) {
       debug('error connecting to MongoDB using collection: [%s]', collection);
@@ -35,8 +40,10 @@ module.exports = function(url, collection, options, cb) {
       } else {
         throw error;
       }
+
       return;
     }
+
     debug('successful connection to MongoDB using collection: [%s]', collection);
     self._db = client;
     self._mdb = client.db();

package/lib/agenda/define.js

@@ -16,6 +16,7 @@ module.exports = function(name, options, processor) {
     processor = options;
     options = {};
   }
+
   this._definitions[name] = {
     fn: processor,
     concurrency: options.concurrency || this._defaultConcurrency,

package/lib/agenda/every.js

@@ -25,7 +25,6 @@ module.exports = async function(interval, names, data, options) {
 
     job.attrs.type = 'single';
     job.repeatEvery(interval, options);
-    job.computeNextRunAt();
     await job.save();
 
     return job;
@@ -46,8 +45,8 @@ module.exports = async function(interval, names, data, options) {
       debug('every() -> all jobs created successfully');
 
       return Promise.all(jobs);
-    } catch (err) { // @TODO: catch - ignore :O
-      debug('every() -> error creating one or more of the jobs');
+    } catch (error) { // @TODO: catch - ignore :O
+      debug('every() -> error creating one or more of the jobs', error);
     }
   };
 

package/lib/agenda/find-and-lock-next-job.js

@@ -8,77 +8,69 @@ const {createJob} = require('../utils');
  * @function
  * @param {String} jobName name of job to try to lock
  * @param {Object} definition definition used to tell how job is run
- * @param {Function} cb called when job lock fails or passes
  * @access protected
  * @caller jobQueueFilling() only
- * @returns {undefined}
+ * @returns Promise
  */
-module.exports = function(jobName, definition, cb) {
+module.exports = async function(jobName, definition) {
   const self = this;
   const now = new Date();
   const lockDeadline = new Date(Date.now().valueOf() - definition.lockLifetime);
-  debug('_findAndLockNextJob(%s, [Function], cb)', jobName);
+  debug('_findAndLockNextJob(%s, [Function])', jobName);
 
   // Don't try and access MongoDB if we've lost connection to it.
   // Trying to resolve crash on Dev PC when it resumes from sleep. NOTE: Does this still happen?
   const s = this._mdb.s || this._mdb.db.s;
-  if (s.topology.connections().length === 0) {
+  if (s.topology.connections && s.topology.connections().length === 0 && !this._mongoUseUnifiedTopology) {
     if (s.topology.autoReconnect && !s.topology.isDestroyed()) {
       // Continue processing but notify that Agenda has lost the connection
       debug('Missing MongoDB connection, not attempting to find and lock a job');
       self.emit('error', new Error('Lost MongoDB connection'));
-      cb();
     } else {
       // No longer recoverable
       debug('topology.autoReconnect: %s, topology.isDestroyed(): %s', s.topology.autoReconnect, s.topology.isDestroyed());
-      cb(new Error('MongoDB connection is not recoverable, application restart required'));
+      throw new Error('MongoDB connection is not recoverable, application restart required');
     }
   } else {
     // /**
     // * Query used to find job to run
-    // * @type {{$or: [*]}}
+    // * @type {{$and: [*]}}
     // */
     const JOB_PROCESS_WHERE_QUERY = {
-      $or: [{
+      $and: [{
         name: jobName,
-        lockedAt: null,
-        nextRunAt: {$lte: this._nextScanAt},
         disabled: {$ne: true}
       }, {
-        name: jobName,
-        lockedAt: {$exists: false},
-        nextRunAt: {$lte: this._nextScanAt},
-        disabled: {$ne: true}
-      }, {
-        name: jobName,
-        lockedAt: {$lte: lockDeadline},
-        disabled: {$ne: true}
+        $or: [{
+          lockedAt: {$eq: null},
+          nextRunAt: {$lte: this._nextScanAt}
+        }, {
+          lockedAt: {$lte: lockDeadline}
+        }]
       }]
     };
 
     /**
-    * Query used to set a job as locked
-    * @type {{$set: {lockedAt: Date}}}
-    */
+     * Query used to set a job as locked
+     * @type {{$set: {lockedAt: Date}}}
+     */
     const JOB_PROCESS_SET_QUERY = {$set: {lockedAt: now}};
 
     /**
-    * Query used to affect what gets returned
-    * @type {{returnOriginal: boolean, sort: object}}
-    */
+     * Query used to affect what gets returned
+     * @type {{returnOriginal: boolean, sort: object}}
+     */
     const JOB_RETURN_QUERY = {returnOriginal: false, sort: this._sort};
 
     // Find ONE and ONLY ONE job and set the 'lockedAt' time so that job begins to be processed
-    this._collection.findOneAndUpdate(JOB_PROCESS_WHERE_QUERY, JOB_PROCESS_SET_QUERY, JOB_RETURN_QUERY, (err, result) => {
-      let job;
-      if (!err && result.value) {
-        debug('found a job available to lock, creating a new job on Agenda with id [%s]', result.value._id);
-        job = createJob(self, result.value);
-      }
-      if (err) {
-        debug('error occurred when running query to find and lock job');
-      }
-      cb(err, job);
-    });
+    const result = await this._collection.findOneAndUpdate(JOB_PROCESS_WHERE_QUERY, JOB_PROCESS_SET_QUERY, JOB_RETURN_QUERY);
+
+    let job;
+    if (result.value) {
+      debug('found a job available to lock, creating a new job on Agenda with id [%s]', result.value._id);
+      job = createJob(self, result.value);
+    }
+
+    return job;
   }
 };

package/lib/agenda/index.js

@@ -1,5 +1,6 @@
 const {EventEmitter} = require('events');
 const humanInterval = require('human-interval');
+const JobProcessingQueue = require('./job-processing-queue');
 
 /**
  * @class Agenda
@@ -22,6 +23,7 @@ const humanInterval = require('human-interval');
  * @property {Array} _jobsToLock
  */
 class Agenda extends EventEmitter {
+  // eslint-disable-next-line default-param-last
   constructor(config = {}, cb) {
     super();
 
@@ -38,10 +40,10 @@ class Agenda extends EventEmitter {
     this._definitions = {};
     this._runningJobs = [];
     this._lockedJobs = [];
-    this._jobQueue = [];
+    this._jobQueue = new JobProcessingQueue();
     this._defaultLockLifetime = config.defaultLockLifetime || 10 * 60 * 1000; // 10 minute default lockLifetime
     this._sort = config.sort || {nextRunAt: 1, priority: -1};
-    this._indices = Object.assign({name: 1}, this._sort, {priority: -1, lockedAt: 1, nextRunAt: 1, disabled: 1});
+    this._indices = {name: 1, ...this._sort, priority: -1, lockedAt: 1, nextRunAt: 1, disabled: 1};
 
     this._isLockingOnTheFly = false;
     this._jobsToLock = [];
@@ -49,6 +51,9 @@ class Agenda extends EventEmitter {
 
     if (config.mongo) {
       this.mongo(config.mongo, config.db ? config.db.collection : undefined, cb);
+      if (config.mongo.s && config.mongo.topology && config.mongo.topology.s) {
+        this._mongoUseUnifiedTopology = Boolean(config.mongo.topology.s.options.useUnifiedTopology);
+      }
     } else if (config.db) {
       this.database(config.db.address, config.db.collection, config.db.options, cb);
     }

package/lib/agenda/job-processing-queue.js

@@ -0,0 +1,81 @@
+/**
+ * @class
+ * @param {Object} args - Job Options
+ * @property {Object} agenda - The Agenda instance
+ * @property {Object} attrs
+ */
+class JobProcessingQueue {
+  constructor() {
+    this._queue = [];
+  }
+
+  get length() {
+    return this._queue.length;
+  }
+}
+
+/**
+ * Pops and returns last queue element (next job to be processed) without checking concurrency.
+ * @returns {Job} Next Job to be processed
+ */
+JobProcessingQueue.prototype.pop = function() {
+  return this._queue.pop();
+};
+
+/**
+ * Inserts job in first queue position
+ * @param {Job} job job to add to queue
+ * @returns {undefined}
+ */
+JobProcessingQueue.prototype.push = function(job) {
+  this._queue.push(job);
+};
+
+/**
+ * Inserts job in queue where it will be order from left to right in decreasing
+ * order of nextRunAt and priority (in case of same nextRunAt), if all values
+ * are even the first jobs to be introduced will have priority
+ * @param {Job} job job to add to queue
+ * @returns {undefined}
+ */
+JobProcessingQueue.prototype.insert = function(job) {
+  const matchIndex = this._queue.findIndex(element => {
+    if (element.attrs.nextRunAt.getTime() <= job.attrs.nextRunAt.getTime()) {
+      if (element.attrs.nextRunAt.getTime() === job.attrs.nextRunAt.getTime()) {
+        if (element.attrs.priority >= job.attrs.priority) {
+          return true;
+        }
+      } else {
+        return true;
+      }
+    }
+
+    return false;
+  });
+
+  if (matchIndex === -1) {
+    this._queue.push(job);
+  } else {
+    this._queue.splice(matchIndex, 0, job);
+  }
+};
+
+/**
+ * Returns (does not pop, element remains in queue) first element (always from the right)
+ * that can be processed (not blocked by concurrency execution)
+ * @param {Object} agendaDefinitions job to add to queue
+ * @returns {Job} Next Job to be processed
+ */
+JobProcessingQueue.prototype.returnNextConcurrencyFreeJob = function(agendaDefinitions) {
+  let next;
+  for (next = this._queue.length - 1; next > 0; next -= 1) {
+    const def = agendaDefinitions[this._queue[next].attrs.name];
+    if (def.concurrency > def.running) {
+      break;
+    }
+  }
+
+  return this._queue[next];
+};
+
+module.exports = JobProcessingQueue;

package/lib/agenda/jobs.js

@@ -8,13 +8,15 @@ const {createJob} = require('../utils');
  * @param {Object} query object for MongoDB
  * @param {Object} sort object for MongoDB
  * @param {Number} limit number of documents to return from MongoDB
+ * @param {Number} number of documents to skip in MongoDB
  * @returns {Promise} resolves when fails or passes
  */
-module.exports = async function(query = {}, sort = {}, limit = 0) {
+module.exports = async function(query = {}, sort = {}, limit = 0, skip = 0) {
   const result = await this._collection
     .find(query)
     .sort(sort)
     .limit(limit)
+    .skip(skip)
     .toArray();
 
   return result.map(job => createJob(this, job));

package/lib/agenda/now.js

@@ -8,16 +8,21 @@ const noCallback = require('../no-callback');
  * @function
  * @param {String} name name of job to schedule
  * @param {Object} data data to pass to job
- * @returns {Job} new job instance created
+ * @returns {Promise} resolves with the new job instance created
  */
 module.exports = async function(name, data) {
-  // eslint-disable-next-line prefer-rest-params
-  noCallback(arguments, 2);
   debug('Agenda.now(%s, [Object])', name);
-  const job = this.create(name, data);
+  try {
+    // eslint-disable-next-line prefer-rest-params
+    noCallback(arguments, 2);
+    const job = this.create(name, data);
 
-  job.schedule(new Date());
-  await job.save();
+    job.schedule(new Date());
+    await job.save();
 
-  return job;
+    return job;
+  } catch (error) {
+    debug('error trying to create a job for this exact moment');
+    throw error;
+  }
 };

package/lib/agenda/save-job.js

@@ -126,12 +126,13 @@ module.exports = async function(job) {
       const result = await this._collection.findOneAndUpdate(query, update, {upsert: true, returnOriginal: false});
       return processDbResult(job, result);
     }
+
     // If all else fails, the job does not exist yet so we just insert it into MongoDB
     debug('using default behavior, inserting new job via insertOne() with props that were set: \n%O', props);
     const result = await this._collection.insertOne(props);
     return processDbResult(job, result);
-  } catch (err) {
+  } catch (error) {
     debug('processDbResult() received an error, job was not updated/created');
-    throw err;
+    throw error;
   }
 };

package/lib/agenda/schedule.js

@@ -40,9 +40,9 @@ module.exports = function(when, names, data) {
       const jobs = await Promise.all(names.map(name => createJob(when, name, data)));
       debug('Agenda.schedule()::createJobs() -> all jobs created successfully');
       return jobs;
-    } catch (err) {
+    } catch (error) {
       debug('Agenda.schedule()::createJobs() -> error creating one or more of the jobs');
-      throw err;
+      throw error;
     }
   };
 

package/lib/agenda/start.js

@@ -14,6 +14,7 @@ module.exports = async function() {
     debug('Agenda.start was already called, ignoring');
     return this._ready;
   }
+
   await this._ready;
   debug('Agenda.start called, creating interval to call processJobs every [%dms]', this._processEvery);
   this._processInterval = setInterval(processJobs.bind(this), this._processEvery);

package/lib/job/compute-next-run-at.js

@@ -15,6 +15,7 @@ module.exports = function() {
   const interval = this.attrs.repeatInterval;
   const timezone = this.attrs.repeatTimezone;
   const {repeatAt} = this.attrs;
+  const previousNextRunAt = this.attrs.nextRunAt || new Date();
   this.attrs.nextRunAt = undefined;
 
   const dateForTimezone = date => {
@@ -22,6 +23,7 @@ module.exports = function() {
     if (timezone !== null) {
       date.tz(timezone);
     }
+
     return date;
   };
 
@@ -36,23 +38,26 @@ module.exports = function() {
     try {
       const cronTime = new CronTime(interval);
       let nextDate = cronTime._getNextDateFrom(lastRun);
-      if (nextDate.valueOf() === lastRun.valueOf()) {
+      if (nextDate.valueOf() === lastRun.valueOf() || nextDate.valueOf() <= previousNextRunAt.valueOf()) {
         // Handle cronTime giving back the same date for the next run time
         nextDate = cronTime._getNextDateFrom(dateForTimezone(new Date(lastRun.valueOf() + 1000)));
       }
+
       this.attrs.nextRunAt = nextDate;
-      debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, this.attrs.nextRunAt.toISOString());
-    } catch (e) {
+      debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, new Date(this.attrs.nextRunAt).toISOString());
+    // Either `xo` linter or Node.js 8 stumble on this line if it isn't just ignored
+    } catch (error) { // eslint-disable-line no-unused-vars
       // Nope, humanInterval then!
       try {
         if (!this.attrs.lastRunAt && humanInterval(interval)) {
           this.attrs.nextRunAt = lastRun.valueOf();
-          debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, this.attrs.nextRunAt.toISOString());
+          debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, new Date(this.attrs.nextRunAt).toISOString());
         } else {
           this.attrs.nextRunAt = lastRun.valueOf() + humanInterval(interval);
-          debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, this.attrs.nextRunAt.toISOString());
+          debug('[%s:%s] nextRunAt set to [%s]', this.attrs.name, this.attrs._id, new Date(this.attrs.nextRunAt).toISOString());
         }
-      } catch (e) {}
+      // Either `xo` linter or Node.js 8 stumble on this line if it isn't just ignored
+      } catch (error) {} // eslint-disable-line no-unused-vars
     } finally {
       if (isNaN(this.attrs.nextRunAt)) {
         this.attrs.nextRunAt = undefined;
@@ -90,5 +95,6 @@ module.exports = function() {
   } else if (repeatAt) {
     computeFromRepeatAt.call(this);
   }
+
   return this;
 };

package/lib/job/fail.js

@@ -12,6 +12,7 @@ module.exports = function(reason) {
   if (reason instanceof Error) {
     reason = reason.message;
   }
+
   this.attrs.failReason = reason;
   this.attrs.failCount = (this.attrs.failCount || 0) + 1;
   const now = new Date();

package/lib/job/index.js

@@ -16,6 +16,7 @@ const parsePriority = priority => {
   if (typeof priority === 'number' || priority instanceof Number) {
     return priority;
   }
+
   return priorityMap[priority];
 };
 

package/lib/job/is-running.js

@@ -13,11 +13,14 @@ module.exports = function() {
   if (!this.attrs.lastRunAt) {
     return false;
   }
+
   if (!this.attrs.lastFinishedAt) {
     return true;
   }
+
   if (this.attrs.lockedAt && this.attrs.lastRunAt.getTime() > this.attrs.lastFinishedAt.getTime()) {
     return true;
   }
+
   return false;
 };

package/lib/job/priority.js

@@ -18,6 +18,7 @@ const parsePriority = priority => {
   if (typeof priority === 'number' || priority instanceof Number) {
     return priority;
   }
+
   return priorityMap[priority];
 };
 

package/lib/job/repeat-every.js

@@ -13,9 +13,13 @@ module.exports = function(interval, options) {
   this.attrs.repeatInterval = interval;
   this.attrs.repeatTimezone = options.timezone ? options.timezone : null;
   if (options.skipImmediate) {
-    this.attrs.lastRunAt = new Date();
+    // Set the lastRunAt time to the nextRunAt so that the new nextRunAt will be computed in reference to the current value.
+    this.attrs.lastRunAt = this.attrs.nextRunAt || new Date();
     this.computeNextRunAt();
     this.attrs.lastRunAt = undefined;
+  } else {
+    this.computeNextRunAt();
   }
+
   return this;
 };

package/lib/job/run.js

@@ -12,6 +12,8 @@ module.exports = function() {
   const {agenda} = self;
   const definition = agenda._definitions[self.attrs.name];
 
+  // @TODO: this lint issue should be looked into: https://eslint.org/docs/rules/no-async-promise-executor
+  // eslint-disable-next-line no-async-promise-executor
   return new Promise(async(resolve, reject) => {
     self.attrs.lastRunAt = new Date();
     debug('[%s:%s] setting lastRunAt to: %s', self.attrs.name, self.attrs._id, self.attrs.lastRunAt.toISOString());
@@ -24,6 +26,7 @@ module.exports = function() {
       if (finished) {
         return;
       }
+
       finished = true;
 
       if (err) {
@@ -34,9 +37,9 @@ module.exports = function() {
 
       self.attrs.lockedAt = null;
 
-      await self.save().catch(err => {
+      await self.save().catch(error => {
         debug('[%s:%s] failed to be saved to MongoDB', self.attrs.name, self.attrs._id);
-        reject(err);
+        reject(error);
       });
       debug('[%s:%s] was saved successfully to MongoDB', self.attrs.name, self.attrs._id);
 
@@ -49,6 +52,7 @@ module.exports = function() {
         agenda.emit('success:' + self.attrs.name, self);
         debug('[%s:%s] has succeeded', self.attrs.name, self.attrs._id);
       }
+
       agenda.emit('complete', self);
       agenda.emit('complete:' + self.attrs.name, self);
       debug('[%s:%s] job finished at [%s] and was unlocked', self.attrs.name, self.attrs._id, self.attrs.lastFinishedAt);
@@ -65,6 +69,7 @@ module.exports = function() {
         debug('[%s:%s] has no definition, can not run', self.attrs.name, self.attrs._id);
         throw new Error('Undefined job');
       }
+
       if (definition.fn.length === 2) {
         debug('[%s:%s] process function being called', self.attrs.name, self.attrs._id);
         await definition.fn(self, jobCallback);
@@ -73,9 +78,9 @@ module.exports = function() {
         await definition.fn(self);
         await jobCallback();
       }
-    } catch (err) {
+    } catch (error) {
       debug('[%s:%s] unknown error occurred', self.attrs.name, self.attrs._id);
-      await jobCallback(err);
+      await jobCallback(error);
     }
   });
 };

package/lib/utils/process-jobs.js

@@ -52,9 +52,11 @@ module.exports = function(extraJob) {
     if (self._lockLimit && self._lockLimit <= self._lockedJobs.length) {
       shouldLock = false;
     }
+
     if (jobDefinition.lockLimit && jobDefinition.lockLimit <= jobDefinition.locked) {
       shouldLock = false;
     }
+
     debug('job [%s] lock status: shouldLock = %s', name, shouldLock);
     return shouldLock;
   }
@@ -65,46 +67,13 @@ module.exports = function(extraJob) {
    * @param {boolean} inFront puts the job in front of queue if true
    * @returns {undefined}
    */
-  function enqueueJobs(jobs, inFront) {
+  function enqueueJobs(jobs) {
     if (!Array.isArray(jobs)) {
       jobs = [jobs];
     }
 
     jobs.forEach(job => {
-      let jobIndex;
-      let start;
-      let loopCondition;
-      let endCondition;
-      let inc;
-
-      if (inFront) {
-        start = jobQueue.length ? jobQueue.length - 1 : 0;
-        inc = -1;
-        loopCondition = function() {
-          return jobIndex >= 0;
-        };
-        endCondition = function(queuedJob) {
-          return !queuedJob || queuedJob.attrs.priority < job.attrs.priority;
-        };
-      } else {
-        start = 0;
-        inc = 1;
-        loopCondition = function() {
-          return jobIndex < jobQueue.length;
-        };
-        endCondition = function(queuedJob) {
-          return queuedJob.attrs.priority >= job.attrs.priority;
-        };
-      }
-
-      for (jobIndex = start; loopCondition(); jobIndex += inc) {
-        if (endCondition(jobQueue[jobIndex])) {
-          break;
-        }
-      }
-
-      // Insert the job to the queue at its prioritized position for processing
-      jobQueue.splice(jobIndex, 0, job);
+      jobQueue.insert(job);
     });
   }
 
@@ -114,7 +83,7 @@ module.exports = function(extraJob) {
    * We do this because sometimes jobs are scheduled but will be run before the next process time
    * @returns {undefined}
    */
-  function lockOnTheFly() {
+  async function lockOnTheFly() {
     // Already running this? Return
     if (self._isLockingOnTheFly) {
       debug('lockOnTheFly() already running, returning');
@@ -158,26 +127,22 @@ module.exports = function(extraJob) {
     const options = {returnOriginal: false};
 
     // Lock the job in MongoDB!
-    self._collection.findOneAndUpdate(criteria, update, options, (err, resp) => {
-      if (err) {
-        throw err;
-      }
-      // Did the "job" get locked? Create a job object and run
-      if (resp.value) {
-        const job = createJob(self, resp.value);
-        debug('found job [%s] that can be locked on the fly', job.attrs.name);
-        self._lockedJobs.push(job);
-        definitions[job.attrs.name].locked++;
-        enqueueJobs(job);
-        jobProcessing();
-      }
+    const resp = await self._collection.findOneAndUpdate(criteria, update, options);
+
+    if (resp.value) {
+      const job = createJob(self, resp.value);
+      debug('found job [%s] that can be locked on the fly', job.attrs.name);
+      self._lockedJobs.push(job);
+      definitions[job.attrs.name].locked++;
+      enqueueJobs(job);
+      jobProcessing();
+    }
 
-      // Mark lock on fly is done for now
-      self._isLockingOnTheFly = false;
+    // Mark lock on fly is done for now
+    self._isLockingOnTheFly = false;
 
-      // Re-run in case anything is in the queue
-      lockOnTheFly();
-    });
+    // Re-run in case anything is in the queue
+    await lockOnTheFly();
   }
 
   /**
@@ -185,7 +150,7 @@ module.exports = function(extraJob) {
    * @param {String} name fill a queue with specific job name
    * @returns {undefined}
    */
-  function jobQueueFilling(name) {
+  async function jobQueueFilling(name) {
     // Don't lock because of a limit we have set (lockLimit, etc)
     if (!shouldLock(name)) {
       debug('lock limit reached in queue filling for [%s]', name);
@@ -197,12 +162,8 @@ module.exports = function(extraJob) {
     self._nextScanAt = new Date(now.valueOf() + self._processEvery);
 
     // For this job name, find the next job to run and lock it!
-    self._findAndLockNextJob(name, definitions[name], (err, job) => {
-      if (err) {
-        debug('[%s] job lock failed while filling queue', name);
-        throw err;
-      }
-
+    try {
+      const job = await self._findAndLockNextJob(name, definitions[name]);
       // Still have the job?
       // 1. Add it to lock list
       // 2. Add count of locked jobs
@@ -213,10 +174,12 @@ module.exports = function(extraJob) {
         self._lockedJobs.push(job);
         definitions[job.attrs.name].locked++;
         enqueueJobs(job);
-        jobQueueFilling(name);
+        await jobQueueFilling(name);
         jobProcessing();
       }
-    });
+    } catch (error) {
+      debug('[%s] job lock failed while filling queue', name, error);
+    }
   }
 
   /**
@@ -233,29 +196,19 @@ module.exports = function(extraJob) {
     const now = new Date();
 
     // Get the next job that is not blocked by concurrency
-    let next;
-    for (next = jobQueue.length - 1; next > 0; next -= 1) {
-      const def = definitions[jobQueue[next].attrs.name];
-      if (def.concurrency > def.running) {
-        break;
-      }
-    }
-
-    // We now have the job we are going to process and its definition
-    const job = jobQueue.splice(next, 1)[0];
-    const jobDefinition = definitions[job.attrs.name];
+    const job = jobQueue.returnNextConcurrencyFreeJob(definitions);
 
     debug('[%s:%s] about to process job', job.attrs.name, job.attrs._id);
 
     // If the 'nextRunAt' time is older than the current time, run the job
     // Otherwise, setTimeout that gets called at the time of 'nextRunAt'
-    if (job.attrs.nextRunAt < now) {
+    if (job.attrs.nextRunAt <= now) {
       debug('[%s:%s] nextRunAt is in the past, run the job immediately', job.attrs.name, job.attrs._id);
       runOrRetry();
     } else {
       const runIn = job.attrs.nextRunAt - now;
       debug('[%s:%s] nextRunAt is in the future, calling setTimeout(%d)', job.attrs.name, job.attrs._id, runIn);
-      setTimeout(runOrRetry, runIn);
+      setTimeout(jobProcessing, runIn);
     }
 
     /**
@@ -264,6 +217,8 @@ module.exports = function(extraJob) {
      */
     async function runOrRetry() {
       if (self._processInterval) {
+        const job = jobQueue.pop();
+        const jobDefinition = definitions[job.attrs.name];
         if (jobDefinition.concurrency > jobDefinition.running && self._runningJobs.length < self._maxConcurrency) {
           // Get the deadline of when the job is not supposed to go past for locking
           const lockDeadline = new Date(Date.now() - jobDefinition.lockLifetime);
@@ -287,15 +242,12 @@ module.exports = function(extraJob) {
           debug('[%s:%s] processing job', job.attrs.name, job.attrs._id);
 
           job.run()
-            .catch(err => [err, job])
+            .catch(error => [error, job])
             .then(job => processJobResult(...Array.isArray(job) ? job : [null, job])); // eslint-disable-line promise/prefer-await-to-then
-
-          // Re-run the loop to check for more jobs to process (locally)
-          jobProcessing();
         } else {
           // Run the job immediately by putting it on the top of the queue
           debug('[%s:%s] concurrency preventing immediate run, pushing job to top of queue', job.attrs.name, job.attrs._id);
-          enqueueJobs(job, true);
+          enqueueJobs(job);
         }
       }
     }
@@ -311,10 +263,11 @@ module.exports = function(extraJob) {
     if (err) {
       return job.agenda.emit('error', err);
     }
+
     const {name} = job.attrs;
 
     // Job isn't in running jobs so throw an error
-    if (self._runningJobs.indexOf(job) === -1) {
+    if (!self._runningJobs.includes(job)) {
       debug('[%s] callback was called, job must have been marked as complete already', job.attrs._id);
       throw new Error('callback already called - job ' + name + ' already marked complete');
     }

package/package.json

@@ -1,13 +1,13 @@
 {
   "name": "agenda",
-  "version": "2.1.0",
+  "version": "3.1.0",
   "description": "Light weight job scheduler for Node.js",
   "main": "index.js",
   "files": [
     "lib"
   ],
   "engines": {
-    "node": ">=8.0.0"
+    "node": ">=10.0.0"
   },
   "scripts": {
     "test": "npm run lint && npm run mocha",
@@ -16,11 +16,13 @@
     "mocha-debug": "DEBUG=agenda:**,-agenda:internal:** mocha --reporter spec --timeout 8000 -b",
     "mocha-debug-internal": "DEBUG=agenda:internal:** mocha --reporter spec --timeout 8000 -b",
     "mocha-debug-all": "DEBUG=agenda:** mocha --reporter spec --timeout 8000 -b",
+    "docs": "jsdoc --configure .jsdoc.json --verbose"
+  },
+  "config": {
     "blanket": {
       "pattern": "lib",
       "data-cover-never": "node_modules"
-    },
-    "docs": "jsdoc --configure .jsdoc.json --verbose"
+    }
   },
   "repository": {
     "type": "git",
@@ -40,29 +42,38 @@
     "url": "https://github.com/agenda/agenda/issues"
   },
   "dependencies": {
-    "cron": "~1.7.2",
+    "cron": "~1.8.0",
     "date.js": "~0.3.3",
     "debug": "~4.1.1",
-    "human-interval": "~0.1.6",
-    "moment-timezone": "~0.5.26",
-    "mongodb": "~3.2.7"
+    "human-interval": "~1.0.0",
+    "moment-timezone": "~0.5.27",
+    "mongodb": "~3.5.0"
   },
   "devDependencies": {
     "blanket": "1.2.3",
-    "coveralls": "3.0.6",
+    "coveralls": "3.0.11",
     "delay": "4.3.0",
-    "eslint": "4.9.0",
+    "eslint": "6.8.0",
+    "eslint-config-xo": "0.27.2",
+    "eslint-plugin-ava": "10.2.0",
+    "eslint-plugin-eslint-comments": "3.1.2",
+    "eslint-plugin-import": "2.20.2",
+    "eslint-plugin-node": "11.1.0",
+    "eslint-plugin-unicorn": "16.1.1",
     "expect.js": "0.3.1",
-    "jsdoc": "3.6.3",
+    "jsdoc": "3.6.4",
     "jsdoc-template": "https://github.com/braintree/jsdoc-template",
-    "mocha": "6.1.4",
+    "mocha": "7.1.1",
     "mocha-lcov-reporter": "1.3.0",
     "q": "1.5.1",
-    "sinon": "7.3.2",
-    "xo": "0.21.1"
+    "sinon": "9.0.1",
+    "xo": "0.27.2"
   },
   "xo": {
     "space": 2,
+    "ignores": [
+      "docs/**"
+    ],
     "rules": {
       "space-before-function-paren": [
         "error",