pixl-server-storage 3.2.9 → 4.0.1

package/README.md

@@ -1,12 +1,12 @@
 # Overview
 
-This module is a component for use in [pixl-server](https://www.github.com/jhuckaby/pixl-server).  It implements a simple key/value storage system that can use multiple back-ends, such as [Amazon S3](https://aws.amazon.com/s3/), [Couchbase](http://www.couchbase.com/nosql-databases/couchbase-server), [Redis](https://redis.io/), or a local filesystem.  It introduces the concept of a "chunked linked list", which supports extremely fast push, pop, shift, unshift, and random reads/writes.  Also provided is a fast hash table implementation with key iteration, a transaction system, and an indexing and search system.
+This module is a component for use in [pixl-server](https://www.github.com/jhuckaby/pixl-server).  It implements a simple key/value storage system that can use multiple back-ends, such as [Amazon S3](https://aws.amazon.com/s3/), [Redis](https://redis.io/), or a local filesystem.  It introduces the concept of a "chunked linked list", which supports extremely fast push, pop, shift, unshift, and random reads/writes.  Also provided is a fast hash table implementation with key iteration, a transaction system, and an indexing and search system.
 
 ## Features at a Glance
 
 * Uses very little memory in most cases.
 * Store JSON or binary (raw) data records.
-* Supports multiple back-ends including Amazon S3, Couchbase, Redis, and local filesystem.
+* Supports multiple back-ends including Amazon S3, Redis, and local filesystem.
 * Linked lists with very fast push, pop, shift, unshift, and random reads/writes.
 * Hash tables with key iterators, and very fast reads / writes.
 * Advisory locking system with shared and exclusive locks.
@@ -55,7 +55,6 @@ Here is the table of contents for this current document:
 		+ [S3 Key Template](#s3-key-template)
 		+ [S3 Cache](#s3-cache)
 		+ [S3 Compatible Services](#s3-compatible-services)
-	* [Couchbase](#couchbase)
 	* [Redis](#redis)
 		+ [RedisCluster](#rediscluster)
 	* [SQLite](#sqlite)
@@ -267,7 +266,7 @@ The `debug` property is only used when using [Standalone Mode](#standalone-mode)
 
 # Engines
 
-The storage system can be backed by a number of different "engines", which actually perform the reads and writes.  A simple local filesystem implementation is included, as well as modules for Amazon S3, Couchbase and Redis.  Each one requires a bit of extra configuration.
+The storage system can be backed by a number of different "engines", which actually perform the reads and writes.  A simple local filesystem implementation is included, as well as modules for Amazon S3 and Redis.  Each one requires a bit of extra configuration.
 
 ## Local Filesystem
 
@@ -573,44 +572,6 @@ Here is a complete example with the new parameters added:
 }
 ```
 
-## Couchbase
-
-Please note that as of this writing (April 2022), pixl-server-storage only supports Couchbase Client v2, so you need to force install version `2.6.12` (see instructions below).  Work is underway to support the v3 API, which has many breaking changes.
-
-If you want to use [Couchbase](http://www.couchbase.com/nosql-databases/couchbase-server) as a backing store, here is how to do so.  First, you need to manually install the [couchbase](https://www.npmjs.com/package/couchbase) module into your app, and it **must be v2**:
-
-```sh
-npm install --save couchbase@2.6.12
-```
-
-Then configure your storage thusly:
-
-```json
-{
-	"engine": "Couchbase",
-	"Couchbase": {
-		"connectString": "couchbase://127.0.0.1",
-		"bucket": "default",
-		"username": "",
-		"password": "",
-		"serialize": false,
-		"keyPrefix": ""
-	}
-}
-```
-
-Set the `connectString` for your own Couchbase server setup.  You can embed a username and password into the string if they are required to connect (this is different from the bucket password), and use `couchbases://` for SSL, if desired.
-
-The `bucket` property should be set to the bucket name.  If you don't know this then `default` is probably correct.  The `password` property is the bucket password, and may or may not be required, depending on your Couchbase server setup.
-
-The `serialize` property, when set to `true`, will cause all object values to be serialized to JSON before storing, and they will also be parsed from JSON when fetching.  When set to `false` (the default), this is left up to Couchbase to handle.
-
-The optional `keyPrefix` property works similarly to the [S3 Key Prefix](#s3-key-prefix) feature.  It allows you to prefix all the Couchbase keys with a common string, to separate your application's data in a shared bucket situation.
-
-The optional `keyTemplate` property works similarly to the [S3 Key Template](#s3-key-template) feature.  It allows you to specify an exact layout of MD5 hash characters, which can be prefixed, mixed in with or postfixed after the key.
-
-Note that for Couchbase Server v5.0+ (Couchbase Node SDK 2.5+), you will have to supply both a `username` and `password` for a valid user created in the Couchbase UI.  Prior to v5+ you could omit the `username` and only specify a `password`, or no password at all if your bucket has no authentication.
-
 ## Redis
 
 If you want to use [Redis](https://redis.io/) as a backing store, here is how to do so.  First, you need to manually install the [ioredis](https://www.npmjs.com/package/ioredis) module into your app:
@@ -681,10 +642,10 @@ The optional `keyTemplate` property works similarly to the [S3 Key Template](#s3
 
 ## SQLite
 
-If you want to use [SQLite](https://sqlite.com/) as a backing store, here is how to do so.  First, you need to manually install the [sqlite3](https://www.npmjs.com/package/sqlite3) module into your app:
+If you want to use [SQLite](https://sqlite.com/) as a backing store, here is how to do so.  First, you need to manually install the [better-sqlite3](https://www.npmjs.com/package/better-sqlite3) module into your app:
 
 ```sh
-npm install --save sqlite3
+npm install --save better-sqlite3
 ```
 
 Then configure your storage thusly:
@@ -852,8 +813,6 @@ storage.head( 'test1', function(err, data) {
 } );
 ```
 
-Note that the [Couchbase](#couchbase) engine does not support `head`, but the [Amazon S3](#amazon-s3) and [Local Filesystem](#local-filesystem) engines both do.
-
 You can fetch multiple records at once by calling `getMulti()` and passing in array of keys.  Example:
 
 ```js
@@ -951,7 +910,7 @@ storage.getStream( 'test1.gif', function(err, readStream) {
 } );
 ```
 
-Please note that not all the storage engines support streams natively, so the content may actually be loaded into RAM in the background.  Namely, as of this writing, the Couchbase and Redis APIs do not support streams, so they are currently simulated in those engines.  Streams *are* supported natively in both the Filesystem and Amazon S3 engines.
+Please note that not all the storage engines support streams natively, so the content may actually be loaded into RAM in the background.  Namely, as of this writing, the Redis APIs do not support streams, so they are currently simulated in those engines.  Streams *are* supported natively in both the Filesystem and Amazon S3 engines.
 
 # Expiring Data
 
@@ -1322,7 +1281,7 @@ If you install the [pixl-unit](https://www.github.com/jhuckaby/pixl-unit) module
 pixl-unit test/test.js --verbose
 ```
 
-This also allows you to specify an alternate test configuration file via the `--configFile` option.  Using this you can load your own test config, which may use a different engine (e.g. S3, Couchbase, etc.):
+This also allows you to specify an alternate test configuration file via the `--configFile` option.  Using this you can load your own test config, which may use a different engine (e.g. S3, etc.):
 
 ```sh
 pixl-unit test/test.js --configFile /path/to/my/config.json
@@ -1332,7 +1291,7 @@ pixl-unit test/test.js --configFile /path/to/my/config.json
 
 **The MIT License (MIT)**
 
-Copyright (c) 2015 - 2018 Joseph Huckaby.
+Copyright (c) 2015 - 2026 Joseph Huckaby.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/docs/Transactions.md

@@ -54,9 +54,9 @@ Choose your `trans_dir` carefully.  This directory is used to store temporary fi
 |----------------|----------------|
 | Local Filesystem | Make sure your `trans_dir` is on the *same mount* as your local storage data, to ensure speed and safety.  This is the default. |
 | NFS Filesystem | Make sure your `trans_dir` is **NOT** on the NFS mount, but instead points to a local fast SSD mount. |
-| S3 or Couchbase | Make sure your `trans_dir` points to a local fast SSD mount. |
+| S3 | Make sure your `trans_dir` points to a local fast SSD mount. |
 
-So basically, if you plan to use the [Filesystem](../README.md#local-filesystem) engine with a local (i.e. non-network) disk, you don't need to make any adjustments.  The default setting for `trans_dir` is a subdirectory just inside your `base_dir` engine property.  But if you are going with a networked (i.e. NFS) filesystem, or you're going to use [S3](../README.md#amazon-s3) or [Couchbase](../README.md#couchbase), then you should set `trans_dir` to a local filesystem path on a fast disk (ideally SSD).
+So basically, if you plan to use the [Filesystem](../README.md#local-filesystem) engine with a local (i.e. non-network) disk, you don't need to make any adjustments.  The default setting for `trans_dir` is a subdirectory just inside your `base_dir` engine property.  But if you are going with a networked (i.e. NFS) filesystem, or you're going to use [S3](../README.md#amazon-s3), then you should set `trans_dir` to a local filesystem path on a fast disk (ideally SSD).
 
 **Pro-Tip:** When using transactions with the Amazon S3 engine, consider setting the `maxAttempts` property in your `S3` configuration object, so the AWS client library makes multiple attempts before failing an operation.  Network hiccups can happen.
 
@@ -180,7 +180,7 @@ storage.begin( 'some_path', function(err, trans) {
 
 It should be noted that if you receive an error from a [commit()](API.md#commit) call, it is *vital* that you call [abort()](API.md#abort) to undo whatever operations may have already executed.  A commit error is typically very bad, and your storage system will be in an unknown state.  Only by calling [abort()](API.md#abort) can you restore it to before the transaction started.
 
-If the [abort()](API.md#abort) also fails, then the database raises a fatal error and exits immediately.  See [Emergency Shutdown](#emergency-shutdown) below for details about what this means, and [Recovery](#recovery) for how to get back up and running.  Examples of fatal errors include your disk running completely out of space, or a major network failure when using NFS, S3 or Couchbase.
+If the [abort()](API.md#abort) also fails, then the database raises a fatal error and exits immediately.  See [Emergency Shutdown](#emergency-shutdown) below for details about what this means, and [Recovery](#recovery) for how to get back up and running.  Examples of fatal errors include your disk running completely out of space, or a major network failure when using NFS, or S3.
 
 ## Automatic API Wrappers
 
@@ -215,7 +215,7 @@ You can still use these methods inside of your own transaction, and they will "j
 
 ## Emergency Shutdown
 
-If a fatal storage error is encountered in the middle of an abort (rollback) operation, the database immediately shuts itself down.  This is because your data will be in an undefined state, stuck in the middle of partial transaction.  This should be a very rare event, only occurring when the underlying storage completely fails to write records.  Examples include a disk running out of space (local or NFS filesystem), or a hard network failure with S3 or Couchbase.  When this happens, the Node.js process will exit (by default), and will need to be recovered (see [Recovery](#recovery) below).
+If a fatal storage error is encountered in the middle of an abort (rollback) operation, the database immediately shuts itself down.  This is because your data will be in an undefined state, stuck in the middle of partial transaction.  This should be a very rare event, only occurring when the underlying storage completely fails to write records.  Examples include a disk running out of space (local or NFS filesystem), or a hard network failure with S3.  When this happens, the Node.js process will exit (by default), and will need to be recovered (see [Recovery](#recovery) below).
 
 Upon fatal error, your application can hook the event and provide its own emergency shutdown procedure.  Simply add an event listener on the storage object for the `fatal` event.  It will be passed an `Error` object describing the actual error that occurred.  It is then up to your code to call `process.exit()`.  Example:
 
@@ -271,7 +271,7 @@ The transaction system is implemented by temporarily "branching" the storage sys
 
 ### Temporary Directory
 
-A local temporary directory on disk is always used for transactions, regardless of the storage engine.  Even if you are using S3 or Couchbase for primary storage, transactions still use temp files on disk before and during commit.  This is to insure both speed and data safety.  You can control where the temporary files live by setting the `trans_dir` configuration property.
+A local temporary directory on disk is always used for transactions, regardless of the storage engine.  Even if you are using S3 for primary storage, transactions still use temp files on disk before and during commit.  This is to insure both speed and data safety.  You can control where the temporary files live by setting the `trans_dir` configuration property.
 
 Inside the temp directory, two subdirectories are created: `data` and `logs`.  The `data` directory is used to hold all modified records during a transaction (before commit).  Each is named using a unique transaction ID (see below) and a hash of the original key.  This ensures no files will collide with each other, even with many concurrent transactions.  The `logs` directory is for rollback logs.  When a transaction is committed, the original state of the mutated records is written to the log, so it can be applied in reverse during a recovery event.
 
@@ -409,7 +409,7 @@ So here is the commit process, step by step:
 
 The idea here is that no matter where a crash or sudden power loss occurs during the commit process, a full rollback can always be achieved.  The actual changes on the main storage system are only made once the rollback log is fully written and flushed to disk, and the log itself is only deleted when the record changes are also made (and flushed, where possible).
 
-This is not a 100% guarantee of data durability, as corruption can happen during a crash or power loss.  The rollback log may only be partially written, or corrupted in some way where a replay is not possible.  This can happen because fsync calls are not guaranteed on all operating systems, filesystems or disk media (e.g. certain SSDs), and really all bets are off if you are using S3 or Couchbase for primary storage.
+This is not a 100% guarantee of data durability, as corruption can happen during a crash or power loss.  The rollback log may only be partially written, or corrupted in some way where a replay is not possible.  This can happen because fsync calls are not guaranteed on all operating systems, filesystems or disk media (e.g. certain SSDs), and really all bets are off if you are using S3 for primary storage.
 
 Basically it's all just a big crap shoot, but we're trying to cover as many error cases as possible.
 

package/engines/SQLite.js

@@ -1,16 +1,15 @@
 // SQLite Storage Plugin
-// Copyright (c) 2023 Joseph Huckaby
+// Copyright (c) 2023 - 2026 Joseph Huckaby
 // Released under the MIT License
 
-// Requires the 'sqlite3' module from npm
-// npm install --save sqlite3
+// Requires the 'better-sqlite3' module
 
 const fs = require('fs');
 const Path = require('path');
 const zlib = require('zlib');
 const Class = require("pixl-class");
 const Component = require("pixl-server/component");
-const SQLite3 = require('sqlite3');
+const BetterSqlite3 = require('better-sqlite3');
 const Tools = require("pixl-tools");
 const Perf = require("pixl-perf");
 const Cache = require("pixl-cache");
@@ -81,36 +80,34 @@ module.exports = Class.create({
 		}
 		
 		var db_file = Path.join( this.baseDir, sql_config.filename );
-		this.logDebug(3, "Opening database file: " + db_file);
-		
-		async.series([
-			function(callback) {
-				self.db = new SQLite3.Database( db_file, callback );
-			},
-			function(callback) {
-				// optionally set pragmas on the db
-				if (!sql_config.pragmas) return process.nextTick(callback);
-				async.eachSeries( Object.keys(sql_config.pragmas),
-					function(key, callback) {
-						var value = sql_config.pragmas[key];
-						self.db.run(`PRAGMA ${key} = ${value};`, callback);
-					},
-					callback
-				); // eachSeries
-			},
-			function(callback) {
-				// create our table if necessary
-				self.db.run( 'CREATE TABLE IF NOT EXISTS items( key TEXT PRIMARY KEY, value BLOB, modified INTEGER )', callback );
-			},
-		], 
-		function(err) {
-			if (err) {
-				self.logError('sqlite', "FATAL ERROR: Database setup failed: " + err);
-				return callback(err);
+		this.logDebug(3, "Opening database file: " + db_file, sql_config.pragmas || {});
+		
+		try {
+			// open sync connection
+			this.db = new BetterSqlite3(db_file);
+			
+			// optionally set pragmas on the db
+			if (sql_config.pragmas) {
+				Object.keys(sql_config.pragmas).forEach(function(key) {
+					var value = sql_config.pragmas[key];
+					// value may be string (e.g. WAL) or number
+					self.db.pragma(`${key} = ${value}`);
+				});
 			}
+			
+			// create our table if necessary
+			this.db.prepare('CREATE TABLE IF NOT EXISTS items( key TEXT PRIMARY KEY, value BLOB, modified INTEGER )').run();
+		}
+		catch (err) {
+			this.logError('sqlite', "FATAL ERROR: Database setup failed: " + err);
+			return callback(err);
+		}
+		
+		// behave asynchronously for API compatibility
+		process.nextTick(function() {
 			self.logDebug(3, "Setup complete");
 			callback();
-		}); // async.series
+		});
 	},
 	
 	prepKey: function(key) {
@@ -148,20 +145,23 @@ module.exports = Class.create({
 			value = Buffer.from( JSON.stringify( value ) );
 		}
 		
-		this.db.run( this.commands.put, { $key: key, $value: value, $now: now }, function(err) {
-			if (err) {
+		process.nextTick(function() {
+			try {
+				self.db.prepare(self.commands.put).run({ key: key, value: value, now: now });
+			}
+			catch (err) {
 				err.message = "Failed to store object: " + key + ": " + err;
 				self.logError('sqlite', '' + err);
+				if (callback) callback(err);
 			}
-			else self.logDebug(9, "Store complete: " + key);
 			
+			self.logDebug(9, "Store complete: " + key);
 			// possibly cache in LRU
 			if (self.cache && !is_binary) {
 				self.cache.set( key, value, { date: Tools.timeNow(true) } );
 			}
-			
-			if (callback) callback(err);
-		} ); // put
+			if (callback) callback(null);
+		});
 	},
 	
 	putStream: function(key, inp, callback) {
@@ -200,24 +200,24 @@ module.exports = Class.create({
 			return;
 		} // cache
 		
-		this.db.get( this.commands.head, { $key: key }, function(err, row) {
-			if (err) {
-				// an actual error
+		process.nextTick(function() {
+			var row = null;
+			try {
+				row = self.db.prepare(self.commands.head).get({ key: key });
+			}
+			catch (err) {
 				err.message = "Failed to head key: " + key + ": " + err;
 				self.logError('sqlite', '' + err);
 				callback(err);
 			}
-			else if (!row) {
-				// record not found
-				// always use "NoSuchKey" in error code
+			
+			if (!row) {
 				var err = new Error("Failed to head key: " + key + ": Not found");
 				err.code = "NoSuchKey";
-				callback( err, null );
-			}
-			else {
-				callback( null, { mod: row.modified, len: row['length(value)'] } );
+				return callback(err, null);
 			}
-		}); // head
+			callback(null, { mod: row.modified, len: row['length(value)'] });
+		});
 	},
 	
 	get: function(key, callback) {
@@ -247,44 +247,38 @@ module.exports = Class.create({
 		
 		this.logDebug(9, "Fetching SQLite Object: " + key);
 		
-		this.db.get( this.commands.get, { $key: key }, function(err, row) {
-			if (err) {
-				// an actual error
+		process.nextTick(function() {
+			var row = null;
+			try {
+				row = self.db.prepare(self.commands.get).get({ key: key });
+			}
+			catch (err) {
 				err.message = "Failed to fetch key: " + key + ": " + err;
 				self.logError('sqlite', '' + err);
 				callback(err);
 			}
-			else if (!row) {
-				// record not found
-				// always use "NoSuchKey" in error code
+			
+			if (!row) {
 				var err = new Error("Failed to fetch key: " + key + ": Not found");
 				err.code = "NoSuchKey";
-				callback( err, null );
+				return callback(err, null);
 			}
-			else {
-				// success
-				if (is_binary) {
-					self.logDebug(9, "Binary fetch complete: " + key, '' + row.value.length + ' bytes');
-					callback( null, row.value );
-				}
-				else {
-					// possibly cache in LRU
-					if (self.cache) {
-						self.cache.set( key, row.value, { date: Tools.timeNow(true) } );
-					}
-					
-					var json = null;
-					try { json = JSON.parse( row.value.toString() ); }
-					catch (err) {
-						self.logError('sqlite', "Failed to parse JSON record: " + key + ": " + err);
-						callback( err, null );
-						return;
-					}
-					self.logDebug(9, "JSON fetch complete: " + key, self.debugLevel(10) ? json : null);
-					callback( null, json );
-				}
+			if (is_binary) {
+				self.logDebug(9, "Binary fetch complete: " + key, '' + row.value.length + ' bytes');
+				return callback(null, row.value);
+			}
+			if (self.cache) {
+				self.cache.set( key, row.value, { date: Tools.timeNow(true) } );
+			}
+			var json = null;
+			try { json = JSON.parse( row.value.toString() ); }
+			catch (err) {
+				self.logError('sqlite', "Failed to parse JSON record: " + key + ": " + err);
+				return callback(err, null);
 			}
-		}); // get
+			self.logDebug(9, "JSON fetch complete: " + key, self.debugLevel(10) ? json : null);
+			callback(null, json);
+		});
 	},
 	
 	getBuffer: function(key, callback) {
@@ -294,26 +288,25 @@ module.exports = Class.create({
 		
 		this.logDebug(9, "Fetching SQLite Object: " + key);
 		
-		this.db.get( this.commands.get, { $key: key }, function(err, row) {
-			if (err) {
-				// an actual error
+		process.nextTick(function() {
+			var row = null;
+			try {
+				row = self.db.prepare(self.commands.get).get({ key: key });
+			}
+			catch (err) {
 				err.message = "Failed to fetch key: " + key + ": " + err;
 				self.logError('sqlite', '' + err);
 				callback(err);
 			}
-			else if (!row) {
-				// record not found
-				// always use "NoSuchKey" in error code
+			
+			if (!row) {
 				var err = new Error("Failed to fetch key: " + key + ": Not found");
 				err.code = "NoSuchKey";
-				callback( err, null );
-			}
-			else {
-				// success
-				self.logDebug(9, "Binary fetch complete: " + key, '' + row.value.length + ' bytes');
-				callback( null, row.value );
+				return callback(err, null);
 			}
-		}); // get
+			self.logDebug(9, "Binary fetch complete: " + key, '' + row.value.length + ' bytes');
+			callback(null, row.value);
+		});
 	},
 	
 	getStream: function(key, callback) {
@@ -390,25 +383,27 @@ module.exports = Class.create({
 		
 		this.logDebug(9, "Deleting SQLite Object: " + key);
 		
-		this.db.run( this.commands.delete, { $key: key }, function(err) {
-			// In sqlite3 callbacks `this` is special, and contains `changes`
-			if (!err && !this.changes) {
-				err = new Error("Failed to delete object: " + key + ": Not found");
-				err.code = "NoSuchKey";
+		process.nextTick(function() {
+			var info = null;
+			try {
+				info = self.db.prepare(self.commands.delete).run({ key: key });
 			}
-			if (err) {
+			catch (err) {
 				self.logError('sqlite', "Failed to delete object: " + key + ": " + err);
+				callback(err);
+			}
+			
+			if (!info.changes) {
+				var err = new Error("Failed to delete object: " + key + ": Not found");
+				err.code = "NoSuchKey";
 				return callback(err);
 			}
 			self.logDebug(9, "Delete complete: " + key);
-			
-			// possibly delete from LRU cache as well
 			if (self.cache && self.cache.has(key)) {
 				self.cache.delete(key);
 			}
-			
 			callback();
-		}); // delete
+		});
 	},
 	
 	runMaintenance: function(callback) {
@@ -528,39 +523,30 @@ module.exports = Class.create({
 			} );
 		}; // compress
 		
-		// Perform the backup -- as long as WAL mode is enabled, this should only acquire a shared lock on the DB
-		// allowing reads and writes to continue (albeit slowly), and NOT be included in the backup
-		// see: https://sqlite.org/forum/forumpost/95c0e12f4c
+		// Perform the backup using better-sqlite3's async backup API
+		// As long as WAL mode is enabled, this should only acquire a shared lock
 		perf.begin('backup');
-		
 		var db = this.db;
-		var backup = db.backup(file);
-		backup.step(-1);
-		
-		var timer = setInterval( function() {
-			if (backup.idle) { 
-				backup.step(-1); 
-			}
-			
-			if (backup.completed) {
-				perf.end('backup');
-				clearTimeout(timer);
-				self.logDebug(6, "SQLite backup operation completed: " + file);
-				
-				// now compress or keep or done
-				if (backups.compress) compress();
-				else if (backups.keep) keep();
-				else finish();
+		db.backup(file, {
+			progress: function(info) {
+				// transfer pages per tick (tuneable)
+				return 100;
 			}
-			
-			if (backup.failed) {
-				perf.end('backup');
-				clearTimeout(timer);
-				self.logError('backup', "SQLite backup operation failed.  Please check permissions and available disk space.");
-				fs.unlink(file, noop);
-				finish();
-			}
-		}, 10 );
+		})
+		.then(function() {
+			perf.end('backup');
+			self.logDebug(6, "SQLite backup operation completed: " + file);
+			// now compress or keep or done
+			if (backups.compress) compress();
+			else if (backups.keep) keep();
+			else finish();
+		})
+		.catch(function(err) {
+			perf.end('backup');
+			self.logError('backup', "SQLite backup operation failed.  Please check permissions and available disk space. " + err);
+			fs.unlink(file, noop);
+			finish();
+		});
 	},
 	
 	shutdown: function(callback) {
@@ -569,15 +555,15 @@ module.exports = Class.create({
 		this.logDebug(2, "Shutting down SQLite");
 		
 		if (this.db) {
-			// close db
-			self.db.close( function(err) {
-				if (err) self.logError('sqlite', "Failed to shutdown database cleanly: " + err);
-				else self.logDebug(3, "Shutdown complete");
-				callback();
-			} );
-			self.db = null;
-			self.commands = null;
+			try { this.db.close(); }
+			catch (err) { this.logError('sqlite', "Failed to shutdown database cleanly: " + err); }
+			this.db = null;
+			this.commands = null;
 		}
+		process.nextTick(function() {
+			self.logDebug(3, "Shutdown complete");
+			callback();
+		});
 	}
 	
 });

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pixl-server-storage",
-	"version": "3.2.9",
+	"version": "4.0.1",
 	"description": "A key/value/list storage component for the pixl-server framework.",
 	"author": "Joseph Huckaby <jhuckaby@gmail.com>",
 	"homepage": "https://github.com/jhuckaby/pixl-server-storage",

package/test/config.json

@@ -76,6 +76,13 @@
 				"enabled": true,
 				"maxItems": 1000,
 				"maxBytes": 10485760
+			},
+			"backups": {
+				"enabled": true,
+				"dir": "data/backups",
+				"filename": "backup-[yyyy]-[mm]-[dd]-[hh]-[mi]-[ss].db",
+				"compress": true,
+				"keep": 7
 			}
 		}
 	}

package/test/test-indexer.js

@@ -9,8 +9,11 @@ var cp = require('child_process');
 var async = require('async');
 var Tools = require('pixl-tools');
 
+var sample_tickets = null;
+var fixtures = null;
+
 var sample_data = require('./sample-data.json');
-var sample_tickets = sample_data.Ticket;
+var orig_sample_tickets = sample_data.Ticket;
 
 var index_config = {
 	base_path: "/index/ontrack",
@@ -50,7 +53,7 @@ var index_config = {
 	remove_words: ["the","of","and","a","to","in","is","you","that","it","he","was","for","on","are","as","with","his","they","I","at","be","this","have","from","or","one","had","by","word","but","not","what","all","were","we","when","your","can","said","there","use","an","each","which","she","do","how","their","if","will","up","other","about","out","many","then","them","these","so","some","her","would","make","like","him","into","time","has","look","two","more","write","go","see","number","no","way","could","people","my","than","first","water","been","call","who","oil","its","now","find","long","down","day","did","get","come","made","may","part"]
 };
 
-var fixtures = {
+var orig_fixtures = {
 	searchRecordsExact2: {
 		'title:"Released to Preproduction"': { "2653": 1, "2654": 1, "2659": 1, "2662": 1, "2665": 1 },
 		'status:open title:"Released to Preproduction"': { "2653": 1, "2654": 1 },
@@ -317,6 +320,13 @@ var fixtures = {
 
 module.exports = {
 	tests: [
+		
+		function resetFixtures(test) {
+			fixtures = Tools.copyHash( orig_fixtures, true );
+			sample_tickets = Tools.copyHash( orig_sample_tickets, true );
+			test.ok( true );
+			test.done();
+		},
 	
 		// insert record
 		

package/test/test.js

@@ -51,6 +51,7 @@ module.exports = {
 			
 			// override Filesystem base dir to go somewhere more sane
 			if (storage_config.Filesystem) storage_config.Filesystem.base_dir = base_data_dir;
+			if (storage_config.SQLite) storage_config.SQLite.base_dir = base_data_dir;
 		});
 		
 		// delete old unit test log
@@ -58,6 +59,7 @@ module.exports = {
 			// startup mock server
 			server.startup( function() {
 				// startup complete
+				server.logger.debug(9, "BASE DIR: " + base_data_dir);
 				
 				// write log in sync mode, for troubleshooting
 				server.logger.set('sync', true);
@@ -90,6 +92,16 @@ module.exports = {
 				// finally add indexer tests
 				self.tests = self.tests.concat( indexerTests.tests );
 				
+				// log memory usage at end
+				self.tests.push( function(test) {
+					if (global.gc) { global.gc(); global.gc(); }
+					test.ok( true );
+					setTimeout( function() { 
+						self.storage.logDebug(9, "MEMORY USAGE: " + Tools.getTextFromBytes( process.memoryUsage.rss() ), process.memoryUsage() );
+						test.done(); 
+					}, 1 );
+				} );
+				
 				// startup complete
 				// delay this by 1ms so the log is in the correct order (pre-start is async)
 				setTimeout( function() { callback(); }, 1 );