pixl-server-storage 4.1.0 → 4.2.0

package/README.md

@@ -58,6 +58,7 @@ Here is the table of contents for this current document:
 	* [Redis](#redis)
 		+ [RedisCluster](#rediscluster)
 	* [SQLite](#sqlite)
+	* [Postgres](#postgres)
 	* [Hybrid](#hybrid)
 - [Key Normalization](#key-normalization)
 - [Basic Functions](#basic-functions)
@@ -66,6 +67,7 @@ Here is the table of contents for this current document:
 	* [Copying Records](#copying-records)
 	* [Renaming Records](#renaming-records)
 	* [Deleting Records](#deleting-records)
+	* [Optimizing Storage](#optimizing-storage)
 - [Storing Binary Blobs](#storing-binary-blobs)
 - [Using Streams](#using-streams)
 - [Expiring Data](#expiring-data)
@@ -589,7 +591,12 @@ Then configure your storage thusly:
 		"host": "127.0.0.1",
 		"port": 6379,
 		"keyPrefix": "",
-		"keyTemplate": ""
+		"keyTemplate": "",
+		"cache": {
+			"enabled": true,
+			"maxItems": 100000,
+			"maxBytes": 104857600
+		}
 	}
 }
 ```
@@ -600,6 +607,8 @@ The optional `keyPrefix` property works similarly to the [S3 Key Prefix](#s3-key
 
 The optional `keyTemplate` property works similarly to the [S3 Key Template](#s3-key-template) feature.  It allows you to specify an exact layout of MD5 hash characters, which can be prefixed, mixed in with or postfixed after the key.
 
+See [S3 Cache](#s3-cache) for details on the `cache` section, as it works in the same way.
+
 ### RedisCluster
 
 If you want to use a Redis cluster (e.g. [AWS ElastiCache](https://aws.amazon.com/elasticache/)), then here is how to do that.  First, you will need to manually install the following module into your app:
@@ -625,7 +634,12 @@ Then configure your storage thusly:
 			}
 		},
 		"keyPrefix": "",
-		"keyTemplate": ""
+		"keyTemplate": "",
+		"cache": {
+			"enabled": true,
+			"maxItems": 100000,
+			"maxBytes": 104857600
+		}
 	}
 }
 ```
@@ -640,6 +654,8 @@ The optional `keyPrefix` property works similarly to the [S3 Key Prefix](#s3-key
 
 The optional `keyTemplate` property works similarly to the [S3 Key Template](#s3-key-template) feature.  It allows you to specify an exact layout of MD5 hash characters, which can be prefixed, mixed in with or postfixed after the key.
 
+See [S3 Cache](#s3-cache) for details on the `cache` section, as it works in the same way.
+
 ## SQLite
 
 If you want to use [SQLite](https://sqlite.com/) as a backing store, here is how to do so.  First, you need to manually install the [better-sqlite3](https://www.npmjs.com/package/better-sqlite3) module into your app:
@@ -782,6 +798,8 @@ The `Hybrid` engine only has two properties, `docEngine` and `binaryEngine`.  Th
 
 Note that all of the engine configuration objects are on the same level as the `Hybrid` object.
 
+When transactions are enabled, the `Hybrid` engine routes transaction commits through the configured `docEngine`, because pixl-server-storage transactions only apply to JSON records.  If the `docEngine` supports native transactions, such as Postgres, Redis or SQLite, `Hybrid` will use that native commit path automatically.  The `binaryEngine` is not involved in transactions, because binary keys are passed straight through.
+
 # Key Normalization
 
 In order to maintain compatibility with all the various engines, keys are "normalized" on all entry points.  Specifically, they undergo the following transformations before being passed along to the engine:
@@ -914,6 +932,19 @@ storage.delete( 'test1', function(err) {
 } );
 ```
 
+## Optimizing Storage
+
+To ask the current engine to run any engine-specific optimization routine, call the [optimize()](https://github.com/jhuckaby/pixl-server-storage/blob/master/docs/API.md#optimize) method.  This waits for the storage queue and advisory locks to drain, then runs whatever is appropriate for the backing store.  SQLite runs `VACUUM`, performs a WAL checkpoint when applicable, and runs an integrity check.  Postgres runs `VACUUM` on the configured storage table.  Redis and engines without an optimization routine return a no-op report.
+
+```js
+storage.optimize( function(err, report) {
+	if (err) throw err;
+	console.log( report );
+} );
+```
+
+The returned report is JSON-friendly and suitable for rendering into logs, Markdown or an admin UI.  It includes a `perf` object from [pixl-perf](https://github.com/jhuckaby/pixl-perf) with named timings for the optimization steps.  See the [API Reference](https://github.com/jhuckaby/pixl-server-storage/blob/master/docs/API.md#optimize) for details.
+
 # Storing Binary Blobs
 
 To store a binary value, pass a filled `Buffer` object as the value, and specify a key ending in a "file extension", e.g. `.gif`.  The latter requirement is so the engine can detect which records are binary and which are JSON, just by looking at the key.  Example:
@@ -1109,7 +1140,7 @@ Other examples of errors include transaction commit failures and transaction rol
 Transactions (well, more specifically, all storage actions) are logged with the `category` column set to `transaction`.  The `code` column will be one of the following constants, denoting which action took place:
 
 ```
-get, put, head, delete, expire_set, perf_sec, perf_min, commit, index, unindex, search, sort, maint
+get, put, head, delete, expire_set, perf_sec, perf_min, commit, index, unindex, search, sort, maint, optimize
 ```
 
 You can control which of these event types are logged, by including a `log_event_types` object in your storage configuration.  Include keys with true values for any log event types you want to see logged.  Example:
@@ -1117,7 +1148,7 @@ You can control which of these event types are logged, by including a `log_event
 ```js
 log_event_types: { 
 	get:0, put:1, head:0, delete:1, expire_set:1, perf_sec:1, perf_min:1,
-	commit:1, index:1, unindex:1, search:0, sort:0, maint:1 
+	commit:1, index:1, unindex:1, search:0, sort:0, maint:1, optimize:1
 }
 ```
 

package/docs/API.md

@@ -32,6 +32,7 @@ let storage = server.Storage;
 	* [shareUnlock](#shareunlock)
 	* [expire](#expire)
 	* [addRecordType](#addrecordtype)
+	* [optimize](#optimize)
 	* [getStats](#getstats)
 - [List Methods](#list-methods)
 	* [listCreate](#listcreate)
@@ -430,6 +431,63 @@ storage.addRecordType( 'my_custom_type', {
 
 See [Custom Record Types](../README.md#custom-record-types) for more details.
 
+## optimize
+
+```js
+storage.optimize( CALLBACK );
+```
+
+The `optimize()` method asks the current storage engine to run any engine-specific compaction or optimization routine, and returns a JSON-friendly report.  This is mostly useful for database engines.  SQLite runs `VACUUM`, performs a WAL checkpoint when applicable, and runs an integrity check.  Postgres runs `VACUUM` on the configured storage table.  Redis and engines without an optimization routine return a no-op report.
+
+The method waits for the storage queue and advisory locks to drain before starting.  Your callback is passed an error if optimization fails, or a report object as the second argument.  The `perf` property is the direct result of a [pixl-perf](https://github.com/jhuckaby/pixl-perf) `metrics()` call using second-scale timing.  Example:
+
+```js
+storage.optimize( function(err, report) {
+	if (err) throw err;
+	console.log( report );
+} );
+```
+
+Example SQLite report:
+
+```json
+{
+	"engine": "SQLite",
+	"optimized": true,
+	"file": "/let/data/sqlite.db",
+	"size_before": 1048576,
+	"size_after": 786432,
+	"bytes_reclaimed": 262144,
+	"integrity_ok": true,
+	"integrity_check": "ok",
+	"perf": {
+		"scale": 1,
+		"perf": {
+			"total": 0.421,
+			"vacuum": 0.31,
+			"checkpoint": 0.001,
+			"integrity": 0.11
+		},
+		"counters": {}
+	},
+	"operations": [
+		{
+			"name": "vacuum",
+			"ok": true,
+			"size_before": 1048576,
+			"size_after": 786432,
+			"bytes_reclaimed": 262144,
+			"reclaimed": "256 KB"
+		},
+		{
+			"name": "integrity_check",
+			"ok": true,
+			"result": "ok"
+		}
+	]
+}
+```
+
 ## getStats
 
 ```js

package/docs/Indexer.md

@@ -56,6 +56,7 @@ let storage = server.Storage;
 	* [Changing Fields](#changing-fields)
 - [Performance Tips](#performance-tips)
 - [Indexer Internals](#indexer-internals)
+	* [Sorters](#sorters)
 	* [Date and Number Fields](#date-and-number-fields)
 	* [Special Metadata](#special-metadata)
 
@@ -1006,6 +1007,67 @@ The `tags` field also has the [master_list](#master-list) property set, so a spe
 }
 ```
 
+### Sorters
+
+Sorters are stored a little differently than searchable field indexes.  A field index is inverted by word, meaning each unique word gets its own hash of matching record IDs.  A sorter is a single paginated [Hash](Hashes.md) containing every indexed record ID for that sorter, with the sorter value stored as the hash value.
+
+For example, a sorter with an ID of `username` is stored here:
+
+```
+index/
+ └ myapp/
+    └ username/
+       ├ sort/
+       │  └ data.json
+       └ sort.json
+```
+
+Internally, the hash path is:
+
+```
+index/myapp/username/sort
+```
+
+The hash keys are your primary record IDs, and the hash values are the values that will be used for sorting:
+
+```js
+{
+	"TICKET0001": "alice",
+	"TICKET0002": "bob",
+	"TICKET0003": "carol"
+}
+```
+
+Number sorters use the same layout, but store numeric values instead:
+
+```js
+{
+	"TICKET0001": 3,
+	"TICKET0002": 12,
+	"TICKET0003": 0
+}
+```
+
+The sorter hash uses `sorter_page_size` from your index configuration, or `1000` if that is not set.  This is separate from the standard index hash page size, because sorter hashes can be very large and are commonly scanned page by page.
+
+When [indexRecord()](API.md#indexrecord) runs, sorter updates happen after the searchable fields have been indexed.  For each sorter, the indexer reads the configured `source` path from the source record, applies `default_value` if needed, and writes the value using [hashPut()](API.md#hashput).  The record's `_data` record also gets a `_sorters` entry with the last indexed sorter value.  This allows the indexer to skip writing a sorter again if the value has not changed.
+
+When a record is unindexed, or when a sorter has `delete` set, the record ID is removed from the sorter hash using [hashDelete()](API.md#hashdelete).
+
+The important thing to understand is how sorting actually happens.  When you call [sortRecords()](API.md#sortrecords), the indexer does not fetch only the matching IDs from the sorter hash.  Instead, it enumerates the entire sorter hash, one page at a time, using [hashEachPage()](API.md#hasheachpage).  As each page is loaded, the indexer checks each hash key against the current search result hash.  If the ID is present in your result set, the sorter value is plucked out and added to an in-memory list of pairs:
+
+```js
+[
+	[ "TICKET0001", "alice" ],
+	[ "TICKET0002", "bob" ],
+	[ "TICKET0003", "carol" ]
+]
+```
+
+Once the full sorter hash has been scanned, only those matching pairs remain.  That final list is sorted in memory, using either a numeric comparison for `type: "number"`, or `localeCompare()` for string sorting.  The sorted record IDs are then returned as a simple array.
+
+This design keeps the persistent storage model simple and compact, but it also means sorting cost is based on the size of the entire sorter hash, not just the size of the current search results.  For example, sorting 50 matching records still requires scanning the whole sorter hash if the sorter contains 100,000 total records.  This is why the [Performance Tips](#performance-tips) recommend only adding sorters for fields you truly need.
+
 ### Date and Number Fields
 
 Date and number fields are still technically indexed as "words", but some internal trickery is applied to allow for range searches.  Essentially numbers and dates are placed into several range buckets.  Along with a [Master List](#master-list) which is automatically enabled for these fields, the system can search across a range of values.