pixl-server-storage 3.1.18 → 3.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +146 -134
- package/docs/API.md +149 -144
- package/docs/Hashes.md +20 -20
- package/docs/Indexer.md +11 -14
- package/docs/Lists.md +37 -37
- package/docs/Transactions.md +11 -11
- package/engines/Redis.js +39 -36
- package/engines/RedisCluster.js +10 -13
- package/engines/S3.js +88 -0
- package/package.json +1 -1
package/docs/Hashes.md
CHANGED
|
@@ -8,8 +8,8 @@ All hash operations will automatically lock the hash using [Advisory Locking](..
|
|
|
8
8
|
|
|
9
9
|
The code examples all assume you have your preloaded `Storage` component instance in a local variable named `storage`. The component instance can be retrieved from a running server like this:
|
|
10
10
|
|
|
11
|
-
```
|
|
12
|
-
|
|
11
|
+
```js
|
|
12
|
+
let storage = server.Storage;
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
## Table of Contents
|
|
@@ -37,7 +37,7 @@ Care should be taken when calculating your hash page sizes. It all depends on h
|
|
|
37
37
|
|
|
38
38
|
To create a hash, call [hashCreate()](API.md#hashcreate). Specify the desired storage path, options, and a callback function. You can optionally pass in a custom page size via the second argument (otherwise it'll use the default size):
|
|
39
39
|
|
|
40
|
-
```
|
|
40
|
+
```js
|
|
41
41
|
storage.hashCreate( 'hash1', { page_size: 100 }, function(err) {
|
|
42
42
|
if (err) throw err;
|
|
43
43
|
} );
|
|
@@ -53,7 +53,7 @@ To store a key/value pair, call [hashPut](API.md#hashput), and to fetch a value
|
|
|
53
53
|
|
|
54
54
|
Here is an example of storing and fetching a hash record. The hash itself is located at the storage path `users`, and we are storing and fetching the hash key `bsanders` within that hash.
|
|
55
55
|
|
|
56
|
-
```
|
|
56
|
+
```js
|
|
57
57
|
// Store a key/value pair
|
|
58
58
|
storage.hashPut( 'users', 'bsanders', { name: 'Bernie', age: 75 }, function(err) {
|
|
59
59
|
if (err) throw err;
|
|
@@ -73,7 +73,7 @@ Note that you do not need to explicitly create the hash via [hashCreate()](API.m
|
|
|
73
73
|
In addition to storing single records, you can specify multiple at a time using the [hashPutMulti()](API.md#hashputmulti) method. Instead of key and value arguments, pass in an object containing as many as you want:
|
|
74
74
|
|
|
75
75
|
```js
|
|
76
|
-
|
|
76
|
+
let records = {
|
|
77
77
|
"bsanders": { name: "Bernie", age: 75 },
|
|
78
78
|
"hclinton": { name: "Hillary", age: 68 },
|
|
79
79
|
"dtrump": { name: "Donald", age: 70 }
|
|
@@ -86,7 +86,7 @@ storage.hashPutMulti( 'users', records, function(err) {
|
|
|
86
86
|
|
|
87
87
|
Similarly, to fetch multiple records, use [hashGetMulti()](API.md#hashgetmulti). Specify your desired keys in an array, and you'll get an array of values with numerical indexes that match up to your keys:
|
|
88
88
|
|
|
89
|
-
```
|
|
89
|
+
```js
|
|
90
90
|
storage.hashGetMulti( 'users', ['bsanders', 'hclinton', 'dtrump'], function(err, values) {
|
|
91
91
|
if (err) throw err;
|
|
92
92
|
// values[0] will be the bsanders record.
|
|
@@ -97,13 +97,13 @@ storage.hashGetMulti( 'users', ['bsanders', 'hclinton', 'dtrump'], function(err,
|
|
|
97
97
|
|
|
98
98
|
Finally, if you simply want to fetch *all* the records in a hash in one fell swoop, and you aren't concerned with memory usage, call [hashGetAll()](API.md#hashgetall).
|
|
99
99
|
|
|
100
|
-
```
|
|
100
|
+
```js
|
|
101
101
|
storage.hashGetAll( 'users', function(err, items) {
|
|
102
102
|
if (err) throw err;
|
|
103
103
|
|
|
104
104
|
// do something with all items
|
|
105
|
-
for (
|
|
106
|
-
|
|
105
|
+
for (let key in items) {
|
|
106
|
+
let value = items[key];
|
|
107
107
|
// do something with key/value pair
|
|
108
108
|
}
|
|
109
109
|
} );
|
|
@@ -126,7 +126,7 @@ function(err) {
|
|
|
126
126
|
|
|
127
127
|
Alternatively, you can use [hashEachSync()](API.md#hasheachsync) in which the iterator is invoked in a *synchronous* manner, i.e. continuing as soon as it returns (similar to [Array.forEach()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/forEach)). However, please note that the full loop operation isn't synchronous, and you need to provide a callback to be fired when every key has been iterated over. Example:
|
|
128
128
|
|
|
129
|
-
```
|
|
129
|
+
```js
|
|
130
130
|
storage.hashEachSync( 'users', function(key, value) {
|
|
131
131
|
// do something with key/value
|
|
132
132
|
// no callback here
|
|
@@ -139,11 +139,11 @@ function(err) {
|
|
|
139
139
|
|
|
140
140
|
Finally, you can use [hashEachPage()](API.md#hasheachpage), which iterates over the internal hash pages, and only fires your iterator function once per page, instead of once per key. This is typically faster as it requires fewer function calls. Your iterator is invoked in an *asynchronous* manner, i.e. it must fire a callback to continue (similar to [async eachSeries()](http://caolan.github.io/async/docs.html#.eachSeries)). Example:
|
|
141
141
|
|
|
142
|
-
```
|
|
142
|
+
```js
|
|
143
143
|
storage.hashEachPage( 'users', function(items, callback) {
|
|
144
144
|
// do something with page of items
|
|
145
|
-
for (
|
|
146
|
-
|
|
145
|
+
for (let key in items) {
|
|
146
|
+
let value = items[key];
|
|
147
147
|
// do something with each key/value pair
|
|
148
148
|
}
|
|
149
149
|
|
|
@@ -160,7 +160,7 @@ function(err) {
|
|
|
160
160
|
|
|
161
161
|
To duplicate a hash and all of its items at a new storage path, call [hashCopy()](API.md#hashcopy), specifying the old and new paths. Example:
|
|
162
162
|
|
|
163
|
-
```
|
|
163
|
+
```js
|
|
164
164
|
storage.hashCopy( 'hash1', 'hash2', function(err) {
|
|
165
165
|
if (err) throw err;
|
|
166
166
|
} );
|
|
@@ -168,7 +168,7 @@ storage.hashCopy( 'hash1', 'hash2', function(err) {
|
|
|
168
168
|
|
|
169
169
|
To rename a hash, call [hashRename()](API.md#hashrename). This is basically just a [hashCopy()](API.md#hashcopy) followed by a [hashDeleteAll()](API.md#hashdeleteall). Example:
|
|
170
170
|
|
|
171
|
-
```
|
|
171
|
+
```js
|
|
172
172
|
storage.hashRename( 'hash1', 'hash2', function(err) {
|
|
173
173
|
if (err) throw err;
|
|
174
174
|
} );
|
|
@@ -180,7 +180,7 @@ With both of these functions, it is highly recommended you make sure the destina
|
|
|
180
180
|
|
|
181
181
|
To delete a single hash key, call [hashDelete()](API.md#hashdelete). Example:
|
|
182
182
|
|
|
183
|
-
```
|
|
183
|
+
```js
|
|
184
184
|
storage.hashDelete( 'users', 'dtrump', function(err) {
|
|
185
185
|
if (err) throw err;
|
|
186
186
|
} );
|
|
@@ -188,7 +188,7 @@ storage.hashDelete( 'users', 'dtrump', function(err) {
|
|
|
188
188
|
|
|
189
189
|
If you delete the last key from a hash, an "empty" hash will remain in storage (this includes metadata such as the options, page size, etc). If you want to delete the *entire* hash when the last key is removed, pass `true` as the 3rd argument before the callback:
|
|
190
190
|
|
|
191
|
-
```
|
|
191
|
+
```js
|
|
192
192
|
storage.hashDelete( 'users', 'dtrump', true, function(err) {
|
|
193
193
|
if (err) throw err;
|
|
194
194
|
} );
|
|
@@ -196,7 +196,7 @@ storage.hashDelete( 'users', 'dtrump', true, function(err) {
|
|
|
196
196
|
|
|
197
197
|
To delete multiple hash keys at once, use [hashDeleteMulti()](API.md#hashdeletemulti). Specify your desired keys in an array:
|
|
198
198
|
|
|
199
|
-
```
|
|
199
|
+
```js
|
|
200
200
|
storage.hashDeleteMulti( 'users', ['bsanders', 'hclinton', 'dtrump'], function(err) {
|
|
201
201
|
if (err) throw err;
|
|
202
202
|
} );
|
|
@@ -204,7 +204,7 @@ storage.hashDeleteMulti( 'users', ['bsanders', 'hclinton', 'dtrump'], function(e
|
|
|
204
204
|
|
|
205
205
|
Finally, to delete an *entire* hash including all its keys, call [hashDeleteAll()](API.md#hashdeleteall):
|
|
206
206
|
|
|
207
|
-
```
|
|
207
|
+
```js
|
|
208
208
|
storage.hashDeleteAll( 'users', function(err) {
|
|
209
209
|
if (err) throw err;
|
|
210
210
|
} );
|
|
@@ -212,7 +212,7 @@ storage.hashDeleteAll( 'users', function(err) {
|
|
|
212
212
|
|
|
213
213
|
As with `hashDelete()`, by default this will only empty a hash, leaving behind an empty header record (with options, page size, etc). To delete that as well, pass `true` as the 2nd argument before the callback:
|
|
214
214
|
|
|
215
|
-
```
|
|
215
|
+
```js
|
|
216
216
|
storage.hashDeleteAll( 'users', true, function(err) {
|
|
217
217
|
if (err) throw err;
|
|
218
218
|
} );
|
package/docs/Indexer.md
CHANGED
|
@@ -10,14 +10,14 @@ Before using the indexer, please keep the following in mind:
|
|
|
10
10
|
- You are expected to store your record data yourself.
|
|
11
11
|
- The indexer **only** indexes fields for searching purposes -- it does not store raw records for retrieval.
|
|
12
12
|
- When performing a search, you **only** get your record IDs back.
|
|
13
|
-
- You are expected to paginate and fetch your own record data (i.e. via [getMulti()](API.md#
|
|
13
|
+
- You are expected to paginate and fetch your own record data (i.e. via [getMulti()](API.md#getmulti) or other).
|
|
14
14
|
|
|
15
15
|
The indexer works with any storage engine, but it is optimized for the local filesystem. Transactions are automatically used for indexing and unindexing records, if they are enabled (this is highly recommended).
|
|
16
16
|
|
|
17
17
|
The code examples all assume you have your preloaded `Storage` component instance in a local variable named `storage`. The component instance can be retrieved from a running server like this:
|
|
18
18
|
|
|
19
|
-
```
|
|
20
|
-
|
|
19
|
+
```js
|
|
20
|
+
let storage = server.Storage;
|
|
21
21
|
```
|
|
22
22
|
|
|
23
23
|
## Table of Contents
|
|
@@ -63,11 +63,9 @@ var storage = server.Storage;
|
|
|
63
63
|
|
|
64
64
|
The indexer only processes words that contain standard U.S. ASCII alphanumeric characters, e.g. A-Z, 0-9, and underscore. All other characters are skipped, and serve as word separators. International (Unicode) characters are converted to ASCII at index and search time (see [Unicode Characters](#unicode-characters) below for more on this).
|
|
65
65
|
|
|
66
|
-
The indexer
|
|
67
|
-
|
|
68
|
-
This system is not designed for large datasets. Thousands of records is probably fine, and maybe even tens of thousands depending on the index types and data size. *Hundreds of thousands* of records would likely end in tears. Also, keep an eye on your [inodes](https://en.wikipedia.org/wiki/Inode), because this thing is hungry for them.
|
|
66
|
+
The indexer can be slow, depending on the storage engine. All index operations involve reading and writing JSON storage records *at the word level* (i.e. a hash for each unique word!), and the system is designed to eat as little memory as possible. Expect hundreds or even thousands of storage operations for indexing a single record. Searching is fairly quick by comparison, because typically you're only searching on a small number of words. This also assumes your storage back-end is fast (preferably local SSD filesystem, or you are using a very fast key/value store like Couchbase or SQLite), and your search queries are simple and straightforward.
|
|
69
67
|
|
|
70
|
-
|
|
68
|
+
This system is not designed for large query results. Thousands of records returned is probably fine, and maybe even tens of thousands depending on the index types and data size. *Hundreds of thousands* of records would likely end in tears. Also, if you are using the Filesystem storage engine, keep an eye on your [inodes](https://en.wikipedia.org/wiki/Inode), because this thing is hungry for them.
|
|
71
69
|
|
|
72
70
|
## Configuration
|
|
73
71
|
|
|
@@ -325,7 +323,7 @@ Here is a simple example:
|
|
|
325
323
|
|
|
326
324
|
```js
|
|
327
325
|
// Index configuration
|
|
328
|
-
|
|
326
|
+
let config = {
|
|
329
327
|
"base_path": "index/myapp",
|
|
330
328
|
"fields": [
|
|
331
329
|
{
|
|
@@ -353,7 +351,7 @@ var config = {
|
|
|
353
351
|
};
|
|
354
352
|
|
|
355
353
|
// Record object
|
|
356
|
-
|
|
354
|
+
let record = {
|
|
357
355
|
"BodyText": "This is the body text of my ticket, which <b>may contain HTML</b> and \nmultiple\nlines.\n",
|
|
358
356
|
"ModifyDate": "2018/01/07",
|
|
359
357
|
"Tags": "bug, assigned, open"
|
|
@@ -804,7 +802,7 @@ If you want to make changes to your index configuration, i.e. add or remove fiel
|
|
|
804
802
|
To add a new field to your index, you simply have to apply the change to your index configuration, then call [indexRecord()](API.md#indexrecord) on all your records again. The indexer is smart enough to know that all your existing field values haven't changed, but you have added a new one which needs to be indexed. Example:
|
|
805
803
|
|
|
806
804
|
```js
|
|
807
|
-
|
|
805
|
+
const async = require('async');
|
|
808
806
|
|
|
809
807
|
// fetch all record ids
|
|
810
808
|
storage.searchRecords( '*', config, function(err, results) {
|
|
@@ -834,7 +832,7 @@ storage.searchRecords( '*', config, function(err, results) {
|
|
|
834
832
|
To delete a field, the process is similar to adding one. You will have to iterate over all your records, and call [indexRecord()](API.md#indexrecord) on each one, except this time we'll add a special `delete` property to the index you want to delete. This tells the library to force a delete on reindex. Here is an example:
|
|
835
833
|
|
|
836
834
|
```js
|
|
837
|
-
|
|
835
|
+
const async = require('async');
|
|
838
836
|
|
|
839
837
|
// mark the field you want to delete with a `delete` property
|
|
840
838
|
config.fields[2].delete = true;
|
|
@@ -880,8 +878,7 @@ Make sure you leave the index field settings set to their old values when you ad
|
|
|
880
878
|
- Make use of the [Stemmer](#stemming) for your full text fields.
|
|
881
879
|
- Only enable `master_list` on fields with a small number of unique words.
|
|
882
880
|
- Sorting is slow, consider using alphabetically sortable IDs, and sorting the records that way.
|
|
883
|
-
- Searching by date range and number range
|
|
884
|
-
- Everything is slow, don't use this stupid library, LOL :)
|
|
881
|
+
- Searching by date range and number range can be very slow.
|
|
885
882
|
|
|
886
883
|
## Indexer Internals
|
|
887
884
|
|
|
@@ -963,7 +960,7 @@ The first thing to notice is that everything is under a base path of `index/myap
|
|
|
963
960
|
So in our example configuration, we have three indexed fields: `body`, `modified` and `tags`. As you can see in the data layout, each one occupies its own "namespace" in storage, and contains hashes for each unique value, and in some cases a [summary](#master-list) record as well. For reference, here was our sample record ("TICKET0001") was was used to populate the indexes:
|
|
964
961
|
|
|
965
962
|
```js
|
|
966
|
-
|
|
963
|
+
let record = {
|
|
967
964
|
"BodyText": "This is the body text of my ticket, which <b>may contain HTML</b> and \nmultiple\nlines.\n",
|
|
968
965
|
"ModifyDate": "2018/01/07",
|
|
969
966
|
"Tags": "bug, assigned, open"
|
package/docs/Lists.md
CHANGED
|
@@ -6,8 +6,8 @@ All list operations will automatically lock the list using [Advisory Locking](..
|
|
|
6
6
|
|
|
7
7
|
The code examples all assume you have your preloaded `Storage` component instance in a local variable named `storage`. The component instance can be retrieved from a running server like this:
|
|
8
8
|
|
|
9
|
-
```
|
|
10
|
-
|
|
9
|
+
```js
|
|
10
|
+
let storage = server.Storage;
|
|
11
11
|
```
|
|
12
12
|
|
|
13
13
|
## Table of Contents
|
|
@@ -40,7 +40,7 @@ Care should be taken when calculating your list page sizes. It all depends on h
|
|
|
40
40
|
|
|
41
41
|
To create a list, call [listCreate()](API.md#listcreate). Specify the desired key, options, and a callback function. You can optionally pass in a custom page size via the second argument (otherwise it'll use the default size):
|
|
42
42
|
|
|
43
|
-
```
|
|
43
|
+
```js
|
|
44
44
|
storage.listCreate( 'list1', { page_size: 100 }, function(err) {
|
|
45
45
|
if (err) throw err;
|
|
46
46
|
} );
|
|
@@ -54,7 +54,7 @@ Lists can be treated as arrays to a certain extent. Methods are provided to [pu
|
|
|
54
54
|
|
|
55
55
|
Examples:
|
|
56
56
|
|
|
57
|
-
```
|
|
57
|
+
```js
|
|
58
58
|
// push onto the end
|
|
59
59
|
storage.listPush( 'list1', { username: 'tsmith', age: 25 }, function(err) {
|
|
60
60
|
if (err) throw err;
|
|
@@ -82,7 +82,7 @@ Furthermore, the [listPush()](API.md#listpush) and [listUnshift()](API.md#listun
|
|
|
82
82
|
|
|
83
83
|
Items can be fetched from lists by calling [listGet()](API.md#listget), and specifying an index offset starting from zero. You can fetch any number of items at a time, and the storage engine will figure out which pages need to be loaded. To fetch items from the end of a list, use a negative index. Example use:
|
|
84
84
|
|
|
85
|
-
```
|
|
85
|
+
```js
|
|
86
86
|
storage.listGet( 'list1', 40, 5, function(err, items) {
|
|
87
87
|
if (err) throw err;
|
|
88
88
|
} );
|
|
@@ -90,7 +90,7 @@ storage.listGet( 'list1', 40, 5, function(err, items) {
|
|
|
90
90
|
|
|
91
91
|
This would fetch 5 items starting at item index 40 (zero-based). To fetch the entire list, set the index and length to zero:
|
|
92
92
|
|
|
93
|
-
```
|
|
93
|
+
```js
|
|
94
94
|
storage.listGet( 'list1', 0, 0, function(err, items) {
|
|
95
95
|
if (err) throw err;
|
|
96
96
|
} );
|
|
@@ -102,8 +102,8 @@ You can "splice" a list just like you would an array. That is, cut a chunk out
|
|
|
102
102
|
|
|
103
103
|
Here is an example which removes 2 items at index 40, and replaces with 2 new items:
|
|
104
104
|
|
|
105
|
-
```
|
|
106
|
-
|
|
105
|
+
```js
|
|
106
|
+
let new_items = [
|
|
107
107
|
{ username: 'jhuckaby', age: 38, gender: 'male' },
|
|
108
108
|
{ username: 'sfields', age: 34, gender: 'female' }
|
|
109
109
|
];
|
|
@@ -121,8 +121,8 @@ As with [listGet()](API.md#listget) you can specify a negative index number to t
|
|
|
121
121
|
|
|
122
122
|
While it is possible to manually sort your list by fetching all the items as an array, sorting it in memory, then rewriting the entire list, this can be quite time consuming. Instead, you can perform a [listInsertSorted()](API.md#listinsertsorted) when adding items to a list. This will find the correct location for a single item based on sorting criteria, and then splice it into place, keeping the list sorted as you go. Example:
|
|
123
123
|
|
|
124
|
-
```
|
|
125
|
-
|
|
124
|
+
```js
|
|
125
|
+
let new_user = {
|
|
126
126
|
username: 'jhuckaby',
|
|
127
127
|
age: 38,
|
|
128
128
|
gender: 'male'
|
|
@@ -136,14 +136,14 @@ storage.listInsertSorted( 'users', new_user, ['username', 1], function(err) {
|
|
|
136
136
|
|
|
137
137
|
That third argument is an array of sort criteria, consisting of the property name to sort by (e.g. `username`) and a sort direction (`1` for ascending, `-1` for descending). You can alternately specify a comparator function here, which is called with the item you are inserting, and the item to compare it to. This is similar to the built-in [Array.sort()](), which should return `1` or `-1` depending on if your item should come after, or before, the second item. Example:
|
|
138
138
|
|
|
139
|
-
```
|
|
140
|
-
|
|
139
|
+
```js
|
|
140
|
+
let new_user = {
|
|
141
141
|
username: 'jhuckaby',
|
|
142
142
|
age: 38,
|
|
143
143
|
gender: 'male'
|
|
144
144
|
};
|
|
145
145
|
|
|
146
|
-
|
|
146
|
+
let comparator = function(a, b) {
|
|
147
147
|
return( (a.username < b.username) ? -1 : 1 );
|
|
148
148
|
};
|
|
149
149
|
|
|
@@ -159,7 +159,7 @@ For large lists, this can still take considerable time, as it is iterating over
|
|
|
159
159
|
|
|
160
160
|
Need to iterate over the items in your list, but don't want to load the entire thing into memory? Use the [listEach()](API.md#listeach) method. Example:
|
|
161
161
|
|
|
162
|
-
```
|
|
162
|
+
```js
|
|
163
163
|
storage.listEach( 'list1', function(item, idx, callback) {
|
|
164
164
|
// do something with item, then fire callback
|
|
165
165
|
callback();
|
|
@@ -174,7 +174,7 @@ Your iterator function is passed the item and a special callback function, which
|
|
|
174
174
|
|
|
175
175
|
Alternatively, you can use [listEachPage()](API.md#listeachpage), which iterates over the internal list [pages](#list-page-size), and only fires your iterator function once per page, instead of once per item. This is typically faster as it requires fewer function calls. Example:
|
|
176
176
|
|
|
177
|
-
```
|
|
177
|
+
```js
|
|
178
178
|
storage.listEachPage( 'list1', function(items, callback) {
|
|
179
179
|
// do something with items, then fire callback
|
|
180
180
|
callback();
|
|
@@ -189,7 +189,7 @@ function(err) {
|
|
|
189
189
|
|
|
190
190
|
To iterate over and possibly update items, you can use the [listEachUpdate()](API.md#listeachupdate) method. Your iterator callback accepts a second boolean argument which can indicate that you made changes. Example:
|
|
191
191
|
|
|
192
|
-
```
|
|
192
|
+
```js
|
|
193
193
|
storage.listEachUpdate( 'list1', function(item, idx, callback) {
|
|
194
194
|
// do something with item, then fire callback
|
|
195
195
|
item.something = "something new!";
|
|
@@ -205,7 +205,7 @@ As you can see, the iterator callback accepts two arguments, an error (or someth
|
|
|
205
205
|
|
|
206
206
|
For a speed optimization, you can optionally iterate over entire list [pages](#list-page-size) rather than individual items. To do this, use the [listEachPageUpdate()](API.md#listeachpageupdate) method. In this case your iterator callback is passed an array of items on each page. Example:
|
|
207
207
|
|
|
208
|
-
```
|
|
208
|
+
```js
|
|
209
209
|
storage.listEachPageUpdate( 'list1', function(items, callback) {
|
|
210
210
|
// do something with items, then fire callback
|
|
211
211
|
items.forEach( function(item) {
|
|
@@ -225,7 +225,7 @@ Several methods are provided for searching through lists for items matching a se
|
|
|
225
225
|
|
|
226
226
|
All of these methods accept a "criteria" object, which may have one or more key/value pairs. These must *all* match a list item for it to be selected. For example, if you have a list of users, and you want to find a male with blue eyes, you would pass a criteria object similar to this:
|
|
227
227
|
|
|
228
|
-
```
|
|
228
|
+
```js
|
|
229
229
|
{
|
|
230
230
|
gender: "male",
|
|
231
231
|
eyes: "blue"
|
|
@@ -234,7 +234,7 @@ All of these methods accept a "criteria" object, which may have one or more key/
|
|
|
234
234
|
|
|
235
235
|
Alternatively, you can use regular expression objects for the criteria values, for more complex matching. Example:
|
|
236
236
|
|
|
237
|
-
```
|
|
237
|
+
```js
|
|
238
238
|
{
|
|
239
239
|
gender: /^MALE$/i,
|
|
240
240
|
eyes: /blu/
|
|
@@ -243,7 +243,7 @@ Alternatively, you can use regular expression objects for the criteria values, f
|
|
|
243
243
|
|
|
244
244
|
Example of finding a single object with [listFind()](API.md#listfind):
|
|
245
245
|
|
|
246
|
-
```
|
|
246
|
+
```js
|
|
247
247
|
storage.listFind( 'list1', { username: 'jhuckaby' }, function(err, item, idx) {
|
|
248
248
|
if (err) throw err;
|
|
249
249
|
} );
|
|
@@ -251,7 +251,7 @@ storage.listFind( 'list1', { username: 'jhuckaby' }, function(err, item, idx) {
|
|
|
251
251
|
|
|
252
252
|
Example of finding and deleting a single object with [listFindCut()](API.md#listfindcut):
|
|
253
253
|
|
|
254
|
-
```
|
|
254
|
+
```js
|
|
255
255
|
storage.listFindCut( 'list1', { username: 'jhuckaby' }, function(err, item) {
|
|
256
256
|
if (err) throw err;
|
|
257
257
|
} );
|
|
@@ -259,9 +259,9 @@ storage.listFindCut( 'list1', { username: 'jhuckaby' }, function(err, item) {
|
|
|
259
259
|
|
|
260
260
|
Example of finding and replacing a single object with [listFindReplace()](API.md#listfindreplace):
|
|
261
261
|
|
|
262
|
-
```
|
|
263
|
-
|
|
264
|
-
|
|
262
|
+
```js
|
|
263
|
+
let criteria = { username: 'jhuckaby' };
|
|
264
|
+
let new_item = { username: 'huckabyj', foo: 'bar' };
|
|
265
265
|
|
|
266
266
|
storage.listFindReplace( 'list1', criteria, new_item, function(err) {
|
|
267
267
|
if (err) throw err;
|
|
@@ -270,9 +270,9 @@ storage.listFindReplace( 'list1', criteria, new_item, function(err) {
|
|
|
270
270
|
|
|
271
271
|
Example of finding and updating a single object with [listFindUpdate()](API.md#listfindupdate):
|
|
272
272
|
|
|
273
|
-
```
|
|
274
|
-
|
|
275
|
-
|
|
273
|
+
```js
|
|
274
|
+
let criteria = { username: 'jhuckaby' };
|
|
275
|
+
let updates = { gender: 'male', age: 38 };
|
|
276
276
|
|
|
277
277
|
storage.listFindUpdate( 'list1', criteria, updates, function(err, item) {
|
|
278
278
|
if (err) throw err;
|
|
@@ -281,9 +281,9 @@ storage.listFindUpdate( 'list1', criteria, updates, function(err, item) {
|
|
|
281
281
|
|
|
282
282
|
You can also increment or decrement numerical properties with [listFindUpdate()](API.md#listfindupdate). If an item key exists and is a number, you can set any update key to a string prefixed with `+` (increment) or `-` (decrement), followed by the delta number (int or float), e.g. `+1`. So for example, imagine a list of users, and an item property such as `number_of_logins`. When a user logs in again, you could increment this counter like this:
|
|
283
283
|
|
|
284
|
-
```
|
|
285
|
-
|
|
286
|
-
|
|
284
|
+
```js
|
|
285
|
+
let criteria = { username: 'jhuckaby' };
|
|
286
|
+
let updates = { number_of_logins: "+1" };
|
|
287
287
|
|
|
288
288
|
storage.listFindUpdate( 'list1', criteria, updates, function(err, item) {
|
|
289
289
|
if (err) throw err;
|
|
@@ -292,7 +292,7 @@ storage.listFindUpdate( 'list1', criteria, updates, function(err, item) {
|
|
|
292
292
|
|
|
293
293
|
And finally, here is an example of finding *all* items that match our criteria using [listFindEach()](API.md#listfindeach), and iterating over them:
|
|
294
294
|
|
|
295
|
-
```
|
|
295
|
+
```js
|
|
296
296
|
storage.listFindEach( 'list1', { gender: 'male' }, function(item, idx, callback) {
|
|
297
297
|
// do something with item, then fire callback
|
|
298
298
|
callback();
|
|
@@ -307,7 +307,7 @@ function(err) {
|
|
|
307
307
|
|
|
308
308
|
To duplicate a list and all of its items, call [listCopy()](API.md#listcopy), specifying the old and new key. Example:
|
|
309
309
|
|
|
310
|
-
```
|
|
310
|
+
```js
|
|
311
311
|
storage.listCopy( 'list1', 'list2', function(err) {
|
|
312
312
|
if (err) throw err;
|
|
313
313
|
} );
|
|
@@ -315,7 +315,7 @@ storage.listCopy( 'list1', 'list2', function(err) {
|
|
|
315
315
|
|
|
316
316
|
To rename a list, call [listRename()](API.md#listrename). This is basically just a [listCopy()](API.md#listcopy) followed by a [listDelete()](API.md#listdelete). Example:
|
|
317
317
|
|
|
318
|
-
```
|
|
318
|
+
```js
|
|
319
319
|
storage.listRename( 'list1', 'list2', function(err) {
|
|
320
320
|
if (err) throw err;
|
|
321
321
|
} );
|
|
@@ -327,7 +327,7 @@ With both of these functions, it is highly recommended you make sure the destina
|
|
|
327
327
|
|
|
328
328
|
To delete a list and all of its items, call [listDelete()](API.md#listdelete). The second argument should be a boolean set to `true` if you want the list *entirely* deleted including the header (options, page size, etc.), or `false` if you only want the list *cleared* (delete the items only, leaving an empty list behind). Example:
|
|
329
329
|
|
|
330
|
-
```
|
|
330
|
+
```js
|
|
331
331
|
storage.listDelete( 'list1', true, function(err) {
|
|
332
332
|
if (err) throw err;
|
|
333
333
|
// list is entirely deleted
|
|
@@ -338,7 +338,7 @@ storage.listDelete( 'list1', true, function(err) {
|
|
|
338
338
|
|
|
339
339
|
Lists consist of a header record, plus additional records for each page. The header is literally just a simple JSON record, stored at the exact key specified for the list. So if you created an empty list with key `mylist`, and then you fetched the `mylist` record using a simple [get()](API.md#get), you'd see this:
|
|
340
340
|
|
|
341
|
-
```
|
|
341
|
+
```js
|
|
342
342
|
{
|
|
343
343
|
type: 'list',
|
|
344
344
|
length: 0,
|
|
@@ -360,7 +360,7 @@ This is the list header record, which defines the list and its pages. Here are
|
|
|
360
360
|
|
|
361
361
|
The list pages are stored as records "under" the main key, by adding a slash, followed by the page number. So if you pushed one item onto the list, the updated header record would look like this:
|
|
362
362
|
|
|
363
|
-
```
|
|
363
|
+
```js
|
|
364
364
|
{
|
|
365
365
|
type: 'list',
|
|
366
366
|
length: 1,
|
|
@@ -374,7 +374,7 @@ Notice that the `first_page` and `last_page` are both still set to `0`, even tho
|
|
|
374
374
|
|
|
375
375
|
So then if you then fetched the key `mylist/0` you'd actually get the raw page data, which is a JSON record with an `items` array:
|
|
376
376
|
|
|
377
|
-
```
|
|
377
|
+
```js
|
|
378
378
|
{
|
|
379
379
|
items: [
|
|
380
380
|
{ username: "jhuckaby", gender: "male" }
|
package/docs/Transactions.md
CHANGED
|
@@ -6,8 +6,8 @@ The transaction system works with any storage engine, but it is optimized for th
|
|
|
6
6
|
|
|
7
7
|
The code examples all assume you have your preloaded `Storage` component instance in a local variable named `storage`. The component instance can be retrieved from a running server like this:
|
|
8
8
|
|
|
9
|
-
```
|
|
10
|
-
|
|
9
|
+
```js
|
|
10
|
+
let storage = server.Storage;
|
|
11
11
|
```
|
|
12
12
|
|
|
13
13
|
## Table of Contents
|
|
@@ -34,9 +34,9 @@ Transaction locking is advisory. Meaning, transactions will all play nice with
|
|
|
34
34
|
|
|
35
35
|
The transaction system is designed for JSON records only. Binary records are not included as part of a transaction, and operations are silently passed straight through. Meaning, any calls accessing binary records inside of a transaction will result in the original record being changed. This behavior may be changed in a future version.
|
|
36
36
|
|
|
37
|
-
Please note that transactions aren't 100% [ACID compliant](https://en.wikipedia.org/wiki/ACID), but they do follow *most* of the rules. When using the local filesystem engine, the real question is durability, as in what happens in the event of a sudden power loss. When a transaction is committed, by design it could involve changing a huge amount of files. While this operation is "effectively atomic" by way of advisory locking, the filesystem may still end up in an unknown state after sudden reboot, for example in the middle of a very large commit. Now, the system does have automatic recovery using a rollback log, and *should* be able to restore the filesystem to the state right before the commit. But really, when we're talking about yanking power cords, lightning strikes and brown-outs, who the hell knows. The rollback log may be incomplete or corrupted (yes, we call [fsync](https://nodejs.org/api/fs.html#
|
|
37
|
+
Please note that transactions aren't 100% [ACID compliant](https://en.wikipedia.org/wiki/ACID), but they do follow *most* of the rules. When using the local filesystem engine, the real question is durability, as in what happens in the event of a sudden power loss. When a transaction is committed, by design it could involve changing a huge amount of files. While this operation is "effectively atomic" by way of advisory locking, the filesystem may still end up in an unknown state after sudden reboot, for example in the middle of a very large commit. Now, the system does have automatic recovery using a rollback log, and *should* be able to restore the filesystem to the state right before the commit. But really, when we're talking about yanking power cords, lightning strikes and brown-outs, who the hell knows. The rollback log may be incomplete or corrupted (yes, we call [fsync](https://nodejs.org/api/fs.html#fsfsyncfd-callback) on it before ever starting the commit, but even then, things can happen -- fsync isn't a 100% guarantee, especially with SSDs).
|
|
38
38
|
|
|
39
|
-
I guess the bottom line is, always keep backups of your data
|
|
39
|
+
I guess the bottom line is, always keep backups of your data!
|
|
40
40
|
|
|
41
41
|
## Configuration
|
|
42
42
|
|
|
@@ -58,7 +58,7 @@ Choose your `trans_dir` carefully. This directory is used to store temporary fi
|
|
|
58
58
|
|
|
59
59
|
So basically, if you plan to use the [Filesystem](../README.md#local-filesystem) engine with a local (i.e. non-network) disk, you don't need to make any adjustments. The default setting for `trans_dir` is a subdirectory just inside your `base_dir` engine property. But if you are going with a networked (i.e. NFS) filesystem, or you're going to use [S3](../README.md#amazon-s3) or [Couchbase](../README.md#couchbase), then you should set `trans_dir` to a local filesystem path on a fast disk (ideally SSD).
|
|
60
60
|
|
|
61
|
-
**Pro-Tip:** When using transactions with the Amazon S3 engine, consider setting the `
|
|
61
|
+
**Pro-Tip:** When using transactions with the Amazon S3 engine, consider setting the `maxAttempts` property in your `S3` configuration object, so the AWS client library makes multiple attempts before failing an operation. Network hiccups can happen.
|
|
62
62
|
|
|
63
63
|
## Basic Use
|
|
64
64
|
|
|
@@ -86,7 +86,7 @@ The [begin()](API.md#begin) method is asynchronous because obtaining a lock may
|
|
|
86
86
|
Here is a more comprehensive example, which deposits a sum of money into a fictional bank account. We are using the [async](https://www.npmjs.com/package/async) module for better flow control.
|
|
87
87
|
|
|
88
88
|
```js
|
|
89
|
-
|
|
89
|
+
const async = require('async');
|
|
90
90
|
|
|
91
91
|
storage.begin( 'accounts', function(err, trans) {
|
|
92
92
|
// transaction has begun, now use 'trans' as storage proxy
|
|
@@ -248,7 +248,7 @@ Database recovery is complete. Please see logs/recovery.log for full details.
|
|
|
248
248
|
|
|
249
249
|
You can then start your application normally, i.e. without the `--recover` flag. Alternatively, if you would prefer that the database automatically recovers and starts up on its own, just add the following property in your `Storage` configuration:
|
|
250
250
|
|
|
251
|
-
```
|
|
251
|
+
```json
|
|
252
252
|
{
|
|
253
253
|
"trans_auto_recover": true
|
|
254
254
|
}
|
|
@@ -279,7 +279,7 @@ Inside the temp directory, two subdirectories are created: `data` and `logs`. T
|
|
|
279
279
|
|
|
280
280
|
When a transaction is first started, the only thing that happens is a unique ID is assigned, and a lock is obtained. Both are based on the path that is passed into [begin()](API.md#begin). Nothing is written to disk at this point -- that is deferred until actual operations are performed on the transaction object.
|
|
281
281
|
|
|
282
|
-
Each record that is mutated (created, updated or deleted) inside a transaction is written to a temporary file, and also a ledger is kept in memory to keep track of which records are changed. The in-memory ledger only contains keys and a "state"
|
|
282
|
+
Each record that is mutated (created, updated or deleted) inside a transaction is written to a temporary file, and also a ledger is kept in memory to keep track of which records are changed. The in-memory ledger only contains keys and a "state", representing a created, updated or deleted record. Only records created or updated have an associated temp file on disk. Deleted records are marked in memory only (no need for a temp file).
|
|
283
283
|
|
|
284
284
|
For example, consider a transaction that begins with path `test1`...
|
|
285
285
|
|
|
@@ -289,7 +289,7 @@ storage.begin( 'test1', function(err, trans) {
|
|
|
289
289
|
} );
|
|
290
290
|
```
|
|
291
291
|
|
|
292
|
-
The transaction ID will be `5a105e8b9d40e1329780d62ea2265d8a` (which is just an MD5 of `test1`). Now, let's say our `trans_dir` was set to `/
|
|
292
|
+
The transaction ID will be `5a105e8b9d40e1329780d62ea2265d8a` (which is just an MD5 of `test1`). Now, let's say our `trans_dir` was set to `/let/tmp/db`, and inside our transaction we store a record with key `test1/record1`...
|
|
293
293
|
|
|
294
294
|
```js
|
|
295
295
|
trans.put( 'test1/record1', { foo: 12345 }, function(err) {
|
|
@@ -300,7 +300,7 @@ trans.put( 'test1/record1', { foo: 12345 }, function(err) {
|
|
|
300
300
|
So at this point we wrote the `test1/record1` record using our `trans` object, but nothing has happened in the outer storage system (i.e. nothing was written to the primary storage engine). Instead, the following temp file was written to disk, containing `{"foo":12345}`:
|
|
301
301
|
|
|
302
302
|
```
|
|
303
|
-
/
|
|
303
|
+
/tmp/db/data/5a105e8b9d40e1329780d62ea2265d8a-0d1454fd1bcdc024acedcbe5cfff4ffd.json
|
|
304
304
|
```
|
|
305
305
|
|
|
306
306
|
The temp filename is made up of the transaction ID (`5a105e8b9d40e1329780d62ea2265d8a`) and the MD5 hash of the record key (`0d1454fd1bcdc024acedcbe5cfff4ffd`). This is to insure it will not collide with any other records in any other concurrent transactions, doesn't require any further subdirectories, and we can easily retrieve it at commit time if we know the plain key.
|
|
@@ -378,7 +378,7 @@ The commit process basically replays all the transaction operations on primary s
|
|
|
378
378
|
The log will contain the original JSON contents of all the records that will be mutated by the transaction. It is basically a "snapshot" of the previous state, before we start the commit. The log goes into the `logs` subdirectory under our `trans_dir` temp directory, and is named using the same Transaction ID hash:
|
|
379
379
|
|
|
380
380
|
```
|
|
381
|
-
/
|
|
381
|
+
/tmp/db/logs/5a105e8b9d40e1329780d62ea2265d8a.log
|
|
382
382
|
```
|
|
383
383
|
|
|
384
384
|
The log file format is plain text with line-delimited JSON for the records. The first line is a header record that describes the transaction. Here is an example log:
|