@zukhruf/sqlite-data-version 0.3.49 → 0.3.51
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -6
- package/dist/data-version.d.ts.map +1 -1
- package/dist/data-version.js +15 -22
- package/dist/data-version.js.map +1 -1
- package/dist/followed-database.d.ts +8 -0
- package/dist/followed-database.d.ts.map +1 -0
- package/dist/followed-database.js +74 -0
- package/dist/followed-database.js.map +1 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -56,8 +56,11 @@ no change
|
|
|
56
56
|
## The rules
|
|
57
57
|
|
|
58
58
|
- `new DataVersion(path, unavailable)` opens its own read-only connection to the database. It does not share a connection with a reader. [ADR 0001](./docs/adr/0001-poll-data-version.md) tells why.
|
|
59
|
-
- `current` gives the data version now. Compare it with the value that you read before
|
|
59
|
+
- `current` gives the data version now. It is the number of `DataVersion`, not the `PRAGMA data_version` of SQLite. Compare it with the value that you read before from the same `DataVersion`. A different value tells that another connection committed, or that the owner replaced the file. The same value tells that no change occurred.
|
|
60
60
|
- The number tells only that a change occurred. It does not count the commits, and it does not tell what changed.
|
|
61
|
+
- `DataVersion` follows the path. The owner can replace the file at the path: it can rename a new file over it, or delete the file and create it again. A replaced file is a change. Then `DataVersion` reads the new file, and each later commit to the new file is a change too. [ADR 0002](./docs/adr/0002-a-replaced-file-is-followed-by-name.md) tells why.
|
|
62
|
+
- A path that this process cannot see is no change. The path is missing, or the mode of a folder denies this process. The owner can delete the file and create it again a short time later. `DataVersion` keeps the old connection until it can see a file at the path again, and then a replaced file is a change.
|
|
63
|
+
- Each read of `current` reads the status of the file at the path one time (`stat`).
|
|
61
64
|
- A commit by the same connection does not change the number. The `DataVersion` connection never writes, so each commit that it sees comes from another connection.
|
|
62
65
|
- Dispose closes the connection. Use `using`, or call `[Symbol.dispose]()`.
|
|
63
66
|
- The caller gives the error class. Thus the error names what the caller needs, for example the access that the user must give.
|
|
@@ -88,20 +91,25 @@ console.log(`The owner last wrote at ${written.toISOString()}`);
|
|
|
88
91
|
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
|
|
89
92
|
| `new unavailable(path, cause)` | SQLite cannot open the file: `SQLITE_CANTOPEN` (14), for a missing file or a file that the system denies, a privacy denial too. |
|
|
90
93
|
| The error of SQLite | All other failures. |
|
|
94
|
+
| The error of the file system | `current` cannot read the status of the path, for a cause other than a missing path or a folder that denies this process. |
|
|
91
95
|
|
|
92
96
|
`lastWrite` gives `new unavailable(path, cause)` when the file is missing (`ENOENT`) or the mode of its folder denies this process (`EACCES`). It gives the error of the file system for all other failures.
|
|
93
97
|
|
|
94
98
|
## The same concepts in other systems
|
|
95
99
|
|
|
96
|
-
| System | What it does
|
|
97
|
-
| -------------------------------------- |
|
|
98
|
-
| SQLite `PRAGMA data_version` | It gives a number that changes when another connection commits. The SQLite documentation tells to use it to find out when to read again.
|
|
99
|
-
| File-change events (FSEvents, inotify) | They tell that a file changed. A process that keeps the file open and writes a WAL can commit many times before a write that an event reports.
|
|
100
|
-
| Datasette `immutable` mode | It opens a file that nobody writes, so the data never changes. This package is the opposite case: the owner writes the file.
|
|
100
|
+
| System | What it does |
|
|
101
|
+
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
102
|
+
| SQLite `PRAGMA data_version` | It gives a number that changes when another connection commits. The SQLite documentation tells to use it to find out when to read again. |
|
|
103
|
+
| File-change events (FSEvents, inotify) | They tell that a file changed. A process that keeps the file open and writes a WAL can commit many times before a write that an event reports. |
|
|
104
|
+
| Datasette `immutable` mode | It opens a file that nobody writes, so the data never changes. This package is the opposite case: the owner writes the file. |
|
|
105
|
+
| GNU `tail -F` | It follows a file by name. `tail.c` compares the device and inode of the name with the open file. When they differ, it prints "has been replaced; following new file" and opens the new file. `DataVersion` does the same. |
|
|
106
|
+
| SQLite `SQLITE_FCNTL_HAS_MOVED` | `fileHasMoved` in `os_unix.c` compares the inode of the path with the inode of the open file. SQLite only gives a warning. It never opens the new file. `node:sqlite` has no `file_control`, so `DataVersion` reads the status of the path itself. |
|
|
107
|
+
| Filebeat native file identity | It identifies a file by its device and inode. After a file closes, the system can give its inode to a new file, and Filebeat then sees no new file. `DataVersion` keeps the old file open until the new file is open, so the system cannot give that inode to the new file. |
|
|
101
108
|
|
|
102
109
|
## Documentation
|
|
103
110
|
|
|
104
111
|
- [0001 Poll data_version](./docs/adr/0001-poll-data-version.md)
|
|
112
|
+
- [0002 A replaced file is followed by name](./docs/adr/0002-a-replaced-file-is-followed-by-name.md)
|
|
105
113
|
- [Code copied into this package](./docs/copied-code.md)
|
|
106
114
|
|
|
107
115
|
## Development
|
|
@@ -119,6 +127,7 @@ npx nx run sqlite-data-version:build # compiles src/ to dist/
|
|
|
119
127
|
src/
|
|
120
128
|
index.ts the exports
|
|
121
129
|
data-version.ts DataVersion and UnavailableError
|
|
130
|
+
followed-database.ts the connection that follows the path to a replaced file
|
|
122
131
|
data-version.test.ts the tests, on real SQLite files
|
|
123
132
|
last-write.ts lastWrite
|
|
124
133
|
last-write.test.ts the tests, on real SQLite files
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"data-version.d.ts","sourceRoot":"","sources":["../src/data-version.ts"],"names":[],"mappings":"AAIA,MAAM,MAAM,gBAAgB,GAAG,KAAK,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,KAAK,KAAK,CAAC;
|
|
1
|
+
{"version":3,"file":"data-version.d.ts","sourceRoot":"","sources":["../src/data-version.ts"],"names":[],"mappings":"AAIA,MAAM,MAAM,gBAAgB,GAAG,KAAK,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO,KAAK,KAAK,CAAC;AAQ3E,qBAAa,WAAY,YAAW,UAAU;;gBAKhC,IAAI,EAAE,MAAM,EAAE,WAAW,EAAE,gBAAgB;IAKvD,IAAI,OAAO,IAAI,MAAM,CAOpB;IAED,CAAC,MAAM,CAAC,OAAO,CAAC,IAAI,IAAI;CAGzB"}
|
package/dist/data-version.js
CHANGED
|
@@ -1,35 +1,28 @@
|
|
|
1
|
-
import {
|
|
2
|
-
function open(path, unavailable) {
|
|
3
|
-
try {
|
|
4
|
-
return new DatabaseSync(path, { readOnly: true });
|
|
5
|
-
}
|
|
6
|
-
catch (cause) {
|
|
7
|
-
if (cause instanceof Error &&
|
|
8
|
-
'errcode' in cause &&
|
|
9
|
-
// SQLite's CANTOPEN (14). A missing or denied file fails the open()
|
|
10
|
-
// syscall, and SQLite reports that as CANTOPEN, a privacy denial too.
|
|
11
|
-
// SQLite gives AUTH only for an authorizer, and this private connection
|
|
12
|
-
// has none. Any other failure is not one that access can fix.
|
|
13
|
-
cause.errcode === 14)
|
|
14
|
-
throw new unavailable(path, cause);
|
|
15
|
-
throw cause;
|
|
16
|
-
}
|
|
17
|
-
}
|
|
1
|
+
import { FollowedDatabase } from "./followed-database.js";
|
|
18
2
|
// The owner commits through a WAL it keeps open, and file-change events
|
|
19
3
|
// report a write only when the file closes. SQLite's data_version changes on
|
|
20
4
|
// every commit by another connection, so polling current sees each one.
|
|
5
|
+
// The owner can also replace the file at the path. A new connection's
|
|
6
|
+
// data_version starts again and can equal an old value, so current is a
|
|
7
|
+
// count of the changes seen, not the data_version itself.
|
|
21
8
|
export class DataVersion {
|
|
22
9
|
#database;
|
|
23
|
-
#
|
|
10
|
+
#seen;
|
|
11
|
+
#current = 0;
|
|
24
12
|
constructor(path, unavailable) {
|
|
25
|
-
this.#database =
|
|
26
|
-
this.#
|
|
13
|
+
this.#database = new FollowedDatabase(path, unavailable);
|
|
14
|
+
this.#seen = this.#database.observe();
|
|
27
15
|
}
|
|
28
16
|
get current() {
|
|
29
|
-
|
|
17
|
+
const observed = this.#database.observe();
|
|
18
|
+
if (observed !== this.#seen) {
|
|
19
|
+
this.#seen = observed;
|
|
20
|
+
this.#current += 1;
|
|
21
|
+
}
|
|
22
|
+
return this.#current;
|
|
30
23
|
}
|
|
31
24
|
[Symbol.dispose]() {
|
|
32
|
-
this.#database.
|
|
25
|
+
this.#database[Symbol.dispose]();
|
|
33
26
|
}
|
|
34
27
|
}
|
|
35
28
|
//# sourceMappingURL=data-version.js.map
|
package/dist/data-version.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"data-version.js","sourceRoot":"","sources":["../src/data-version.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,
|
|
1
|
+
{"version":3,"file":"data-version.js","sourceRoot":"","sources":["../src/data-version.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAM1D,wEAAwE;AACxE,6EAA6E;AAC7E,wEAAwE;AACxE,sEAAsE;AACtE,wEAAwE;AACxE,0DAA0D;AAC1D,MAAM,OAAO,WAAW;IACb,SAAS,CAAmB;IACrC,KAAK,CAAS;IACd,QAAQ,GAAG,CAAC,CAAC;IAEb,YAAY,IAAY,EAAE,WAA6B;QACrD,IAAI,CAAC,SAAS,GAAG,IAAI,gBAAgB,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;QACzD,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,CAAC;IACxC,CAAC;IAED,IAAI,OAAO;QACT,MAAM,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,EAAE,CAAC;QAC1C,IAAI,QAAQ,KAAK,IAAI,CAAC,KAAK,EAAE,CAAC;YAC5B,IAAI,CAAC,KAAK,GAAG,QAAQ,CAAC;YACtB,IAAI,CAAC,QAAQ,IAAI,CAAC,CAAC;QACrB,CAAC;QACD,OAAO,IAAI,CAAC,QAAQ,CAAC;IACvB,CAAC;IAED,CAAC,MAAM,CAAC,OAAO,CAAC;QACd,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,CAAC;IACnC,CAAC;CACF"}
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
import type { UnavailableError } from './data-version.ts';
|
|
2
|
+
export declare class FollowedDatabase implements Disposable {
|
|
3
|
+
#private;
|
|
4
|
+
constructor(path: string, unavailable: UnavailableError);
|
|
5
|
+
observe(): string;
|
|
6
|
+
[Symbol.dispose](): void;
|
|
7
|
+
}
|
|
8
|
+
//# sourceMappingURL=followed-database.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"followed-database.d.ts","sourceRoot":"","sources":["../src/followed-database.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,mBAAmB,CAAC;AAqC1D,qBAAa,gBAAiB,YAAW,UAAU;;gBAOrC,IAAI,EAAE,MAAM,EAAE,WAAW,EAAE,gBAAgB;IAgBvD,OAAO,IAAI,MAAM;IAcjB,CAAC,MAAM,CAAC,OAAO,CAAC,IAAI,IAAI;CAGzB"}
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
import { statSync } from 'node:fs';
|
|
2
|
+
import { DatabaseSync } from 'node:sqlite';
|
|
3
|
+
function open(path, unavailable) {
|
|
4
|
+
try {
|
|
5
|
+
return new DatabaseSync(path, { readOnly: true });
|
|
6
|
+
}
|
|
7
|
+
catch (cause) {
|
|
8
|
+
if (cause instanceof Error &&
|
|
9
|
+
'errcode' in cause &&
|
|
10
|
+
// SQLite's CANTOPEN (14). A missing or denied file fails the open()
|
|
11
|
+
// syscall, and SQLite reports that as CANTOPEN, a privacy denial too.
|
|
12
|
+
// SQLite gives AUTH only for an authorizer, and this private connection
|
|
13
|
+
// has none. Any other failure is not one that access can fix.
|
|
14
|
+
cause.errcode === 14)
|
|
15
|
+
throw new unavailable(path, cause);
|
|
16
|
+
throw cause;
|
|
17
|
+
}
|
|
18
|
+
}
|
|
19
|
+
// The device and inode of the file at the path, or undefined when this
|
|
20
|
+
// process cannot see the path: it is missing, or a folder's mode denies it.
|
|
21
|
+
// Bigint, because inode numbers on NTFS and ReFS exceed 2^53.
|
|
22
|
+
function identify(path) {
|
|
23
|
+
try {
|
|
24
|
+
const stats = statSync(path, { bigint: true, throwIfNoEntry: false });
|
|
25
|
+
return stats === undefined ? undefined : `${stats.dev}:${stats.ino}`;
|
|
26
|
+
}
|
|
27
|
+
catch (cause) {
|
|
28
|
+
if (cause instanceof Error && 'code' in cause && cause.code === 'EACCES')
|
|
29
|
+
return undefined;
|
|
30
|
+
throw cause;
|
|
31
|
+
}
|
|
32
|
+
}
|
|
33
|
+
// A connection keeps reading the file it opened, also after the owner
|
|
34
|
+
// replaces the file at the path. This follows the path by name, as tail -F
|
|
35
|
+
// does: when the path names a different file, it opens that file.
|
|
36
|
+
export class FollowedDatabase {
|
|
37
|
+
#path;
|
|
38
|
+
#unavailable;
|
|
39
|
+
#identity;
|
|
40
|
+
#database;
|
|
41
|
+
#version;
|
|
42
|
+
constructor(path, unavailable) {
|
|
43
|
+
this.#path = path;
|
|
44
|
+
this.#unavailable = unavailable;
|
|
45
|
+
// Stat before open: a file replaced between the two is one extra change
|
|
46
|
+
// at the next observation. Stat after open could give the new file's
|
|
47
|
+
// identity to the old connection, and then the replace is never seen.
|
|
48
|
+
// A path this process cannot see fails the open too, as CANTOPEN.
|
|
49
|
+
const identity = identify(path);
|
|
50
|
+
this.#database = open(path, unavailable);
|
|
51
|
+
this.#identity = identity ?? '';
|
|
52
|
+
this.#version = this.#database.prepare('PRAGMA data_version');
|
|
53
|
+
}
|
|
54
|
+
// The identity of the held file and its data version. A path this process
|
|
55
|
+
// cannot see keeps the old connection: the owner may be about to create the
|
|
56
|
+
// file again, and the open connection still reads.
|
|
57
|
+
observe() {
|
|
58
|
+
const identity = identify(this.#path);
|
|
59
|
+
if (identity !== undefined && identity !== this.#identity) {
|
|
60
|
+
// Open the new file before the old one closes. While the old file is
|
|
61
|
+
// open its inode stays in use, so the new file cannot get the same one.
|
|
62
|
+
const database = open(this.#path, this.#unavailable);
|
|
63
|
+
this.#database.close();
|
|
64
|
+
this.#database = database;
|
|
65
|
+
this.#identity = identity;
|
|
66
|
+
this.#version = database.prepare('PRAGMA data_version');
|
|
67
|
+
}
|
|
68
|
+
return `${this.#identity}|${String(this.#version.get()?.data_version)}`;
|
|
69
|
+
}
|
|
70
|
+
[Symbol.dispose]() {
|
|
71
|
+
this.#database.close();
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
//# sourceMappingURL=followed-database.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"followed-database.js","sourceRoot":"","sources":["../src/followed-database.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AACnC,OAAO,EAAE,YAAY,EAAsB,MAAM,aAAa,CAAC;AAI/D,SAAS,IAAI,CAAC,IAAY,EAAE,WAA6B;IACvD,IAAI,CAAC;QACH,OAAO,IAAI,YAAY,CAAC,IAAI,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,CAAC;IACpD,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IACE,KAAK,YAAY,KAAK;YACtB,SAAS,IAAI,KAAK;YAClB,oEAAoE;YACpE,sEAAsE;YACtE,wEAAwE;YACxE,8DAA8D;YAC9D,KAAK,CAAC,OAAO,KAAK,EAAE;YAEpB,MAAM,IAAI,WAAW,CAAC,IAAI,EAAE,KAAK,CAAC,CAAC;QACrC,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,uEAAuE;AACvE,4EAA4E;AAC5E,8DAA8D;AAC9D,SAAS,QAAQ,CAAC,IAAY;IAC5B,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,QAAQ,CAAC,IAAI,EAAE,EAAE,MAAM,EAAE,IAAI,EAAE,cAAc,EAAE,KAAK,EAAE,CAAC,CAAC;QACtE,OAAO,KAAK,KAAK,SAAS,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,GAAG,KAAK,CAAC,GAAG,IAAI,KAAK,CAAC,GAAG,EAAE,CAAC;IACvE,CAAC;IAAC,OAAO,KAAK,EAAE,CAAC;QACf,IAAI,KAAK,YAAY,KAAK,IAAI,MAAM,IAAI,KAAK,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ;YACtE,OAAO,SAAS,CAAC;QACnB,MAAM,KAAK,CAAC;IACd,CAAC;AACH,CAAC;AAED,sEAAsE;AACtE,2EAA2E;AAC3E,kEAAkE;AAClE,MAAM,OAAO,gBAAgB;IAClB,KAAK,CAAS;IACd,YAAY,CAAmB;IACxC,SAAS,CAAS;IAClB,SAAS,CAAe;IACxB,QAAQ,CAAgB;IAExB,YAAY,IAAY,EAAE,WAA6B;QACrD,IAAI,CAAC,KAAK,GAAG,IAAI,CAAC;QAClB,IAAI,CAAC,YAAY,GAAG,WAAW,CAAC;QAChC,wEAAwE;QACxE,qEAAqE;QACrE,sEAAsE;QACtE,kEAAkE;QAClE,MAAM,QAAQ,GAAG,QAAQ,CAAC,IAAI,CAAC,CAAC;QAChC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC,IAAI,EAAE,WAAW,CAAC,CAAC;QACzC,IAAI,CAAC,SAAS,GAAG,QAAQ,IAAI,EAAE,CAAC;QAChC,IAAI,CAAC,QAAQ,GAAG,IAAI,CAAC,SAAS,CAAC,OAAO,CAAC,qBAAqB,CAAC,CAAC;IAChE,CAAC;IAED,0EAA0E;IAC1E,4EAA4E;IAC5E,mDAAmD;IACnD,OAAO;QACL,MAAM,QAAQ,GAAG,QAAQ,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACtC,IAAI,QAAQ,KAAK,SAAS,IAAI,QAAQ,KAAK,IAAI,CAAC,SAAS,EAAE,CAAC;YAC1D,qEAAqE;YACrE,wEAAwE;YACxE,MAAM,QAAQ,GAAG,IAAI,CAAC,IAAI,CAAC,KAAK,EAAE,IAAI,CAAC,YAAY,CAAC,CAAC;YACrD,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,CAAC;YACvB,IAAI,CAAC,SAAS,GAAG,QAAQ,CAAC;YAC1B,IAAI,CAAC,SAAS,GAAG,QAAQ,CAAC;YAC1B,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC,OAAO,CAAC,qBAAqB,CAAC,CAAC;QAC1D,CAAC;QACD,OAAO,GAAG,IAAI,CAAC,SAAS,IAAI,MAAM,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,EAAE,EAAE,YAAY,CAAC,EAAE,CAAC;IAC1E,CAAC;IAED,CAAC,MAAM,CAAC,OAAO,CAAC;QACd,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,CAAC;IACzB,CAAC;CACF"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@zukhruf/sqlite-data-version",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.51",
|
|
4
4
|
"description": "The data version of a SQLite database that another process owns and writes. It polls PRAGMA data_version on its own read-only connection, which changes on each commit by another connection, also while the owner keeps the file open.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -40,6 +40,6 @@
|
|
|
40
40
|
"access": "public"
|
|
41
41
|
},
|
|
42
42
|
"devDependencies": {
|
|
43
|
-
"@zukhruf/testing": "^0.3.
|
|
43
|
+
"@zukhruf/testing": "^0.3.51"
|
|
44
44
|
}
|
|
45
45
|
}
|