@fayzanx/mmap-io 1.6.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +22 -0
- package/README.md +117 -0
- package/binding.gyp +32 -0
- package/dist/mmap-io.d.ts +29 -0
- package/dist/mmap-io.js +24 -0
- package/dist/test.js +153 -0
- package/dist/test.types.d.ts +1 -0
- package/dist/test.types.js +17 -0
- package/mmap-io.d.ts +29 -0
- package/mmap-io.js +1 -0
- package/package.json +91 -0
- package/src/mman.h +109 -0
- package/src/mmap-io.cc +340 -0
- package/src/mmap-io.ts +132 -0
- package/src/test.types.ts +22 -0
package/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2015 Oscar Campbell
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE.
|
22
|
+
|
package/README.md
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
|
2
|
+
# mmap-io - Node.js memory mapping bindings
|
3
|
+
|
4
|
+
## Note
|
5
|
+
- This is a fork of [ARyaskov/mmap-io](https://github.com/ARyaskov/mmap-io) -> [ozra/mmap-io](https://github.com/ozra/mmap-io) and [ipinfo/mmap-utils](https://github.com/ipinfo/mmap-utils)
|
6
|
+
|
7
|
+
## Quick info
|
8
|
+
|
9
|
+
- Long story short: with this addon you can increase performance of your file interactions by memory mapping files,
|
10
|
+
you are "project" some file to RAM and push data to RAM without any interactions with hdd/ssd subsystem -- this module will
|
11
|
+
facilitate reliable syncing between RAM-file and disk-file.
|
12
|
+
- **mmap-io** is written in C++17 and TypeScript, when you're installing it in your project, you're automatically getting
|
13
|
+
a precompiled binary (.node-file) for your platform from Downloads section of this project.
|
14
|
+
Otherwise it requires a C++17 compiler and Python 3.12+ on your machine to build.
|
15
|
+
- **mmap-io** is tested on Node.js 16 (16.14+), 18, 19, 20, 21, 22. Sorry, but there is no Node.js 17 because of some compilation stage issues.
|
16
|
+
- **mmap-io** has built binaries for Windows x86_64, Linux x86_64, Mac x86_64, Mac ARM.
|
17
|
+
- Potential use cases: working with big files (like highly volatile game map files), pushing data to cache files, video/audio-processing, messaging mechanics for inter-process communication.
|
18
|
+
|
19
|
+
## Quick TS example
|
20
|
+
|
21
|
+
```typescript
|
22
|
+
import fs from "fs"
|
23
|
+
import mmap from "@fayzanx/mmap-io"
|
24
|
+
|
25
|
+
const file = fs.openSync("/home/ubuntu/some-file-here", "r+")
|
26
|
+
const buf = mmap.map(fs.fstatSync(file).size, mmap.PROT_WRITE, mmap.MAP_SHARED, file)
|
27
|
+
mmap.advise(buf, mmap.MADV_RANDOM)
|
28
|
+
// Now you can work with "buf" as with a regular Buffer object.
|
29
|
+
// All your changes will be synced with the file "some-file-here" on disk.
|
30
|
+
// For example, add number 1024 at 0 position of buffer:
|
31
|
+
// buf.writeUInt32LE(1024, 0)
|
32
|
+
|
33
|
+
```
|
34
|
+
|
35
|
+
|
36
|
+
# Fork Notice
|
37
|
+
|
38
|
+
This is a fork of mmap-io (https://github.com/Widdershin/mmap-io/), as the upstream repo is no longer maintained and it didn't compile well on my machine for newer Node.js.
|
39
|
+
|
40
|
+
This version of mmap-io builds on Node up to 22, and provides binaries for Windows and macOS via @mapbox/node-pre-gyp.
|
41
|
+
|
42
|
+
# Author's notice:
|
43
|
+
## Mmap for Node.js
|
44
|
+
mmap(2) / madvise(2) / msync(2) / mincore(2) for node.js revisited.
|
45
|
+
|
46
|
+
**NOTE**: this is a fork of https://github.com/ozra/mmap-io as that repo is
|
47
|
+
unmaintained.
|
48
|
+
|
49
|
+
# Installation
|
50
|
+
|
51
|
+
## npm
|
52
|
+
|
53
|
+
This is my first node.js addon and after hours wasted reading up on V8 API I luckily stumbled upon [Native Abstractions for Node](https://github.com/rvagg/nan). Makes life so much easier. Hot tip!
|
54
|
+
|
55
|
+
_mmap-io_ is written in C++11 and ~~[LiveScript](https://github.com/gkz/LiveScript)~~ — _although I love LS, it's more prudent to use TypeScript for a library, so I've rewritten that code._
|
56
|
+
|
57
|
+
It should be noted that mem-mapping is by nature potentially blocking, and _should not be used in concurrent serving/processing applications_, but rather has it's niche where multiple processes are working on the same giant sets of data (thereby sparing physical memory, and load times if the kernel does it's job for read ahead), preferably multiple readers and single or none concurrent writer, to not spoil the gains by shitloads of spin-locks, mutexes or such. _And your noble specific use case of course._
|
58
|
+
|
59
|
+
|
60
|
+
# News and Updates
|
61
|
+
|
62
|
+
### 2024-05-12: version 1.4.3
|
63
|
+
- Add support for Node 22
|
64
|
+
|
65
|
+
# Install
|
66
|
+
Use npm or yarn or git.
|
67
|
+
|
68
|
+
```bash
|
69
|
+
$ npm install @fayzanx/mmap-io
|
70
|
+
```
|
71
|
+
```bash
|
72
|
+
$ yarn add @fayzanx/mmap-io
|
73
|
+
```
|
74
|
+
|
75
|
+
```bash
|
76
|
+
$ git clone https://github.com/fayzanx/mmap-io.git
|
77
|
+
$ cd mmap-io
|
78
|
+
$ yarn build
|
79
|
+
$ yarn install
|
80
|
+
```
|
81
|
+
|
82
|
+
### Good to Know (TM)
|
83
|
+
|
84
|
+
- Checkout man pages mmap(2), madvise(2), msync(2), mincore(2) for more detailed intell.
|
85
|
+
- The mappings are automatically unmapped when the buffer is garbage collected.
|
86
|
+
- Write-mappings need the fd to be opened with "r+", or you'll get a permission error (13).
|
87
|
+
- If you make a read-only mapping and then ignorantly set a value in the buffer, all hell previously unknown to a JS'er breaks loose (segmentation fault). It is possible to write some devilous code to intercept the SIGSEGV and throw an exception, but let's not do that!
|
88
|
+
- `Offset`, and in some cases `length` needs to be a multiple of mmap-io.PAGESIZE (which commonly is 4096)
|
89
|
+
- Huge pages are only supported for anonymous / private mappings (well, in Linux), so I didn't throw in flags for that since I found no use.
|
90
|
+
- As Ben Noordhuis previously has stated: Supplying hint for a fixed virtual memory adress is kinda moot point in JS, so not supported.
|
91
|
+
- If you miss a feature - contribute! Or request it in an issue.
|
92
|
+
- If documentation isn't clear, make an issue.
|
93
|
+
|
94
|
+
|
95
|
+
# Tests
|
96
|
+
|
97
|
+
```bash
|
98
|
+
$ yarn test
|
99
|
+
```
|
100
|
+
|
101
|
+
### Misc
|
102
|
+
|
103
|
+
- Checkout man pages `mmap(2)`, `madvise(2)`, `msync(2)`, `mincore(2)` for more
|
104
|
+
detailed intel.
|
105
|
+
- The mappings are automatically unmapped when the buffer is garbage collected.
|
106
|
+
- Write-mappings need the fd to be opened with "r+", or you'll get a permission
|
107
|
+
error (13).
|
108
|
+
- If you make a read-only mapping and then ignorantly set a value in the
|
109
|
+
buffer, all hell previously unknown to a JS'er breaks loose (segmentation
|
110
|
+
fault). It is possible to write some devilous code to intercept the SIGSEGV
|
111
|
+
and throw an exception, but let's not do that!
|
112
|
+
- `Offset`, and in some cases `length` needs to be a multiple of `PAGESIZE`
|
113
|
+
(which commonly is 4096)
|
114
|
+
- Huge pages are only supported for anonymous / private mappings (well, in
|
115
|
+
Linux), so I didn't throw in flags for that since I found no use.
|
116
|
+
- As Ben Noordhuis previously has stated: Supplying hint for a fixed virtual
|
117
|
+
memory adress is kinda moot point in JS, so not supported.
|
package/binding.gyp
ADDED
@@ -0,0 +1,32 @@
|
|
1
|
+
{
|
2
|
+
"targets": [{
|
3
|
+
"target_name": "mmap_io",
|
4
|
+
"sources": [ "src/mmap-io.cc" ],
|
5
|
+
"include_dirs": [
|
6
|
+
"<!(node -e \"require('nan')\")"
|
7
|
+
],
|
8
|
+
"cflags_cc": [
|
9
|
+
"-std=c++17"
|
10
|
+
],
|
11
|
+
"conditions": [
|
12
|
+
[ 'OS=="mac"',
|
13
|
+
{ "xcode_settings": {
|
14
|
+
'OTHER_CPLUSPLUSFLAGS' : ['-std=c++17','-stdlib=libc++'],
|
15
|
+
'OTHER_LDFLAGS': ['-stdlib=libc++'],
|
16
|
+
'MACOSX_DEPLOYMENT_TARGET': '10.8'
|
17
|
+
}}
|
18
|
+
]
|
19
|
+
]
|
20
|
+
},
|
21
|
+
{
|
22
|
+
"target_name": "action_after_build",
|
23
|
+
"type": "none",
|
24
|
+
"dependencies": [ "<(module_name)" ],
|
25
|
+
"copies": [
|
26
|
+
{
|
27
|
+
"files": [ "<(PRODUCT_DIR)/<(module_name).node" ],
|
28
|
+
"destination": "<(module_path)"
|
29
|
+
}
|
30
|
+
]
|
31
|
+
}]
|
32
|
+
}
|
@@ -0,0 +1,29 @@
|
|
1
|
+
type FileDescriptor = number;
|
2
|
+
type MapProtectionFlags = MmapIo["PROT_NONE"] | MmapIo["PROT_READ"] | MmapIo["PROT_WRITE"] | MmapIo["PROT_EXEC"] | 3 | 5 | 6 | 7;
|
3
|
+
type MapFlags = MmapIo["MAP_SHARED"] | MmapIo["MAP_PRIVATE"] | MmapIo["MAP_ANONYMOUS"] | MmapIo["MAP_POPULATE"] | MmapIo["MAP_NONBLOCK"] | number;
|
4
|
+
type MapAdvise = MmapIo["MADV_NORMAL"] | MmapIo["MADV_RANDOM"] | MmapIo["MADV_SEQUENTIAL"] | MmapIo["MADV_WILLNEED"] | MmapIo["MADV_DONTNEED"];
|
5
|
+
type MmapIo = {
|
6
|
+
map(size: number, protection: MapProtectionFlags, flags: MapFlags, fd: FileDescriptor, offset?: number, advise?: MapAdvise, name?: Buffer): Buffer;
|
7
|
+
advise(buffer: Buffer, offset: number, length: number, advise: MapAdvise): void;
|
8
|
+
advise(buffer: Buffer, advise: MapAdvise): void;
|
9
|
+
incore(buffer: Buffer): [number, number];
|
10
|
+
sync(buffer: Buffer, offset?: number, size?: number, blocking_sync?: boolean, invalidate_pages?: boolean): void;
|
11
|
+
sync(buffer: Buffer, blocking_sync: boolean, invalidate_pages?: boolean): void;
|
12
|
+
readonly PROT_READ: 1;
|
13
|
+
readonly PROT_WRITE: 2;
|
14
|
+
readonly PROT_EXEC: 4;
|
15
|
+
readonly PROT_NONE: 0;
|
16
|
+
readonly MAP_SHARED: 1;
|
17
|
+
readonly MAP_PRIVATE: 2;
|
18
|
+
readonly MAP_ANONYMOUS: 32;
|
19
|
+
readonly MAP_POPULATE: 32768;
|
20
|
+
readonly MAP_NONBLOCK: 65536;
|
21
|
+
readonly MADV_NORMAL: 0;
|
22
|
+
readonly MADV_RANDOM: 1;
|
23
|
+
readonly MADV_SEQUENTIAL: 2;
|
24
|
+
readonly MADV_WILLNEED: 3;
|
25
|
+
readonly MADV_DONTNEED: 4;
|
26
|
+
readonly PAGESIZE: number;
|
27
|
+
};
|
28
|
+
declare const mmap: MmapIo;
|
29
|
+
export default mmap;
|
package/dist/mmap-io.js
ADDED
@@ -0,0 +1,24 @@
|
|
1
|
+
"use strict";
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
3
|
+
const binary = require('@mapbox/node-pre-gyp');
|
4
|
+
const path = require('path');
|
5
|
+
const binding_path = binary.find(path.resolve(path.join(__dirname, '../package.json')));
|
6
|
+
const mmap_lib_raw_ = require(binding_path);
|
7
|
+
// snatch the raw C++-sync func
|
8
|
+
const raw_sync_fn_ = mmap_lib_raw_.sync_lib_private__;
|
9
|
+
// Hide the original C++11 func from users
|
10
|
+
delete mmap_lib_raw_.sync_lib_private__;
|
11
|
+
// Take care of all the param juggling here instead of in C++ code, by making
|
12
|
+
// some overloads, and doing some argument defaults
|
13
|
+
mmap_lib_raw_.sync = function (buf, par_a, par_b, par_c, par_d) {
|
14
|
+
if (typeof par_a === "boolean") {
|
15
|
+
raw_sync_fn_(buf, 0, buf.length, par_a, par_b || false);
|
16
|
+
}
|
17
|
+
else {
|
18
|
+
raw_sync_fn_(buf, par_a || 0, par_b || buf.length, par_c || false, par_d || false);
|
19
|
+
}
|
20
|
+
};
|
21
|
+
// mmap_lib_raw_.sync = sync_
|
22
|
+
const mmap = mmap_lib_raw_;
|
23
|
+
module.exports = mmap;
|
24
|
+
exports.default = mmap;
|
package/dist/test.js
ADDED
@@ -0,0 +1,153 @@
|
|
1
|
+
// Generated by LiveScript 1.6.0
|
2
|
+
var fs, os, mmap, assert, constants, say, PAGESIZE, PROT_READ, PROT_WRITE, MAP_SHARED, e, fd, size, buffer, out, i$, ix, incore_stats, WRONG_PAGE_SIZE, testFile, testSize, fdW, wBuffer, fdR, rBuffer, i, val;
|
3
|
+
fs = require("fs");
|
4
|
+
os = require("os");
|
5
|
+
mmap = require("../");
|
6
|
+
assert = require("assert");
|
7
|
+
constants = require("constants");
|
8
|
+
say = function(){
|
9
|
+
var args, res$, i$, to$;
|
10
|
+
res$ = [];
|
11
|
+
for (i$ = 0, to$ = arguments.length; i$ < to$; ++i$) {
|
12
|
+
res$.push(arguments[i$]);
|
13
|
+
}
|
14
|
+
args = res$;
|
15
|
+
return console.log.apply(console, args);
|
16
|
+
};
|
17
|
+
say("mmap in test is", mmap);
|
18
|
+
PAGESIZE = mmap.PAGESIZE, PROT_READ = mmap.PROT_READ, PROT_WRITE = mmap.PROT_WRITE, MAP_SHARED = mmap.MAP_SHARED;
|
19
|
+
try {
|
20
|
+
say("mmap.PAGESIZE = ", mmap.PAGESIZE, "tries to overwrite it with 47");
|
21
|
+
mmap.PAGESIZE = 47;
|
22
|
+
say("now mmap.PAGESIZE should be the same:", mmap.PAGESIZE, "silently kept");
|
23
|
+
} catch (e$) {
|
24
|
+
e = e$;
|
25
|
+
say("Caught trying to modify the mmap-object. Does this ever happen?", e);
|
26
|
+
}
|
27
|
+
fd = fs.openSync(process.argv[1], 'r');
|
28
|
+
size = fs.fstatSync(fd).size;
|
29
|
+
say("file size", size);
|
30
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd, 0, mmap.MADV_SEQUENTIAL);
|
31
|
+
say("buflen 1 = ", buffer.length);
|
32
|
+
assert.equal(buffer.length, size);
|
33
|
+
say("Give advise with 2 args");
|
34
|
+
mmap.advise(buffer, mmap.MADV_NORMAL);
|
35
|
+
say("Give advise with 4 args");
|
36
|
+
mmap.advise(buffer, 0, mmap.PAGESIZE, mmap.MADV_NORMAL);
|
37
|
+
say("\n\nBuffer contents, read byte for byte backwards and see that nothing explodes:\n");
|
38
|
+
try {
|
39
|
+
out = "";
|
40
|
+
for (i$ = size - 1; i$ >= 0; --i$) {
|
41
|
+
ix = i$;
|
42
|
+
out += String.fromCharCode(buffer[ix]);
|
43
|
+
}
|
44
|
+
incore_stats = mmap.incore(buffer);
|
45
|
+
assert.equal(incore_stats[0], 0);
|
46
|
+
assert.equal(incore_stats[1], 2);
|
47
|
+
} catch (e$) {
|
48
|
+
e = e$;
|
49
|
+
if (e.message !== 'mincore() not implemented') {
|
50
|
+
assert(false, "Shit happened while reading from buffer");
|
51
|
+
}
|
52
|
+
}
|
53
|
+
try {
|
54
|
+
say("read out of bounds test");
|
55
|
+
assert.equal(typeof buffer[size + 47], "undefined");
|
56
|
+
} catch (e$) {
|
57
|
+
e = e$;
|
58
|
+
say("deliberate out of bounds, caught exception - does this thing happen?", e.code, 'err-obj = ', e);
|
59
|
+
}
|
60
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd, 0);
|
61
|
+
say("buflen test 5-arg map call = ", buffer.length);
|
62
|
+
assert.equal(buffer.length, size);
|
63
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd);
|
64
|
+
say("buflen test 4-arg map call = ", buffer.length);
|
65
|
+
assert.equal(buffer.length, size);
|
66
|
+
if (os.type() !== 'Windows_NT') {
|
67
|
+
fd = fs.openSync(process.argv[1], 'r');
|
68
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd, PAGESIZE);
|
69
|
+
say("buflen test 3 = ", buffer.length);
|
70
|
+
assert.equal(buffer.length, size);
|
71
|
+
}
|
72
|
+
fd = fs.openSync(process.argv[1], 'r');
|
73
|
+
try {
|
74
|
+
buffer = mmap.map("foo", PROT_READ, MAP_SHARED, fd, 0);
|
75
|
+
} catch (e$) {
|
76
|
+
e = e$;
|
77
|
+
say("Pass faulty arg - caught deliberate exception: " + e.message);
|
78
|
+
}
|
79
|
+
fd = fs.openSync(process.argv[1], 'r');
|
80
|
+
try {
|
81
|
+
buffer = mmap.map(0, PROT_READ, MAP_SHARED, fd, 0);
|
82
|
+
} catch (e$) {
|
83
|
+
e = e$;
|
84
|
+
say("Pass zero size - caught deliberate exception: " + e.message);
|
85
|
+
}
|
86
|
+
WRONG_PAGE_SIZE = PAGESIZE - 1;
|
87
|
+
fd = fs.openSync(process.argv[1], 'r');
|
88
|
+
try {
|
89
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd, WRONG_PAGE_SIZE);
|
90
|
+
} catch (e$) {
|
91
|
+
e = e$;
|
92
|
+
say("Pass wrong page-size as offset - caught deliberate exception: " + e.message);
|
93
|
+
}
|
94
|
+
fd = fs.openSync(process.argv[1], 'r');
|
95
|
+
try {
|
96
|
+
buffer = mmap.map(size, PROT_READ, MAP_SHARED, fd);
|
97
|
+
mmap.advise(buffer, "fuck off");
|
98
|
+
} catch (e$) {
|
99
|
+
e = e$;
|
100
|
+
say("Pass faulty arg to advise() - caught deliberate exception: " + e.message);
|
101
|
+
}
|
102
|
+
say("Now for some write/read tests");
|
103
|
+
try {
|
104
|
+
say("Creates file");
|
105
|
+
testFile = "./tmp-mmap-file";
|
106
|
+
testSize = 47474;
|
107
|
+
fs.writeFileSync(testFile, "");
|
108
|
+
fs.truncateSync(testFile, testSize);
|
109
|
+
say("open write buffer");
|
110
|
+
fdW = fs.openSync(testFile, 'r+');
|
111
|
+
say("fd-write = ", fdW);
|
112
|
+
wBuffer = mmap.map(testSize, PROT_WRITE, MAP_SHARED, fdW);
|
113
|
+
fs.closeSync(fdW);
|
114
|
+
mmap.advise(wBuffer, mmap.MADV_SEQUENTIAL);
|
115
|
+
say("open read bufer");
|
116
|
+
fdR = fs.openSync(testFile, 'r');
|
117
|
+
rBuffer = mmap.map(testSize, PROT_READ, MAP_SHARED, fdR);
|
118
|
+
fs.closeSync(fdR);
|
119
|
+
mmap.advise(rBuffer, mmap.MADV_SEQUENTIAL);
|
120
|
+
say("verify write and read");
|
121
|
+
for (i$ = 0; i$ < testSize; ++i$) {
|
122
|
+
i = i$;
|
123
|
+
val = 32 + i % 60;
|
124
|
+
wBuffer[i] = val;
|
125
|
+
assert.equal(rBuffer[i], val);
|
126
|
+
}
|
127
|
+
say("Write/read verification seemed to work out");
|
128
|
+
} catch (e$) {
|
129
|
+
e = e$;
|
130
|
+
say("Something fucked up in the write/read test::", e.message);
|
131
|
+
}
|
132
|
+
try {
|
133
|
+
say("sync() tests x 4");
|
134
|
+
say("1. Does explicit blocking sync to disk");
|
135
|
+
mmap.sync(wBuffer, 0, testSize, true, false);
|
136
|
+
say("2. Does explicit blocking sync without offset/length arguments");
|
137
|
+
mmap.sync(wBuffer, true, false);
|
138
|
+
say("3. Does explicit sync to disk without blocking/invalidate flags");
|
139
|
+
mmap.sync(wBuffer, 0, testSize);
|
140
|
+
say("4. Does explicit sync with no additional arguments");
|
141
|
+
mmap.sync(wBuffer);
|
142
|
+
} catch (e$) {
|
143
|
+
e = e$;
|
144
|
+
say("Something fucked up for syncs::", e.message);
|
145
|
+
}
|
146
|
+
try {
|
147
|
+
fs.unlinkSync(testFile);
|
148
|
+
} catch (e$) {
|
149
|
+
e = e$;
|
150
|
+
say("Failed to remove test-file", testFile);
|
151
|
+
}
|
152
|
+
say("\nAll done");
|
153
|
+
process.exit(0);
|
@@ -0,0 +1 @@
|
|
1
|
+
export {};
|
@@ -0,0 +1,17 @@
|
|
1
|
+
"use strict";
|
2
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
3
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
4
|
+
};
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
6
|
+
const mmap_io_1 = __importDefault(require("./mmap-io"));
|
7
|
+
const fs_1 = __importDefault(require("fs"));
|
8
|
+
const fd = fs_1.default.openSync(__filename, "r+");
|
9
|
+
/**
|
10
|
+
* Writing, for instance `4` instead of ie `2` (MADV_SEQUENTIAL), here, will
|
11
|
+
* give an error already while typing (provided language-server is setup)
|
12
|
+
* because of the specific types
|
13
|
+
*
|
14
|
+
* A problem arise with the traditional dirty technique of binary-or, widening
|
15
|
+
* the type from literal to number-field
|
16
|
+
*/
|
17
|
+
mmap_io_1.default.map(100, (mmap_io_1.default.PROT_EXEC | mmap_io_1.default.PROT_READ), mmap_io_1.default.MAP_SHARED, fd, 0, mmap_io_1.default.MADV_SEQUENTIAL);
|
package/mmap-io.d.ts
ADDED
@@ -0,0 +1,29 @@
|
|
1
|
+
type FileDescriptor = number;
|
2
|
+
type MapProtectionFlags = MmapIo["PROT_NONE"] | MmapIo["PROT_READ"] | MmapIo["PROT_WRITE"] | MmapIo["PROT_EXEC"] | 3 | 5 | 6 | 7;
|
3
|
+
type MapFlags = MmapIo["MAP_SHARED"] | MmapIo["MAP_PRIVATE"] | MmapIo["MAP_ANONYMOUS"] | MmapIo["MAP_POPULATE"] | MmapIo["MAP_NONBLOCK"] | number;
|
4
|
+
type MapAdvise = MmapIo["MADV_NORMAL"] | MmapIo["MADV_RANDOM"] | MmapIo["MADV_SEQUENTIAL"] | MmapIo["MADV_WILLNEED"] | MmapIo["MADV_DONTNEED"];
|
5
|
+
type MmapIo = {
|
6
|
+
map(size: number, protection: MapProtectionFlags, flags: MapFlags, fd: FileDescriptor, offset?: number, advise?: MapAdvise, name?: Buffer): Buffer;
|
7
|
+
advise(buffer: Buffer, offset: number, length: number, advise: MapAdvise): void;
|
8
|
+
advise(buffer: Buffer, advise: MapAdvise): void;
|
9
|
+
incore(buffer: Buffer): [number, number];
|
10
|
+
sync(buffer: Buffer, offset?: number, size?: number, blocking_sync?: boolean, invalidate_pages?: boolean): void;
|
11
|
+
sync(buffer: Buffer, blocking_sync: boolean, invalidate_pages?: boolean): void;
|
12
|
+
readonly PROT_READ: 1;
|
13
|
+
readonly PROT_WRITE: 2;
|
14
|
+
readonly PROT_EXEC: 4;
|
15
|
+
readonly PROT_NONE: 0;
|
16
|
+
readonly MAP_SHARED: 1;
|
17
|
+
readonly MAP_PRIVATE: 2;
|
18
|
+
readonly MAP_ANONYMOUS: 32;
|
19
|
+
readonly MAP_POPULATE: 32768;
|
20
|
+
readonly MAP_NONBLOCK: 65536;
|
21
|
+
readonly MADV_NORMAL: 0;
|
22
|
+
readonly MADV_RANDOM: 1;
|
23
|
+
readonly MADV_SEQUENTIAL: 2;
|
24
|
+
readonly MADV_WILLNEED: 3;
|
25
|
+
readonly MADV_DONTNEED: 4;
|
26
|
+
readonly PAGESIZE: number;
|
27
|
+
};
|
28
|
+
declare const mmap: MmapIo;
|
29
|
+
export default mmap;
|
package/mmap-io.js
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
module.exports = require("./dist/mmap-utils")
|
package/package.json
ADDED
@@ -0,0 +1,91 @@
|
|
1
|
+
{
|
2
|
+
"name": "@fayzanx/mmap-io",
|
3
|
+
"version": "1.6.12",
|
4
|
+
"license": "MIT",
|
5
|
+
"author": {
|
6
|
+
"name": "Fayzan Ahmad",
|
7
|
+
"email": "fayzanx@gmail.com",
|
8
|
+
"url": "https://github.com/fayzanx"
|
9
|
+
},
|
10
|
+
"contributors": [
|
11
|
+
{
|
12
|
+
"name": "Oscar Campbell",
|
13
|
+
"email": "oscar@campbell.nu",
|
14
|
+
"url": "https://github.com/ozra"
|
15
|
+
}
|
16
|
+
],
|
17
|
+
"binary": {
|
18
|
+
"module_name": "mmap_io",
|
19
|
+
"module_path": "./build/binding/{configuration}/{node_abi}-{platform}-{arch}/",
|
20
|
+
"remote_path": "./{version}/",
|
21
|
+
"package_name": "{module_name}-v{version}-{node_abi}-{platform}-{arch}.tar.gz",
|
22
|
+
"host": "https://github.com/fayzanx/mmap-io/releases/download/"
|
23
|
+
},
|
24
|
+
"description": "Node.js mmap bindings revisited.",
|
25
|
+
"homepage": "https://github.com/fayzanx/mmap-io",
|
26
|
+
"keywords": [
|
27
|
+
"low level",
|
28
|
+
"file",
|
29
|
+
"memory mapped",
|
30
|
+
"mmap",
|
31
|
+
"madvise",
|
32
|
+
"sync",
|
33
|
+
"shared memory",
|
34
|
+
"C++",
|
35
|
+
"performance"
|
36
|
+
],
|
37
|
+
"repository": {
|
38
|
+
"type": "git",
|
39
|
+
"url": "https://github.com/fayzanx/mmap-io.git"
|
40
|
+
},
|
41
|
+
"bugs": {
|
42
|
+
"url": "http://github.com/fayzanx/mmap-io/issues"
|
43
|
+
},
|
44
|
+
"main": "mmap-io.js",
|
45
|
+
"files": [
|
46
|
+
"binding.gyp",
|
47
|
+
"LICENSE",
|
48
|
+
"dist",
|
49
|
+
"mmap-io.d.ts",
|
50
|
+
"mmap-io.js",
|
51
|
+
"package.json",
|
52
|
+
"package-lock.json",
|
53
|
+
"README.md",
|
54
|
+
"src"
|
55
|
+
],
|
56
|
+
"bin": {
|
57
|
+
"node-pre-gyp-github": "./bin/node-pre-gyp-github"
|
58
|
+
},
|
59
|
+
"scripts": {
|
60
|
+
"clean": "rm -rf build dist",
|
61
|
+
"build": "npm run build-addon && npm run build-es",
|
62
|
+
"build-addon": "node-pre-gyp install --fallback-to-build",
|
63
|
+
"build-es": "run-script-os",
|
64
|
+
"build-es:win32": "tsc && copy .\\dist\\mmap-io.d.ts .\\",
|
65
|
+
"build-es:default": "tsc && cp ./dist/mmap-io.d.ts ./",
|
66
|
+
"rebuild": "node-pre-gyp reinstall --build-from-source",
|
67
|
+
"prepare": "npm run build",
|
68
|
+
"install": "npm run build-addon",
|
69
|
+
"build-binary-x64": "node-pre-gyp rebuild && node-pre-gyp package",
|
70
|
+
"build-binary-x86": "node-pre-gyp rebuild --target_arch=ia32 && node-pre-gyp package --target_arch=ia32",
|
71
|
+
"publish:github": "node-pre-gyp-github publish",
|
72
|
+
"publish:npm": "npm publish --access public --registry=https://registry.npmjs.org",
|
73
|
+
"publish:all": "npm run publish:github || npm run publish:npm",
|
74
|
+
"test": "echo 'tests disabled\n'",
|
75
|
+
"watch": "while true; do (npm run build; inotifywait -qre close_write,moved_to --exclude '\\.git' ./src/; ) done;"
|
76
|
+
},
|
77
|
+
"devDependencies": {
|
78
|
+
"@types/node": "^18.19.41",
|
79
|
+
"node-pre-gyp-github": "^2.0.0",
|
80
|
+
"tsx": "^4.16.2",
|
81
|
+
"typescript": "^5.5.3"
|
82
|
+
},
|
83
|
+
"dependencies": {
|
84
|
+
"@mapbox/node-pre-gyp": "^1.0.11",
|
85
|
+
"errno": "*",
|
86
|
+
"nan": "^2.20.0",
|
87
|
+
"node-gyp": "^10.1.0",
|
88
|
+
"run-script-os": "^1.1.1",
|
89
|
+
"yarn": "^1.22.22"
|
90
|
+
}
|
91
|
+
}
|
package/src/mman.h
ADDED
@@ -0,0 +1,109 @@
|
|
1
|
+
#include <io.h>
|
2
|
+
#include <errno.h>
|
3
|
+
#include <windows.h>
|
4
|
+
#include <sys/types.h>
|
5
|
+
|
6
|
+
#define PROT_NONE 0x00
|
7
|
+
#define PROT_READ 0x01
|
8
|
+
#define PROT_WRITE 0x02
|
9
|
+
#define PROT_EXEC 0x04
|
10
|
+
|
11
|
+
#define MAP_FILE 0x00
|
12
|
+
#define MAP_SHARED 0x01
|
13
|
+
#define MAP_PRIVATE 0x02
|
14
|
+
#define MAP_TYPE 0x0F
|
15
|
+
#define MAP_ANONYMOUS 0x20
|
16
|
+
#define MAP_FAILED ((void*) -1)
|
17
|
+
|
18
|
+
#define MS_ASYNC 0x01
|
19
|
+
#define MS_SYNC 0x02
|
20
|
+
#define MS_INVALIDATE 0x04
|
21
|
+
|
22
|
+
#define MADV_NORMAL 0x00
|
23
|
+
#define MADV_RANDOM 0x01
|
24
|
+
#define MADV_SEQUENTIAL 0x02
|
25
|
+
#define MADV_WILLNEED 0x03
|
26
|
+
#define MADV_DONTNEED 0x04
|
27
|
+
|
28
|
+
inline void* mmap(void* addr, size_t length, int prot, int flags, int fd, size_t offset, LPCSTR name) {
|
29
|
+
if (prot & ~(PROT_READ | PROT_WRITE | PROT_EXEC))
|
30
|
+
return MAP_FAILED;
|
31
|
+
if (fd == -1) {
|
32
|
+
if (!(flags & MAP_ANONYMOUS) || offset)
|
33
|
+
return MAP_FAILED;
|
34
|
+
} else if (flags & MAP_ANONYMOUS)
|
35
|
+
return MAP_FAILED;
|
36
|
+
|
37
|
+
DWORD protect;
|
38
|
+
if (prot & PROT_WRITE) {
|
39
|
+
if (prot & PROT_EXEC)
|
40
|
+
protect = PAGE_EXECUTE_READWRITE;
|
41
|
+
else
|
42
|
+
protect = PAGE_READWRITE;
|
43
|
+
} else if (prot & PROT_EXEC) {
|
44
|
+
if (prot & PROT_READ)
|
45
|
+
protect = PAGE_EXECUTE_READ;
|
46
|
+
else
|
47
|
+
protect = PAGE_EXECUTE;
|
48
|
+
} else
|
49
|
+
protect = PAGE_READONLY;
|
50
|
+
|
51
|
+
size_t end = length + offset;
|
52
|
+
|
53
|
+
#if _WIN64
|
54
|
+
const DWORD dwEndLow = DWORD(end & 0xFFFFFFFFL);
|
55
|
+
const DWORD dwEndHigh = DWORD((end >> 32) & 0xFFFFFFFFL);
|
56
|
+
const DWORD dwOffsetLow = DWORD(offset & 0xFFFFFFFFL);
|
57
|
+
const DWORD dwOffsetHigh = DWORD((offset >> 32) & 0xFFFFFFFFL);
|
58
|
+
#else
|
59
|
+
const DWORD dwEndLow = DWORD(end);
|
60
|
+
const DWORD dwEndHigh = DWORD(0);
|
61
|
+
const DWORD dwOffsetLow = DWORD(offset);
|
62
|
+
const DWORD dwOffsetHigh = DWORD(0);
|
63
|
+
#endif
|
64
|
+
|
65
|
+
HANDLE h = (fd != -1) ? HANDLE(uv_get_osfhandle(fd)) : INVALID_HANDLE_VALUE;
|
66
|
+
HANDLE fm = CreateFileMapping(h, nullptr, protect, dwEndHigh, dwEndLow, name);
|
67
|
+
if (fm == nullptr)
|
68
|
+
return MAP_FAILED;
|
69
|
+
|
70
|
+
DWORD dwDesiredAccess;
|
71
|
+
if (prot & PROT_WRITE)
|
72
|
+
dwDesiredAccess = FILE_MAP_WRITE;
|
73
|
+
else
|
74
|
+
dwDesiredAccess = FILE_MAP_READ;
|
75
|
+
if (prot & PROT_EXEC)
|
76
|
+
dwDesiredAccess |= FILE_MAP_EXECUTE;
|
77
|
+
if (flags & MAP_PRIVATE)
|
78
|
+
dwDesiredAccess |= FILE_MAP_COPY;
|
79
|
+
|
80
|
+
void *map = MapViewOfFile(fm, dwDesiredAccess, dwOffsetHigh, dwOffsetLow, length);
|
81
|
+
CloseHandle(fm);
|
82
|
+
|
83
|
+
return (map != nullptr) ? map : MAP_FAILED;
|
84
|
+
}
|
85
|
+
|
86
|
+
inline int munmap(void *addr, size_t length) {
|
87
|
+
if (UnmapViewOfFile(addr))
|
88
|
+
return 0;
|
89
|
+
|
90
|
+
errno = GetLastError();
|
91
|
+
return -1;
|
92
|
+
}
|
93
|
+
|
94
|
+
inline int msync(void *addr, size_t length, int flags) {
|
95
|
+
if (FlushViewOfFile(addr, length))
|
96
|
+
return 0;
|
97
|
+
|
98
|
+
errno = GetLastError();
|
99
|
+
return -1;
|
100
|
+
}
|
101
|
+
|
102
|
+
inline int madvise(void *addr, size_t length, int advice) {
|
103
|
+
return 0; // Unsupported on Windows
|
104
|
+
}
|
105
|
+
|
106
|
+
inline int mincore(void *addr, size_t length, unsigned char *vec) {
|
107
|
+
errno = ENOSYS;
|
108
|
+
return -1;
|
109
|
+
}
|
package/src/mmap-io.cc
ADDED
@@ -0,0 +1,340 @@
|
|
1
|
+
/*
|
2
|
+
Licensed under The MIT License (MIT)
|
3
|
+
You will find the full license legal mumbo jumbo in file "LICENSE"
|
4
|
+
|
5
|
+
Copyright (c) 2015 - 2018 Oscar Campbell
|
6
|
+
|
7
|
+
Inspired by Ben Noordhuis module node-mmap - which does the same thing for older node
|
8
|
+
versions, sans advise and sync.
|
9
|
+
*/
|
10
|
+
#include <nan.h>
|
11
|
+
#include <errno.h>
|
12
|
+
#include <string>
|
13
|
+
|
14
|
+
#ifdef _WIN32
|
15
|
+
#include <windows.h>
|
16
|
+
#include "mman.h"
|
17
|
+
#else
|
18
|
+
#include <unistd.h>
|
19
|
+
#include <sys/mman.h>
|
20
|
+
#endif
|
21
|
+
|
22
|
+
using namespace v8;
|
23
|
+
|
24
|
+
// Just a bit more clear as to intent
|
25
|
+
#define JS_FN(a) NAN_METHOD(a)
|
26
|
+
|
27
|
+
// This lib is one of those pieces of code where clarity is better then puny
|
28
|
+
// micro opts (in comparison to the massive blocking that will occur when the
|
29
|
+
// data is first read from disk) Since casting `size` to `void*` feels a little
|
30
|
+
// "out there" considering that `void *` may be 32b or 64b (or, I dunno, 47b on
|
31
|
+
// some quant particle system), we throw this struct in.
|
32
|
+
struct MMap {
|
33
|
+
MMap(char *data, size_t size) : data(data), size(size) {}
|
34
|
+
char *data = nullptr;
|
35
|
+
size_t size = 0;
|
36
|
+
};
|
37
|
+
|
38
|
+
void do_mmap_cleanup(char *data, void *hint) {
|
39
|
+
auto map_info = static_cast<MMap*>(hint);
|
40
|
+
munmap(data, map_info->size);
|
41
|
+
delete map_info;
|
42
|
+
}
|
43
|
+
|
44
|
+
inline int do_mmap_advice(char *addr, size_t length, int advise) {
|
45
|
+
return madvise(static_cast<void*>(addr), length, advise);
|
46
|
+
}
|
47
|
+
|
48
|
+
// Make it simpler the next time V8 breaks API's and such with a wrapper fn.
|
49
|
+
template <typename T, typename VT>
|
50
|
+
inline auto get_v(VT v8_value) -> T {
|
51
|
+
return Nan::To<T>(v8_value).FromJust();
|
52
|
+
}
|
53
|
+
|
54
|
+
// Make it simpler the next time V8 breaks API's and such with a wrapper fn.
|
55
|
+
template <typename T, typename VT>
|
56
|
+
inline auto get_v(VT v8_value, T default_value) -> T {
|
57
|
+
return Nan::To<T>(v8_value).FromMaybe(default_value);
|
58
|
+
}
|
59
|
+
|
60
|
+
template <typename VT>
|
61
|
+
inline auto get_obj(VT v8_obj) -> Local<Object> {
|
62
|
+
return Nan::To<Object>(v8_obj).ToLocalChecked();
|
63
|
+
}
|
64
|
+
|
65
|
+
JS_FN(mmap_map) {
|
66
|
+
Nan::HandleScope();
|
67
|
+
|
68
|
+
if (info.Length() < 4 && info.Length() > 7) {
|
69
|
+
return Nan::ThrowError(
|
70
|
+
"map() takes 4, 5, 6 or 7 arguments: (size :int, protection :int, flags :int, fd :int [, offset :int [, advise :int [, name :string ]])."
|
71
|
+
);
|
72
|
+
}
|
73
|
+
|
74
|
+
if (!info[0]->IsNumber()) {
|
75
|
+
return Nan::ThrowError("mmap: size (arg[0]) must be an integer");
|
76
|
+
}
|
77
|
+
if (!info[1]->IsNumber()) {
|
78
|
+
return Nan::ThrowError("mmap: protection_flags (arg[1]) must be an integer");
|
79
|
+
}
|
80
|
+
if (!info[2]->IsNumber()) {
|
81
|
+
return Nan::ThrowError("mmap: flags (arg[2]) must be an integer");
|
82
|
+
}
|
83
|
+
if (!info[3]->IsNumber()) {
|
84
|
+
return Nan::ThrowError("mmap: fd (arg[3]) must be an integer (a file descriptor)");
|
85
|
+
}
|
86
|
+
// offset and advise are optional.
|
87
|
+
|
88
|
+
constexpr void *hinted_address = nullptr;
|
89
|
+
const size_t size = static_cast<size_t>(get_v<int64_t>(info[0]));
|
90
|
+
const int protection = get_v<int>(info[1]);
|
91
|
+
const int flags = get_v<int>(info[2]);
|
92
|
+
const int fd = get_v<int>(info[3]);
|
93
|
+
const size_t offset = static_cast<size_t>(get_v<int64_t>(info[4], 0));
|
94
|
+
const int advise = get_v<int>(info[5], 0);
|
95
|
+
|
96
|
+
#ifdef _WIN32
|
97
|
+
char* nameData = nullptr;
|
98
|
+
|
99
|
+
if (info.Length() > 6) {
|
100
|
+
Local<Object> nameBuf = get_obj(info[6]);
|
101
|
+
nameData = node::Buffer::Data(nameBuf);
|
102
|
+
}
|
103
|
+
|
104
|
+
char* data = static_cast<char*>( mmap( hinted_address, size, protection, flags, fd, offset, nameData) );
|
105
|
+
#else
|
106
|
+
char* data = static_cast<char*>( mmap( hinted_address, size, protection, flags, fd, offset) );
|
107
|
+
#endif
|
108
|
+
|
109
|
+
if (data == MAP_FAILED) {
|
110
|
+
return Nan::ThrowError((std::string("mmap failed, ") + std::to_string(errno)).c_str());
|
111
|
+
}
|
112
|
+
else {
|
113
|
+
if (advise != 0) {
|
114
|
+
auto ret = do_mmap_advice(data, size, advise);
|
115
|
+
if (ret) {
|
116
|
+
return Nan::ThrowError((std::string("madvise() failed, ") + std::to_string(errno)).c_str());
|
117
|
+
}
|
118
|
+
}
|
119
|
+
|
120
|
+
auto map_info = new MMap(data, size);
|
121
|
+
Nan::MaybeLocal<Object> buf = node::Buffer::New(
|
122
|
+
v8::Isolate::GetCurrent(), data, size, do_mmap_cleanup, static_cast<void*>(map_info));
|
123
|
+
if (buf.IsEmpty()) {
|
124
|
+
return Nan::ThrowError(std::string("couldn't allocate Node Buffer()").c_str());
|
125
|
+
} else {
|
126
|
+
info.GetReturnValue().Set(buf.ToLocalChecked());
|
127
|
+
}
|
128
|
+
}
|
129
|
+
}
|
130
|
+
|
131
|
+
JS_FN(mmap_advise) {
|
132
|
+
Nan::HandleScope();
|
133
|
+
|
134
|
+
if (info.Length() != 2 && info.Length() != 4) {
|
135
|
+
return Nan::ThrowError(
|
136
|
+
"advise() takes 2 or 4 arguments: (buffer :Buffer, advise :int) | (buffer :Buffer, offset :int, length :int, advise :int)."
|
137
|
+
);
|
138
|
+
}
|
139
|
+
if (!info[0]->IsObject()) return Nan::ThrowError("advice(): buffer (arg[0]) must be a Buffer");
|
140
|
+
if (!info[1]->IsNumber()) return Nan::ThrowError("advice(): (arg[1]) must be an integer");
|
141
|
+
|
142
|
+
Local<Object> buf = get_obj(info[0]); // info[0]->ToObject(); // get_v<Local<Object>>(info[0]);
|
143
|
+
char *data = node::Buffer::Data(buf);
|
144
|
+
size_t size = node::Buffer::Length(buf);
|
145
|
+
|
146
|
+
int ret = ([&]() -> int {
|
147
|
+
if (info.Length() == 2) {
|
148
|
+
int advise = get_v<int>(info[1], 0);
|
149
|
+
return do_mmap_advice(data, size, advise);
|
150
|
+
}
|
151
|
+
else {
|
152
|
+
size_t offset = get_v<int64_t>(info[1], 0);
|
153
|
+
size_t length = get_v<int64_t>(info[2], 0);
|
154
|
+
int advise = get_v<int>(info[3], 0);
|
155
|
+
return do_mmap_advice(data + offset, length, advise);
|
156
|
+
}
|
157
|
+
})();
|
158
|
+
|
159
|
+
if (ret) {
|
160
|
+
return Nan::ThrowError((std::string("madvise() failed, ") + std::to_string(errno)).c_str());
|
161
|
+
}
|
162
|
+
|
163
|
+
//Nan::ReturnUndefined();
|
164
|
+
}
|
165
|
+
|
166
|
+
JS_FN(mmap_incore) {
|
167
|
+
Nan::HandleScope();
|
168
|
+
|
169
|
+
if (info.Length() != 1) {
|
170
|
+
return Nan::ThrowError(
|
171
|
+
"incore() takes 1 argument: (buffer :Buffer) ."
|
172
|
+
);
|
173
|
+
}
|
174
|
+
|
175
|
+
if (!info[0]->IsObject()) return Nan::ThrowError("advice(): buffer (arg[0]) must be a Buffer");
|
176
|
+
|
177
|
+
Local<Object> buf = get_obj(info[0]); // info[0]->ToObject(); // get_v<Local<Object>>(info[0]);
|
178
|
+
char *data = node::Buffer::Data(buf);
|
179
|
+
size_t size = node::Buffer::Length(buf);
|
180
|
+
|
181
|
+
#ifdef _WIN32
|
182
|
+
SYSTEM_INFO sysinfo;
|
183
|
+
GetSystemInfo(&sysinfo);
|
184
|
+
size_t page_size = sysinfo.dwPageSize;
|
185
|
+
#else
|
186
|
+
size_t page_size = sysconf(_SC_PAGESIZE);
|
187
|
+
#endif
|
188
|
+
|
189
|
+
size_t needed_bytes = (size+page_size-1) / page_size;
|
190
|
+
size_t pages = size / page_size;
|
191
|
+
|
192
|
+
#ifdef __APPLE__
|
193
|
+
char *result_data = static_cast<char *>(malloc(needed_bytes));
|
194
|
+
#else
|
195
|
+
unsigned char *result_data = static_cast<unsigned char *>(malloc(needed_bytes));
|
196
|
+
#endif
|
197
|
+
|
198
|
+
if (size % page_size > 0) {
|
199
|
+
pages++;
|
200
|
+
}
|
201
|
+
|
202
|
+
int ret = mincore(data, size, result_data);
|
203
|
+
|
204
|
+
if (ret) {
|
205
|
+
free(result_data);
|
206
|
+
if (errno == ENOSYS) {
|
207
|
+
return Nan::ThrowError("mincore() not implemented");
|
208
|
+
} else {
|
209
|
+
return Nan::ThrowError((std::string("mincore() failed, ") + std::to_string(errno)).c_str());
|
210
|
+
}
|
211
|
+
}
|
212
|
+
|
213
|
+
// Now we want to check all of the pages
|
214
|
+
uint32_t pages_mapped = 0;
|
215
|
+
uint32_t pages_unmapped = 0;
|
216
|
+
|
217
|
+
for(size_t i = 0; i < pages; i++) {
|
218
|
+
if(!(result_data[i] & 0x1)) {
|
219
|
+
pages_unmapped++;
|
220
|
+
} else {
|
221
|
+
pages_mapped++;
|
222
|
+
}
|
223
|
+
}
|
224
|
+
|
225
|
+
free(result_data);
|
226
|
+
|
227
|
+
v8::Local<v8::Array> arr = Nan::New<v8::Array>(2);
|
228
|
+
Nan::Set(arr, 0, Nan::New(pages_unmapped));
|
229
|
+
Nan::Set(arr, 1, Nan::New(pages_mapped));
|
230
|
+
info.GetReturnValue().Set(arr);
|
231
|
+
}
|
232
|
+
|
233
|
+
JS_FN(mmap_sync_lib_private_) {
|
234
|
+
Nan::HandleScope();
|
235
|
+
|
236
|
+
// I barfed at the thought of implementing all variants of info-combos in
|
237
|
+
// C++, so the arg-shuffling and checking is done in a ES wrapper function
|
238
|
+
// - see "mmap-io.ts"
|
239
|
+
if (info.Length() != 5) {
|
240
|
+
return Nan::ThrowError(
|
241
|
+
"sync() takes 5 arguments: (buffer :Buffer, offset :int, length :int, do_blocking_sync :bool, invalidate_pages_and_signal_refresh_to_consumers :bool)."
|
242
|
+
);
|
243
|
+
}
|
244
|
+
|
245
|
+
if (!info[0]->IsObject()) return Nan::ThrowError("sync(): buffer (arg[0]) must be a Buffer");
|
246
|
+
|
247
|
+
Local<Object> buf = get_obj(info[0]); // info[0]->ToObject(); // get_v<Local<Object>>(info[0]);
|
248
|
+
char *data = node::Buffer::Data(buf);
|
249
|
+
|
250
|
+
size_t offset = get_v<int64_t>(info[1], 0);
|
251
|
+
size_t length = get_v<int64_t>(info[2], 0);
|
252
|
+
bool blocking_sync = get_v<bool>(info[3], false);
|
253
|
+
bool invalidate = get_v<bool>(info[4], false);
|
254
|
+
int flags = ( (blocking_sync ? MS_SYNC : MS_ASYNC) | (invalidate ? MS_INVALIDATE : 0) );
|
255
|
+
|
256
|
+
int ret = msync(data + offset, length, flags);
|
257
|
+
|
258
|
+
if (ret) {
|
259
|
+
return Nan::ThrowError((std::string("msync() failed, ") + std::to_string(errno)).c_str());
|
260
|
+
}
|
261
|
+
//Nan::ReturnUndefined();
|
262
|
+
}
|
263
|
+
|
264
|
+
NAN_MODULE_INIT(Init) {
|
265
|
+
auto exports = target;
|
266
|
+
|
267
|
+
constexpr auto std_property_attrs = static_cast<PropertyAttribute>(
|
268
|
+
ReadOnly | DontDelete
|
269
|
+
);
|
270
|
+
|
271
|
+
using JsFnType = decltype(mmap_map);
|
272
|
+
|
273
|
+
auto set_int_prop = [&](const char *key, int val) -> void {
|
274
|
+
Nan::DefineOwnProperty(
|
275
|
+
exports,
|
276
|
+
Nan::New(key).ToLocalChecked(),
|
277
|
+
Nan::New(val),
|
278
|
+
std_property_attrs
|
279
|
+
);
|
280
|
+
};
|
281
|
+
|
282
|
+
auto set_fn_prop = [&](const char *key, JsFnType fn) -> void {
|
283
|
+
Nan::DefineOwnProperty(
|
284
|
+
exports,
|
285
|
+
Nan::New<v8::String>(key).ToLocalChecked(),
|
286
|
+
Nan::GetFunction(Nan::New<FunctionTemplate>(fn)).ToLocalChecked(),
|
287
|
+
std_property_attrs
|
288
|
+
);
|
289
|
+
};
|
290
|
+
|
291
|
+
set_int_prop("PROT_READ", PROT_READ);
|
292
|
+
set_int_prop("PROT_WRITE", PROT_WRITE);
|
293
|
+
set_int_prop("PROT_EXEC", PROT_EXEC);
|
294
|
+
set_int_prop("PROT_NONE", PROT_NONE);
|
295
|
+
|
296
|
+
set_int_prop("MAP_SHARED", MAP_SHARED);
|
297
|
+
set_int_prop("MAP_ANONYMOUS", MAP_ANONYMOUS);
|
298
|
+
set_int_prop("MAP_PRIVATE", MAP_PRIVATE);
|
299
|
+
|
300
|
+
#ifdef MAP_NONBLOCK
|
301
|
+
set_int_prop("MAP_NONBLOCK", MAP_NONBLOCK);
|
302
|
+
#endif
|
303
|
+
|
304
|
+
#ifdef MAP_POPULATE
|
305
|
+
set_int_prop("MAP_POPULATE", MAP_POPULATE);
|
306
|
+
#endif
|
307
|
+
|
308
|
+
set_int_prop("MADV_NORMAL", MADV_NORMAL);
|
309
|
+
set_int_prop("MADV_RANDOM", MADV_RANDOM);
|
310
|
+
set_int_prop("MADV_SEQUENTIAL", MADV_SEQUENTIAL);
|
311
|
+
set_int_prop("MADV_WILLNEED", MADV_WILLNEED);
|
312
|
+
set_int_prop("MADV_DONTNEED", MADV_DONTNEED);
|
313
|
+
|
314
|
+
//set_int_prop("MS_ASYNC", MS_ASYNC);
|
315
|
+
//set_int_prop("MS_SYNC", MS_SYNC);
|
316
|
+
//set_int_prop("MS_INVALIDATE", MS_INVALIDATE);
|
317
|
+
|
318
|
+
#ifdef _WIN32
|
319
|
+
SYSTEM_INFO sysinfo;
|
320
|
+
GetSystemInfo(&sysinfo);
|
321
|
+
set_int_prop("PAGESIZE", sysinfo.dwPageSize);
|
322
|
+
#else
|
323
|
+
set_int_prop("PAGESIZE", sysconf(_SC_PAGESIZE));
|
324
|
+
#endif
|
325
|
+
|
326
|
+
set_fn_prop("map", mmap_map);
|
327
|
+
set_fn_prop("advise", mmap_advise);
|
328
|
+
set_fn_prop("incore", mmap_incore);
|
329
|
+
|
330
|
+
// This one is wrapped by a JS-function and deleted from obj to hide from user
|
331
|
+
Nan::DefineOwnProperty(
|
332
|
+
exports,
|
333
|
+
Nan::New<v8::String>("sync_lib_private__").ToLocalChecked(),
|
334
|
+
Nan::GetFunction(Nan::New<FunctionTemplate>(mmap_sync_lib_private_)).ToLocalChecked(),
|
335
|
+
static_cast<PropertyAttribute>(0)
|
336
|
+
);
|
337
|
+
|
338
|
+
}
|
339
|
+
|
340
|
+
NAN_MODULE_WORKER_ENABLED(mmap_io, Init);
|
package/src/mmap-io.ts
ADDED
@@ -0,0 +1,132 @@
|
|
1
|
+
const binary = require('@mapbox/node-pre-gyp');
|
2
|
+
const path = require('path');
|
3
|
+
const binding_path = binary.find(path.resolve(path.join(__dirname,'../package.json')));
|
4
|
+
const mmap_lib_raw_ = require(binding_path);
|
5
|
+
|
6
|
+
type FileDescriptor = number
|
7
|
+
|
8
|
+
type MapProtectionFlags =
|
9
|
+
| MmapIo["PROT_NONE"] // 0
|
10
|
+
| MmapIo["PROT_READ"] // 1
|
11
|
+
| MmapIo["PROT_WRITE"] // 2
|
12
|
+
| MmapIo["PROT_EXEC"] // 4
|
13
|
+
| 3 // R+W
|
14
|
+
| 5 // R+X
|
15
|
+
| 6 // W+X
|
16
|
+
| 7 // R+W+X
|
17
|
+
|
18
|
+
// making `map` a wrapper around the C++ `map`-implementation and allowing an
|
19
|
+
// array of flags would be clean, perfectly literally typed, and the dirtier
|
20
|
+
// binary-or can be done in the wrapper.
|
21
|
+
//
|
22
|
+
// type MapProtectionFlagsList = Array<
|
23
|
+
// | MmapIo["PROT_NONE"]
|
24
|
+
// | MmapIo["PROT_READ"]
|
25
|
+
// | MmapIo["PROT_WRITE"]
|
26
|
+
// | MmapIo["PROT_EXEC"]
|
27
|
+
// >
|
28
|
+
|
29
|
+
type MapFlags =
|
30
|
+
| MmapIo["MAP_SHARED"]
|
31
|
+
| MmapIo["MAP_PRIVATE"]
|
32
|
+
| MmapIo["MAP_ANONYMOUS"]
|
33
|
+
| MmapIo["MAP_POPULATE"]
|
34
|
+
| MmapIo["MAP_NONBLOCK"]
|
35
|
+
| number
|
36
|
+
|
37
|
+
type MapAdvise =
|
38
|
+
| MmapIo["MADV_NORMAL"]
|
39
|
+
| MmapIo["MADV_RANDOM"]
|
40
|
+
| MmapIo["MADV_SEQUENTIAL"]
|
41
|
+
| MmapIo["MADV_WILLNEED"]
|
42
|
+
| MmapIo["MADV_DONTNEED"]
|
43
|
+
|
44
|
+
type MmapIo = {
|
45
|
+
map(
|
46
|
+
size: number,
|
47
|
+
protection: MapProtectionFlags,
|
48
|
+
flags: MapFlags,
|
49
|
+
fd: FileDescriptor,
|
50
|
+
offset?: number,
|
51
|
+
advise?: MapAdvise,
|
52
|
+
name?: Buffer
|
53
|
+
): Buffer
|
54
|
+
|
55
|
+
advise(
|
56
|
+
buffer: Buffer,
|
57
|
+
offset: number,
|
58
|
+
length: number,
|
59
|
+
advise: MapAdvise
|
60
|
+
): void
|
61
|
+
advise(buffer: Buffer, advise: MapAdvise): void
|
62
|
+
|
63
|
+
/// Returns tuple of [ unmapped-pages-count, mapped-pages-count ]
|
64
|
+
incore(buffer: Buffer): [number, number]
|
65
|
+
|
66
|
+
sync(
|
67
|
+
buffer: Buffer,
|
68
|
+
offset?: number,
|
69
|
+
size?: number,
|
70
|
+
blocking_sync?: boolean,
|
71
|
+
invalidate_pages?: boolean
|
72
|
+
): void
|
73
|
+
|
74
|
+
sync(
|
75
|
+
buffer: Buffer,
|
76
|
+
blocking_sync: boolean,
|
77
|
+
invalidate_pages?: boolean
|
78
|
+
): void
|
79
|
+
|
80
|
+
readonly PROT_READ: 1
|
81
|
+
readonly PROT_WRITE: 2
|
82
|
+
readonly PROT_EXEC: 4
|
83
|
+
readonly PROT_NONE: 0
|
84
|
+
|
85
|
+
readonly MAP_SHARED: 1
|
86
|
+
readonly MAP_PRIVATE: 2
|
87
|
+
readonly MAP_ANONYMOUS: 32
|
88
|
+
readonly MAP_POPULATE: 32768
|
89
|
+
readonly MAP_NONBLOCK: 65536
|
90
|
+
|
91
|
+
readonly MADV_NORMAL: 0
|
92
|
+
readonly MADV_RANDOM: 1
|
93
|
+
readonly MADV_SEQUENTIAL: 2
|
94
|
+
readonly MADV_WILLNEED: 3
|
95
|
+
readonly MADV_DONTNEED: 4
|
96
|
+
|
97
|
+
readonly PAGESIZE: number
|
98
|
+
}
|
99
|
+
|
100
|
+
// snatch the raw C++-sync func
|
101
|
+
const raw_sync_fn_ = mmap_lib_raw_.sync_lib_private__
|
102
|
+
|
103
|
+
// Hide the original C++11 func from users
|
104
|
+
delete mmap_lib_raw_.sync_lib_private__
|
105
|
+
|
106
|
+
// Take care of all the param juggling here instead of in C++ code, by making
|
107
|
+
// some overloads, and doing some argument defaults
|
108
|
+
mmap_lib_raw_.sync = function(
|
109
|
+
buf: Buffer,
|
110
|
+
par_a?: any,
|
111
|
+
par_b?: any,
|
112
|
+
par_c?: any,
|
113
|
+
par_d?: any
|
114
|
+
): void {
|
115
|
+
if (typeof par_a === "boolean") {
|
116
|
+
raw_sync_fn_(buf, 0, buf.length, par_a, par_b || false)
|
117
|
+
} else {
|
118
|
+
raw_sync_fn_(
|
119
|
+
buf,
|
120
|
+
par_a || 0,
|
121
|
+
par_b || buf.length,
|
122
|
+
par_c || false,
|
123
|
+
par_d || false
|
124
|
+
)
|
125
|
+
}
|
126
|
+
}
|
127
|
+
|
128
|
+
// mmap_lib_raw_.sync = sync_
|
129
|
+
|
130
|
+
const mmap = mmap_lib_raw_ as MmapIo
|
131
|
+
module.exports = mmap
|
132
|
+
export default mmap
|
@@ -0,0 +1,22 @@
|
|
1
|
+
import mmap from "./mmap-io"
|
2
|
+
import fs from "fs"
|
3
|
+
|
4
|
+
const fd = fs.openSync(__filename, "r+")
|
5
|
+
|
6
|
+
/**
|
7
|
+
* Writing, for instance `4` instead of ie `2` (MADV_SEQUENTIAL), here, will
|
8
|
+
* give an error already while typing (provided language-server is setup)
|
9
|
+
* because of the specific types
|
10
|
+
*
|
11
|
+
* A problem arise with the traditional dirty technique of binary-or, widening
|
12
|
+
* the type from literal to number-field
|
13
|
+
*/
|
14
|
+
|
15
|
+
mmap.map(
|
16
|
+
100,
|
17
|
+
(mmap.PROT_EXEC | mmap.PROT_READ) as 5,
|
18
|
+
mmap.MAP_SHARED,
|
19
|
+
fd,
|
20
|
+
0,
|
21
|
+
mmap.MADV_SEQUENTIAL
|
22
|
+
)
|