@shieldfive/cli 0.2.1 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +65 -0
- package/README.md +113 -9
- package/SECURITY.md +9 -5
- package/package.json +1 -1
- package/src/agent/client.mjs +102 -0
- package/src/agent/ledger.mjs +254 -0
- package/src/agent/main.mjs +72 -0
- package/src/agent/paths.mjs +132 -0
- package/src/agent/protocol.mjs +125 -0
- package/src/agent/server.mjs +525 -0
- package/src/auth.mjs +79 -1
- package/src/cli.mjs +365 -36
- package/src/prompt.mjs +79 -0
- package/src/upload.mjs +11 -5
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,71 @@ All notable changes to `@shieldfive/cli` will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## Unreleased
|
|
9
|
+
|
|
10
|
+
## 0.3.0 - 2026-09-16
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
|
|
14
|
+
- **`sf login`, `sf status`, `sf logout`.** `sf login` prompts for your password
|
|
15
|
+
and, if two-factor is enabled, a code, then starts an agent that holds the
|
|
16
|
+
signed-in session and the unlocked vault key in memory only. `sf push` and
|
|
17
|
+
`sf sync` use it when it is running and need nothing in the environment. The
|
|
18
|
+
agent locks itself after 8 hours without use (`--idle=<hours>` to change it),
|
|
19
|
+
and `sf logout` revokes the session on the server.
|
|
20
|
+
- **`sf verify <file>...`** reports whether each file is safely stored in your
|
|
21
|
+
vault. The bytes on disk now must match an upload the agent performed, and the
|
|
22
|
+
server must confirm that upload at the moment of asking. Names and sizes play
|
|
23
|
+
no part. Exits 0 only when every file is confirmed, so a script can gate on it.
|
|
24
|
+
- **Unattended sync on two-factor accounts.** One interactive `sf login` is
|
|
25
|
+
enough: the agent refreshes the session without prompting again until it locks
|
|
26
|
+
or you log out.
|
|
27
|
+
|
|
28
|
+
- **`sf logout --everywhere`** signs out every session on the account, including
|
|
29
|
+
the browser and phone.
|
|
30
|
+
|
|
31
|
+
### Security
|
|
32
|
+
|
|
33
|
+
- The agent's socket accepts six operations: `status`, `upload`, `sync`,
|
|
34
|
+
`verify`, `lock`, `logout`. None lists, downloads or decrypts vault contents,
|
|
35
|
+
or returns key material, and a test pins the list.
|
|
36
|
+
- `sf login` never reads `SF_PASSWORD`. The agent is started with an explicit
|
|
37
|
+
environment allowlist, so a password exported for the older flow does not
|
|
38
|
+
reach it.
|
|
39
|
+
- Uploads are recorded in `~/.shieldfive/ledger/<account>.jsonl` with an HMAC of
|
|
40
|
+
each file's contents under a key derived from the vault key rather than a
|
|
41
|
+
plain hash, so the ledger cannot confirm which documents you hold to anyone
|
|
42
|
+
without that key.
|
|
43
|
+
- Hardened before release after a security review:
|
|
44
|
+
- The ledger records the MAC of the bytes actually encrypted, read in the
|
|
45
|
+
same pass as the upload. Hashing the file before and after could not see a
|
|
46
|
+
change made and undone mid-upload, and `sf verify` would then have called a
|
|
47
|
+
file backed up that the vault did not hold.
|
|
48
|
+
- Ledger records carry a tag under the vault-derived key and a device id;
|
|
49
|
+
forged lines, and lines copied from another machine, are ignored.
|
|
50
|
+
- `SF_EMAIL` and `SF_PASSWORD` take precedence over a running agent, so a
|
|
51
|
+
script cannot be redirected to a different account.
|
|
52
|
+
- A lock file stops two agents from starting at once, and an agent that is
|
|
53
|
+
still revoking its session cannot delete the socket of the one that replaced
|
|
54
|
+
it.
|
|
55
|
+
- `sf status` no longer postpones the idle lock.
|
|
56
|
+
- Stopping wipes keys before touching the network, and its token refresh and
|
|
57
|
+
revoke share a 5 second budget, so an unreachable server cannot keep the
|
|
58
|
+
agent alive. An upload already running keeps its own copy of the key.
|
|
59
|
+
- The socket lives in `~/.shieldfive/run` everywhere, so cron jobs and other
|
|
60
|
+
minimal environments find the agent.
|
|
61
|
+
|
|
62
|
+
### Notes
|
|
63
|
+
|
|
64
|
+
- `sf login` has not been run against production. The pieces are covered by
|
|
65
|
+
offline tests, including one that runs the agent as its own process against a
|
|
66
|
+
local stub of the auth and verify endpoints. Sign-in, unlock and session
|
|
67
|
+
refresh against the live backend are not.
|
|
68
|
+
- The agent does not run on Windows. `SF_EMAIL` and `SF_PASSWORD` keep working
|
|
69
|
+
there, and everywhere, exactly as before.
|
|
70
|
+
- The ledger starts empty. Existing `.shieldfive-sync.json` manifests are not
|
|
71
|
+
imported: they hold no content hash to check a claim against.
|
|
72
|
+
|
|
8
73
|
## 0.2.1 - 2026-09-14
|
|
9
74
|
|
|
10
75
|
### Fixed
|
package/README.md
CHANGED
|
@@ -41,6 +41,11 @@ size, uploading nothing.
|
|
|
41
41
|
large file.
|
|
42
42
|
- `sf sync` is **append-only**: it uploads new and changed files; it does not
|
|
43
43
|
yet mirror local deletions or renames into the vault.
|
|
44
|
+
- **`sf login` and the agent have not been run against production.** They are
|
|
45
|
+
covered by offline tests, including one that runs the agent as its own process
|
|
46
|
+
and reaches it from separate `sf` processes against a local stub of the auth
|
|
47
|
+
and verify endpoints. Sign-in, unlock and session refresh against the live
|
|
48
|
+
backend are not covered.
|
|
44
49
|
|
|
45
50
|
## Requirements
|
|
46
51
|
|
|
@@ -66,27 +71,58 @@ trust it):
|
|
|
66
71
|
git clone https://github.com/shieldfive/cli.git
|
|
67
72
|
cd cli
|
|
68
73
|
npm install # @shieldfive/crypto (+ libsodium), @supabase/supabase-js, @noble/*
|
|
69
|
-
npm test #
|
|
74
|
+
npm test # 107 tests
|
|
70
75
|
```
|
|
71
76
|
|
|
72
77
|
## Commands
|
|
73
78
|
|
|
74
79
|
```
|
|
75
|
-
sf
|
|
76
|
-
sf
|
|
80
|
+
sf login [--idle=<hours>] sign in once; an agent holds the session in memory (locks after 8 h idle)
|
|
81
|
+
sf status is an agent running, and for which account
|
|
82
|
+
sf logout [--everywhere] revoke the session and stop the agent; --everywhere signs out every device
|
|
83
|
+
sf push <folder> encrypt and upload every file once
|
|
77
84
|
sf sync <folder> [--watch] [--interval=N] upload new/changed files; --watch keeps a poll loop (N seconds, default 5)
|
|
85
|
+
sf verify <file>... report whether each file is safely stored in your vault
|
|
86
|
+
sf encrypt <folder> encrypt each file locally, upload nothing (no account needed)
|
|
78
87
|
```
|
|
79
88
|
|
|
80
89
|
`sf push` and `sf sync` pick the direct or multipart upload path automatically
|
|
81
|
-
from the file size.
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
90
|
+
from the file size.
|
|
91
|
+
|
|
92
|
+
With an agent running, `sf sync` records uploads in the agent's ledger (see
|
|
93
|
+
[What the agent can and cannot do](#what-the-agent-can-and-cannot-do)) and
|
|
94
|
+
`sf verify` works. Without one, `sf sync` writes a `.shieldfive-sync.json`
|
|
95
|
+
manifest in the target folder that records what has already been uploaded (by
|
|
96
|
+
size and mtime), so unchanged files are skipped and an interrupted run does not
|
|
97
|
+
re-upload everything; `sf verify` needs the agent.
|
|
85
98
|
|
|
86
99
|
## Running the live commands
|
|
87
100
|
|
|
88
|
-
|
|
89
|
-
|
|
101
|
+
### With `sf login`
|
|
102
|
+
|
|
103
|
+
```sh
|
|
104
|
+
sf login
|
|
105
|
+
sf push ./my-folder
|
|
106
|
+
sf sync ./my-folder --watch
|
|
107
|
+
sf verify ./my-folder/tax-return-2025.pdf
|
|
108
|
+
sf logout
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
`sf login` prompts for your email, your password and, if it differs, your vault
|
|
112
|
+
password. It never reads them from the environment and there is no flag for
|
|
113
|
+
them. It then starts an agent in the background and exits; every later command
|
|
114
|
+
talks to that agent and needs nothing exported.
|
|
115
|
+
|
|
116
|
+
If `SF_EMAIL` and `SF_PASSWORD` are also set, `sf push` and `sf sync` sign in
|
|
117
|
+
with those instead and tell you an agent is running, so a script written for one
|
|
118
|
+
account is never sent to another. `sf push` exits 2 if a file changed while it
|
|
119
|
+
was being uploaded: the vault then holds the version that was read, not the file
|
|
120
|
+
as it is now.
|
|
121
|
+
|
|
122
|
+
### Without an agent
|
|
123
|
+
|
|
124
|
+
`sf push` and `sf sync` still work from environment variables, exactly as
|
|
125
|
+
before. Your password is typed, never passed on the command line:
|
|
90
126
|
|
|
91
127
|
```sh
|
|
92
128
|
export SF_EMAIL=you@example.com
|
|
@@ -116,6 +152,68 @@ Enter the current code from your authenticator. In a non-interactive context
|
|
|
116
152
|
(scripts, CI) set `SF_TOTP_CODE` to the current code instead of being prompted.
|
|
117
153
|
Only TOTP is supported; SMS and other factors are not.
|
|
118
154
|
|
|
155
|
+
With `sf login` you are asked once. The agent refreshes the stepped-up session
|
|
156
|
+
on its own, so `sf sync --watch` keeps working on a two-factor account until the
|
|
157
|
+
agent locks or you log out.
|
|
158
|
+
|
|
159
|
+
## What the agent can and cannot do
|
|
160
|
+
|
|
161
|
+
The agent keeps two things in memory: your signed-in session and your unlocked
|
|
162
|
+
vault key. Neither is written to disk. It listens on a Unix socket in
|
|
163
|
+
`~/.shieldfive/run`, a directory only your user can enter, and refuses to start
|
|
164
|
+
if that directory is a symlink, owned by someone else or open to other users.
|
|
165
|
+
Only one agent runs at a time.
|
|
166
|
+
|
|
167
|
+
It accepts six requests: `status`, `upload`, `sync`, `verify`, `lock` and
|
|
168
|
+
`logout`. It cannot list what is in your vault, download a file, decrypt
|
|
169
|
+
anything, or hand out the key, because there is no request for any of those. A
|
|
170
|
+
process that reaches the socket can put files into your vault and ask whether
|
|
171
|
+
files are backed up. It cannot read your vault through the agent.
|
|
172
|
+
|
|
173
|
+
Uploads are recorded in `~/.shieldfive/ledger/<account>.jsonl`: the path, size,
|
|
174
|
+
time, vault file id, and an HMAC of the bytes that were actually encrypted, under
|
|
175
|
+
a key derived from your vault key. A plain hash would let anyone holding the
|
|
176
|
+
ledger confirm whether you have a particular known document; the HMAC does not,
|
|
177
|
+
without your vault key. Each record also carries a tag under that key, so a line
|
|
178
|
+
added to the ledger by anything other than the agent is ignored, and records
|
|
179
|
+
made on another machine do not count on this one.
|
|
180
|
+
|
|
181
|
+
`sf verify` reports a file as **backed up** only when the bytes on disk now
|
|
182
|
+
match an upload in the ledger and the server confirms, at that moment, that the
|
|
183
|
+
upload is stored, complete, intact and not in your Bin. It never decides by name
|
|
184
|
+
or size. The other answers:
|
|
185
|
+
|
|
186
|
+
| Answer | Meaning |
|
|
187
|
+
|---|---|
|
|
188
|
+
| `NOT confirmed` | these bytes were uploaded, but the server cannot confirm they are still safely stored |
|
|
189
|
+
| `changed since upload` | this path was uploaded, but the file has changed since |
|
|
190
|
+
| `no upload record` | this agent never uploaded these bytes; they may still be in your vault from another device |
|
|
191
|
+
| `unreadable` | the local file could not be read |
|
|
192
|
+
|
|
193
|
+
`no upload record` does not mean the file is missing from your vault. Do not
|
|
194
|
+
delete anything on the strength of that answer.
|
|
195
|
+
|
|
196
|
+
The agent locks itself after 8 hours without an upload, sync or verify
|
|
197
|
+
(`sf login --idle=<hours>` to change it; `sf status` does not count). Locking and
|
|
198
|
+
`sf logout` both overwrite the key in memory first and then sign the session out
|
|
199
|
+
on the server. Sessions on ShieldFive do not expire on their own, so a session
|
|
200
|
+
that is forgotten rather than signed out would stay valid. If the server cannot
|
|
201
|
+
be reached when you log out, `sf logout` says the session is still valid and
|
|
202
|
+
exits 1; run `sf login` and then `sf logout --everywhere` once you are online.
|
|
203
|
+
|
|
204
|
+
Limits:
|
|
205
|
+
|
|
206
|
+
- Windows is not supported. Use `SF_EMAIL` and `SF_PASSWORD` there.
|
|
207
|
+
- Node cannot see which process connected to the socket, so directory
|
|
208
|
+
permissions are the only gate. This is also how `ssh-agent` works on macOS.
|
|
209
|
+
- Overwriting the key in memory is best-effort. A garbage-collected runtime can
|
|
210
|
+
copy key material during crypto operations.
|
|
211
|
+
- The ledger is per device. A file uploaded from another machine shows as
|
|
212
|
+
`no upload record` here.
|
|
213
|
+
|
|
214
|
+
The full design, including the threat model, is in
|
|
215
|
+
[`docs/agent-design.md`](docs/agent-design.md).
|
|
216
|
+
|
|
119
217
|
## How the upload works (for auditors)
|
|
120
218
|
|
|
121
219
|
The upload uses **chunked AES-GCM** for file content and issues a
|
|
@@ -167,6 +265,12 @@ file — a machine-checked demonstration that only ciphertext is uploaded.
|
|
|
167
265
|
finalize; the streaming chunk reader
|
|
168
266
|
- `src/sync.mjs` — `sf sync`: manifest, change detection, reconcile pass, watch
|
|
169
267
|
loop (reuses `upload.mjs`)
|
|
268
|
+
- `src/prompt.mjs` — terminal prompts; the password prompt echoes nothing
|
|
269
|
+
- `src/agent/` — `sf login`'s agent: `server.mjs` (the process and its six
|
|
270
|
+
requests), `client.mjs`, `protocol.mjs` (the request allowlist), `ledger.mjs`
|
|
271
|
+
(HMAC upload ledger and `verify` classification), `paths.mjs` (socket
|
|
272
|
+
directory checks), `main.mjs` (entry point and session handoff)
|
|
273
|
+
- `docs/agent-design.md` — design record and threat model for the agent
|
|
170
274
|
- `src/sfCrypto.mjs` — the post-quantum hybrid encrypt/decrypt used by the
|
|
171
275
|
`sf encrypt` demo
|
|
172
276
|
- `test/*.test.mjs` — round-trip guarantees plus the mock end-to-end upload and
|
package/SECURITY.md
CHANGED
|
@@ -63,9 +63,13 @@ not pursue civil claims or refer law enforcement against researchers who:
|
|
|
63
63
|
|
|
64
64
|
## Bug bounty
|
|
65
65
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
66
|
+
**There is no bug bounty.** ShieldFive ran one until 2026-07; it closed, and
|
|
67
|
+
`/security/bug-bounty` now redirects to the security page. Reports are answered
|
|
68
|
+
on the timelines above, and credited if you want credit. They are not paid.
|
|
69
|
+
|
|
70
|
+
This section previously said the opposite and pointed at that redirected URL.
|
|
71
|
+
It is corrected here rather than removed, so anyone who read the old text finds
|
|
72
|
+
out what changed.
|
|
69
73
|
|
|
70
74
|
## Scope
|
|
71
75
|
|
|
@@ -79,6 +83,6 @@ Out of scope for *this* repository (report elsewhere or not at all):
|
|
|
79
83
|
[`@shieldfive/crypto`](https://github.com/shieldfive/crypto).
|
|
80
84
|
- Vulnerabilities in dependencies (`@noble/*`, `@supabase/supabase-js`) — report
|
|
81
85
|
those upstream.
|
|
82
|
-
- Server-side issues in the ShieldFive backend —
|
|
83
|
-
|
|
86
|
+
- Server-side issues in the ShieldFive backend — same address,
|
|
87
|
+
`security@shieldfive.com`, different codebase.
|
|
84
88
|
- Attacks that require an attacker to already control the user's device.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@shieldfive/cli",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Open-source command-line client for ShieldFive: encrypt a folder on your machine and sync it to your EU-hosted, post-quantum vault. Read the code; it encrypts before upload.",
|
|
6
6
|
"license": "Apache-2.0",
|
|
@@ -0,0 +1,102 @@
|
|
|
1
|
+
// ShieldFive CLI — talk to a running agent.
|
|
2
|
+
|
|
3
|
+
import { connect } from 'node:net'
|
|
4
|
+
|
|
5
|
+
import { socketPath } from './paths.mjs'
|
|
6
|
+
import { encode, lineSplitter } from './protocol.mjs'
|
|
7
|
+
|
|
8
|
+
export class AgentUnavailableError extends Error {
|
|
9
|
+
constructor(message) {
|
|
10
|
+
super(message)
|
|
11
|
+
this.name = 'AgentUnavailableError'
|
|
12
|
+
this.code = 'agent_unavailable'
|
|
13
|
+
}
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
export class AgentRequestError extends Error {
|
|
17
|
+
constructor(code, message) {
|
|
18
|
+
super(message)
|
|
19
|
+
this.name = 'AgentRequestError'
|
|
20
|
+
this.code = code
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
/**
|
|
25
|
+
* Send one request and resolve with the result.
|
|
26
|
+
*
|
|
27
|
+
* `timeoutMs: 0` disables the timeout, which uploads and syncs need: a large
|
|
28
|
+
* file can take longer than any fixed number.
|
|
29
|
+
*/
|
|
30
|
+
export function request(op, args = {}, { path, timeoutMs = 10_000 } = {}) {
|
|
31
|
+
return new Promise((resolve, reject) => {
|
|
32
|
+
// Resolved here rather than as a default parameter, so a platform the agent
|
|
33
|
+
// does not support rejects this promise instead of throwing out of it.
|
|
34
|
+
let socketFile
|
|
35
|
+
try {
|
|
36
|
+
socketFile = path ?? socketPath()
|
|
37
|
+
} catch (err) {
|
|
38
|
+
reject(err)
|
|
39
|
+
return
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
let settled = false
|
|
43
|
+
const socket = connect({ path: socketFile })
|
|
44
|
+
|
|
45
|
+
const settle = (fn, value) => {
|
|
46
|
+
if (settled) return
|
|
47
|
+
settled = true
|
|
48
|
+
clearTimeout(timer)
|
|
49
|
+
socket.destroy()
|
|
50
|
+
fn(value)
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
const timer = timeoutMs
|
|
54
|
+
? setTimeout(
|
|
55
|
+
() => settle(reject, new AgentRequestError('timeout', `Agent did not answer within ${timeoutMs} ms.`)),
|
|
56
|
+
timeoutMs,
|
|
57
|
+
)
|
|
58
|
+
: null
|
|
59
|
+
|
|
60
|
+
socket.on('error', (err) => {
|
|
61
|
+
if (err.code === 'ENOENT' || err.code === 'ECONNREFUSED') {
|
|
62
|
+
settle(reject, new AgentUnavailableError('The ShieldFive agent is not running. Run sf login.'))
|
|
63
|
+
} else {
|
|
64
|
+
settle(reject, err)
|
|
65
|
+
}
|
|
66
|
+
})
|
|
67
|
+
|
|
68
|
+
socket.setEncoding('utf8')
|
|
69
|
+
socket.on(
|
|
70
|
+
'data',
|
|
71
|
+
lineSplitter((line) => {
|
|
72
|
+
let message
|
|
73
|
+
try {
|
|
74
|
+
message = JSON.parse(line)
|
|
75
|
+
} catch {
|
|
76
|
+
return settle(reject, new AgentRequestError('bad_response', 'Agent sent something that is not JSON.'))
|
|
77
|
+
}
|
|
78
|
+
if (message.ok) settle(resolve, message.result)
|
|
79
|
+
else settle(reject, new AgentRequestError(message.error?.code ?? 'error', message.error?.message ?? 'Agent refused.'))
|
|
80
|
+
}),
|
|
81
|
+
)
|
|
82
|
+
socket.on('close', () =>
|
|
83
|
+
settle(reject, new AgentRequestError('closed', 'Agent closed the connection without answering.')),
|
|
84
|
+
)
|
|
85
|
+
|
|
86
|
+
socket.on('connect', () => socket.write(encode({ id: 1, op, args })))
|
|
87
|
+
})
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
/**
|
|
91
|
+
* The agent's status, or null when no agent is listening or the agent cannot
|
|
92
|
+
* run on this platform. Callers fall back to environment credentials on null,
|
|
93
|
+
* so an unsupported platform must answer null, not throw.
|
|
94
|
+
*/
|
|
95
|
+
export async function agentStatus(options = {}) {
|
|
96
|
+
try {
|
|
97
|
+
return await request('status', {}, { timeoutMs: 2_000, ...options })
|
|
98
|
+
} catch (err) {
|
|
99
|
+
if (err instanceof AgentUnavailableError || err.code === 'unsupported_platform') return null
|
|
100
|
+
throw err
|
|
101
|
+
}
|
|
102
|
+
}
|
|
@@ -0,0 +1,254 @@
|
|
|
1
|
+
// ShieldFive CLI — the upload ledger.
|
|
2
|
+
//
|
|
3
|
+
// One JSON line per upload the agent performed:
|
|
4
|
+
//
|
|
5
|
+
// {"v":1,"path":"/abs/path","size":48211,"mtimeMs":…,"mac":"<hex>",
|
|
6
|
+
// "fileId":"…","uploadedAt":"…","device":"<uuid>","tag":"<hex>"}
|
|
7
|
+
//
|
|
8
|
+
// `mac` is HMAC-SHA256 over exactly the plaintext bytes that were encrypted and
|
|
9
|
+
// uploaded, keyed by a key derived from the vault root key. A plain SHA-256
|
|
10
|
+
// would let anyone who reads this file confirm whether you hold a specific
|
|
11
|
+
// known document; keyed, the entries are unlinkable without the root key.
|
|
12
|
+
//
|
|
13
|
+
// `tag` is HMAC-SHA256 over the record's other fields under the same key. The
|
|
14
|
+
// ledger is an ordinary file the user's processes can write, so without it a
|
|
15
|
+
// forged line naming a missing file's MAC and some real fileId would make that
|
|
16
|
+
// file report as backed up. Records whose tag does not verify are dropped.
|
|
17
|
+
//
|
|
18
|
+
// Nothing here decides that a file is backed up by its name, path or size. A
|
|
19
|
+
// match on the MAC says those exact bytes were uploaded; whether they are still
|
|
20
|
+
// safely stored is a question for the server, asked at the moment it matters
|
|
21
|
+
// (see classify()).
|
|
22
|
+
|
|
23
|
+
import { createHmac, hkdfSync, timingSafeEqual } from 'node:crypto'
|
|
24
|
+
import { createReadStream } from 'node:fs'
|
|
25
|
+
import { mkdir, open, readFile, stat } from 'node:fs/promises'
|
|
26
|
+
import { dirname, isAbsolute, join } from 'node:path'
|
|
27
|
+
|
|
28
|
+
export const LEDGER_VERSION = 1
|
|
29
|
+
const LEDGER_INFO = 'shieldfive/cli/ledger/v1'
|
|
30
|
+
const TAGGED_FIELDS = ['v', 'path', 'size', 'mtimeMs', 'mac', 'fileId', 'uploadedAt', 'device']
|
|
31
|
+
export const VERIFY_BATCH = 100
|
|
32
|
+
|
|
33
|
+
/** Derive the ledger MAC key from the 32-byte vault root key. */
|
|
34
|
+
export function deriveLedgerKey(rootKey) {
|
|
35
|
+
if (!(rootKey instanceof Uint8Array) || rootKey.length !== 32) {
|
|
36
|
+
throw new Error('deriveLedgerKey needs the 32-byte root key')
|
|
37
|
+
}
|
|
38
|
+
return Buffer.from(hkdfSync('sha256', rootKey, Buffer.alloc(0), LEDGER_INFO, 32))
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
/** HMAC-SHA256 of a file's bytes, streamed. Returns lowercase hex. */
|
|
42
|
+
export function macFile(path, ledgerKey) {
|
|
43
|
+
return new Promise((resolve, reject) => {
|
|
44
|
+
const hmac = createHmac('sha256', ledgerKey)
|
|
45
|
+
const stream = createReadStream(path)
|
|
46
|
+
stream.on('error', reject)
|
|
47
|
+
stream.on('data', (chunk) => hmac.update(chunk))
|
|
48
|
+
stream.on('end', () => resolve(hmac.digest('hex')))
|
|
49
|
+
})
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
function tagFor(ledgerKey, record) {
|
|
53
|
+
const fields = TAGGED_FIELDS.map((f) => record[f] ?? null)
|
|
54
|
+
return createHmac('sha256', ledgerKey).update(JSON.stringify(fields)).digest('hex')
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
export function ledgerPath(dir, accountId) {
|
|
58
|
+
// Supabase user ids are UUIDs. Validating the shape keeps an account id from
|
|
59
|
+
// ever being read as a path.
|
|
60
|
+
if (typeof accountId !== 'string' || !/^[0-9a-f-]{36}$/i.test(accountId)) {
|
|
61
|
+
throw new Error(`Refusing ledger for malformed account id ${JSON.stringify(accountId)}`)
|
|
62
|
+
}
|
|
63
|
+
return join(dir, `${accountId.toLowerCase()}.jsonl`)
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
/** Tag, append and fsync one record. Creates the file 0600 inside a 0700 dir. */
|
|
67
|
+
export async function appendEntry(file, entry, { ledgerKey } = {}) {
|
|
68
|
+
if (!ledgerKey) throw new Error('appendEntry needs the ledger key to tag the record')
|
|
69
|
+
|
|
70
|
+
// Build and tag the record before the first await. The agent overwrites the
|
|
71
|
+
// key on logout, and a tag computed after that would be a tag under zeros.
|
|
72
|
+
const record = { v: LEDGER_VERSION, ...entry }
|
|
73
|
+
record.tag = tagFor(ledgerKey, record)
|
|
74
|
+
const line = JSON.stringify(record) + '\n'
|
|
75
|
+
|
|
76
|
+
await mkdir(dirname(file), { recursive: true, mode: 0o700 })
|
|
77
|
+
const handle = await open(file, 'a+', 0o600)
|
|
78
|
+
try {
|
|
79
|
+
// If a previous write was torn and left no trailing newline, start on a new
|
|
80
|
+
// line. Otherwise this record would be glued to the broken one and both
|
|
81
|
+
// would be discarded as malformed.
|
|
82
|
+
const { size } = await handle.stat()
|
|
83
|
+
let prefix = ''
|
|
84
|
+
if (size > 0) {
|
|
85
|
+
const { buffer } = await handle.read(Buffer.alloc(1), 0, 1, size - 1)
|
|
86
|
+
if (buffer[0] !== 0x0a) prefix = '\n'
|
|
87
|
+
}
|
|
88
|
+
await handle.write(prefix + line)
|
|
89
|
+
await handle.sync()
|
|
90
|
+
} finally {
|
|
91
|
+
await handle.close()
|
|
92
|
+
}
|
|
93
|
+
return record
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
/**
|
|
97
|
+
* Every well-formed, correctly tagged record, oldest first.
|
|
98
|
+
*
|
|
99
|
+
* A malformed line or a record whose tag does not verify is counted and
|
|
100
|
+
* skipped rather than failing the read, so one torn write or one forged line
|
|
101
|
+
* does not hide every other upload.
|
|
102
|
+
*/
|
|
103
|
+
export async function readEntries(file, { ledgerKey } = {}) {
|
|
104
|
+
if (!ledgerKey) throw new Error('readEntries needs the ledger key to check record tags')
|
|
105
|
+
|
|
106
|
+
let raw
|
|
107
|
+
try {
|
|
108
|
+
raw = await readFile(file, 'utf8')
|
|
109
|
+
} catch (err) {
|
|
110
|
+
if (err.code === 'ENOENT') return { entries: [], malformed: 0, unauthenticated: 0 }
|
|
111
|
+
throw err
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
const entries = []
|
|
115
|
+
let malformed = 0
|
|
116
|
+
let unauthenticated = 0
|
|
117
|
+
for (const line of raw.split('\n')) {
|
|
118
|
+
if (!line.trim()) continue
|
|
119
|
+
let rec
|
|
120
|
+
try {
|
|
121
|
+
rec = JSON.parse(line)
|
|
122
|
+
} catch {
|
|
123
|
+
malformed++
|
|
124
|
+
continue
|
|
125
|
+
}
|
|
126
|
+
const wellFormed =
|
|
127
|
+
rec?.v === LEDGER_VERSION &&
|
|
128
|
+
typeof rec.path === 'string' &&
|
|
129
|
+
typeof rec.mac === 'string' &&
|
|
130
|
+
/^[0-9a-f]{64}$/.test(rec.mac) &&
|
|
131
|
+
typeof rec.fileId === 'string' &&
|
|
132
|
+
rec.fileId &&
|
|
133
|
+
typeof rec.tag === 'string' &&
|
|
134
|
+
/^[0-9a-f]{64}$/.test(rec.tag)
|
|
135
|
+
if (!wellFormed) {
|
|
136
|
+
malformed++
|
|
137
|
+
continue
|
|
138
|
+
}
|
|
139
|
+
const expected = Buffer.from(tagFor(ledgerKey, rec), 'hex')
|
|
140
|
+
if (!timingSafeEqual(expected, Buffer.from(rec.tag, 'hex'))) {
|
|
141
|
+
unauthenticated++
|
|
142
|
+
continue
|
|
143
|
+
}
|
|
144
|
+
entries.push(rec)
|
|
145
|
+
}
|
|
146
|
+
return { entries, malformed, unauthenticated }
|
|
147
|
+
}
|
|
148
|
+
|
|
149
|
+
/**
|
|
150
|
+
* Where each local path stands.
|
|
151
|
+
*
|
|
152
|
+
* verified the bytes on disk now are bytes that were uploaded,
|
|
153
|
+
* and the server confirms that upload is stored, ready,
|
|
154
|
+
* outside the Bin and intact
|
|
155
|
+
* missing the bytes were uploaded, but the server cannot confirm
|
|
156
|
+
* it is still safely stored
|
|
157
|
+
* changed_since_upload this path was uploaded, but the file has changed since
|
|
158
|
+
* not_in_ledger no record of these bytes being uploaded from here
|
|
159
|
+
* unreadable the local file could not be read
|
|
160
|
+
*
|
|
161
|
+
* `not_in_ledger` is not `missing`. It means this agent never uploaded these
|
|
162
|
+
* bytes; the file may well be in the vault from another device.
|
|
163
|
+
*
|
|
164
|
+
* When `device` is given, only records made on that device count.
|
|
165
|
+
*
|
|
166
|
+
* `verifyFileIds(ids)` must resolve to the ids the server reports verified. It
|
|
167
|
+
* is called in batches of at most VERIFY_BATCH, and an id in an answer only
|
|
168
|
+
* counts if that batch asked about it.
|
|
169
|
+
*/
|
|
170
|
+
export async function classify(paths, { entries, ledgerKey, verifyFileIds, device }) {
|
|
171
|
+
const usable = device ? entries.filter((e) => e.device === device) : entries
|
|
172
|
+
|
|
173
|
+
const byMac = new Map()
|
|
174
|
+
const byPath = new Map()
|
|
175
|
+
for (const e of usable) {
|
|
176
|
+
const macList = byMac.get(e.mac) ?? []
|
|
177
|
+
macList.push(e)
|
|
178
|
+
byMac.set(e.mac, macList)
|
|
179
|
+
byPath.set(e.path, e) // later records win: entries are oldest first
|
|
180
|
+
}
|
|
181
|
+
|
|
182
|
+
const pending = []
|
|
183
|
+
const results = []
|
|
184
|
+
|
|
185
|
+
for (const path of paths) {
|
|
186
|
+
if (typeof path !== 'string' || !isAbsolute(path)) {
|
|
187
|
+
results.push({ path, state: 'unreadable', reason: 'not an absolute path' })
|
|
188
|
+
continue
|
|
189
|
+
}
|
|
190
|
+
|
|
191
|
+
let st
|
|
192
|
+
try {
|
|
193
|
+
st = await stat(path)
|
|
194
|
+
} catch (err) {
|
|
195
|
+
results.push({ path, state: 'unreadable', reason: err.code ?? 'stat failed' })
|
|
196
|
+
continue
|
|
197
|
+
}
|
|
198
|
+
if (!st.isFile()) {
|
|
199
|
+
results.push({ path, state: 'unreadable', reason: 'not a regular file' })
|
|
200
|
+
continue
|
|
201
|
+
}
|
|
202
|
+
|
|
203
|
+
let mac
|
|
204
|
+
try {
|
|
205
|
+
mac = await macFile(path, ledgerKey)
|
|
206
|
+
} catch (err) {
|
|
207
|
+
results.push({ path, state: 'unreadable', reason: err.code ?? 'read failed' })
|
|
208
|
+
continue
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
const sameBytes = byMac.get(mac)
|
|
212
|
+
if (sameBytes?.length) {
|
|
213
|
+
// Prefer the record for this exact path; otherwise any upload of the same
|
|
214
|
+
// bytes counts, because identity here is the content, not the location.
|
|
215
|
+
const own = sameBytes.filter((e) => e.path === path)
|
|
216
|
+
const candidates = (own.length ? own : sameBytes).map((e) => e.fileId)
|
|
217
|
+
const result = { path, state: null, size: st.size, fileIds: [...new Set(candidates)] }
|
|
218
|
+
if (!own.length) result.matchedPath = sameBytes[sameBytes.length - 1].path
|
|
219
|
+
results.push(result)
|
|
220
|
+
pending.push(result)
|
|
221
|
+
continue
|
|
222
|
+
}
|
|
223
|
+
|
|
224
|
+
if (byPath.has(path)) {
|
|
225
|
+
results.push({ path, state: 'changed_since_upload', size: st.size })
|
|
226
|
+
continue
|
|
227
|
+
}
|
|
228
|
+
|
|
229
|
+
results.push({ path, state: 'not_in_ledger', size: st.size })
|
|
230
|
+
}
|
|
231
|
+
|
|
232
|
+
const ids = [...new Set(pending.flatMap((r) => r.fileIds))]
|
|
233
|
+
const verified = new Set()
|
|
234
|
+
for (let i = 0; i < ids.length; i += VERIFY_BATCH) {
|
|
235
|
+
const batch = ids.slice(i, i + VERIFY_BATCH)
|
|
236
|
+
const asked = new Set(batch)
|
|
237
|
+
const answer = await verifyFileIds(batch)
|
|
238
|
+
for (const id of answer ?? []) {
|
|
239
|
+
// An id the server volunteered without being asked in this batch is not
|
|
240
|
+
// an answer to this question. On a check that gates deletion, that must
|
|
241
|
+
// fail closed.
|
|
242
|
+
if (asked.has(id)) verified.add(id)
|
|
243
|
+
}
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
for (const r of pending) {
|
|
247
|
+
const ok = r.fileIds.find((id) => verified.has(id))
|
|
248
|
+
r.state = ok ? 'verified' : 'missing'
|
|
249
|
+
if (ok) r.fileId = ok
|
|
250
|
+
delete r.fileIds
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
return results
|
|
254
|
+
}
|