querysub 0.721.0 → 0.723.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/flushlogs.js +7 -0
- package/debugging.md +43 -0
- package/package.json +10 -3
- package/src/-f-node-discovery/NodeDiscovery.ts +1 -1
- package/src/0-path-value-core/AuthorityLookup.ts +0 -1
- package/src/0-path-value-core/PathRouter.ts +0 -1
- package/src/0-path-value-core/PathWatcher.ts +0 -3
- package/src/0-path-value-core/ValidStateComputer.ts +0 -3
- package/src/2-proxy/PathValueProxyWatcher.ts +0 -3
- package/src/2-proxy/TransactionDelayer.ts +0 -1
- package/src/4-deploy/edgeNodes.ts +0 -1
- package/src/4-querysub/querysubPrediction.ts +0 -1
- package/src/config.ts +0 -1
- package/src/deployManager/machineSchema.ts +2 -5
- package/src/deployManager/publishQuerysub.ts +87 -0
- package/src/deployManager/publishQuerysubMain.ts +38 -0
- package/src/diagnostics/StatsHeader.tsx +2 -1
- package/src/diagnostics/StatsOverrides.ts +35 -0
- package/src/diagnostics/logs/IndexedLogs/BufferUnitIndex.ts +0 -1
- package/src/diagnostics/logs/IndexedLogs/MCPIndexedLogs.ts +36 -23
- package/src/diagnostics/logs/IndexedLogs/flushLogsEntry.ts +42 -0
- package/src/diagnostics/logs/IndexedLogs/moveIndexLogsToPublic.ts +6 -0
- package/src/diagnostics/network/logFilesMain.ts +57 -0
- package/src/diagnostics/network/logSearchMain.ts +86 -0
- package/src/diagnostics/network/networkNodesMain.ts +225 -0
- package/src/diagnostics/network/networkProbe.ts +31 -0
- package/src/diagnostics/network/networkServicesMain.ts +170 -0
- package/src/diagnostics/network/portMappingsMain.ts +28 -0
- package/src/diagnostics/statsDefinitions.tsx +0 -1
- package/src/library-components/LazyComponent.tsx +0 -1
- package/src/library-components/URLParam.ts +0 -2
package/bin/flushlogs.js
ADDED
package/debugging.md
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Debugging the Network
|
|
2
|
+
|
|
3
|
+
When the site won't load, the debug pages are unreachable, or nodes keep dying or being removed, work through these steps in order. Each step needs less of the network to be up than the one after it, so start at the top and only move on when the current step doesn't explain the problem.
|
|
4
|
+
|
|
5
|
+
Every script takes `--cwd <project root>` so it runs with that project's domain, keys, and certs (for the cyoa network: `--cwd D:/repos/qs-cyoa/`). Both scripts are read-only, except that, like every server-mode querysub process, they run the node-discovery audit, which deletes node list entries whose port has provably been taken by a newer thread.
|
|
6
|
+
|
|
7
|
+
## 1. What nodes exist
|
|
8
|
+
|
|
9
|
+
```
|
|
10
|
+
yarn network-nodes --cwd D:/repos/qs-cyoa/
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
Reads the node list (the `nodes/` folder of the fast archive bucket, which falls back to Backblaze if the storage servers are down), then calls every node and logs each one as it answers: heartbeat age, ping, service folder, entry point, uptime, git ref, and how many nodes it knows about. It ends with a summary:
|
|
14
|
+
|
|
15
|
+
- **Unreachable but still writing heartbeats**: the process is alive but nobody can call it. Usually networking (a port not forwarded, a firewall, another process answering on that port).
|
|
16
|
+
- **Stale or missing heartbeats**: dead processes waiting to be removed. Entries that stay listed with no heartbeat file across several runs mean deletes aren't sticking in storage.
|
|
17
|
+
- **Several threads on the same machine and port**: all but one are dead threads.
|
|
18
|
+
- **By machine**: which ports each machine is running and what is on them.
|
|
19
|
+
- **Ghosts / missing**: disagreement between the node list and what the nodes themselves hold in memory.
|
|
20
|
+
|
|
21
|
+
The node list lives on the storage servers, so if this step is slow or fails, the storage servers are the first suspect.
|
|
22
|
+
|
|
23
|
+
## 2. Which services are healthy
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
yarn network-services --cwd D:/repos/qs-cyoa/
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Reads the service configs and machine heartbeats (also from the storage servers, falling back to Backblaze, so it can be slow when they are down), works out every instance that should be running, and checks each instance's recorded node against the node list and with a call. It reports machines with stale heartbeats, instances a machine never reported, instances with no node id yet, and instances whose node is unreachable or missing from the node list. A node id that the machine recorded but whose port is now held by another process usually means the machine's record is stale, not that the service is down; check the step 1 output for a reachable node with the same service folder.
|
|
30
|
+
|
|
31
|
+
`overview.md` describes the roles that should exist (path value servers, function runners, the client-facing querysub server, error notifications, logging, machine management, storage); every one of them should show up here as a healthy service.
|
|
32
|
+
|
|
33
|
+
## 3. Look at the machine directly
|
|
34
|
+
|
|
35
|
+
If the problem still isn't apparent, SSH into the machine (`ssh root@<machine>`). Every service instance runs in its own tmux session named `<service key>-<instance index>-dply` (for example `cyoa-0-dply`, `serverall-2-dply`), and the machine's own deploy process runs in `machine-alwaysup`.
|
|
36
|
+
|
|
37
|
+
```
|
|
38
|
+
tmux ls
|
|
39
|
+
tmux attach -t cyoa-0-dply
|
|
40
|
+
tmux capture-pane -t cyoa-0-dply -p -S -2000
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Each instance's folder is `~/machine-services/<session name>/`, holding the `parameters.json` it was started with and `serviceNodeId.txt`, the node id it registered as.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "querysub",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.723.0",
|
|
4
4
|
"main": "index.js",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"note1": "note on node-forge fork, see https://github.com/digitalbazaar/forge/issues/744 for details",
|
|
@@ -28,6 +28,12 @@
|
|
|
28
28
|
"thin": "yarn typenode ./src/archiveapps/snapshotThinningEntry.ts --nobreak --public",
|
|
29
29
|
"thin-watch": "yarn typenode ./src/archiveapps/snapshotThinningEntry.ts --nobreak --public --watch",
|
|
30
30
|
"thin-dry": "yarn typenode ./src/archiveapps/snapshotThinningEntry.ts --nobreak --public --dry-run",
|
|
31
|
+
"network-nodes": "yarn typenode ./src/diagnostics/network/networkNodesMain.ts",
|
|
32
|
+
"network-services": "yarn typenode ./src/diagnostics/network/networkServicesMain.ts",
|
|
33
|
+
"port-mappings": "yarn typenode ./src/diagnostics/network/portMappingsMain.ts",
|
|
34
|
+
"log-search": "yarn typenode ./src/diagnostics/network/logSearchMain.ts",
|
|
35
|
+
"log-files": "yarn typenode ./src/diagnostics/network/logFilesMain.ts",
|
|
36
|
+
"publish-querysub": "yarn typenode ./src/deployManager/publishQuerysubMain.ts",
|
|
31
37
|
"parameters-timeline": "yarn typenode ./src/deployManager/parametersTimelineMain.ts",
|
|
32
38
|
"setup-machine": "yarn typenode ./src/deployManager/setupMachineMain.ts",
|
|
33
39
|
"stop-machine": "yarn typenode ./src/deployManager/stopMachineMain.ts",
|
|
@@ -50,6 +56,7 @@
|
|
|
50
56
|
"join": "./bin/join.js",
|
|
51
57
|
"join-public": "./bin/join-public.js",
|
|
52
58
|
"movelogs": "./bin/movelogs.js",
|
|
59
|
+
"flushlogs": "./bin/flushlogs.js",
|
|
53
60
|
"addsuperuser": "./bin/addsuperuser.js",
|
|
54
61
|
"error-watch": "./bin/error-watch.js",
|
|
55
62
|
"error-watch-public": "./bin/error-watch-public.js",
|
|
@@ -81,10 +88,10 @@
|
|
|
81
88
|
"node-forge": "https://github.com/sliftist/forge#e618181b469b07bdc70b968b0391beb8ef5fecd6",
|
|
82
89
|
"pako": "^2.1.0",
|
|
83
90
|
"peggy": "^5.0.6",
|
|
84
|
-
"sliftutils": "^1.9.
|
|
91
|
+
"sliftutils": "^1.9.17",
|
|
85
92
|
"socket-function": "^1.3.1",
|
|
86
93
|
"terser": "^5.31.0",
|
|
87
|
-
"typenode": "^6.6.
|
|
94
|
+
"typenode": "^6.6.3",
|
|
88
95
|
"typesafecss": "^0.32.0",
|
|
89
96
|
"yaml": "^2.5.0",
|
|
90
97
|
"yargs": "^15.3.1"
|
|
@@ -28,7 +28,7 @@ let HEARTBEAT_INTERVAL = timeInMinute * 1;
|
|
|
28
28
|
// Interval which we check other heartbeats
|
|
29
29
|
let CHECK_INTERVAL = HEARTBEAT_INTERVAL;
|
|
30
30
|
// If the heartbeat is older than thing, it fails the dead check
|
|
31
|
-
|
|
31
|
+
export const DEAD_THRESHOLD = HEARTBEAT_INTERVAL * 2;
|
|
32
32
|
// If we find another node is dead this number of checks, we remove it from storage, and tell
|
|
33
33
|
// all other nodes to remove from their memory.
|
|
34
34
|
let DEAD_CHECK_COUNT = 4;
|
|
@@ -231,7 +231,6 @@ export class PathRouter {
|
|
|
231
231
|
if (prefix) {
|
|
232
232
|
let key = getPathIndex(path, prefix.childKeyIndex);
|
|
233
233
|
if (key === undefined) {
|
|
234
|
-
require("debugbreak")(2);
|
|
235
234
|
debugger;
|
|
236
235
|
throw new Error(`Impossible, hash index ${prefix.childKeyIndex} is out of range for path ${path}, but it matched the prefix ${prefix.originalPrefix}`);
|
|
237
236
|
}
|
|
@@ -52,8 +52,6 @@ function incrementWatcherSequence() {
|
|
|
52
52
|
|
|
53
53
|
if (!registerResource) {
|
|
54
54
|
debugger;
|
|
55
|
-
require("debugbreak")(2);
|
|
56
|
-
debugger;
|
|
57
55
|
}
|
|
58
56
|
|
|
59
57
|
// WATCH CASES
|
|
@@ -112,7 +110,6 @@ class PathWatcher {
|
|
|
112
110
|
}) {
|
|
113
111
|
this.ensureUnwatchingOnDisconnect(config.nodeId);
|
|
114
112
|
if (config.nodeId.startsWith(getInternalValueWatchPrefix()) && (!config.noInitialTrigger || config.initialTrigger)) {
|
|
115
|
-
require("debugbreak")(2);
|
|
116
113
|
debugger;
|
|
117
114
|
// No initial trigger because if it's a remote value, then we won't have it. If it's a local value, then we've already processed it in valid state computers, there's no reason to trigger it again.
|
|
118
115
|
throw new Error("Internal value watches (INTERNAL_VALUE_WATCH_PREFIX) cannot have initial triggers");
|
|
@@ -209,7 +209,6 @@ class ValidStateComputer {
|
|
|
209
209
|
while (dependenciesChanged.size > 0) {
|
|
210
210
|
loopCount++;
|
|
211
211
|
if (loopCount > 3000) {
|
|
212
|
-
require("debugbreak")(2);
|
|
213
212
|
debugger;
|
|
214
213
|
console.error("Too many loops in valid state loop. aborting.", {
|
|
215
214
|
loopCount,
|
|
@@ -350,7 +349,6 @@ class ValidStateComputer {
|
|
|
350
349
|
let lockTime = locks[0].endTime;
|
|
351
350
|
for (let lock of locks) {
|
|
352
351
|
if (compareTime(lock.endTime, lockTime) !== 0) {
|
|
353
|
-
require("debugbreak")(2);
|
|
354
352
|
debugger;
|
|
355
353
|
console.error(`Lock has inconsistent end times. ${debugTime(lock.endTime)} !== ${debugTime(lockTime)}. Path: ${lock.path}`);
|
|
356
354
|
valid = false;
|
|
@@ -359,7 +357,6 @@ class ValidStateComputer {
|
|
|
359
357
|
|
|
360
358
|
for (let value of valueGroup) {
|
|
361
359
|
if (!PathRouter.isSelfAuthority(value.path)) {
|
|
362
|
-
require("debugbreak")(2);
|
|
363
360
|
debugger;
|
|
364
361
|
console.error(`Somehow we're attempting to compute the valid state of a value we don't own. This should be impossible. ${debugPathValue(value)}`);
|
|
365
362
|
valid = false;
|
|
@@ -1739,7 +1739,6 @@ export class PathValueProxyWatcher {
|
|
|
1739
1739
|
for (let lock of locks) {
|
|
1740
1740
|
if (compareTime(lock.endTime, writeTime) > 0) {
|
|
1741
1741
|
// I'm curious how this could even happen?
|
|
1742
|
-
require("debugbreak")(2);
|
|
1743
1742
|
debugger;
|
|
1744
1743
|
console.error(`Write time is before the end time of a lock. This is not allowed. Shifting write time forward ${debugTime(writeTime)} < ${debugTime(lock.endTime)}`);
|
|
1745
1744
|
writeTime = clientWatcher.getFreeWriteTime({
|
|
@@ -2464,8 +2463,6 @@ export function noAtomicSchema<T>(code: () => T) {
|
|
|
2464
2463
|
|
|
2465
2464
|
export const proxyWatcher = new PathValueProxyWatcher();
|
|
2466
2465
|
if ((globalThis as any).proxyWatcher) {
|
|
2467
|
-
debugger;
|
|
2468
|
-
require("debugbreak")(2);
|
|
2469
2466
|
debugger;
|
|
2470
2467
|
console.error(`Loaded PathValueProxyWatcher twice. This will break this. In ${module.filename}`);
|
|
2471
2468
|
}
|
|
@@ -14,7 +14,6 @@ export function isMissingTransactionPart(pathValues: PathValue[], timeoutCheck?:
|
|
|
14
14
|
for (const pathValue of pathValues) {
|
|
15
15
|
if (!pathValue.valid) {
|
|
16
16
|
if (timeoutCheck) return undefined;
|
|
17
|
-
require("debugbreak")(2);
|
|
18
17
|
debugger;
|
|
19
18
|
throw new Error(`Cannot validate a transaction where part of it is invalid. We shouldn't have even been passed this value. Value: ${debugPathValue(pathValue)}`);
|
|
20
19
|
}
|
|
@@ -452,7 +452,6 @@ export function getCallWrites(config: {
|
|
|
452
452
|
if (config.useFinishReordering && !config.metadata?.noFinishInStartOrder && call.fromProxy) {
|
|
453
453
|
let triggerCaller = getCurrentCallCreationProxy();
|
|
454
454
|
if (!triggerCaller) {
|
|
455
|
-
require("debugbreak")(2);
|
|
456
455
|
debugger;
|
|
457
456
|
throw new Error(`getCallWrites did not happen synchronously when triggering a call. It's very important that we synchronously start the proxy watcher, so we can make sure that predictions happen in a consistent order. FIX THIS! Find the async weighting that was added and move it inside the watch function using cacheAsync to make it watched. For: ${debugName}`);
|
|
458
457
|
}
|
package/src/config.ts
CHANGED
|
@@ -23,6 +23,7 @@ import { FunctionSpec, functionSchema } from "../3-path-functions/PathFunctionRu
|
|
|
23
23
|
import { Querysub } from "../4-querysub/Querysub";
|
|
24
24
|
import { green, red } from "socket-function/src/formatting/logColors";
|
|
25
25
|
import { fsExistsAsync } from "../fs";
|
|
26
|
+
import { publishQuerysubAsync, waitForPublishOtpAsync } from "./publishQuerysub";
|
|
26
27
|
|
|
27
28
|
const SERVICE_FOLDER_NAME = "machine-services";
|
|
28
29
|
export const SERVICE_FOLDER = `${SERVICE_FOLDER_NAME}/`;
|
|
@@ -521,13 +522,9 @@ export class MachineServiceControllerBase {
|
|
|
521
522
|
throw new Error(`Querysub folder does not exist at ${querysubFolder}`);
|
|
522
523
|
}
|
|
523
524
|
|
|
524
|
-
|
|
525
|
-
await runPromise("yarn version --minor --no-git-tag-version", { cwd: querysubFolder });
|
|
526
|
-
await runPromise("yarn publish --non-interactive", { cwd: querysubFolder });
|
|
525
|
+
let newVersion = await publishQuerysubAsync({ querysubFolder, getOtp: waitForPublishOtpAsync });
|
|
527
526
|
|
|
528
527
|
// Update package.json reference in our actual repo (version only, no install)
|
|
529
|
-
let querysubPackageJson = JSON.parse(fs.readFileSync(path.join(querysubFolder, "package.json"), "utf8"));
|
|
530
|
-
let newVersion = querysubPackageJson.version;
|
|
531
528
|
|
|
532
529
|
let currentPackageJsonPath = path.join(".", "package.json");
|
|
533
530
|
let currentPackageJson = JSON.parse(fs.readFileSync(currentPackageJsonPath, "utf8"));
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
import child_process from "child_process";
|
|
2
|
+
import fs from "fs";
|
|
3
|
+
import os from "os";
|
|
4
|
+
import path from "path";
|
|
5
|
+
import readline from "readline";
|
|
6
|
+
import { runPromise } from "socket-function/src/runPromise";
|
|
7
|
+
import { magenta, yellow } from "socket-function/src/formatting/logColors";
|
|
8
|
+
import { delay } from "socket-function/src/batching";
|
|
9
|
+
|
|
10
|
+
export const PUBLISH_OTP_FILE = path.join(os.tmpdir(), "querysub-publish-otp.txt");
|
|
11
|
+
const OTP_FILE_POLL_INTERVAL = 1000;
|
|
12
|
+
const LOGIN_URL_PATTERN = /Open (https:\/\/www\.npmjs\.com\/login\/\S+)/;
|
|
13
|
+
const PUBLISH_SUCCESS_TEXT = "success Published";
|
|
14
|
+
|
|
15
|
+
/** Bumps the minor version (unless the current one was never published), then publishes with yarn. npm requires a one-time code for every publish: yarn prints a login link, and the code that page hands out has to come back through getOtp. */
|
|
16
|
+
export async function publishQuerysubAsync(config: {
|
|
17
|
+
querysubFolder: string;
|
|
18
|
+
getOtp: (loginUrl: string) => Promise<string>;
|
|
19
|
+
}): Promise<string> {
|
|
20
|
+
let { querysubFolder } = config;
|
|
21
|
+
let version = readVersion(querysubFolder);
|
|
22
|
+
let published = (await runPromise(`npm view querysub@${version} version`, { quiet: true, nothrow: true })).trim();
|
|
23
|
+
if (published === version) {
|
|
24
|
+
await runPromise("yarn version --minor --no-git-tag-version", { cwd: querysubFolder });
|
|
25
|
+
version = readVersion(querysubFolder);
|
|
26
|
+
}
|
|
27
|
+
console.log(magenta(`Publishing querysub ${version}`));
|
|
28
|
+
|
|
29
|
+
let output = await runPromise("yarn publish --non-interactive", { cwd: querysubFolder, nothrow: true, includeStderrInOutput: true });
|
|
30
|
+
if (output.includes(PUBLISH_SUCCESS_TEXT)) return version;
|
|
31
|
+
let loginUrl = LOGIN_URL_PATTERN.exec(output)?.[1];
|
|
32
|
+
if (!loginUrl) {
|
|
33
|
+
throw new Error(`yarn publish of querysub ${version} failed without giving a login link:\n${output}`);
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
let otp = await config.getOtp(loginUrl);
|
|
37
|
+
output = await runPromise(`yarn publish --non-interactive --otp ${otp}`, { cwd: querysubFolder, nothrow: true, includeStderrInOutput: true });
|
|
38
|
+
if (!output.includes(PUBLISH_SUCCESS_TEXT)) {
|
|
39
|
+
throw new Error(`yarn publish of querysub ${version} with the one-time code failed:\n${output}`);
|
|
40
|
+
}
|
|
41
|
+
return version;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
function readVersion(querysubFolder: string): string {
|
|
45
|
+
return String(JSON.parse(fs.readFileSync(path.join(querysubFolder, "package.json"), "utf8")).version);
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/** Opens the login link, explains the LastPass passkey dance, and waits for the code: typed into this terminal, or written to PUBLISH_OTP_FILE (for when something other than a person is running the publish). */
|
|
49
|
+
export async function waitForPublishOtpAsync(loginUrl: string): Promise<string> {
|
|
50
|
+
await fs.promises.rm(PUBLISH_OTP_FILE, { force: true });
|
|
51
|
+
openInBrowser(loginUrl);
|
|
52
|
+
console.log(yellow(`npm needs a one-time code to publish. Opened ${loginUrl}`));
|
|
53
|
+
console.log(yellow(`LastPass is NOT enabled on npmjs.com by default: click the LastPass icon, enable it for this site, then sign in with the passkey from LastPass. Copy the code the page gives you.`));
|
|
54
|
+
console.log(yellow(`Type the code here and press enter, or write it to ${PUBLISH_OTP_FILE}`));
|
|
55
|
+
|
|
56
|
+
let done = false;
|
|
57
|
+
let fromTerminal = new Promise<string>(resolve => {
|
|
58
|
+
if (!process.stdin.isTTY) return;
|
|
59
|
+
let reader = readline.createInterface({ input: process.stdin, output: process.stdout });
|
|
60
|
+
reader.question("One-time code: ", answer => {
|
|
61
|
+
reader.close();
|
|
62
|
+
resolve(answer);
|
|
63
|
+
});
|
|
64
|
+
});
|
|
65
|
+
let fromFile = (async () => {
|
|
66
|
+
while (!done) {
|
|
67
|
+
let text = await fs.promises.readFile(PUBLISH_OTP_FILE, "utf8").catch(() => "");
|
|
68
|
+
if (text.trim()) return text;
|
|
69
|
+
await delay(OTP_FILE_POLL_INTERVAL);
|
|
70
|
+
}
|
|
71
|
+
return "";
|
|
72
|
+
})();
|
|
73
|
+
let code = (await Promise.race([fromTerminal, fromFile])).trim();
|
|
74
|
+
done = true;
|
|
75
|
+
await fs.promises.rm(PUBLISH_OTP_FILE, { force: true });
|
|
76
|
+
if (!/^\w+$/.test(code)) {
|
|
77
|
+
throw new Error(`Expected a one-time code of letters and digits, was ${JSON.stringify(code)}`);
|
|
78
|
+
}
|
|
79
|
+
return code;
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
function openInBrowser(url: string) {
|
|
83
|
+
let child = process.platform === "win32"
|
|
84
|
+
? child_process.spawn("cmd", ["/c", "start", "", url], { detached: true, stdio: "ignore" })
|
|
85
|
+
: child_process.spawn("xdg-open", [url], { detached: true, stdio: "ignore" });
|
|
86
|
+
child.unref();
|
|
87
|
+
}
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
module.hotreload = false;
|
|
2
|
+
|
|
3
|
+
{
|
|
4
|
+
let argv = process.argv;
|
|
5
|
+
let cwdIdx = argv.indexOf("--cwd");
|
|
6
|
+
if (cwdIdx >= 0 && argv[cwdIdx + 1]) {
|
|
7
|
+
process.chdir(argv[cwdIdx + 1]);
|
|
8
|
+
argv.splice(cwdIdx, 2);
|
|
9
|
+
}
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
import "../inject";
|
|
13
|
+
|
|
14
|
+
import yargs from "yargs";
|
|
15
|
+
import { green } from "socket-function/src/formatting/logColors";
|
|
16
|
+
import { disablePathAuditer } from "../diagnostics/pathAuditerCallback";
|
|
17
|
+
import { MachineServiceControllerBase } from "./machineSchema";
|
|
18
|
+
|
|
19
|
+
let args = yargs(process.argv.slice(2))
|
|
20
|
+
.option("message", { type: "string", demandOption: true, describe: "commit message for both querysub and the service repo" })
|
|
21
|
+
.strict()
|
|
22
|
+
.argv;
|
|
23
|
+
|
|
24
|
+
async function main() {
|
|
25
|
+
disablePathAuditer();
|
|
26
|
+
let controller = new MachineServiceControllerBase();
|
|
27
|
+
await controller.commitPushAndPublishQuerysub(args.message);
|
|
28
|
+
await controller.commitPushService(args.message);
|
|
29
|
+
console.log(green(`Published querysub, pointed ${process.cwd()} at it, and pushed both repos`));
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
main().then(
|
|
33
|
+
() => process.exit(),
|
|
34
|
+
e => {
|
|
35
|
+
console.error((e as Error).stack || e);
|
|
36
|
+
process.exit(1);
|
|
37
|
+
},
|
|
38
|
+
);
|
|
@@ -5,7 +5,7 @@ import { timeInHour } from "socket-function/src/misc";
|
|
|
5
5
|
import { isCurrentUserSuperUser } from "../user-implementation/userData";
|
|
6
6
|
import { Querysub } from "../4-querysub/Querysub";
|
|
7
7
|
import { StatWarning, StatWarningAction } from "./StatWarning";
|
|
8
|
-
import { getStatOverridesSync, setStatOverride, statOffenderKey } from "./StatsOverrides";
|
|
8
|
+
import { getStatOverridesSync, isStatStoragePersistentSync, setStatOverride, statOffenderKey } from "./StatsOverrides";
|
|
9
9
|
|
|
10
10
|
module.hotreload = true;
|
|
11
11
|
|
|
@@ -181,6 +181,7 @@ export class StatsHeader extends qreact.Component<{ getStats: () => StatDefiniti
|
|
|
181
181
|
|
|
182
182
|
render() {
|
|
183
183
|
if (!isCurrentUserSuperUser()) return undefined;
|
|
184
|
+
if (!isStatStoragePersistentSync()) return undefined;
|
|
184
185
|
|
|
185
186
|
let { categories, byCategory, infos } = this.buildInfos();
|
|
186
187
|
let activeWarnings = infos.filter(info =>
|
|
@@ -2,6 +2,8 @@ import { Querysub } from "../4-querysub/Querysub";
|
|
|
2
2
|
import { t } from "../2-proxy/schema2";
|
|
3
3
|
|
|
4
4
|
const LOCAL_STORAGE_KEY = "StatsHeader.overrides.v2";
|
|
5
|
+
const DEFAULT_HEAP_LIMIT_BYTES = 1024 * 1024 * 1024;
|
|
6
|
+
const INCOGNITO_QUOTA_HEAP_MULTIPLIER = 2;
|
|
5
7
|
|
|
6
8
|
export type StatOverride = { thresholdOverride?: number; ignoreUntilMs?: number };
|
|
7
9
|
type OverridesMap = { [key: string]: StatOverride };
|
|
@@ -10,8 +12,41 @@ let cachedOverrides: OverridesMap | undefined;
|
|
|
10
12
|
|
|
11
13
|
let overridesSchema = Querysub.createLocalSchema("statsOverrides", {
|
|
12
14
|
sequenceNumber: t.number,
|
|
15
|
+
storagePersists: t.boolean,
|
|
13
16
|
});
|
|
14
17
|
|
|
18
|
+
if (typeof window !== "undefined") {
|
|
19
|
+
void detectStoragePersists().then(persists => {
|
|
20
|
+
if (!persists) {
|
|
21
|
+
console.log(`StatsOverrides: private browsing detected, stat warnings are hidden because their overrides would not persist`);
|
|
22
|
+
return;
|
|
23
|
+
}
|
|
24
|
+
Querysub.localCommit(() => {
|
|
25
|
+
overridesSchema().storagePersists = true;
|
|
26
|
+
});
|
|
27
|
+
});
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
async function detectStoragePersists(): Promise<boolean> {
|
|
31
|
+
if (typeof navigator.storage?.getDirectory === "function") {
|
|
32
|
+
try {
|
|
33
|
+
await navigator.storage.getDirectory();
|
|
34
|
+
} catch {
|
|
35
|
+
return false;
|
|
36
|
+
}
|
|
37
|
+
}
|
|
38
|
+
if (typeof navigator.storage?.estimate !== "function") return true;
|
|
39
|
+
let { quota } = await navigator.storage.estimate();
|
|
40
|
+
if (!quota) return true;
|
|
41
|
+
let memory = (performance as unknown as { memory?: { jsHeapSizeLimit: number } }).memory;
|
|
42
|
+
let heapLimit = memory && memory.jsHeapSizeLimit || DEFAULT_HEAP_LIMIT_BYTES;
|
|
43
|
+
return quota >= heapLimit * INCOGNITO_QUOTA_HEAP_MULTIPLIER;
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
export function isStatStoragePersistentSync(): boolean {
|
|
47
|
+
return overridesSchema().storagePersists;
|
|
48
|
+
}
|
|
49
|
+
|
|
15
50
|
function loadFromStorage(): OverridesMap {
|
|
16
51
|
if (cachedOverrides) {
|
|
17
52
|
return cachedOverrides;
|
|
@@ -24,6 +24,7 @@ const MOVE_GRACE = timeInMinute;
|
|
|
24
24
|
const PATHS_CACHE_TTL = timeInMinute * 15;
|
|
25
25
|
// During the read/scan loops, emit a progress line at most this often.
|
|
26
26
|
const PROGRESS_LOG_INTERVAL = timeInSecond * 5;
|
|
27
|
+
const FLUSH_CALL_TIMEOUT = timeInSecond * 30;
|
|
27
28
|
|
|
28
29
|
const LOGGER_NAMES = ["logs/log", "logs/info", "logs/warn", "logs/error"] as const;
|
|
29
30
|
type LoggerName = typeof LOGGER_NAMES[number];
|
|
@@ -195,6 +196,7 @@ export class MCPIndexedLogs {
|
|
|
195
196
|
|
|
196
197
|
let pathsStart = Date.now();
|
|
197
198
|
let totalPathsSeen = 0;
|
|
199
|
+
let pathsByMachine = new Map<string, number>();
|
|
198
200
|
await Promise.all(LOGGER_NAMES.map(async (loggerName) => {
|
|
199
201
|
let logger = this.getLoggerByName(loggers, loggerName);
|
|
200
202
|
let archives = logger.debugGetCachedLogs({ type: useType });
|
|
@@ -210,13 +212,17 @@ export class MCPIndexedLogs {
|
|
|
210
212
|
totalPathsSeen += paths.length;
|
|
211
213
|
|
|
212
214
|
for (let p of paths) {
|
|
213
|
-
if (p.machineId !== machineId) continue;
|
|
214
215
|
if (p.endTime < startTime || p.startTime > endTime) continue;
|
|
216
|
+
pathsByMachine.set(p.machineId || "(none)", (pathsByMachine.get(p.machineId || "(none)") || 0) + 1);
|
|
217
|
+
if (p.machineId !== machineId) continue;
|
|
215
218
|
allFiles.push({ path: p, loggerName, archives });
|
|
216
219
|
fileCounts[loggerName].total++;
|
|
217
220
|
}
|
|
218
221
|
}));
|
|
219
222
|
console.log(`[search] paths gathered in ${formatTime(Date.now() - pathsStart)} (totalSeen=${totalPathsSeen} matchedMachine=${allFiles.length})`);
|
|
223
|
+
if (!allFiles.length) {
|
|
224
|
+
console.warn(`[search] no ${useType} log files in range are from machine ${machineId}. Files in range by machine: ${[...pathsByMachine].map(([id, count]) => `${id}=${count}`).join(", ") || "(none at all)"}`);
|
|
225
|
+
}
|
|
220
226
|
|
|
221
227
|
if (config.direction === "fromStart") {
|
|
222
228
|
allFiles.sort((a, b) => a.path.startTime - b.path.startTime);
|
|
@@ -482,52 +488,59 @@ export class MCPIndexedLogs {
|
|
|
482
488
|
// (e.g. older versions still running). Records moved-through up to
|
|
483
489
|
// now - MOVE_GRACE so we skip this on subsequent calls covering the same
|
|
484
490
|
// window.
|
|
485
|
-
private async ensureMovedThrough(machineId: string, endTime: number, forceRefresh?: boolean): Promise<"cached" | "no-node" | "moved"> {
|
|
491
|
+
private async ensureMovedThrough(machineId: string, endTime: number, forceRefresh?: boolean): Promise<"cached" | "no-node" | "unreachable" | "moved"> {
|
|
486
492
|
let lastMoved = this.movedThroughByMachine.get(machineId) ?? 0;
|
|
487
493
|
if (!forceRefresh && lastMoved >= endTime) return "cached";
|
|
488
494
|
|
|
489
495
|
let nodeIds = await this.findRemoteNodesOnMachine(machineId);
|
|
490
496
|
if (nodeIds.length === 0) {
|
|
491
|
-
console.warn(`MCPIndexedLogs: no
|
|
497
|
+
console.warn(`MCPIndexedLogs: no nodes in the node list are on machine ${machineId}, so its pending logs could NOT be flushed. Searching only what it already moved to public storage.`);
|
|
492
498
|
return "no-node";
|
|
493
499
|
}
|
|
500
|
+
console.log(`MCPIndexedLogs: flushing machine ${machineId} through ${formatDateTime(endTime)} via one of ${nodeIds.length} node(s): ${nodeIds.join(", ")}`);
|
|
494
501
|
|
|
502
|
+
let unflushed: string[] = [];
|
|
495
503
|
for (let loggerName of LOGGER_NAMES) {
|
|
496
|
-
let
|
|
504
|
+
let flushedBy = "";
|
|
505
|
+
let failures: string[] = [];
|
|
497
506
|
for (let nodeId of nodeIds) {
|
|
507
|
+
let start = Date.now();
|
|
498
508
|
try {
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
await IndexedLogShimController.nodes[nodeId].forceMoveLogsToPublic({
|
|
502
|
-
indexedLogsName: loggerName,
|
|
503
|
-
});
|
|
504
|
-
answered = true;
|
|
505
|
-
break;
|
|
506
|
-
}
|
|
507
|
-
let hasPending = await timeoutToUndefinedSilent(
|
|
508
|
-
5000,
|
|
509
|
+
let hasPending = !!forceRefresh || await timeoutToError(
|
|
510
|
+
FLUSH_CALL_TIMEOUT,
|
|
509
511
|
IndexedLogShimController.nodes[nodeId].hasPendingInRange({
|
|
510
512
|
indexedLogsName: loggerName,
|
|
511
513
|
startTime: 0,
|
|
512
514
|
endTime,
|
|
513
515
|
}),
|
|
516
|
+
() => new Error(`hasPendingInRange timed out after ${formatTime(FLUSH_CALL_TIMEOUT)}`),
|
|
514
517
|
);
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
518
|
+
if (hasPending) {
|
|
519
|
+
await timeoutToError(
|
|
520
|
+
FLUSH_CALL_TIMEOUT,
|
|
521
|
+
IndexedLogShimController.nodes[nodeId].forceMoveLogsToPublic({ indexedLogsName: loggerName }),
|
|
522
|
+
() => new Error(`forceMoveLogsToPublic timed out after ${formatTime(FLUSH_CALL_TIMEOUT)}`),
|
|
523
|
+
);
|
|
524
|
+
}
|
|
525
|
+
flushedBy = nodeId;
|
|
526
|
+
console.log(`MCPIndexedLogs: ${loggerName} on ${nodeId}: ${hasPending && "had pending logs, moved them to public" || "nothing pending"} (${formatTime(Date.now() - start)})`);
|
|
521
527
|
break;
|
|
522
528
|
} catch (e) {
|
|
523
|
-
|
|
529
|
+
failures.push(`${nodeId} after ${formatTime(Date.now() - start)}: ${String((e as Error).stack ?? e).split("\n")[0]}`);
|
|
524
530
|
}
|
|
525
531
|
}
|
|
526
|
-
if (!
|
|
527
|
-
|
|
532
|
+
if (!flushedBy) {
|
|
533
|
+
unflushed.push(loggerName);
|
|
534
|
+
console.warn(`MCPIndexedLogs: could NOT flush ${loggerName} on machine ${machineId}, every node failed:\n ${failures.join("\n ")}`);
|
|
535
|
+
} else if (failures.length) {
|
|
536
|
+
console.warn(`MCPIndexedLogs: ${loggerName} needed ${failures.length} failed node(s) before ${flushedBy} answered:\n ${failures.join("\n ")}`);
|
|
528
537
|
}
|
|
529
538
|
}
|
|
530
539
|
|
|
540
|
+
if (unflushed.length) {
|
|
541
|
+
console.warn(`MCPIndexedLogs: machine ${machineId} could not be flushed for ${unflushed.join(", ")}. Results only include what it had already moved to public storage on its own.`);
|
|
542
|
+
return "unreachable";
|
|
543
|
+
}
|
|
531
544
|
this.movedThroughByMachine.set(machineId, Date.now() - MOVE_GRACE);
|
|
532
545
|
return "moved";
|
|
533
546
|
}
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
import "../../../inject";
|
|
2
|
+
|
|
3
|
+
import { formatTime } from "socket-function/src/formatting/format";
|
|
4
|
+
import { green, magenta, red } from "socket-function/src/formatting/logColors";
|
|
5
|
+
import { isPublic } from "../../../config";
|
|
6
|
+
import { getOwnMachineId } from "../../../-f-node-discovery/NodeDiscovery";
|
|
7
|
+
import { getLoggers2Async, LogDatum } from "../diskLogger";
|
|
8
|
+
import { IndexedLogs } from "./IndexedLogs";
|
|
9
|
+
|
|
10
|
+
async function countPending(logs: IndexedLogs<LogDatum>): Promise<number> {
|
|
11
|
+
let paths = await logs.getPaths({ startTime: 0, endTime: Date.now(), only: "local" });
|
|
12
|
+
return paths.filter(x => x.logCount === undefined).length;
|
|
13
|
+
}
|
|
14
|
+
|
|
15
|
+
async function main() {
|
|
16
|
+
if (!isPublic()) {
|
|
17
|
+
throw new Error(`flushlogs must run with --public, or the logs would be "moved" into a folder on this machine's own disk instead of public storage`);
|
|
18
|
+
}
|
|
19
|
+
console.log(magenta(`Flushing every pending log file on machine ${getOwnMachineId()} to public storage`));
|
|
20
|
+
let loggers = await getLoggers2Async();
|
|
21
|
+
let stuck = 0;
|
|
22
|
+
for (let [name, logs] of Object.entries(loggers)) {
|
|
23
|
+
let before = await countPending(logs);
|
|
24
|
+
let start = Date.now();
|
|
25
|
+
await logs.moveLogsToPublic(true);
|
|
26
|
+
let after = await countPending(logs);
|
|
27
|
+
let color = after && red || green;
|
|
28
|
+
console.log(color(`${name}: ${before} pending files before, ${after} after (${formatTime(Date.now() - start)})`));
|
|
29
|
+
stuck += after;
|
|
30
|
+
}
|
|
31
|
+
if (stuck) {
|
|
32
|
+
throw new Error(`${stuck} log files are still pending after the flush. See the move logs above for why.`);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
main().then(
|
|
37
|
+
() => process.exit(),
|
|
38
|
+
e => {
|
|
39
|
+
console.error((e as Error).stack || e);
|
|
40
|
+
process.exit(1);
|
|
41
|
+
},
|
|
42
|
+
);
|
|
@@ -77,6 +77,9 @@ export async function moveLogsToPublic(config: {
|
|
|
77
77
|
let validMovingFiles = await checkAndCleanMovingFiles();
|
|
78
78
|
// 2) If they're not past our timeout threshold at the top of the file, Then we just return (NOOP).
|
|
79
79
|
if (validMovingFiles.length > 0) {
|
|
80
|
+
if (forceAll) {
|
|
81
|
+
console.warn(`Forced log move for ${config.loggerName} skipped: another thread holds the move lock (${validMovingFiles.join(", ")}), which goes stale after ${formatTime(movingTimeout)}`);
|
|
82
|
+
}
|
|
80
83
|
return false;
|
|
81
84
|
}
|
|
82
85
|
|
|
@@ -94,6 +97,9 @@ export async function moveLogsToPublic(config: {
|
|
|
94
97
|
// 7) Also, out of all of the moving files which aren't beyond the timeout, we need to make sure ours is the oldest, but not older than half the timeout. If this isn't true, then we just return.
|
|
95
98
|
let oldestMovingFile = validMovingFiles[0];
|
|
96
99
|
if (oldestMovingFile !== ourMovingFileName) {
|
|
100
|
+
if (forceAll) {
|
|
101
|
+
console.warn(`Forced log move for ${config.loggerName} skipped: ${oldestMovingFile} took the move lock before us (${ourMovingFileName})`);
|
|
102
|
+
}
|
|
97
103
|
// Clean up our moving file since we didn't get the lock
|
|
98
104
|
await localLogs.del(ourMovingFileName);
|
|
99
105
|
return false;
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
module.hotreload = false;
|
|
2
|
+
|
|
3
|
+
{
|
|
4
|
+
let argv = process.argv;
|
|
5
|
+
let cwdIdx = argv.indexOf("--cwd");
|
|
6
|
+
if (cwdIdx >= 0 && argv[cwdIdx + 1]) {
|
|
7
|
+
process.chdir(argv[cwdIdx + 1]);
|
|
8
|
+
argv.splice(cwdIdx, 2);
|
|
9
|
+
}
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
import "../../inject";
|
|
13
|
+
|
|
14
|
+
import yargs from "yargs";
|
|
15
|
+
import { formatTime } from "socket-function/src/formatting/format";
|
|
16
|
+
import { magenta } from "socket-function/src/formatting/logColors";
|
|
17
|
+
import { getArchives2 } from "../../-a-archives/archives2";
|
|
18
|
+
import { decodeLogFilePath } from "../logs/IndexedLogs/TimeFileTree";
|
|
19
|
+
import { disablePathAuditer } from "../pathAuditerCallback";
|
|
20
|
+
|
|
21
|
+
let args = yargs(process.argv.slice(2))
|
|
22
|
+
.option("day", { type: "string", demandOption: true, describe: "UTC day folder, ex 2026/11/20" })
|
|
23
|
+
.option("logger", { type: "string", default: "logs/log" })
|
|
24
|
+
.strict()
|
|
25
|
+
.argv;
|
|
26
|
+
|
|
27
|
+
async function listByMachine(config: { label: string; list: () => Promise<string[]> }) {
|
|
28
|
+
let start = Date.now();
|
|
29
|
+
let files = await config.list();
|
|
30
|
+
let byMachine = new Map<string, number>();
|
|
31
|
+
for (let file of files) {
|
|
32
|
+
let decoded = decodeLogFilePath(file);
|
|
33
|
+
if (!decoded) continue;
|
|
34
|
+
let machineId = decoded.machineId || "(none)";
|
|
35
|
+
byMachine.set(machineId, (byMachine.get(machineId) || 0) + 1);
|
|
36
|
+
}
|
|
37
|
+
console.log(magenta(`${config.label}: ${files.length} files in ${formatTime(Date.now() - start)}`), [...byMachine].map(([id, count]) => `${id}=${count}`).join(", "));
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
async function main() {
|
|
41
|
+
disablePathAuditer();
|
|
42
|
+
let archives = getArchives2(`final-indexed-logs/${args.logger}`);
|
|
43
|
+
let folder = args.day.endsWith("/") ? args.day : args.day + "/";
|
|
44
|
+
console.log(magenta(`Listing ${archives.getDebugName()}${folder}`));
|
|
45
|
+
await Promise.all([
|
|
46
|
+
listByMachine({ label: "find", list: () => archives.find(folder, { shallow: true, type: "files" }) }),
|
|
47
|
+
listByMachine({ label: "find with fallbacks", list: () => archives.find(folder, { shallow: true, type: "files", fallbacks: true }) }),
|
|
48
|
+
]);
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
main().then(
|
|
52
|
+
() => process.exit(),
|
|
53
|
+
e => {
|
|
54
|
+
console.error((e as Error).stack || e);
|
|
55
|
+
process.exit(1);
|
|
56
|
+
},
|
|
57
|
+
);
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
module.hotreload = false;
|
|
2
|
+
|
|
3
|
+
{
|
|
4
|
+
let argv = process.argv;
|
|
5
|
+
let cwdIdx = argv.indexOf("--cwd");
|
|
6
|
+
if (cwdIdx >= 0 && argv[cwdIdx + 1]) {
|
|
7
|
+
process.chdir(argv[cwdIdx + 1]);
|
|
8
|
+
argv.splice(cwdIdx, 2);
|
|
9
|
+
}
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
import "../../inject";
|
|
13
|
+
|
|
14
|
+
import yargs from "yargs";
|
|
15
|
+
import { formatDateTime } from "socket-function/src/formatting/format";
|
|
16
|
+
import { blue, magenta, yellow } from "socket-function/src/formatting/logColors";
|
|
17
|
+
import { timeInMinute } from "socket-function/src/misc";
|
|
18
|
+
import { MCPIndexedLogs, normalizeTime } from "../logs/IndexedLogs/MCPIndexedLogs";
|
|
19
|
+
import { machineNames } from "../../deployManager/machineSchema";
|
|
20
|
+
import { disablePathAuditer } from "../pathAuditerCallback";
|
|
21
|
+
|
|
22
|
+
const DEFAULT_LIMIT = 200;
|
|
23
|
+
const DEFAULT_COLUMNS = ["__entry", "__threadId", "param0", "param1", "param2", "param3"];
|
|
24
|
+
const DEFAULT_END_AGE = timeInMinute * 2;
|
|
25
|
+
|
|
26
|
+
let args = yargs(process.argv.slice(2))
|
|
27
|
+
.option("machine", { type: "string", demandOption: true, describe: "machine id, a substring of a machine name (ex \"5600x\"), or \"local\"" })
|
|
28
|
+
.option("query", { type: "string", default: "", describe: "| = OR, & = AND, * = ordered wildcard; case-insensitive substrings" })
|
|
29
|
+
.option("start", { type: "string", demandOption: true, describe: "epoch ms or any Date string (no timezone = local time)" })
|
|
30
|
+
.option("end", { type: "string", describe: `epoch ms or any Date string (default ${DEFAULT_END_AGE / timeInMinute} minutes ago)` })
|
|
31
|
+
.option("direction", { choices: ["fromStart", "fromEnd"] as const, default: "fromEnd" as const })
|
|
32
|
+
.option("columns", { type: "array", string: true, describe: `fields to print (default ${DEFAULT_COLUMNS.join(", ")})` })
|
|
33
|
+
.option("limit", { type: "number", default: DEFAULT_LIMIT })
|
|
34
|
+
.option("refresh", { type: "boolean", default: false, describe: "re-walk the log folders instead of using the cached file listing" })
|
|
35
|
+
.strict()
|
|
36
|
+
.argv;
|
|
37
|
+
|
|
38
|
+
async function resolveMachineId(machine: string): Promise<string> {
|
|
39
|
+
if (machine === "local") return machine;
|
|
40
|
+
let names = await machineNames.values();
|
|
41
|
+
if (names.some(x => x.machineId === machine)) return machine;
|
|
42
|
+
let matches = names.filter(x => x.name.toLowerCase().includes(machine.toLowerCase()));
|
|
43
|
+
if (matches.length === 1) {
|
|
44
|
+
console.log(magenta(`Machine "${machine}" is ${matches[0].name} (${matches[0].machineId})`));
|
|
45
|
+
return matches[0].machineId;
|
|
46
|
+
}
|
|
47
|
+
if (matches.length > 1) {
|
|
48
|
+
throw new Error(`Machine "${machine}" matches several machines: ${matches.map(x => `${x.name} (${x.machineId})`).join(", ")}`);
|
|
49
|
+
}
|
|
50
|
+
console.warn(yellow(`No machine is named like "${machine}", treating it as a machine id`));
|
|
51
|
+
return machine;
|
|
52
|
+
}
|
|
53
|
+
|
|
54
|
+
async function main() {
|
|
55
|
+
disablePathAuditer();
|
|
56
|
+
let machine = await resolveMachineId(args.machine);
|
|
57
|
+
let columns = (args.columns || DEFAULT_COLUMNS).map(String);
|
|
58
|
+
let endTime = args.end && normalizeTime(args.end, "end") || Date.now() - DEFAULT_END_AGE;
|
|
59
|
+
let result = await new MCPIndexedLogs().search({
|
|
60
|
+
query: args.query,
|
|
61
|
+
machine,
|
|
62
|
+
startTime: normalizeTime(args.start, "start"),
|
|
63
|
+
endTime,
|
|
64
|
+
direction: args.direction,
|
|
65
|
+
columns,
|
|
66
|
+
limit: args.limit,
|
|
67
|
+
forceRefresh: args.refresh,
|
|
68
|
+
});
|
|
69
|
+
let rows = result.results.slice();
|
|
70
|
+
rows.sort((a, b) => Number(a.time) - Number(b.time));
|
|
71
|
+
for (let row of rows) {
|
|
72
|
+
let fields = columns.filter(x => row[x] !== undefined).map(x => x.startsWith("param") ? row[x] : `${x}=${row[x]}`);
|
|
73
|
+
console.log(blue(formatDateTime(Number(row.time))), ...fields);
|
|
74
|
+
}
|
|
75
|
+
console.log(magenta(`${rows.length} rows`), JSON.stringify(result.files));
|
|
76
|
+
if (result.note) console.log(yellow(result.note));
|
|
77
|
+
console.log(magenta(`Columns seen: ${result.allColumns.join(", ")}`));
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
main().then(
|
|
81
|
+
() => process.exit(),
|
|
82
|
+
e => {
|
|
83
|
+
console.error((e as Error).stack || e);
|
|
84
|
+
process.exit(1);
|
|
85
|
+
},
|
|
86
|
+
);
|
|
@@ -0,0 +1,225 @@
|
|
|
1
|
+
module.hotreload = false;
|
|
2
|
+
|
|
3
|
+
{
|
|
4
|
+
let argv = process.argv;
|
|
5
|
+
let cwdIdx = argv.indexOf("--cwd");
|
|
6
|
+
if (cwdIdx >= 0 && argv[cwdIdx + 1]) {
|
|
7
|
+
process.chdir(argv[cwdIdx + 1]);
|
|
8
|
+
argv.splice(cwdIdx, 2);
|
|
9
|
+
}
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
import "../../inject";
|
|
13
|
+
|
|
14
|
+
import { decodeNodeId } from "sliftutils/misc/https/certs";
|
|
15
|
+
import { formatDateTime, formatTime } from "socket-function/src/formatting/format";
|
|
16
|
+
import { blue, green, magenta, red, yellow } from "socket-function/src/formatting/logColors";
|
|
17
|
+
import { sort } from "socket-function/src/misc";
|
|
18
|
+
import { getArchives2Buffered } from "../../-a-archives/archives2";
|
|
19
|
+
import { DEAD_THRESHOLD, NodeDiscoveryController } from "../../-f-node-discovery/NodeDiscovery";
|
|
20
|
+
import { NodeCapabilitiesController, NodeMetadata } from "../../-g-core-values/NodeCapabilities";
|
|
21
|
+
import { getDomain } from "../../config";
|
|
22
|
+
import { disablePathAuditer } from "../pathAuditerCallback";
|
|
23
|
+
import { probeAsync, ProbeResult, serviceFolderFromEntryPoint, shortEntryPoint } from "./networkProbe";
|
|
24
|
+
|
|
25
|
+
const NODE_CALL_TIMEOUT_MS = 10_000;
|
|
26
|
+
|
|
27
|
+
type NodeReport = {
|
|
28
|
+
nodeId: string;
|
|
29
|
+
machineId: string;
|
|
30
|
+
port: number;
|
|
31
|
+
heartbeatTime: number;
|
|
32
|
+
alive: ProbeResult<boolean>;
|
|
33
|
+
metadata?: ProbeResult<NodeMetadata>;
|
|
34
|
+
knownNodes?: ProbeResult<string[]>;
|
|
35
|
+
};
|
|
36
|
+
|
|
37
|
+
async function main() {
|
|
38
|
+
disablePathAuditer();
|
|
39
|
+
let domain = getDomain();
|
|
40
|
+
let archive = getArchives2Buffered("nodes/");
|
|
41
|
+
|
|
42
|
+
console.log(magenta(`Reading the node list from ${archive.getDebugName()} (domain ${domain})`));
|
|
43
|
+
let listStart = Date.now();
|
|
44
|
+
let nodeIds = await archive.find("", { fallbacks: true });
|
|
45
|
+
console.log(magenta(`Node list has ${nodeIds.length} nodes, read in ${formatTime(Date.now() - listStart)}`));
|
|
46
|
+
|
|
47
|
+
let reports = await Promise.all(nodeIds.map(async nodeId => {
|
|
48
|
+
let parts = decodeNodeId(nodeId, domain);
|
|
49
|
+
let heartbeatPromise = archive.get(nodeId).then(x => Number(x?.toString()) || 0, () => 0);
|
|
50
|
+
let alive = await probeAsync({ timeoutMs: NODE_CALL_TIMEOUT_MS, call: () => NodeDiscoveryController.nodes[nodeId].isAlive() });
|
|
51
|
+
let report: NodeReport = {
|
|
52
|
+
nodeId,
|
|
53
|
+
machineId: parts?.machineId || "",
|
|
54
|
+
port: parts?.port || 0,
|
|
55
|
+
heartbeatTime: 0,
|
|
56
|
+
alive,
|
|
57
|
+
};
|
|
58
|
+
if (alive.value) {
|
|
59
|
+
let [metadata, knownNodes] = await Promise.all([
|
|
60
|
+
probeAsync({ timeoutMs: NODE_CALL_TIMEOUT_MS, call: () => NodeCapabilitiesController.nodes[nodeId].getMetadata() }),
|
|
61
|
+
probeAsync({ timeoutMs: NODE_CALL_TIMEOUT_MS, call: () => NodeDiscoveryController.nodes[nodeId].getAllNodeIds() }),
|
|
62
|
+
]);
|
|
63
|
+
report.metadata = metadata;
|
|
64
|
+
report.knownNodes = knownNodes;
|
|
65
|
+
}
|
|
66
|
+
report.heartbeatTime = await heartbeatPromise;
|
|
67
|
+
logNodeReport(report);
|
|
68
|
+
return report;
|
|
69
|
+
}));
|
|
70
|
+
|
|
71
|
+
logSummary({ reports, nodeIds });
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
function heartbeatText(heartbeatTime: number): string {
|
|
75
|
+
if (!heartbeatTime) return red("no heartbeat file");
|
|
76
|
+
let age = Date.now() - heartbeatTime;
|
|
77
|
+
let text = `heartbeat ${formatTime(age)} ago`;
|
|
78
|
+
if (age > DEAD_THRESHOLD) return red(`${text} (stale, dead threshold ${formatTime(DEAD_THRESHOLD)})`);
|
|
79
|
+
return text;
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
function logNodeReport(report: NodeReport) {
|
|
83
|
+
let where = `${report.machineId}:${report.port}`;
|
|
84
|
+
if (!report.alive.value) {
|
|
85
|
+
console.log(red(`✗ ${report.nodeId}`), where, heartbeatText(report.heartbeatTime), red(`unreachable after ${formatTime(report.alive.ms)}: ${report.alive.error}`));
|
|
86
|
+
return;
|
|
87
|
+
}
|
|
88
|
+
let parts = [green(`✓ ${report.nodeId}`), where, heartbeatText(report.heartbeatTime), `ping ${formatTime(report.alive.ms)}`];
|
|
89
|
+
let metadata = report.metadata?.value;
|
|
90
|
+
if (metadata) {
|
|
91
|
+
let service = serviceFolderFromEntryPoint(metadata.entryPoint);
|
|
92
|
+
if (service) parts.push(`service ${service}`);
|
|
93
|
+
parts.push(`entry ${shortEntryPoint(metadata.entryPoint)}`, `up ${formatTime(Date.now() - metadata.startupTime)}`, `pid ${metadata.processId}`, `git ${metadata.gitRef.slice(0, 8)}`);
|
|
94
|
+
if (metadata.scheduledShutdownTime) parts.push(yellow(`scheduled shutdown ${formatDateTime(metadata.scheduledShutdownTime)}`));
|
|
95
|
+
} else if (report.metadata) {
|
|
96
|
+
parts.push(yellow(`metadata failed: ${report.metadata.error}`));
|
|
97
|
+
}
|
|
98
|
+
let known = report.knownNodes?.value;
|
|
99
|
+
if (known) {
|
|
100
|
+
parts.push(`knows ${known.length} nodes`);
|
|
101
|
+
} else if (report.knownNodes) {
|
|
102
|
+
parts.push(yellow(`node list failed: ${report.knownNodes.error}`));
|
|
103
|
+
}
|
|
104
|
+
console.log(...parts);
|
|
105
|
+
}
|
|
106
|
+
|
|
107
|
+
function logSummary(config: { reports: NodeReport[]; nodeIds: string[] }) {
|
|
108
|
+
let { reports, nodeIds } = config;
|
|
109
|
+
let listed = new Set(nodeIds);
|
|
110
|
+
let reachable = reports.filter(x => x.alive.value);
|
|
111
|
+
let unreachable = reports.filter(x => !x.alive.value);
|
|
112
|
+
let isFresh = (x: NodeReport) => !!x.heartbeatTime && Date.now() - x.heartbeatTime <= DEAD_THRESHOLD;
|
|
113
|
+
|
|
114
|
+
console.log();
|
|
115
|
+
console.log(magenta(`===== Summary =====`));
|
|
116
|
+
console.log(`${nodeIds.length} listed, ${green(`${reachable.length} reachable`)}, ${unreachable.length && red(`${unreachable.length} unreachable`) || "0 unreachable"}`);
|
|
117
|
+
|
|
118
|
+
let struggling = unreachable.filter(isFresh);
|
|
119
|
+
if (struggling.length) {
|
|
120
|
+
console.log(red(`Unreachable but still writing heartbeats (alive, but not answering calls):`));
|
|
121
|
+
for (let report of struggling) {
|
|
122
|
+
console.log(red(` ${report.nodeId}`), heartbeatText(report.heartbeatTime), report.alive.error?.split("\n")[0]);
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
let stale = reports.filter(x => !isFresh(x));
|
|
126
|
+
if (stale.length) {
|
|
127
|
+
console.log(yellow(`Stale or missing heartbeats (dead, waiting to be removed):`));
|
|
128
|
+
for (let report of stale) {
|
|
129
|
+
console.log(yellow(` ${report.nodeId}`), heartbeatText(report.heartbeatTime), report.alive.value && green("but it answered calls") || "");
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
let byPort = new Map<string, NodeReport[]>();
|
|
134
|
+
for (let report of reports) {
|
|
135
|
+
let key = `${report.machineId}:${report.port}`;
|
|
136
|
+
let list = byPort.get(key);
|
|
137
|
+
if (!list) {
|
|
138
|
+
list = [];
|
|
139
|
+
byPort.set(key, list);
|
|
140
|
+
}
|
|
141
|
+
list.push(report);
|
|
142
|
+
}
|
|
143
|
+
let duplicates = [...byPort].filter(([, list]) => list.length > 1);
|
|
144
|
+
if (duplicates.length) {
|
|
145
|
+
console.log(yellow(`Several threads listed on the same machine and port (all but one are dead threads):`));
|
|
146
|
+
for (let [key, list] of duplicates) {
|
|
147
|
+
console.log(yellow(` ${key}`), list.map(x => `${x.nodeId} (${x.alive.value && "reachable" || "unreachable"})`).join(", "));
|
|
148
|
+
}
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
let byMachine = new Map<string, NodeReport[]>();
|
|
152
|
+
for (let report of reports) {
|
|
153
|
+
let list = byMachine.get(report.machineId);
|
|
154
|
+
if (!list) {
|
|
155
|
+
list = [];
|
|
156
|
+
byMachine.set(report.machineId, list);
|
|
157
|
+
}
|
|
158
|
+
list.push(report);
|
|
159
|
+
}
|
|
160
|
+
console.log(blue(`By machine:`));
|
|
161
|
+
let machines = [...byMachine];
|
|
162
|
+
sort(machines, ([machineId]) => machineId);
|
|
163
|
+
for (let [machineId, list] of machines) {
|
|
164
|
+
let up = list.filter(x => x.alive.value).length;
|
|
165
|
+
let entries = list.map(x => {
|
|
166
|
+
let entryPoint = x.metadata?.value?.entryPoint;
|
|
167
|
+
let label = entryPoint && (serviceFolderFromEntryPoint(entryPoint) || shortEntryPoint(entryPoint)) || "?";
|
|
168
|
+
return `${x.port}=${label}${x.alive.value ? "" : "✗"}`;
|
|
169
|
+
});
|
|
170
|
+
sort(entries, x => x);
|
|
171
|
+
let color = up === list.length && green || up && yellow || red;
|
|
172
|
+
console.log(color(` ${machineId || "(undecodable node id)"} ${up}/${list.length} reachable`), entries.join(" "));
|
|
173
|
+
}
|
|
174
|
+
|
|
175
|
+
let gitRefs = new Map<string, number>();
|
|
176
|
+
for (let report of reachable) {
|
|
177
|
+
let gitRef = report.metadata?.value?.gitRef;
|
|
178
|
+
if (!gitRef) continue;
|
|
179
|
+
gitRefs.set(gitRef, (gitRefs.get(gitRef) || 0) + 1);
|
|
180
|
+
}
|
|
181
|
+
if (gitRefs.size > 1) {
|
|
182
|
+
console.log(blue(`Git refs running:`), [...gitRefs].map(([ref, count]) => `${ref.slice(0, 8)} x${count}`).join(", "));
|
|
183
|
+
}
|
|
184
|
+
|
|
185
|
+
let reporters = reachable.filter(x => x.knownNodes?.value);
|
|
186
|
+
let ghosts = new Map<string, string[]>();
|
|
187
|
+
let missingFromPeers = new Map<string, number>();
|
|
188
|
+
for (let report of reporters) {
|
|
189
|
+
let known = new Set(report.knownNodes?.value || []);
|
|
190
|
+
for (let nodeId of known) {
|
|
191
|
+
if (listed.has(nodeId)) continue;
|
|
192
|
+
let list = ghosts.get(nodeId);
|
|
193
|
+
if (!list) {
|
|
194
|
+
list = [];
|
|
195
|
+
ghosts.set(nodeId, list);
|
|
196
|
+
}
|
|
197
|
+
list.push(report.nodeId);
|
|
198
|
+
}
|
|
199
|
+
for (let nodeId of nodeIds) {
|
|
200
|
+
if (known.has(nodeId)) continue;
|
|
201
|
+
missingFromPeers.set(nodeId, (missingFromPeers.get(nodeId) || 0) + 1);
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
if (ghosts.size) {
|
|
205
|
+
console.log(yellow(`Nodes peers still know about that are not in the node list (ghosts in their memory):`));
|
|
206
|
+
for (let [nodeId, by] of ghosts) {
|
|
207
|
+
console.log(yellow(` ${nodeId}`), `known by ${by.length}/${reporters.length} peers`);
|
|
208
|
+
}
|
|
209
|
+
}
|
|
210
|
+
let missing = [...missingFromPeers];
|
|
211
|
+
if (missing.length) {
|
|
212
|
+
console.log(yellow(`Listed nodes that some peers don't know about:`));
|
|
213
|
+
for (let [nodeId, count] of missing) {
|
|
214
|
+
console.log(yellow(` ${nodeId}`), `missing from ${count}/${reporters.length} peers`);
|
|
215
|
+
}
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
main().then(
|
|
220
|
+
() => process.exit(),
|
|
221
|
+
e => {
|
|
222
|
+
console.error((e as Error).stack || e);
|
|
223
|
+
process.exit(1);
|
|
224
|
+
},
|
|
225
|
+
);
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import path from "path";
|
|
2
|
+
import { SERVICE_FOLDER } from "../../deployManager/machineSchema";
|
|
3
|
+
|
|
4
|
+
export type ProbeResult<T> = { value?: T; error?: string; ms: number };
|
|
5
|
+
|
|
6
|
+
export async function probeAsync<T>(config: { timeoutMs: number; call: () => Promise<T> }): Promise<ProbeResult<T>> {
|
|
7
|
+
let start = Date.now();
|
|
8
|
+
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
9
|
+
let timeout = new Promise<never>((resolve, reject) => {
|
|
10
|
+
timer = setTimeout(() => reject(new Error(`Timed out after ${config.timeoutMs}ms`)), config.timeoutMs);
|
|
11
|
+
});
|
|
12
|
+
try {
|
|
13
|
+
let value = await Promise.race([config.call(), timeout]);
|
|
14
|
+
return { value, ms: Date.now() - start };
|
|
15
|
+
} catch (e) {
|
|
16
|
+
return { error: String((e as Error).stack || e), ms: Date.now() - start };
|
|
17
|
+
} finally {
|
|
18
|
+
clearTimeout(timer);
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
export function serviceFolderFromEntryPoint(entryPoint: string): string {
|
|
23
|
+
let normalized = entryPoint.replaceAll("\\", "/");
|
|
24
|
+
let index = normalized.indexOf(SERVICE_FOLDER);
|
|
25
|
+
if (index < 0) return "";
|
|
26
|
+
return normalized.slice(index + SERVICE_FOLDER.length).split("/")[0];
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
export function shortEntryPoint(entryPoint: string): string {
|
|
30
|
+
return path.basename(entryPoint || "");
|
|
31
|
+
}
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
module.hotreload = false;
|
|
2
|
+
|
|
3
|
+
{
|
|
4
|
+
let argv = process.argv;
|
|
5
|
+
let cwdIdx = argv.indexOf("--cwd");
|
|
6
|
+
if (cwdIdx >= 0 && argv[cwdIdx + 1]) {
|
|
7
|
+
process.chdir(argv[cwdIdx + 1]);
|
|
8
|
+
argv.splice(cwdIdx, 2);
|
|
9
|
+
}
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
import "../../inject";
|
|
13
|
+
|
|
14
|
+
import { formatTime } from "socket-function/src/formatting/format";
|
|
15
|
+
import { blue, green, magenta, red, yellow } from "socket-function/src/formatting/logColors";
|
|
16
|
+
import { sort } from "socket-function/src/misc";
|
|
17
|
+
import { getArchives2Buffered } from "../../-a-archives/archives2";
|
|
18
|
+
import { NodeDiscoveryController } from "../../-f-node-discovery/NodeDiscovery";
|
|
19
|
+
import { MACHINE_RESYNC_INTERVAL, MachineInfo, getAllServiceConfigs, getLiveServiceParameters, getMachineIdList, machineConfigs, machineInfos, machineNames } from "../../deployManager/machineSchema";
|
|
20
|
+
import { disablePathAuditer } from "../pathAuditerCallback";
|
|
21
|
+
import { probeAsync, ProbeResult } from "./networkProbe";
|
|
22
|
+
|
|
23
|
+
const NODE_CALL_TIMEOUT_MS = 10_000;
|
|
24
|
+
const DEAD_MACHINE_THRESHOLD = MACHINE_RESYNC_INTERVAL * 4;
|
|
25
|
+
|
|
26
|
+
async function timedAsync<T>(label: string, call: () => Promise<T>): Promise<T> {
|
|
27
|
+
let start = Date.now();
|
|
28
|
+
console.log(magenta(`Reading ${label}...`));
|
|
29
|
+
let value = await call();
|
|
30
|
+
console.log(magenta(`Read ${label} in ${formatTime(Date.now() - start)}`));
|
|
31
|
+
return value;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
async function main() {
|
|
35
|
+
disablePathAuditer();
|
|
36
|
+
|
|
37
|
+
let [nodeIds, configs, infos, names, disabledMachines] = await Promise.all([
|
|
38
|
+
timedAsync("node list", () => getArchives2Buffered("nodes/").find("", { fallbacks: true })),
|
|
39
|
+
timedAsync("service configs", () => getAllServiceConfigs()),
|
|
40
|
+
timedAsync("machine heartbeats", () => machineInfos.entries()),
|
|
41
|
+
timedAsync("machine names", () => machineNames.values()),
|
|
42
|
+
timedAsync("machine configs", async () => new Set((await machineConfigs.values()).filter(x => x.disabled).map(x => x.machineId))),
|
|
43
|
+
]);
|
|
44
|
+
let listedNodes = new Set(nodeIds);
|
|
45
|
+
let infoByMachine = new Map<string, MachineInfo>(infos);
|
|
46
|
+
let nameByMachine = new Map(names.map(x => [x.machineId, x.name]));
|
|
47
|
+
let machineLabel = (machineId: string) => {
|
|
48
|
+
let name = nameByMachine.get(machineId);
|
|
49
|
+
return name && `${name} (${machineId})` || machineId;
|
|
50
|
+
};
|
|
51
|
+
|
|
52
|
+
let referencedNodeIds = new Set<string>();
|
|
53
|
+
for (let info of infoByMachine.values()) {
|
|
54
|
+
for (let service of Object.values(info.services || {})) {
|
|
55
|
+
for (let nodeId of Object.values(service.nodeIds || {})) {
|
|
56
|
+
if (nodeId) referencedNodeIds.add(nodeId.trim());
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
let aliveByNode = new Map<string, ProbeResult<boolean>>();
|
|
61
|
+
await Promise.all([...referencedNodeIds].map(async nodeId => {
|
|
62
|
+
aliveByNode.set(nodeId, await probeAsync({ timeoutMs: NODE_CALL_TIMEOUT_MS, call: () => NodeDiscoveryController.nodes[nodeId].isAlive() }));
|
|
63
|
+
}));
|
|
64
|
+
|
|
65
|
+
console.log();
|
|
66
|
+
console.log(magenta(`===== Machines =====`));
|
|
67
|
+
let machineIds = [...new Set([...infoByMachine.keys(), ...configs.flatMap(x => getMachineIdList(getLiveServiceParameters(x)))])];
|
|
68
|
+
sort(machineIds, x => machineLabel(x));
|
|
69
|
+
for (let machineId of machineIds) {
|
|
70
|
+
let info = infoByMachine.get(machineId);
|
|
71
|
+
if (!info) {
|
|
72
|
+
console.log(red(`✗ ${machineLabel(machineId)} has no machine heartbeat, but services are configured to run on it`));
|
|
73
|
+
continue;
|
|
74
|
+
}
|
|
75
|
+
let age = Date.now() - info.heartbeat;
|
|
76
|
+
let disabled = disabledMachines.has(machineId) && yellow(" (disabled)") || "";
|
|
77
|
+
if (age > DEAD_MACHINE_THRESHOLD) {
|
|
78
|
+
console.log(red(`✗ ${machineLabel(machineId)} heartbeat ${formatTime(age)} ago (dead threshold ${formatTime(DEAD_MACHINE_THRESHOLD)})${disabled}`));
|
|
79
|
+
continue;
|
|
80
|
+
}
|
|
81
|
+
console.log(green(`✓ ${machineLabel(machineId)} heartbeat ${formatTime(age)} ago${disabled}`), `git ${info.gitRef.slice(0, 8)}`);
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
console.log();
|
|
85
|
+
console.log(magenta(`===== Services =====`));
|
|
86
|
+
let problems = 0;
|
|
87
|
+
sort(configs, x => x.parameters.key);
|
|
88
|
+
for (let config of configs) {
|
|
89
|
+
let parameters = getLiveServiceParameters(config);
|
|
90
|
+
if (!parameters.deploy) {
|
|
91
|
+
console.log(blue(`- ${parameters.key} (not deployed)`));
|
|
92
|
+
continue;
|
|
93
|
+
}
|
|
94
|
+
let instanceCounts = new Map<string, number>();
|
|
95
|
+
for (let machineId of getMachineIdList(parameters)) {
|
|
96
|
+
instanceCounts.set(machineId, (instanceCounts.get(machineId) || 0) + 1);
|
|
97
|
+
}
|
|
98
|
+
let lines: string[] = [];
|
|
99
|
+
let healthy = 0;
|
|
100
|
+
let expected = 0;
|
|
101
|
+
for (let [machineId, count] of instanceCounts) {
|
|
102
|
+
if (disabledMachines.has(machineId)) {
|
|
103
|
+
lines.push(yellow(` ${machineLabel(machineId)}: machine disabled, ${count} instance(s) not expected`));
|
|
104
|
+
continue;
|
|
105
|
+
}
|
|
106
|
+
let info = infoByMachine.get(machineId);
|
|
107
|
+
let service = info?.services?.[config.serviceId];
|
|
108
|
+
for (let index = 0; index < count; index++) {
|
|
109
|
+
expected++;
|
|
110
|
+
let where = ` ${machineLabel(machineId)} #${index}`;
|
|
111
|
+
if (!info) {
|
|
112
|
+
lines.push(red(`${where}: no machine heartbeat`));
|
|
113
|
+
continue;
|
|
114
|
+
}
|
|
115
|
+
if (Date.now() - info.heartbeat > DEAD_MACHINE_THRESHOLD) {
|
|
116
|
+
lines.push(red(`${where}: machine heartbeat is ${formatTime(Date.now() - info.heartbeat)} old`));
|
|
117
|
+
continue;
|
|
118
|
+
}
|
|
119
|
+
if (!service) {
|
|
120
|
+
lines.push(red(`${where}: the machine has not reported this service at all`));
|
|
121
|
+
continue;
|
|
122
|
+
}
|
|
123
|
+
let launch = `launched ${service.totalTimesLaunched}x, last ${service.lastLaunchedTime && `${formatTime(Date.now() - service.lastLaunchedTime)} ago` || "never"}`;
|
|
124
|
+
let error = service.errorFromLastRun && red(` last run error: ${service.errorFromLastRun.split("\n").find(x => x.trim()) || ""}`) || "";
|
|
125
|
+
let nodeId = (service.nodeIds?.[index] || "").trim();
|
|
126
|
+
if (!nodeId) {
|
|
127
|
+
lines.push(yellow(`${where}: no node id recorded yet (${launch})${error}`));
|
|
128
|
+
continue;
|
|
129
|
+
}
|
|
130
|
+
let alive = aliveByNode.get(nodeId);
|
|
131
|
+
let inList = listedNodes.has(nodeId) && "in node list" || red("NOT in node list");
|
|
132
|
+
if (!alive?.value) {
|
|
133
|
+
lines.push(red(`${where}: ${nodeId} unreachable after ${formatTime(alive?.ms || 0)}, ${inList} (${launch})${error} ${alive?.error?.split("\n")[0] || ""}`));
|
|
134
|
+
continue;
|
|
135
|
+
}
|
|
136
|
+
if (!listedNodes.has(nodeId)) {
|
|
137
|
+
lines.push(yellow(`${where}: ${nodeId} answers calls but is ${inList} (${launch})${error}`));
|
|
138
|
+
continue;
|
|
139
|
+
}
|
|
140
|
+
healthy++;
|
|
141
|
+
lines.push(green(`${where}: ${nodeId} ok, ping ${formatTime(alive.ms)}`) + ` (${launch})${error}`);
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
problems += expected - healthy;
|
|
145
|
+
let color = healthy === expected && green || healthy && yellow || red;
|
|
146
|
+
console.log(color(`${healthy === expected && "✓" || "✗"} ${parameters.key} ${healthy}/${expected} healthy`), config.info?.title || "");
|
|
147
|
+
for (let line of lines) {
|
|
148
|
+
console.log(line);
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
let unattributed = nodeIds.filter(nodeId => !referencedNodeIds.has(nodeId));
|
|
153
|
+
console.log();
|
|
154
|
+
console.log(magenta(`===== Summary =====`));
|
|
155
|
+
console.log(`${configs.length} service configs, ${problems && red(`${problems} unhealthy instance(s)`) || green("every deployed instance healthy")}`);
|
|
156
|
+
if (unattributed.length) {
|
|
157
|
+
console.log(blue(`${unattributed.length}/${nodeIds.length} listed nodes are not recorded as any service's instance (dev machines, scripts, or instances whose node id file has not been read yet):`));
|
|
158
|
+
for (let nodeId of unattributed) {
|
|
159
|
+
console.log(blue(` ${nodeId}`));
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
|
|
164
|
+
main().then(
|
|
165
|
+
() => process.exit(),
|
|
166
|
+
e => {
|
|
167
|
+
console.error((e as Error).stack || e);
|
|
168
|
+
process.exit(1);
|
|
169
|
+
},
|
|
170
|
+
);
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
import { listPortMappings, resolveGateway } from "socket-function/src/forwardPort";
|
|
2
|
+
import { sort } from "socket-function/src/misc";
|
|
3
|
+
|
|
4
|
+
async function main() {
|
|
5
|
+
let ports = new Set(process.argv.slice(2).map(Number).filter(x => x));
|
|
6
|
+
let gateway = await resolveGateway();
|
|
7
|
+
console.log(`Gateway ${gateway.gatewayIP}, we are ${gateway.internalIP}`);
|
|
8
|
+
let mappings = await listPortMappings();
|
|
9
|
+
sort(mappings, x => x.externalPort);
|
|
10
|
+
console.log(`${mappings.length} UPnP mappings${ports.size && `, showing ports ${[...ports].join(", ")}` || ""}`);
|
|
11
|
+
for (let mapping of mappings) {
|
|
12
|
+
if (ports.size && !ports.has(mapping.externalPort)) continue;
|
|
13
|
+
let lease = mapping.leaseDuration && `lease ${mapping.leaseDuration}s left` || "permanent";
|
|
14
|
+
console.log(`${mapping.protocol} ${mapping.externalPort} -> ${mapping.internalClient}:${mapping.internalPort} ${mapping.enabled && "enabled" || "DISABLED"} ${lease} ${JSON.stringify(mapping.description)}`);
|
|
15
|
+
}
|
|
16
|
+
for (let port of ports) {
|
|
17
|
+
if (mappings.some(x => x.externalPort === port)) continue;
|
|
18
|
+
console.log(`${port}: NO MAPPING`);
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
|
|
22
|
+
main().then(
|
|
23
|
+
() => process.exit(),
|
|
24
|
+
e => {
|
|
25
|
+
console.error((e as Error).stack || e);
|
|
26
|
+
process.exit(1);
|
|
27
|
+
},
|
|
28
|
+
);
|