@lincoln504/pi-research 1.3.5 → 1.3.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +30 -50
- package/dist/cli.mjs +47 -21
- package/dist/thread-worker.mjs +49 -14
- package/docs/CHANGELOG.md +26 -0
- package/package.json +9 -5
- package/skills/pi-research/SKILL.md +1 -1
- package/src/infrastructure/browser/thread-worker-messaging.ts +24 -2
- package/src/infrastructure/browser/thread-worker.mjs +49 -14
- package/src/skill-install/skill-installer.ts +16 -2
- package/src/utils/log-rotation.ts +26 -19
- package/src/utils/log-utils.ts +26 -2
- package/src/youtube/potoken.ts +25 -3
package/README.md
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
<p align="center">
|
|
2
|
-
<img src="https://raw.githubusercontent.com/Lincoln504/pi-research/main/docs/media/banner.jpg" alt="pi-research" width="100%" />
|
|
2
|
+
<img src="https://raw.githubusercontent.com/Lincoln504/pi-research/main/docs/media/banner.jpg" alt="pi-research — free, unlimited web search and deep research for coding agents" width="100%" />
|
|
3
3
|
</p>
|
|
4
4
|
|
|
5
5
|
<p align="center">
|
|
6
|
-
<strong>
|
|
6
|
+
<strong>Free, unlimited web search & deep research for coding agents — no API key, no quota</strong>
|
|
7
7
|
</p>
|
|
8
8
|
|
|
9
9
|
<p align="center">
|
|
@@ -24,75 +24,55 @@
|
|
|
24
24
|
|
|
25
25
|
<a href="https://github.com/Lincoln504/pi-research/actions/workflows/ci.yml"><img alt="Build status" src="https://img.shields.io/github/actions/workflow/status/Lincoln504/pi-research/ci.yml?style=flat-square&branch=main" /></a> <a href="https://www.npmjs.com/package/@lincoln504/pi-research"><img alt="npm version" src="https://img.shields.io/npm/v/@lincoln504/pi-research.svg?style=flat-square" /></a>
|
|
26
26
|
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
27
|
+
Search and scraping run locally through a stealth browser — no search provider, no monthly cap. You pay only for the LLM tokens you use.
|
|
28
|
+
|
|
29
|
+
### Install
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
pi install npm:@lincoln504/pi-research
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
In [pi](https://github.com/earendil-works/pi) it works out of the box, on your session's model and pi's configuration. Standalone use — [agent skill](docs/AGENT-SKILL.md), CLI, or [SDK](docs/SDK.md) — needs a model configured; see [Configuration](docs/CONFIGURATION.md). The first install pulls the stealth browser engine, which takes a few minutes.
|
|
30
36
|
|
|
31
37
|
### How it works
|
|
32
38
|
|
|
33
|
-
A research run
|
|
34
|
-
opening search, parallel researcher agents scrape and read sources through a fixed,
|
|
35
|
-
sandboxed toolkit (no shell access, and no ability to write or modify files), and an evaluator decides whether
|
|
36
|
-
to run another round or synthesize — returning one cited Markdown report while its
|
|
37
|
-
sources are saved to the knowledge store in the background.
|
|
39
|
+
A research run loops through agent teams: a coordinator plans and opens with a search, researcher agents scrape and read in parallel, and an evaluator decides whether to go another round or synthesize. The result is one cited Markdown report, with findings optionally saved to the knowledge store.
|
|
38
40
|
|
|
39
|
-
Three depth levels — normal, deep, ultra — set the team size and number of rounds
|
|
40
|
-
ask in natural language and the tool picks the right one:
|
|
41
|
+
Three depth levels — normal, deep, ultra — set the team size and number of rounds. Ask in natural language and the tool picks the right one:
|
|
41
42
|
|
|
42
43
|

|
|
43
44
|
|
|
44
45
|
### Use cases
|
|
45
46
|
|
|
46
|
-
-
|
|
47
|
-
-
|
|
48
|
-
-
|
|
49
|
-
-
|
|
50
|
-
- Using the pi-research agent skill as OpenClaw's web access.
|
|
51
|
-
- Building agent systems that identify and examine web sources.
|
|
47
|
+
- Researching for your [Pi](https://github.com/earendil-works/pi) Coding Agent — plug-and-play, no API key needed.
|
|
48
|
+
- Researching from Claude Code, Codex, OpenClaw, or another coding agent, with a cheaper or local model driving the run so it doesn't spend your main agent's budget.
|
|
49
|
+
- Keeping a persistent, searchable knowledge store of findings — global or per-project.
|
|
50
|
+
- Building agent systems that find and read web content, or populating a dataset of web sources.
|
|
52
51
|
|
|
53
52
|
### Why pi-research
|
|
54
53
|
|
|
55
|
-
- **
|
|
56
|
-
- **You own your index.** Every finding is written
|
|
57
|
-
- **Cited reports, not raw dumps.**
|
|
58
|
-
- **Read-only by design.**
|
|
59
|
-
- **Search a little or a lot
|
|
54
|
+
- **No quota, no monthly fee.** Most AI-search providers cap free searches and then charge, with results served from *their* index on *their* servers. pi-research searches and scrapes locally — DuckDuckGo via a stealth browser.
|
|
55
|
+
- **You own your index.** Every finding is written to a local [LanceDB](https://lancedb.com) knowledge store that seeds future runs, so repeat questions get faster, cheaper answers.
|
|
56
|
+
- **Cited reports, not raw dumps.** One synthesized, sourced Markdown report instead of raw web content pasted into the conversation — context-efficient and easy to verify.
|
|
57
|
+
- **Read-only by design.** The research agent cannot run shell commands or write, edit, or delete anything. It can read and search files, so treat it as you would any read-only agent with access to your machine.
|
|
58
|
+
- **Search a little or a lot.** Depth levels span a quick pass to a large-scale investigation.
|
|
60
59
|
|
|
61
60
|
### Requirements / limitations
|
|
62
61
|
|
|
63
62
|
- Node.js >= 22.19.0
|
|
64
|
-
- An LLM with a 100k+ context window
|
|
65
|
-
- Internet access
|
|
66
|
-
-
|
|
67
|
-
-
|
|
68
|
-
- Cloudflare and similar anti-automation systems block scraping on some sites, so runs will identify sources they cannot reach. pi-research compensates with volume: the search tool rapidly pulls a large set of results off free DuckDuckGo, giving the model a wide selection of reachable content to cite.
|
|
69
|
-
|
|
70
|
-
### Install
|
|
71
|
-
|
|
72
|
-
```bash
|
|
73
|
-
pi install npm:@lincoln504/pi-research
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
See [Agent skill](docs/AGENT-SKILL.md) or [SDK](docs/SDK.md) for standalone (no pi extension) installation.
|
|
77
|
-
|
|
78
|
-
The pi extension works out of the box — it runs on your pi session's model and pi's own configuration. Standalone use (agent skill / CLI) requires a configured model: set `PI_RESEARCH_MODEL` (e.g. `openai/gpt-4o`) as an environment variable or in `~/.pi/research/config.env` (the CLI also accepts a per-run `--model`), plus `PI_RESEARCH_API_KEY` if you don't use pi; the SDK takes a `model` option. See [Configuration](docs/CONFIGURATION.md).
|
|
79
|
-
|
|
80
|
-
The first install pulls the stealth browser engine, which takes a few minutes.
|
|
81
|
-
|
|
82
|
-
### Stability (v1.x.x)
|
|
63
|
+
- An LLM with a 100k+ context window — bring your own key or use a local model
|
|
64
|
+
- Internet access on a residential IP — search, scraping, and YouTube transcripts rely on a residential connection; a datacenter/VPS/cloud IP gets bot-blocked
|
|
65
|
+
- The [pi runtime](https://github.com/earendil-works/pi) — the pi extension uses the host's copy; the standalone CLI and agent skill install it as a dependency
|
|
66
|
+
- Cloudflare and similar systems block scraping on some sites — a run flags pages it can't reach and compensates with a wide pool of search results to cite
|
|
83
67
|
|
|
84
|
-
|
|
85
|
-
and the stealth-browser stack. v1.0.0 is the first stable release since. Channels:
|
|
68
|
+
### Channels
|
|
86
69
|
|
|
87
|
-
|
|
88
|
-
- A git install is the development channel: latest commits, first to break.
|
|
70
|
+
npm (`npm:@lincoln504/pi-research`) is the stable channel, kept current with breaking pi changes. A git install is the development channel: latest commits, first to break.
|
|
89
71
|
|
|
90
72
|
### License
|
|
91
73
|
|
|
92
|
-
MIT. Bundled third-party licenses are listed in
|
|
93
|
-
[docs/THIRD-PARTY-NOTICES.md](docs/THIRD-PARTY-NOTICES.md).
|
|
74
|
+
MIT. Bundled third-party licenses are listed in [docs/THIRD-PARTY-NOTICES.md](docs/THIRD-PARTY-NOTICES.md).
|
|
94
75
|
|
|
95
76
|
### Package identity
|
|
96
77
|
|
|
97
|
-
This project is always `@lincoln504/pi-research` on npm. The
|
|
98
|
-
package is unrelated and deprecated.
|
|
78
|
+
This project is always `@lincoln504/pi-research` on npm. The `pi-research` (unscoped) package is unrelated and deprecated.
|
package/dist/cli.mjs
CHANGED
|
@@ -216,7 +216,14 @@ function formatArg(arg) {
|
|
|
216
216
|
}
|
|
217
217
|
function redactSecrets(message) {
|
|
218
218
|
const rawLength = message.length;
|
|
219
|
-
let
|
|
219
|
+
let scanEnd = REDACT_SCAN_LENGTH;
|
|
220
|
+
if (rawLength > scanEnd) {
|
|
221
|
+
const extensionLimit = Math.min(rawLength, scanEnd + REDACT_SCAN_EXTENSION_MAX);
|
|
222
|
+
while (scanEnd < extensionLimit && TOKEN_CHAR_PATTERN.test(message.charAt(scanEnd))) {
|
|
223
|
+
scanEnd += 1;
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
let out = rawLength > scanEnd ? message.slice(0, scanEnd) : message;
|
|
220
227
|
out = out.replace(ANSI_PATTERN, "");
|
|
221
228
|
out = out.replace(URL_CREDENTIALS_PATTERN, "$1[REDACTED]@");
|
|
222
229
|
out = out.replace(JWT_PATTERN, "[REDACTED]");
|
|
@@ -233,14 +240,14 @@ function redactSecrets(message) {
|
|
|
233
240
|
if (lastKeptUnit >= 55296 && lastKeptUnit <= 56319) keep -= 1;
|
|
234
241
|
}
|
|
235
242
|
if (out.length > MAX_LOG_MESSAGE_LENGTH) {
|
|
236
|
-
out = `${out.slice(0, keep)}\u2026[truncated ${rawLength - keep} chars]`;
|
|
243
|
+
out = `${out.slice(0, keep)}\u2026[truncated ${Math.max(rawLength, out.length) - keep} chars]`;
|
|
237
244
|
}
|
|
238
245
|
return out;
|
|
239
246
|
}
|
|
240
247
|
function stripTerminalEscapes(message) {
|
|
241
248
|
return message.replace(TERMINAL_ESCAPE_PATTERN, "").replace(CONTROL_CHARS_PATTERN, " ");
|
|
242
249
|
}
|
|
243
|
-
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TERMINAL_ESCAPE_PATTERN;
|
|
250
|
+
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
|
|
244
251
|
var init_log_utils = __esm({
|
|
245
252
|
"src/utils/log-utils.ts"() {
|
|
246
253
|
"use strict";
|
|
@@ -258,6 +265,8 @@ var init_log_utils = __esm({
|
|
|
258
265
|
LONG_HEX_SECRET_PATTERN = /\b[0-9a-fA-F]{32,}\b/g;
|
|
259
266
|
CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
|
|
260
267
|
REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
|
|
268
|
+
TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
|
|
269
|
+
REDACT_SCAN_EXTENSION_MAX = 2048;
|
|
261
270
|
TERMINAL_ESCAPE_PATTERN = new RegExp(
|
|
262
271
|
"\\x1b\\[[0-?]*[ -/]*[@-~]?|\\x1b\\][^\\x07\\x1b]*(?:\\x07|\\x1b\\\\)?|\\x1b[PX^_][^\\x1b]*(?:\\x1b\\\\)?|\\x1b.?",
|
|
263
272
|
"g"
|
|
@@ -269,9 +278,18 @@ var init_log_utils = __esm({
|
|
|
269
278
|
import * as fs from "node:fs";
|
|
270
279
|
import * as path2 from "node:path";
|
|
271
280
|
function touchOwnerOnly(filePath, mode) {
|
|
281
|
+
let fd;
|
|
272
282
|
try {
|
|
273
|
-
fs.
|
|
283
|
+
fd = fs.openSync(filePath, APPEND_CREATE_NOFOLLOW_FLAGS, mode);
|
|
284
|
+
fs.fchmodSync(fd, mode);
|
|
274
285
|
} catch {
|
|
286
|
+
} finally {
|
|
287
|
+
if (fd !== void 0) {
|
|
288
|
+
try {
|
|
289
|
+
fs.closeSync(fd);
|
|
290
|
+
} catch {
|
|
291
|
+
}
|
|
292
|
+
}
|
|
275
293
|
}
|
|
276
294
|
}
|
|
277
295
|
var APPEND_CREATE_NOFOLLOW_FLAGS, LogRotation;
|
|
@@ -297,10 +315,6 @@ var init_log_rotation = __esm({
|
|
|
297
315
|
if (fs.existsSync(logFile)) {
|
|
298
316
|
fs.unlinkSync(logFile);
|
|
299
317
|
touchOwnerOnly(logFile, 384);
|
|
300
|
-
try {
|
|
301
|
-
fs.chmodSync(logFile, 384);
|
|
302
|
-
} catch {
|
|
303
|
-
}
|
|
304
318
|
}
|
|
305
319
|
const files = fs.readdirSync(logDir);
|
|
306
320
|
const baseName = path2.basename(logFile);
|
|
@@ -337,10 +351,6 @@ var init_log_rotation = __esm({
|
|
|
337
351
|
return;
|
|
338
352
|
}
|
|
339
353
|
touchOwnerOnly(logFile, 384);
|
|
340
|
-
try {
|
|
341
|
-
fs.chmodSync(logFile, 384);
|
|
342
|
-
} catch {
|
|
343
|
-
}
|
|
344
354
|
try {
|
|
345
355
|
const files = fs.readdirSync(logDir);
|
|
346
356
|
const logFiles = files.filter((f) => f.startsWith(path2.basename(logFile)) && f !== path2.basename(logFile)).sort();
|
|
@@ -9914,19 +9924,19 @@ function reconcileSkillInstalls(opts = {}) {
|
|
|
9914
9924
|
if (e.type !== "symlink") {
|
|
9915
9925
|
const stale2 = currentVersion !== null && e.version !== currentVersion;
|
|
9916
9926
|
if (stale2 && source !== null && fs16.existsSync(e.path) && isOwnedCopy(e.path)) {
|
|
9917
|
-
const
|
|
9918
|
-
const
|
|
9927
|
+
const dir2 = path17.dirname(e.path);
|
|
9928
|
+
const base2 = path17.basename(e.path);
|
|
9919
9929
|
try {
|
|
9920
|
-
for (const name of fs16.readdirSync(
|
|
9921
|
-
if (name.startsWith(`.${
|
|
9922
|
-
fs16.rmSync(path17.join(
|
|
9930
|
+
for (const name of fs16.readdirSync(dir2)) {
|
|
9931
|
+
if (name.startsWith(`.${base2}.new-`) || name.startsWith(`.${base2}.old-`)) {
|
|
9932
|
+
fs16.rmSync(path17.join(dir2, name), { recursive: true, force: true });
|
|
9923
9933
|
}
|
|
9924
9934
|
}
|
|
9925
9935
|
} catch {
|
|
9926
9936
|
}
|
|
9927
9937
|
const suffix = crypto10.randomBytes(4).toString("hex");
|
|
9928
|
-
const fresh = path17.join(
|
|
9929
|
-
const retired = path17.join(
|
|
9938
|
+
const fresh = path17.join(dir2, `.${base2}.new-${suffix}`);
|
|
9939
|
+
const retired = path17.join(dir2, `.${base2}.old-${suffix}`);
|
|
9930
9940
|
try {
|
|
9931
9941
|
copyDir(source, fresh);
|
|
9932
9942
|
} catch {
|
|
@@ -9972,6 +9982,16 @@ function reconcileSkillInstalls(opts = {}) {
|
|
|
9972
9982
|
}
|
|
9973
9983
|
continue;
|
|
9974
9984
|
}
|
|
9985
|
+
const dir = path17.dirname(e.path);
|
|
9986
|
+
const base = path17.basename(e.path);
|
|
9987
|
+
try {
|
|
9988
|
+
for (const name of fs16.readdirSync(dir)) {
|
|
9989
|
+
if (name.startsWith(`.${base}.new-`)) {
|
|
9990
|
+
fs16.rmSync(path17.join(dir, name), { recursive: true, force: true });
|
|
9991
|
+
}
|
|
9992
|
+
}
|
|
9993
|
+
} catch {
|
|
9994
|
+
}
|
|
9975
9995
|
if (!isSymlinkPresent(e.path)) {
|
|
9976
9996
|
result.pruned.push(e.path);
|
|
9977
9997
|
continue;
|
|
@@ -10000,8 +10020,6 @@ function reconcileSkillInstalls(opts = {}) {
|
|
|
10000
10020
|
keep.push({ ...e, type: "symlink", source });
|
|
10001
10021
|
result.repointed.push(e.path);
|
|
10002
10022
|
} catch {
|
|
10003
|
-
const dir = path17.dirname(e.path);
|
|
10004
|
-
const base = path17.basename(e.path);
|
|
10005
10023
|
const suffix = crypto10.randomBytes(4).toString("hex");
|
|
10006
10024
|
const fresh = path17.join(dir, `.${base}.new-${suffix}`);
|
|
10007
10025
|
try {
|
|
@@ -20677,14 +20695,22 @@ function describeError(error) {
|
|
|
20677
20695
|
return String(error);
|
|
20678
20696
|
}
|
|
20679
20697
|
var mintChain = Promise.resolve();
|
|
20698
|
+
var wedged = false;
|
|
20680
20699
|
var MINT_HARD_RELEASE_MS = 5 * 60 * 1e3;
|
|
20681
20700
|
async function mintPoTokens(identifiers, opts = {}) {
|
|
20701
|
+
if (wedged) {
|
|
20702
|
+
throw new Error("PoToken minter is wedged by a prior mint that never settled; refusing to start a new mint until it resolves.");
|
|
20703
|
+
}
|
|
20682
20704
|
const run = mintChain.then(() => doMint(identifiers, opts), () => doMint(identifiers, opts));
|
|
20683
20705
|
const settled = run.then(() => void 0, () => void 0);
|
|
20706
|
+
settled.then(() => {
|
|
20707
|
+
wedged = false;
|
|
20708
|
+
});
|
|
20684
20709
|
let hardTimer;
|
|
20685
20710
|
const hardRelease = new Promise((resolve9) => {
|
|
20686
20711
|
hardTimer = setTimeout(() => {
|
|
20687
20712
|
logger.warn("[youtube] potoken mint mutex force-released \u2014 doMint() did not settle within the hard timeout");
|
|
20713
|
+
wedged = true;
|
|
20688
20714
|
resolve9();
|
|
20689
20715
|
}, MINT_HARD_RELEASE_MS);
|
|
20690
20716
|
safeUnref(hardTimer);
|
package/dist/thread-worker.mjs
CHANGED
|
@@ -46,7 +46,14 @@ function formatArg(arg) {
|
|
|
46
46
|
}
|
|
47
47
|
function redactSecrets(message) {
|
|
48
48
|
const rawLength = message.length;
|
|
49
|
-
let
|
|
49
|
+
let scanEnd = REDACT_SCAN_LENGTH;
|
|
50
|
+
if (rawLength > scanEnd) {
|
|
51
|
+
const extensionLimit = Math.min(rawLength, scanEnd + REDACT_SCAN_EXTENSION_MAX);
|
|
52
|
+
while (scanEnd < extensionLimit && TOKEN_CHAR_PATTERN.test(message.charAt(scanEnd))) {
|
|
53
|
+
scanEnd += 1;
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
let out = rawLength > scanEnd ? message.slice(0, scanEnd) : message;
|
|
50
57
|
out = out.replace(ANSI_PATTERN, "");
|
|
51
58
|
out = out.replace(URL_CREDENTIALS_PATTERN, "$1[REDACTED]@");
|
|
52
59
|
out = out.replace(JWT_PATTERN, "[REDACTED]");
|
|
@@ -63,14 +70,14 @@ function redactSecrets(message) {
|
|
|
63
70
|
if (lastKeptUnit >= 55296 && lastKeptUnit <= 56319) keep -= 1;
|
|
64
71
|
}
|
|
65
72
|
if (out.length > MAX_LOG_MESSAGE_LENGTH) {
|
|
66
|
-
out = `${out.slice(0, keep)}\u2026[truncated ${rawLength - keep} chars]`;
|
|
73
|
+
out = `${out.slice(0, keep)}\u2026[truncated ${Math.max(rawLength, out.length) - keep} chars]`;
|
|
67
74
|
}
|
|
68
75
|
return out;
|
|
69
76
|
}
|
|
70
77
|
function stripTerminalEscapes(message) {
|
|
71
78
|
return message.replace(TERMINAL_ESCAPE_PATTERN, "").replace(CONTROL_CHARS_PATTERN, " ");
|
|
72
79
|
}
|
|
73
|
-
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TERMINAL_ESCAPE_PATTERN;
|
|
80
|
+
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
|
|
74
81
|
var init_log_utils = __esm({
|
|
75
82
|
"src/utils/log-utils.ts"() {
|
|
76
83
|
"use strict";
|
|
@@ -88,6 +95,8 @@ var init_log_utils = __esm({
|
|
|
88
95
|
LONG_HEX_SECRET_PATTERN = /\b[0-9a-fA-F]{32,}\b/g;
|
|
89
96
|
CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
|
|
90
97
|
REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
|
|
98
|
+
TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
|
|
99
|
+
REDACT_SCAN_EXTENSION_MAX = 2048;
|
|
91
100
|
TERMINAL_ESCAPE_PATTERN = new RegExp(
|
|
92
101
|
"\\x1b\\[[0-?]*[ -/]*[@-~]?|\\x1b\\][^\\x07\\x1b]*(?:\\x07|\\x1b\\\\)?|\\x1b[PX^_][^\\x1b]*(?:\\x1b\\\\)?|\\x1b.?",
|
|
93
102
|
"g"
|
|
@@ -360,9 +369,18 @@ var init_error_tracker = __esm({
|
|
|
360
369
|
import * as fs2 from "node:fs";
|
|
361
370
|
import * as path2 from "node:path";
|
|
362
371
|
function touchOwnerOnly(filePath, mode) {
|
|
372
|
+
let fd;
|
|
363
373
|
try {
|
|
364
|
-
fs2.
|
|
374
|
+
fd = fs2.openSync(filePath, APPEND_CREATE_NOFOLLOW_FLAGS, mode);
|
|
375
|
+
fs2.fchmodSync(fd, mode);
|
|
365
376
|
} catch {
|
|
377
|
+
} finally {
|
|
378
|
+
if (fd !== void 0) {
|
|
379
|
+
try {
|
|
380
|
+
fs2.closeSync(fd);
|
|
381
|
+
} catch {
|
|
382
|
+
}
|
|
383
|
+
}
|
|
366
384
|
}
|
|
367
385
|
}
|
|
368
386
|
var APPEND_CREATE_NOFOLLOW_FLAGS, LogRotation;
|
|
@@ -388,10 +406,6 @@ var init_log_rotation = __esm({
|
|
|
388
406
|
if (fs2.existsSync(logFile)) {
|
|
389
407
|
fs2.unlinkSync(logFile);
|
|
390
408
|
touchOwnerOnly(logFile, 384);
|
|
391
|
-
try {
|
|
392
|
-
fs2.chmodSync(logFile, 384);
|
|
393
|
-
} catch {
|
|
394
|
-
}
|
|
395
409
|
}
|
|
396
410
|
const files = fs2.readdirSync(logDir);
|
|
397
411
|
const baseName = path2.basename(logFile);
|
|
@@ -428,10 +442,6 @@ var init_log_rotation = __esm({
|
|
|
428
442
|
return;
|
|
429
443
|
}
|
|
430
444
|
touchOwnerOnly(logFile, 384);
|
|
431
|
-
try {
|
|
432
|
-
fs2.chmodSync(logFile, 384);
|
|
433
|
-
} catch {
|
|
434
|
-
}
|
|
435
445
|
try {
|
|
436
446
|
const files = fs2.readdirSync(logDir);
|
|
437
447
|
const logFiles = files.filter((f) => f.startsWith(path2.basename(logFile)) && f !== path2.basename(logFile)).sort();
|
|
@@ -1920,11 +1930,21 @@ import { platform as platform2 } from "node:os";
|
|
|
1920
1930
|
|
|
1921
1931
|
// src/infrastructure/browser/thread-worker-messaging.ts
|
|
1922
1932
|
init_log_utils();
|
|
1923
|
-
init_scraper_types();
|
|
1924
1933
|
import { appendFileSync as appendFileSync2 } from "node:fs";
|
|
1934
|
+
|
|
1935
|
+
// src/utils/safe-unref.ts
|
|
1936
|
+
function safeUnref(timer) {
|
|
1937
|
+
if (timer && typeof timer.unref === "function") {
|
|
1938
|
+
timer.unref();
|
|
1939
|
+
}
|
|
1940
|
+
}
|
|
1941
|
+
|
|
1942
|
+
// src/infrastructure/browser/thread-worker-messaging.ts
|
|
1943
|
+
init_scraper_types();
|
|
1925
1944
|
var workerId2 = "";
|
|
1926
1945
|
var pageCreationLock = Promise.resolve();
|
|
1927
1946
|
var PAGE_CREATION_HARD_RELEASE_MS = 5 * 60 * 1e3;
|
|
1947
|
+
var wedgedContext = null;
|
|
1928
1948
|
async function createPageSafe(context2) {
|
|
1929
1949
|
let release;
|
|
1930
1950
|
const nextLock = new Promise((resolve) => {
|
|
@@ -1933,6 +1953,10 @@ async function createPageSafe(context2) {
|
|
|
1933
1953
|
const currentLock = pageCreationLock;
|
|
1934
1954
|
pageCreationLock = currentLock.then(() => nextLock);
|
|
1935
1955
|
await currentLock;
|
|
1956
|
+
if (context2 === wedgedContext) {
|
|
1957
|
+
release();
|
|
1958
|
+
throw new Error("Browser context is wedged from a prior page-creation call that never settled; a new browser context is required.");
|
|
1959
|
+
}
|
|
1936
1960
|
const pagePromise = context2.newPage();
|
|
1937
1961
|
let hardReleased = false;
|
|
1938
1962
|
const releaseOnce = () => {
|
|
@@ -1943,9 +1967,20 @@ async function createPageSafe(context2) {
|
|
|
1943
1967
|
};
|
|
1944
1968
|
const hardReleaseTimer = setTimeout(() => {
|
|
1945
1969
|
logToDebugFile2("WARN", "[ThreadWorker] page-creation mutex force-released \u2014 newPage() did not settle within the hard timeout");
|
|
1970
|
+
wedgedContext = context2;
|
|
1946
1971
|
releaseOnce();
|
|
1947
1972
|
}, PAGE_CREATION_HARD_RELEASE_MS);
|
|
1948
|
-
|
|
1973
|
+
safeUnref(hardReleaseTimer);
|
|
1974
|
+
pagePromise.then(
|
|
1975
|
+
() => {
|
|
1976
|
+
if (wedgedContext === context2) wedgedContext = null;
|
|
1977
|
+
releaseOnce();
|
|
1978
|
+
},
|
|
1979
|
+
() => {
|
|
1980
|
+
if (wedgedContext === context2) wedgedContext = null;
|
|
1981
|
+
releaseOnce();
|
|
1982
|
+
}
|
|
1983
|
+
);
|
|
1949
1984
|
pagePromise.catch((err) => logToDebugFile2("DEBUG", `[ThreadWorker] Background page creation rejection: ${err.message}`));
|
|
1950
1985
|
let timeoutId;
|
|
1951
1986
|
try {
|
package/docs/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.3.7] - 2026-08-11
|
|
9
|
+
|
|
10
|
+
Discovery and packaging metadata only — no engine changes.
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- **README rewritten for concision and effectiveness.** The pitch ("free, unlimited web search & deep research for coding agents — no API key, no quota") now leads the page, install moved up to directly under the badges, each claim is stated once, and detail that only matters after adoption moved to `docs/`. 88 → 78 lines.
|
|
15
|
+
- **npm description and keywords rewritten.** The description is now benefit-led and survives catalog truncation; keywords gained `pi-coding-agent`, `web-research`, `deep-research`, `knowledge-store`, `duckduckgo`, and `skills`, and dropped `security` and `stack-exchange`.
|
|
16
|
+
|
|
17
|
+
## [1.3.6] - 2026-08-10
|
|
18
|
+
|
|
19
|
+
An independent fresh-eyes review of the entire 1.3.4→1.3.5 diff — the previous
|
|
20
|
+
release's own fixes had only ever been reviewed by their author. It found seven
|
|
21
|
+
further defects, several in the exact hard-release-timer and redaction-boundary
|
|
22
|
+
patterns 1.3.5 introduced. Every fix below is covered by a regression test
|
|
23
|
+
confirmed (via a local revert-and-rerun) to fail against the pre-fix code.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- **The page-creation and PoToken-mint hard-release timers (new in 1.3.5) could still let two calls race.** Both mutexes force-release after a 5-minute bound so a truly wedged call can't livelock the queue forever — but the bound has no way to tell "permanently hung" apart from "just very slow." A call that resolves a moment after the hard release fires was already let straight through to the next queued caller, who then ran its own `newPage()`/`doMint()` concurrently with the still-in-flight first one: the exact race each mutex exists to prevent (Playwright/Firefox deadlock on concurrent `newPage()`; corrupted `globalThis.window`/`document`/`navigator` from two overlapping bridge/unbridge pairs). Both now mark the wedged resource (the browser context; the whole PoToken minter, since `globalThis` is process-global) and reject further calls against it fast — no second `newPage()`/`doMint()` call — until the original stuck call actually settles.
|
|
28
|
+
- **`redactSecrets()`'s own cost-bounding pre-slice (new in 1.3.5) could itself cut a secret in half.** The 4×`MAX_LOG_MESSAGE_LENGTH` pre-slice added to bound redaction's cost on huge inputs runs before any pattern sees the text — the same class of mid-token truncation the truncate-before-redact fix (1.3.4) closed at the *final* boundary, just reopened at this new earlier one. A secret straddling that cut, in a message dense enough with other redactable content that the cut point's post-redaction position falls inside the final visible window, leaked its un-redacted fragment. The slice boundary now extends forward past any run of token-shaped characters (bounded to 2KB extra) so it never lands mid-secret.
|
|
29
|
+
- **`fs.chmodSync` right after the new `O_NOFOLLOW`-hardened `touchOwnerOnly()` (1.3.5) still followed symlinks.** The O_NOFOLLOW open correctly refused to write through a symlink planted at the log path, but the very next line re-stamped the file's mode with a plain, path-based `chmodSync` — which happily dereferences a symlink and silently chmods whatever it points at, undoing the hardening one line later. `touchOwnerOnly()` now stamps the mode itself via `fchmodSync` on the same already-open, already-verified file descriptor, and the separate path-based chmod calls are gone.
|
|
30
|
+
- **The truncation-count in a redacted log message could go negative.** 1.3.5 switched the reported "truncated N chars" count to the pre-redaction raw length so it would reflect content discarded by the earlier cost-bounding slice — but several redaction patterns *expand* text (`key=1` → `key=[REDACTED]`), so a short, redaction-dense message could end up with a post-redaction length exceeding the raw length, making the subtraction negative and printing the literal `…[truncated -1 chars]`. Now takes the larger of the raw and post-redaction lengths.
|
|
31
|
+
- **`reconcileSkillInstalls()`'s new (1.3.5) symlink-repoint copy fallback never swept its own staging leftovers.** The fallback stages a fresh copy into a dot-prefixed temp dir before an atomic rename into place; a process killed between the copy succeeding and the rename could leave that temp dir behind. The sibling copy-refresh branch already sweeps its own leftovers before every attempt — this branch didn't, and since a crash also frequently makes the manifest entry look deleted (pruning it before the branch runs again), repeated failures could accumulate orphaned staging directories with nothing ever revisiting them. Now sweeps unconditionally on every visit to a symlink entry, independent of whether that entry survives.
|
|
32
|
+
- **The page-creation mutex's hard-release timer (1.3.5) was never unref'd**, unlike the identical PoToken-mint timer in the same release. A bare `setTimeout` keeps the event loop alive for up to 5 minutes; any shutdown path that doesn't route through the cluster worker's forced `process.exit(0)` would hang waiting on it instead of exiting promptly. Now unref'd like its sibling.
|
|
33
|
+
|
|
8
34
|
## [1.3.5] - 2026-08-10
|
|
9
35
|
|
|
10
36
|
Two rounds of workflow-backed code review on the clean 1.3.4 tree (high-effort,
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lincoln504/pi-research",
|
|
3
|
-
"version": "1.3.
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "1.3.7",
|
|
4
|
+
"description": "Free, unlimited web search and deep research for the Pi coding agent — no search API key, no quotas. Local knowledge store; also works from Claude Code, Codex, and OpenClaw.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"private": false,
|
|
7
7
|
"main": "./src/index.ts",
|
|
@@ -52,16 +52,20 @@
|
|
|
52
52
|
"pi",
|
|
53
53
|
"pi-package",
|
|
54
54
|
"pi-extension",
|
|
55
|
+
"pi-coding-agent",
|
|
55
56
|
"openclaw",
|
|
56
57
|
"agent-skill",
|
|
58
|
+
"skills",
|
|
57
59
|
"research",
|
|
60
|
+
"web-research",
|
|
61
|
+
"deep-research",
|
|
58
62
|
"agent",
|
|
59
63
|
"web-search",
|
|
60
64
|
"scraping",
|
|
65
|
+
"knowledge-store",
|
|
66
|
+
"duckduckgo",
|
|
61
67
|
"stealth-browser",
|
|
62
|
-
"camoufox"
|
|
63
|
-
"security",
|
|
64
|
-
"stack-exchange"
|
|
68
|
+
"camoufox"
|
|
65
69
|
],
|
|
66
70
|
"author": "Lincoln Deen",
|
|
67
71
|
"contributors": [
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
name: pi-research
|
|
3
3
|
description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
|
|
4
4
|
license: MIT
|
|
5
|
-
metadata: { "author": "Lincoln Deen", "version": "1.3.
|
|
5
|
+
metadata: { "author": "Lincoln Deen", "version": "1.3.7", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
|
|
6
6
|
allowed-tools: Bash(node:*)
|
|
7
7
|
---
|
|
8
8
|
|
|
@@ -8,6 +8,7 @@ let workerId: string = '';
|
|
|
8
8
|
|
|
9
9
|
import { appendFileSync } from 'node:fs';
|
|
10
10
|
import { redactSecrets } from '../../utils/log-utils.ts';
|
|
11
|
+
import { safeUnref } from '../../utils/safe-unref.ts';
|
|
11
12
|
import { MAX_HTML_SIZE, MAX_PDF_SIZE } from '../../web-research/scraper-types.ts';
|
|
12
13
|
|
|
13
14
|
/**
|
|
@@ -21,6 +22,13 @@ let pageCreationLock = Promise.resolve();
|
|
|
21
22
|
* bound the pathological case where newPage() never settles at all (see comment below). */
|
|
22
23
|
const PAGE_CREATION_HARD_RELEASE_MS = 5 * 60 * 1000;
|
|
23
24
|
|
|
25
|
+
/** Set when a hard-release fires while its newPage() call is still pending —
|
|
26
|
+
* identifies the context that call was creating a page on. A concurrent
|
|
27
|
+
* newPage() on that SAME context is exactly what pageCreationLock exists to
|
|
28
|
+
* prevent, so any call for this context is rejected fast (without ever
|
|
29
|
+
* calling newPage()) until the stuck call actually settles and clears this. */
|
|
30
|
+
let wedgedContext: any = null;
|
|
31
|
+
|
|
24
32
|
/** Exported for unit testing of the page-creation mutex's timeout behavior. */
|
|
25
33
|
export async function createPageSafe(context: any): Promise<any> {
|
|
26
34
|
let release!: () => void;
|
|
@@ -29,6 +37,13 @@ export async function createPageSafe(context: any): Promise<any> {
|
|
|
29
37
|
pageCreationLock = currentLock.then(() => nextLock);
|
|
30
38
|
|
|
31
39
|
await currentLock;
|
|
40
|
+
if (context === wedgedContext) {
|
|
41
|
+
// Release immediately so the queue keeps flowing for OTHER contexts (or
|
|
42
|
+
// this one, once the stuck call clears wedgedContext) instead of stalling
|
|
43
|
+
// every future caller behind the still-pending call.
|
|
44
|
+
release();
|
|
45
|
+
throw new Error('Browser context is wedged from a prior page-creation call that never settled; a new browser context is required.');
|
|
46
|
+
}
|
|
32
47
|
const pagePromise = context.newPage();
|
|
33
48
|
// Release the mutex once newPage() itself has actually settled — NOT when the
|
|
34
49
|
// timeout race below settles. Releasing in an outer `finally` (the old shape)
|
|
@@ -43,7 +58,9 @@ export async function createPageSafe(context: any): Promise<any> {
|
|
|
43
58
|
// A hard release timer bounds the case where newPage() never settles at all
|
|
44
59
|
// (rather than just being slow) — without it, a single wedged newPage() call
|
|
45
60
|
// would hold this lock forever and livelock every future createPageSafe() call
|
|
46
|
-
// on this worker thread.
|
|
61
|
+
// on this worker thread. It marks the context wedged rather than letting the
|
|
62
|
+
// next caller straight through: a merely-slow (not hung) newPage() would
|
|
63
|
+
// otherwise still be in flight when a new one starts on the same context.
|
|
47
64
|
let hardReleased = false;
|
|
48
65
|
const releaseOnce = (): void => {
|
|
49
66
|
if (hardReleased) return;
|
|
@@ -53,9 +70,14 @@ export async function createPageSafe(context: any): Promise<any> {
|
|
|
53
70
|
};
|
|
54
71
|
const hardReleaseTimer = setTimeout(() => {
|
|
55
72
|
logToDebugFile('WARN', '[ThreadWorker] page-creation mutex force-released — newPage() did not settle within the hard timeout');
|
|
73
|
+
wedgedContext = context;
|
|
56
74
|
releaseOnce();
|
|
57
75
|
}, PAGE_CREATION_HARD_RELEASE_MS);
|
|
58
|
-
|
|
76
|
+
safeUnref(hardReleaseTimer);
|
|
77
|
+
pagePromise.then(
|
|
78
|
+
() => { if (wedgedContext === context) wedgedContext = null; releaseOnce(); },
|
|
79
|
+
() => { if (wedgedContext === context) wedgedContext = null; releaseOnce(); },
|
|
80
|
+
);
|
|
59
81
|
pagePromise.catch((err: Error) => logToDebugFile('DEBUG', `[ThreadWorker] Background page creation rejection: ${err.message}`));
|
|
60
82
|
|
|
61
83
|
let timeoutId: NodeJS.Timeout | undefined;
|
|
@@ -46,7 +46,14 @@ function formatArg(arg) {
|
|
|
46
46
|
}
|
|
47
47
|
function redactSecrets(message) {
|
|
48
48
|
const rawLength = message.length;
|
|
49
|
-
let
|
|
49
|
+
let scanEnd = REDACT_SCAN_LENGTH;
|
|
50
|
+
if (rawLength > scanEnd) {
|
|
51
|
+
const extensionLimit = Math.min(rawLength, scanEnd + REDACT_SCAN_EXTENSION_MAX);
|
|
52
|
+
while (scanEnd < extensionLimit && TOKEN_CHAR_PATTERN.test(message.charAt(scanEnd))) {
|
|
53
|
+
scanEnd += 1;
|
|
54
|
+
}
|
|
55
|
+
}
|
|
56
|
+
let out = rawLength > scanEnd ? message.slice(0, scanEnd) : message;
|
|
50
57
|
out = out.replace(ANSI_PATTERN, "");
|
|
51
58
|
out = out.replace(URL_CREDENTIALS_PATTERN, "$1[REDACTED]@");
|
|
52
59
|
out = out.replace(JWT_PATTERN, "[REDACTED]");
|
|
@@ -63,14 +70,14 @@ function redactSecrets(message) {
|
|
|
63
70
|
if (lastKeptUnit >= 55296 && lastKeptUnit <= 56319) keep -= 1;
|
|
64
71
|
}
|
|
65
72
|
if (out.length > MAX_LOG_MESSAGE_LENGTH) {
|
|
66
|
-
out = `${out.slice(0, keep)}\u2026[truncated ${rawLength - keep} chars]`;
|
|
73
|
+
out = `${out.slice(0, keep)}\u2026[truncated ${Math.max(rawLength, out.length) - keep} chars]`;
|
|
67
74
|
}
|
|
68
75
|
return out;
|
|
69
76
|
}
|
|
70
77
|
function stripTerminalEscapes(message) {
|
|
71
78
|
return message.replace(TERMINAL_ESCAPE_PATTERN, "").replace(CONTROL_CHARS_PATTERN, " ");
|
|
72
79
|
}
|
|
73
|
-
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TERMINAL_ESCAPE_PATTERN;
|
|
80
|
+
var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
|
|
74
81
|
var init_log_utils = __esm({
|
|
75
82
|
"src/utils/log-utils.ts"() {
|
|
76
83
|
"use strict";
|
|
@@ -88,6 +95,8 @@ var init_log_utils = __esm({
|
|
|
88
95
|
LONG_HEX_SECRET_PATTERN = /\b[0-9a-fA-F]{32,}\b/g;
|
|
89
96
|
CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
|
|
90
97
|
REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
|
|
98
|
+
TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
|
|
99
|
+
REDACT_SCAN_EXTENSION_MAX = 2048;
|
|
91
100
|
TERMINAL_ESCAPE_PATTERN = new RegExp(
|
|
92
101
|
"\\x1b\\[[0-?]*[ -/]*[@-~]?|\\x1b\\][^\\x07\\x1b]*(?:\\x07|\\x1b\\\\)?|\\x1b[PX^_][^\\x1b]*(?:\\x1b\\\\)?|\\x1b.?",
|
|
93
102
|
"g"
|
|
@@ -360,9 +369,18 @@ var init_error_tracker = __esm({
|
|
|
360
369
|
import * as fs2 from "node:fs";
|
|
361
370
|
import * as path2 from "node:path";
|
|
362
371
|
function touchOwnerOnly(filePath, mode) {
|
|
372
|
+
let fd;
|
|
363
373
|
try {
|
|
364
|
-
fs2.
|
|
374
|
+
fd = fs2.openSync(filePath, APPEND_CREATE_NOFOLLOW_FLAGS, mode);
|
|
375
|
+
fs2.fchmodSync(fd, mode);
|
|
365
376
|
} catch {
|
|
377
|
+
} finally {
|
|
378
|
+
if (fd !== void 0) {
|
|
379
|
+
try {
|
|
380
|
+
fs2.closeSync(fd);
|
|
381
|
+
} catch {
|
|
382
|
+
}
|
|
383
|
+
}
|
|
366
384
|
}
|
|
367
385
|
}
|
|
368
386
|
var APPEND_CREATE_NOFOLLOW_FLAGS, LogRotation;
|
|
@@ -388,10 +406,6 @@ var init_log_rotation = __esm({
|
|
|
388
406
|
if (fs2.existsSync(logFile)) {
|
|
389
407
|
fs2.unlinkSync(logFile);
|
|
390
408
|
touchOwnerOnly(logFile, 384);
|
|
391
|
-
try {
|
|
392
|
-
fs2.chmodSync(logFile, 384);
|
|
393
|
-
} catch {
|
|
394
|
-
}
|
|
395
409
|
}
|
|
396
410
|
const files = fs2.readdirSync(logDir);
|
|
397
411
|
const baseName = path2.basename(logFile);
|
|
@@ -428,10 +442,6 @@ var init_log_rotation = __esm({
|
|
|
428
442
|
return;
|
|
429
443
|
}
|
|
430
444
|
touchOwnerOnly(logFile, 384);
|
|
431
|
-
try {
|
|
432
|
-
fs2.chmodSync(logFile, 384);
|
|
433
|
-
} catch {
|
|
434
|
-
}
|
|
435
445
|
try {
|
|
436
446
|
const files = fs2.readdirSync(logDir);
|
|
437
447
|
const logFiles = files.filter((f) => f.startsWith(path2.basename(logFile)) && f !== path2.basename(logFile)).sort();
|
|
@@ -1920,11 +1930,21 @@ import { platform as platform2 } from "node:os";
|
|
|
1920
1930
|
|
|
1921
1931
|
// src/infrastructure/browser/thread-worker-messaging.ts
|
|
1922
1932
|
init_log_utils();
|
|
1923
|
-
init_scraper_types();
|
|
1924
1933
|
import { appendFileSync as appendFileSync2 } from "node:fs";
|
|
1934
|
+
|
|
1935
|
+
// src/utils/safe-unref.ts
|
|
1936
|
+
function safeUnref(timer) {
|
|
1937
|
+
if (timer && typeof timer.unref === "function") {
|
|
1938
|
+
timer.unref();
|
|
1939
|
+
}
|
|
1940
|
+
}
|
|
1941
|
+
|
|
1942
|
+
// src/infrastructure/browser/thread-worker-messaging.ts
|
|
1943
|
+
init_scraper_types();
|
|
1925
1944
|
var workerId2 = "";
|
|
1926
1945
|
var pageCreationLock = Promise.resolve();
|
|
1927
1946
|
var PAGE_CREATION_HARD_RELEASE_MS = 5 * 60 * 1e3;
|
|
1947
|
+
var wedgedContext = null;
|
|
1928
1948
|
async function createPageSafe(context2) {
|
|
1929
1949
|
let release;
|
|
1930
1950
|
const nextLock = new Promise((resolve) => {
|
|
@@ -1933,6 +1953,10 @@ async function createPageSafe(context2) {
|
|
|
1933
1953
|
const currentLock = pageCreationLock;
|
|
1934
1954
|
pageCreationLock = currentLock.then(() => nextLock);
|
|
1935
1955
|
await currentLock;
|
|
1956
|
+
if (context2 === wedgedContext) {
|
|
1957
|
+
release();
|
|
1958
|
+
throw new Error("Browser context is wedged from a prior page-creation call that never settled; a new browser context is required.");
|
|
1959
|
+
}
|
|
1936
1960
|
const pagePromise = context2.newPage();
|
|
1937
1961
|
let hardReleased = false;
|
|
1938
1962
|
const releaseOnce = () => {
|
|
@@ -1943,9 +1967,20 @@ async function createPageSafe(context2) {
|
|
|
1943
1967
|
};
|
|
1944
1968
|
const hardReleaseTimer = setTimeout(() => {
|
|
1945
1969
|
logToDebugFile2("WARN", "[ThreadWorker] page-creation mutex force-released \u2014 newPage() did not settle within the hard timeout");
|
|
1970
|
+
wedgedContext = context2;
|
|
1946
1971
|
releaseOnce();
|
|
1947
1972
|
}, PAGE_CREATION_HARD_RELEASE_MS);
|
|
1948
|
-
|
|
1973
|
+
safeUnref(hardReleaseTimer);
|
|
1974
|
+
pagePromise.then(
|
|
1975
|
+
() => {
|
|
1976
|
+
if (wedgedContext === context2) wedgedContext = null;
|
|
1977
|
+
releaseOnce();
|
|
1978
|
+
},
|
|
1979
|
+
() => {
|
|
1980
|
+
if (wedgedContext === context2) wedgedContext = null;
|
|
1981
|
+
releaseOnce();
|
|
1982
|
+
}
|
|
1983
|
+
);
|
|
1949
1984
|
pagePromise.catch((err) => logToDebugFile2("DEBUG", `[ThreadWorker] Background page creation rejection: ${err.message}`));
|
|
1950
1985
|
let timeoutId;
|
|
1951
1986
|
try {
|
|
@@ -631,6 +631,22 @@ export function reconcileSkillInstalls(opts: InstallOptions = {}): ReconcileResu
|
|
|
631
631
|
continue;
|
|
632
632
|
}
|
|
633
633
|
|
|
634
|
+
// Sweep any `.{base}.new-*` staging leftover from a previous crashed
|
|
635
|
+
// symlink-repoint attempt at this path, unconditionally — a process kill
|
|
636
|
+
// between copyDir succeeding and the final renameSync below can leave one
|
|
637
|
+
// behind, and this branch is not guaranteed to run again for the same
|
|
638
|
+
// entry (a crash also often leaves isSymlinkPresent() false, pruning the
|
|
639
|
+
// entry on the very next line before it would otherwise get a chance to).
|
|
640
|
+
const dir = path.dirname(e.path);
|
|
641
|
+
const base = path.basename(e.path);
|
|
642
|
+
try {
|
|
643
|
+
for (const name of fs.readdirSync(dir)) {
|
|
644
|
+
if (name.startsWith(`.${base}.new-`)) {
|
|
645
|
+
fs.rmSync(path.join(dir, name), { recursive: true, force: true });
|
|
646
|
+
}
|
|
647
|
+
}
|
|
648
|
+
} catch { /* hygiene only */ }
|
|
649
|
+
|
|
634
650
|
// Link the user deleted by hand — nothing on disk; drop the stale entry.
|
|
635
651
|
if (!isSymlinkPresent(e.path)) { result.pruned.push(e.path); continue; }
|
|
636
652
|
|
|
@@ -662,8 +678,6 @@ export function reconcileSkillInstalls(opts: InstallOptions = {}): ReconcileResu
|
|
|
662
678
|
// partial, SKILL.md-less directory permanently orphaned on disk
|
|
663
679
|
// (unrecognized by isOwnedCopy) when copyDir failed partway (disk
|
|
664
680
|
// full, interrupted process).
|
|
665
|
-
const dir = path.dirname(e.path);
|
|
666
|
-
const base = path.basename(e.path);
|
|
667
681
|
const suffix = crypto.randomBytes(4).toString('hex');
|
|
668
682
|
const fresh = path.join(dir, `.${base}.new-${suffix}`);
|
|
669
683
|
try {
|
|
@@ -21,12 +21,27 @@ const APPEND_CREATE_NOFOLLOW_FLAGS =
|
|
|
21
21
|
(typeof fs.constants.O_NOFOLLOW === 'number' ? fs.constants.O_NOFOLLOW : 0);
|
|
22
22
|
|
|
23
23
|
/** Best-effort (re)create `filePath` with `mode`, refusing to follow a symlink
|
|
24
|
-
* planted at the path
|
|
24
|
+
* planted at the path, and unconditionally stamp `mode` on it. Uses fchmodSync
|
|
25
|
+
* on the already-open (and O_NOFOLLOW-verified) descriptor rather than
|
|
26
|
+
* fs.chmodSync(filePath, ...) — chmodSync operates on the path and follows
|
|
27
|
+
* symlinks, which would silently re-open the exact hole O_NOFOLLOW closes if
|
|
28
|
+
* a symlink appears at this path between the open above and a path-based
|
|
29
|
+
* chmod. mode is only applied by open() when O_CREAT actually creates the
|
|
30
|
+
* file, so callers relying on the file always ending up at `mode` (even when
|
|
31
|
+
* a concurrent writer already recreated it) still need this explicit stamp.
|
|
32
|
+
* Failures are swallowed by design — see call sites.
|
|
25
33
|
* Exported for unit testing of the symlink-refusal behavior. */
|
|
26
34
|
export function touchOwnerOnly(filePath: string, mode: number): void {
|
|
35
|
+
let fd: number | undefined;
|
|
27
36
|
try {
|
|
28
|
-
fs.
|
|
37
|
+
fd = fs.openSync(filePath, APPEND_CREATE_NOFOLLOW_FLAGS, mode);
|
|
38
|
+
fs.fchmodSync(fd, mode);
|
|
29
39
|
} catch { /* best-effort — the next append recreates the file */ }
|
|
40
|
+
finally {
|
|
41
|
+
if (fd !== undefined) {
|
|
42
|
+
try { fs.closeSync(fd); } catch { /* best-effort */ }
|
|
43
|
+
}
|
|
44
|
+
}
|
|
30
45
|
}
|
|
31
46
|
|
|
32
47
|
export class LogRotation {
|
|
@@ -48,13 +63,11 @@ export class LogRotation {
|
|
|
48
63
|
fs.unlinkSync(logFile);
|
|
49
64
|
// Same rationale as the post-rotation recreate in rotateLogFile: the
|
|
50
65
|
// unlink discarded the 0o600 file, and a umask-default recreate by the
|
|
51
|
-
// next append would drop the owner-only posture.
|
|
66
|
+
// next append would drop the owner-only posture. touchOwnerOnly stamps
|
|
67
|
+
// the mode unconditionally (via fchmod on its own fd), covering the
|
|
68
|
+
// case where a concurrent append already recreated the file at umask
|
|
69
|
+
// perms between the unlink and this call.
|
|
52
70
|
touchOwnerOnly(logFile, 0o600);
|
|
53
|
-
// And, as there, re-stamp the mode: a concurrent append landing between
|
|
54
|
-
// the unlink and the open makes the creation-time 0o600 a no-op.
|
|
55
|
-
try {
|
|
56
|
-
fs.chmodSync(logFile, 0o600);
|
|
57
|
-
} catch { /* best-effort */ }
|
|
58
71
|
}
|
|
59
72
|
|
|
60
73
|
const files = fs.readdirSync(logDir);
|
|
@@ -109,18 +122,12 @@ export class LogRotation {
|
|
|
109
122
|
// Rotation moved the 0o600 file aside; without an explicit recreate the next
|
|
110
123
|
// append materializes it at the umask default (typically 0644) in a
|
|
111
124
|
// world-traversable tmpdir, silently dropping the owner-only posture Logger
|
|
112
|
-
// establishes at construction.
|
|
113
|
-
//
|
|
114
|
-
|
|
115
|
-
//
|
|
116
|
-
// processes append to this consolidated log, so a concurrent async append
|
|
117
|
-
// can recreate it at umask perms between the rename and the open, turning
|
|
118
|
-
// the 0o600 into a no-op — stamp the mode unconditionally to close that
|
|
119
|
-
// window for good. ENOENT (nothing recreated it and the open failed) is
|
|
125
|
+
// establishes at construction. touchOwnerOnly stamps the mode
|
|
126
|
+
// unconditionally (via fchmod on its own fd) regardless of whether this
|
|
127
|
+
// open created the file or a concurrent append already recreated it
|
|
128
|
+
// between the rename and this call — ENOENT (open itself failed) is
|
|
120
129
|
// tolerable: best-effort posture, same as the recreate itself.
|
|
121
|
-
|
|
122
|
-
fs.chmodSync(logFile, 0o600);
|
|
123
|
-
} catch { /* best-effort */ }
|
|
130
|
+
touchOwnerOnly(logFile, 0o600);
|
|
124
131
|
|
|
125
132
|
|
|
126
133
|
// Clean up old archives
|
package/src/utils/log-utils.ts
CHANGED
|
@@ -143,6 +143,15 @@ const CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
|
|
|
143
143
|
// O(input length) on multi-megabyte scraped payloads.
|
|
144
144
|
const REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
|
|
145
145
|
|
|
146
|
+
// Characters that can appear inside an unbroken secret token (hex, base64url,
|
|
147
|
+
// JWT segments, API keys). Used to push the REDACT_SCAN_LENGTH cut point past
|
|
148
|
+
// the end of whatever token it would otherwise land in the middle of.
|
|
149
|
+
const TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
|
|
150
|
+
// Hard cap on how far the boundary can be pushed past REDACT_SCAN_LENGTH —
|
|
151
|
+
// bounds the extra work to a fixed amount even against an adversarial run of
|
|
152
|
+
// token-like characters.
|
|
153
|
+
const REDACT_SCAN_EXTENSION_MAX = 2048;
|
|
154
|
+
|
|
146
155
|
export function redactSecrets(message: string): string {
|
|
147
156
|
// Redact BEFORE truncating. Every pattern below needs to see a secret in
|
|
148
157
|
// full to match it (e.g. LONG_HEX_SECRET_PATTERN requires 32+ contiguous
|
|
@@ -151,7 +160,17 @@ export function redactSecrets(message: string): string {
|
|
|
151
160
|
// browser auth secret) too short for any pattern to recognize, so it was
|
|
152
161
|
// written out in the clear right before the "…[truncated N chars]" marker.
|
|
153
162
|
const rawLength = message.length;
|
|
154
|
-
|
|
163
|
+
// The REDACT_SCAN_LENGTH cut below is itself a truncation, subject to the
|
|
164
|
+
// exact same mid-token risk — extend it forward past any run of token
|
|
165
|
+
// characters straddling the cut point so it never slices a secret in half.
|
|
166
|
+
let scanEnd = REDACT_SCAN_LENGTH;
|
|
167
|
+
if (rawLength > scanEnd) {
|
|
168
|
+
const extensionLimit = Math.min(rawLength, scanEnd + REDACT_SCAN_EXTENSION_MAX);
|
|
169
|
+
while (scanEnd < extensionLimit && TOKEN_CHAR_PATTERN.test(message.charAt(scanEnd))) {
|
|
170
|
+
scanEnd += 1;
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
let out = rawLength > scanEnd ? message.slice(0, scanEnd) : message;
|
|
155
174
|
out = out.replace(ANSI_PATTERN, '');
|
|
156
175
|
out = out.replace(URL_CREDENTIALS_PATTERN, '$1[REDACTED]@');
|
|
157
176
|
// Redact whole-token formats BEFORE the key/value pass: the KV value class
|
|
@@ -183,7 +202,12 @@ export function redactSecrets(message: string): string {
|
|
|
183
202
|
if (lastKeptUnit >= 0xd800 && lastKeptUnit <= 0xdbff) keep -= 1;
|
|
184
203
|
}
|
|
185
204
|
if (out.length > MAX_LOG_MESSAGE_LENGTH) {
|
|
186
|
-
|
|
205
|
+
// rawLength can be SMALLER than out.length when redaction net-expands text
|
|
206
|
+
// (e.g. "key=1" → "key=[REDACTED]"), which would make `rawLength - keep`
|
|
207
|
+
// negative. out.length is always > keep in this branch, so the max() is
|
|
208
|
+
// never negative; it still prefers the larger rawLength when the
|
|
209
|
+
// REDACT_SCAN_LENGTH cut discarded raw content redaction then shrank.
|
|
210
|
+
out = `${out.slice(0, keep)}…[truncated ${Math.max(rawLength, out.length) - keep} chars]`;
|
|
187
211
|
}
|
|
188
212
|
return out;
|
|
189
213
|
}
|
package/src/youtube/potoken.ts
CHANGED
|
@@ -77,6 +77,13 @@ function describeError(error: unknown): string {
|
|
|
77
77
|
// Module-level mutex: serialize all globalThis mutation across concurrent callers.
|
|
78
78
|
let mintChain: Promise<unknown> = Promise.resolve();
|
|
79
79
|
|
|
80
|
+
// Set when a hard-release fires while its doMint() is still bridging globals —
|
|
81
|
+
// globalThis is a single process-global resource, so unlike a per-context lock
|
|
82
|
+
// ANY new bridge while one is still installed is unsafe, not just one from the
|
|
83
|
+
// same caller. Blocks new mints until the stuck call's own doMint() actually
|
|
84
|
+
// settles (success or failure) and unbridges for real.
|
|
85
|
+
let wedged = false;
|
|
86
|
+
|
|
80
87
|
// How long the mutex waits for a straggling doMint() before force-releasing
|
|
81
88
|
// the chain for the next caller. doMint() awaits non-fetch operations with no
|
|
82
89
|
// abort signal (interpreter eval, BotGuard VM execution against a
|
|
@@ -98,19 +105,34 @@ export async function mintPoTokens(
|
|
|
98
105
|
identifiers: string[],
|
|
99
106
|
opts: MintOptions = {},
|
|
100
107
|
): Promise<Map<string, string>> {
|
|
108
|
+
if (wedged) {
|
|
109
|
+
throw new Error('PoToken minter is wedged by a prior mint that never settled; refusing to start a new mint until it resolves.');
|
|
110
|
+
}
|
|
101
111
|
// Chain on the mutex so only one bridged section runs at a time.
|
|
102
112
|
const run = mintChain.then(() => doMint(identifiers, opts), () => doMint(identifiers, opts));
|
|
103
113
|
// Keep the chain alive regardless of this call's success/failure, but bound
|
|
104
114
|
// how long the NEXT caller waits on it: if `run` never settles, force the
|
|
105
115
|
// chain open after the hard timeout rather than wedging every future
|
|
106
|
-
// mintPoTokens() call behind it forever.
|
|
107
|
-
//
|
|
108
|
-
//
|
|
116
|
+
// mintPoTokens() call behind it forever. Marking `wedged` (rather than
|
|
117
|
+
// letting the next caller straight through to its own doMint()) prevents
|
|
118
|
+
// two calls from bridging globalThis at once — the abandoned `run` may
|
|
119
|
+
// still be mid-bridge in the background, and a second bridge/unbridge pair
|
|
120
|
+
// racing it would restore the WRONG saved descriptors, permanently
|
|
121
|
+
// corrupting globalThis.window/document/navigator for the process.
|
|
122
|
+
//
|
|
123
|
+
// Derived from `settled` (which never rejects) rather than attaching
|
|
124
|
+
// directly to `run` — `run.finally(...)` would return a new, unconsumed
|
|
125
|
+
// promise that adopts `run`'s rejection, becoming an unhandled rejection
|
|
126
|
+
// whenever doMint() throws.
|
|
109
127
|
const settled = run.then(() => undefined, () => undefined);
|
|
128
|
+
// Once doMint() actually settles (success or failure), its own `finally`
|
|
129
|
+
// always runs unbridge() for real — safe for a new mint to bridge again.
|
|
130
|
+
settled.then(() => { wedged = false; });
|
|
110
131
|
let hardTimer: NodeJS.Timeout;
|
|
111
132
|
const hardRelease = new Promise<void>(resolve => {
|
|
112
133
|
hardTimer = setTimeout(() => {
|
|
113
134
|
logger.warn('[youtube] potoken mint mutex force-released — doMint() did not settle within the hard timeout');
|
|
135
|
+
wedged = true;
|
|
114
136
|
resolve();
|
|
115
137
|
}, MINT_HARD_RELEASE_MS);
|
|
116
138
|
safeUnref(hardTimer);
|