@apmantza/greedysearch-pi 2.1.3 → 2.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -1
- package/README.md +52 -175
- package/bin/search.mjs +2 -0
- package/docs/analysis.md +233 -0
- package/docs/banner.svg +86 -0
- package/docs/development.md +63 -0
- package/docs/inspiration2.md +190 -0
- package/docs/releases.md +88 -0
- package/docs/research.md +82 -0
- package/docs/runtime.md +65 -0
- package/docs/source-fetching.md +33 -0
- package/docs/stealthbrowsermcp.md +807 -0
- package/docs/usage.md +69 -0
- package/extractors/chatgpt.mjs +127 -23
- package/extractors/common.mjs +116 -28
- package/extractors/gemini.mjs +63 -38
- package/package.json +13 -7
- package/scripts/backfill-github-releases.mjs +139 -0
- package/scripts/changelog-extract.mjs +48 -0
- package/scripts/changelog-release.mjs +65 -0
- package/scripts/check-lockfile.mjs +45 -0
- package/scripts/lib/changelog.mjs +168 -0
- package/scripts/lint.mjs +62 -0
- package/scripts/run-tests.ps1 +179 -0
- package/scripts/stealth-check.mjs +663 -0
- package/src/fetcher.mjs +9 -5
- package/src/search/constants.mjs +1 -1
- package/src/search/research.mjs +57 -11
- package/src/search/simple-research.mjs +4 -1
- package/test.mjs +16 -3
- package/skills/greedy-search/skill.md +0 -20
package/CHANGELOG.md
CHANGED
|
@@ -1,6 +1,53 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
## [Unreleased]
|
|
3
|
+
## [Unreleased]
|
|
4
|
+
|
|
5
|
+
### Added
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
### Fixed
|
|
10
|
+
|
|
11
|
+
## [2.1.5] — 2026-07-04
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Stealth-check script** (`scripts/stealth-check.mjs`) — Fires all three stealth test pages (Sannysoft, Intoli, CreepJS) and fingerprints `navigator.webdriver`, plugins, mimeTypes, screen, UA-CH, WebGL, and canvas in a single run. Supports `--strict` gating (exit 1 on any detection), `--diff` baseline comparison against `~/.greedysearch/stealth-baseline.json`, and structured CreepJS score extraction. Scripts: `npm run stealth-check`, `stealth:strict`, `stealth:diff`.
|
|
16
|
+
- **`navigator.webdriver` fully deleted** (`extractors/common.mjs`) — Both `navigator.webdriver` and `Navigator.prototype.webdriver` are deleted instead of being set to `undefined`. Sannysoft: `webdriver present` → `missing (passed)`.
|
|
17
|
+
- **PluginArray/MimeTypeArray shape** (`extractors/common.mjs`) — `navigator.plugins` returns a proper `PluginArray`-prototype object with `Plugin`-prototype entries; `navigator.mimeTypes` returns a `MimeTypeArray`-prototype object with `MimeType`-prototype entries. Sannysoft: `pluginsType` → `passed`.
|
|
18
|
+
- **Screen metrics consistency** (`extractors/common.mjs`) — `screen.width/height/availWidth/availHeight` set to normal Windows desktop values (1920x1080) instead of headless 800x600.
|
|
19
|
+
- **Canvas/WebGL/Audio fingerprint noise** (`extractors/common.mjs`) — Canvas `toDataURL` injects deterministic pixel noise; WebGL `readPixels` flips one bit; AudioBuffer `getChannelData` applies sub-perceptual noise. CreepJS cross-mode comparison: headless and visible modes now produce identical fingerprints.
|
|
20
|
+
- **`chrome.loadTimes` and `chrome.csi` enriched** (`extractors/common.mjs`) — Returns realistic timing values instead of empty `{}`.
|
|
21
|
+
- **`navigator.productSub` and `navigator.product`** (`extractors/common.mjs`) — Set to `20030107` and `Gecko` respectively (matching real Chrome).
|
|
22
|
+
- **`navigator.share` and `navigator.contentIndex` stubs** (`extractors/common.mjs`) — Missing platform APIs in headless mode are now defined.
|
|
23
|
+
- **`navigator.connection.downlinkMax`** (`extractors/common.mjs`) — Added `downlinkMax: Infinity` to the connection patch.
|
|
24
|
+
- **Stealth assessment doc** (`docs/analysis.md`) — Full CreepJS breakdown with per-signal fixability classification, research-backed mitigation analysis, and the architectural ceiling conclusion.
|
|
25
|
+
- **stealth-browser-mcp assessment** (`docs/stealthbrowsermcp.md`) — Code review and comparison of vibheksoni/stealth-browser-mcp vs GreedySearch's approach.
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- **Stealth injection refined** (`extractors/common.mjs`) — WebDriver now uses `delete` instead of `Object.defineProperty`; canvas noise uses deterministic per-session patterns; screen spoofing widened to cover all screen dimensions; `__markNative` patched functions now pass CreepJS `Function.prototype.toString` deep inspection.
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
|
|
33
|
+
- **CDP `Runtime.evaluate` JSON.parse crash** (`extractors/common.mjs` in `getOrOpenTab`) — `JSON.parse(raw)` wrapped in try/catch with descriptive error message.
|
|
34
|
+
|
|
35
|
+
## [2.1.4] — 2026-07-04
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- **Changelog-driven release notes** (`scripts/changelog-extract.mjs`, `scripts/changelog-release.mjs`, `scripts/backfill-github-releases.mjs`, `.github/workflows/release.yml`) — GitHub releases now use curated CHANGELOG sections as the release body, with helper scripts for promoting `Unreleased`, extracting summary notes, and backfilling historical releases. The parser supports both current `## [x.y.z] — date` headings and legacy `## vx.y.z (date)` headings.
|
|
40
|
+
|
|
41
|
+
### Changed
|
|
42
|
+
|
|
43
|
+
- **README simplified like pi-lens and package skill removed** (`README.md`, `docs/*.md`, `package.json`) — The root README is now a concise project overview with installation, quick usage, and links to focused docs for usage, research mode, runtime/Chrome, source fetching, development, and releases. The duplicated Pi skill file was removed so docs live in one place.
|
|
44
|
+
- **Normal all-engine config excludes Semantic Scholar** (`src/search/constants.mjs`, `~/.pi/greedyconfig`) — Default and local all-engine fan-out now use Perplexity, Google, ChatGPT, and Gemini while academic/research engines remain opt-in.
|
|
45
|
+
|
|
46
|
+
### Fixed
|
|
47
|
+
|
|
48
|
+
- **Headless ChatGPT and Gemini extraction in all-mode** (`extractors/chatgpt.mjs`, `extractors/gemini.mjs`, `bin/search.mjs`) — ChatGPT now keeps its tab foregrounded during headless streaming, waits for streaming indicators to clear, and rejects repeated-domain citation stubs. Gemini now targets the latest `model-response`, avoids nested code-block copy buttons, and prefers a complete DOM fallback when clipboard copy captures only a snippet.
|
|
49
|
+
- **Research-mode floor and citation handling** (`src/search/research.mjs`, `src/search/simple-research.mjs`) — Explicit `breadth` / `iterations` values now bypass scale-aware overrides, duplicate fetch targets are skipped, cited-source URL checks only evaluate cited sources, and bot-protected HEAD responses are skipped instead of treated as dead links.
|
|
50
|
+
- **Fetcher return style consistency** (`src/fetcher.mjs`) — Multi-line formatting for the protocol-not-allowed return object in `isPrivateUrl` to match the project's code style conventions.
|
|
4
51
|
|
|
5
52
|
## [2.1.3] — 2026-06-21
|
|
6
53
|
|
package/README.md
CHANGED
|
@@ -1,14 +1,19 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<img src="docs/banner.svg" alt="GreedySearch for Pi" width="1100">
|
|
3
|
+
</p>
|
|
4
|
+
|
|
1
5
|
# GreedySearch for Pi
|
|
2
6
|
|
|
3
|
-
|
|
7
|
+
GreedySearch registers the `greedy_search` tool for Pi: headless-first,
|
|
8
|
+
no-API-key AI/web search through a dedicated Chrome profile.
|
|
4
9
|
|
|
5
|
-
|
|
10
|
+
## What It Does
|
|
6
11
|
|
|
7
|
-
-
|
|
8
|
-
-
|
|
9
|
-
-
|
|
10
|
-
-
|
|
11
|
-
- Chrome
|
|
12
|
+
- Multi-engine search across Perplexity, Google AI, ChatGPT, and Gemini
|
|
13
|
+
- Source-grounded `engine: "all"` results with fetched source content
|
|
14
|
+
- Optional synthesis over engine answers and fetched sources
|
|
15
|
+
- Iterative `depth: "research"` runs with citation audit and research bundles
|
|
16
|
+
- Visible Chrome fallback for login/captcha/cookie setup when needed
|
|
12
17
|
|
|
13
18
|
## Install
|
|
14
19
|
|
|
@@ -22,16 +27,12 @@ Or from git:
|
|
|
22
27
|
pi install git:github.com/apmantza/GreedySearch-pi
|
|
23
28
|
```
|
|
24
29
|
|
|
25
|
-
##
|
|
26
|
-
|
|
27
|
-
- `greedy_search` — multi-engine AI web search, source-grounded synthesis, and deep research
|
|
28
|
-
|
|
29
|
-
## Quick usage
|
|
30
|
+
## Quick Usage
|
|
30
31
|
|
|
31
32
|
```js
|
|
32
|
-
greedy_search({ query: "React 19 changes" });
|
|
33
|
-
greedy_search({ query: "React 19 changes", synthesize: true });
|
|
34
|
-
greedy_search({ query: "Prisma vs Drizzle", engine: "perplexity" });
|
|
33
|
+
greedy_search({ query: "React 19 changes" });
|
|
34
|
+
greedy_search({ query: "React 19 changes", synthesize: true });
|
|
35
|
+
greedy_search({ query: "Prisma vs Drizzle", engine: "perplexity" });
|
|
35
36
|
greedy_search({
|
|
36
37
|
query: "Evaluate browser automation options for AI agents",
|
|
37
38
|
depth: "research",
|
|
@@ -39,181 +40,57 @@ greedy_search({
|
|
|
39
40
|
iterations: 2,
|
|
40
41
|
maxSources: 8,
|
|
41
42
|
});
|
|
42
|
-
// Research mode writes a dataroom-style bundle under .pi/greedysearch-research/ by default.
|
|
43
|
-
// Headless is the default — no window. To force visible Chrome:
|
|
44
|
-
greedy_search({ query: "Visible browser setup", engine: "perplexity", visible: true });
|
|
45
43
|
```
|
|
46
44
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
### Common
|
|
50
|
-
|
|
51
|
-
- `query` (required)
|
|
52
|
-
- `fullAnswer`: return full single-engine output instead of preview
|
|
53
|
-
- `headless`: set to `false` to show Chrome window (default: `true`)
|
|
54
|
-
- `visible` / `alwaysVisible`: set to `true` to always use visible Chrome for this search
|
|
55
|
-
|
|
56
|
-
### Normal search
|
|
57
|
-
|
|
58
|
-
- `engine`: `all` (default web/search fan-out), `perplexity`, `google`, `chatgpt`, `gemini`; opt-in research engines: `semantic-scholar`, `logically`; `bing` still works for signed-in users
|
|
59
|
-
- `synthesize`: for `engine: "all"`, synthesize fetched sources with the configured synthesizer (default false)
|
|
60
|
-
- `synthesizer`: override the configured synthesis engine for this call (`gemini` default, `chatgpt` supported)
|
|
61
|
-
- `depth`: legacy `fast`/`standard`/`deep` aliases are still accepted for compatibility; prefer `synthesize` for normal search
|
|
62
|
-
|
|
63
|
-
### Deep research
|
|
64
|
-
|
|
65
|
-
- `depth`: set to `"research"` to run the iterative research workflow
|
|
66
|
-
- `breadth`: number of research actions per round, 1-5 (default 3)
|
|
67
|
-
- `iterations`: research rounds, 1-3 (default 2)
|
|
68
|
-
- `maxSources`: fetched source cap for the final report, 3-12
|
|
69
|
-
- `researchOutDir`: optional directory for the research bundle
|
|
70
|
-
- `writeResearchBundle`: write the research bundle to disk (default true for research mode)
|
|
71
|
-
|
|
72
|
-
Deep research uses the configured `~/.pi/greedyconfig.engines` list for child searches and Gemini for planning/final synthesis.
|
|
73
|
-
|
|
74
|
-
## Environment variables
|
|
75
|
-
|
|
76
|
-
| Variable | Default | Description |
|
|
77
|
-
| ------------------------------------ | ------------- | ------------------------------------------------------------- |
|
|
78
|
-
| `GREEDY_SEARCH_VISIBLE` | (unset) | Set to `1` to show Chrome window instead of headless |
|
|
79
|
-
| `GREEDY_SEARCH_ALWAYS_VISIBLE` | (unset) | Set to `1` to force visible Chrome for all GreedySearch runs |
|
|
80
|
-
| `GREEDY_SEARCH_IDLE_TIMEOUT_MINUTES` | `5` | Minutes of inactivity before auto-killing GreedySearch Chrome |
|
|
81
|
-
| `GREEDY_SEARCH_LOCALE` | `en` | Default result language (en, de, fr, es, ja, etc.) |
|
|
82
|
-
| `CHROME_PATH` | auto-detected | Path to Chrome/Chromium executable |
|
|
45
|
+
Headless is the default. Use `visible: true` only when you need to establish a
|
|
46
|
+
session, solve a challenge, or inspect the browser:
|
|
83
47
|
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
- **Individual engine search/research** — `engine: "perplexity" | "google" | "chatgpt" | "gemini" | "semantic-scholar" | "logically" | "bing"`; returns that engine's answer and sources.
|
|
87
|
-
- **Grounded multi-engine search** — default `engine: "all"`; fans out to configured engines, ranks sources, fetches top source content, and reports confidence metadata.
|
|
88
|
-
- **All + synthesis** — add `synthesize: true` (or CLI `--synthesize`) to ask the configured synthesizer to combine engine answers and fetched source evidence.
|
|
89
|
-
- **Deep research** — `depth: "research"`; iterative action planning, direct URL fetches, fast multi-engine searches, source fetching, learning extraction, deterministic floor checks, citation audit, a final cited report, and a structured on-disk bundle.
|
|
90
|
-
|
|
91
|
-
Legacy `depth: "fast" | "standard" | "deep"` values remain accepted for compatibility: `fast` skips source fetching; `standard`/`deep` request synthesis.
|
|
92
|
-
|
|
93
|
-
Configure all-engine fan-out and synthesis in `~/.pi/greedyconfig`:
|
|
94
|
-
|
|
95
|
-
```json
|
|
96
|
-
{
|
|
97
|
-
"engines": ["perplexity", "google", "chatgpt", "gemini"],
|
|
98
|
-
"synthesizer": "gemini"
|
|
99
|
-
}
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
Gemini is a normal search engine and can participate in `engine: "all"`. `semantic-scholar` and `logically` are opt-in academic/research engines; include them in `~/.pi/greedyconfig` only when you want the all-engine fan-out to include academic paper discovery or research-assistant workflows. Default `engine: "all"` excludes them because their results are noisy for casual web search — they shine in `depth: "research"` mode instead. Deep research child searches reuse the same configured `engines` list and keep query text on stdin; Gemini remains the research planner/final-report synthesizer. If `synthesize: true` and `"synthesizer": "gemini"`, Gemini runs once as a search engine and again as the synthesizer; set `"synthesizer": "chatgpt"` to separate those roles for normal all-search synthesis.
|
|
103
|
-
|
|
104
|
-
Research bundles are written by default to `.pi/greedysearch-research/<timestamp>_<query>/` and include:
|
|
105
|
-
|
|
106
|
-
```text
|
|
107
|
-
STATUS.md # floor status, open/closed question ledger, and gaps
|
|
108
|
-
OUTLINE.md # bundle table of contents
|
|
109
|
-
reports/SUMMARY.md # final cited report
|
|
110
|
-
reports/CLAIMS.md # extracted claims mapped to source IDs
|
|
111
|
-
reports/EVIDENCE.md # goal-based evidence extracted from fetched sources
|
|
112
|
-
reports/GAPS.md # caveats and remaining uncertainties
|
|
113
|
-
sources/ # fetched source markdown files
|
|
114
|
-
data/manifest.json # run metadata, stop reason, floor checks, citation audit
|
|
115
|
-
data/rounds.json # per-round actions/learnings/gaps
|
|
116
|
-
data/sources.json # ranked source registry
|
|
117
|
-
data/questions.json # STATUS-style question ledger with evidence/source IDs
|
|
118
|
-
data/evidence.json # structured rational/evidence/summary per useful source
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
CLI controls:
|
|
122
|
-
|
|
123
|
-
```bash
|
|
124
|
-
node bin/search.mjs all --inline --stdin --depth research --breadth 3 --iterations 2 --max-sources 8 <<'EOF'
|
|
125
|
-
Evaluate browser automation options for AI agents
|
|
126
|
-
EOF
|
|
127
|
-
node bin/search.mjs all "topic" --depth research --research-out-dir ./research-topic
|
|
128
|
-
node bin/search.mjs all "topic" --depth research --no-research-bundle
|
|
129
|
-
```
|
|
130
|
-
|
|
131
|
-
## Runtime commands
|
|
132
|
-
|
|
133
|
-
Inside Pi, prefer the extension commands (no package path needed):
|
|
134
|
-
|
|
135
|
-
```text
|
|
136
|
-
/greedy-visible # launch visible Chrome for captcha/login/cookie setup
|
|
137
|
-
/greedy-status # show GreedySearch Chrome status
|
|
138
|
-
/greedy-kill # stop GreedySearch Chrome
|
|
139
|
-
/set-greedy-locale # set default result language (de, fr, es, ja, etc.)
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
Git install path:
|
|
143
|
-
|
|
144
|
-
```bash
|
|
145
|
-
GS=~/.pi/agent/git/github.com/apmantza/GreedySearch-pi
|
|
146
|
-
node "$GS/bin/launch.mjs" --status
|
|
147
|
-
node "$GS/bin/visible.mjs" # visible mode
|
|
148
|
-
node "$GS/bin/visible.mjs" --kill # strong visible/port cleanup
|
|
149
|
-
node "$GS/bin/kill-visible.mjs" # same as visible.mjs --kill
|
|
150
|
-
node "$GS/bin/cdp-visible.mjs" list # safe CDP: GreedySearch visible Chrome only
|
|
151
|
-
node "$GS/bin/cdp-headless.mjs" list # safe CDP: GreedySearch headless Chrome only
|
|
152
|
-
node "$GS/bin/cdp-greedy.mjs" list # safe CDP: any GreedySearch Chrome mode
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
npm global install path:
|
|
156
|
-
|
|
157
|
-
```bash
|
|
158
|
-
GS="$(npm root -g)/@apmantza/greedysearch-pi"
|
|
159
|
-
node "$GS/bin/launch.mjs" --status
|
|
160
|
-
node "$GS/bin/visible.mjs"
|
|
161
|
-
node "$GS/bin/visible.mjs" --kill
|
|
162
|
-
node "$GS/bin/kill-visible.mjs"
|
|
163
|
-
node "$GS/bin/cdp-visible.mjs" list
|
|
164
|
-
node "$GS/bin/cdp-headless.mjs" list
|
|
165
|
-
node "$GS/bin/cdp-greedy.mjs" list
|
|
48
|
+
```js
|
|
49
|
+
greedy_search({ query: "Visible setup", engine: "perplexity", visible: true });
|
|
166
50
|
```
|
|
167
51
|
|
|
168
|
-
|
|
52
|
+
## Documentation
|
|
169
53
|
|
|
170
|
-
|
|
54
|
+
- [Usage guide](docs/usage.md) — tool parameters, search modes, and config
|
|
55
|
+
- [Research mode](docs/research.md) — iterative workflow and bundle layout
|
|
56
|
+
- [Runtime and Chrome](docs/runtime.md) — slash commands, env vars, CDP safety
|
|
57
|
+
- [Source fetching](docs/source-fetching.md) — supported source types and metadata
|
|
58
|
+
- [Development](docs/development.md) — project layout, tests, and smoke checks
|
|
59
|
+
- [Release workflow](docs/releases.md) — changelog scripts and GitHub releases
|
|
171
60
|
|
|
172
61
|
## Requirements
|
|
173
62
|
|
|
174
|
-
- Chrome
|
|
63
|
+
- Chrome or Chromium
|
|
175
64
|
- Node.js 20.11.0+
|
|
176
65
|
|
|
177
|
-
##
|
|
178
|
-
|
|
179
|
-
When using `engine: "all"`, top source content is fetched by default. Add `synthesize: true` to synthesize with the configured synthesizer:
|
|
180
|
-
|
|
181
|
-
- **PDFs** — Direct PDF links are parsed to markdown text for source-grounded synthesis
|
|
182
|
-
- **Semantic Scholar** — Discovers academic papers and prefers direct PDF/external paper links when available
|
|
183
|
-
- **Reddit** — Uses Reddit's public `.json` API for posts and comments
|
|
184
|
-
- **GitHub** — Uses GitHub REST API for repos, READMEs, and file trees
|
|
185
|
-
- **General web** — Mozilla Readability extraction with browser fallback when needed
|
|
186
|
-
- **Metadata** — title, author/byline, site name, publish date, language, excerpt
|
|
66
|
+
## Contributing
|
|
187
67
|
|
|
188
|
-
|
|
68
|
+
See [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`AGENTS.md`](AGENTS.md) for the
|
|
69
|
+
extractor, recovery, and release checklists.
|
|
189
70
|
|
|
190
|
-
|
|
191
|
-
- `extractors/` — engine-specific automation + stealth/consent handling
|
|
192
|
-
- `src/` — search pipeline, chrome management, source fetching, formatting
|
|
193
|
-
- `skills/` — Pi skill metadata
|
|
71
|
+
GreedySearch is released under the [MIT License](LICENSE).
|
|
194
72
|
|
|
195
|
-
##
|
|
196
|
-
|
|
197
|
-
Cross-platform test runner (Windows + Unix):
|
|
198
|
-
|
|
199
|
-
```bash
|
|
200
|
-
npm test # run all tests
|
|
201
|
-
npm run test:quick # skip slow tests
|
|
202
|
-
npm run test:smoke # basic health check
|
|
203
|
-
```
|
|
204
|
-
|
|
205
|
-
Full bash test suite (Unix only):
|
|
206
|
-
|
|
207
|
-
```bash
|
|
208
|
-
npm run test:bash # comprehensive tests
|
|
209
|
-
./test.sh parallel # race condition tests
|
|
210
|
-
./test.sh flags # flag/option tests
|
|
211
|
-
```
|
|
73
|
+
## Contributors
|
|
212
74
|
|
|
213
|
-
|
|
75
|
+
Thanks goes to these wonderful people:
|
|
214
76
|
|
|
215
|
-
|
|
77
|
+
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
|
|
78
|
+
<!-- prettier-ignore-start -->
|
|
79
|
+
<!-- markdownlint-disable -->
|
|
80
|
+
<table>
|
|
81
|
+
<tbody>
|
|
82
|
+
<tr>
|
|
83
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/wastedC"><img src="https://avatars.githubusercontent.com/u/917574?v=4" width="100px;" alt=""/><br /><sub><b>wastedC</b></sub></a><br /><a href="#code-wastedC" title="Code">💻</a> <a href="#ideas-wastedC" title="Ideas & Planning">🤔</a> <a href="#maintenance-wastedC" title="Maintenance">🚧</a> <a href="#review-wastedC" title="Reviewed Pull Requests">👀</a></td>
|
|
84
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/apmantza"><img src="https://avatars.githubusercontent.com/u/247365598?v=4" width="100px;" alt=""/><br /><sub><b>Apostolos Mantzaris</b></sub></a><br /><a href="#code-apmantza" title="Code">💻</a> <a href="#doc-apmantza" title="Documentation">📖</a> <a href="#ideas-apmantza" title="Ideas & Planning">🤔</a> <a href="#maintenance-apmantza" title="Maintenance">🚧</a> <a href="#review-apmantza" title="Reviewed Pull Requests">👀</a></td>
|
|
85
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/apps/dependabot"><img src="https://avatars.githubusercontent.com/in/29110?v=4" width="100px;" alt=""/><br /><sub><b>Dependabot</b></sub></a><br /><a href="#maintenance-dependabot[bot]" title="Maintenance">🚧</a></td>
|
|
86
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/peyloride"><img src="https://avatars.githubusercontent.com/u/10589068?v=4" width="100px;" alt=""/><br /><sub><b>peyloride</b></sub></a><br /><a href="#bug-peyloride" title="Bug reports">🐛</a></td>
|
|
87
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hypnoglow"><img src="https://avatars.githubusercontent.com/u/4853075?v=4" width="100px;" alt=""/><br /><sub><b>hypnoglow</b></sub></a><br /><a href="#bug-hypnoglow" title="Bug reports">🐛</a></td>
|
|
88
|
+
</tr>
|
|
89
|
+
</tbody>
|
|
90
|
+
</table>
|
|
216
91
|
|
|
217
|
-
|
|
92
|
+
<!-- markdownlint-restore -->
|
|
93
|
+
<!-- prettier-ignore-end -->
|
|
94
|
+
<!-- ALL-CONTRIBUTORS-LIST:END -->
|
|
218
95
|
|
|
219
|
-
|
|
96
|
+
If you land a pull request or report an issue that gets fixed, we'll add you here.
|
package/bin/search.mjs
CHANGED
|
@@ -314,6 +314,8 @@ async function main() {
|
|
|
314
314
|
const ENGINE_START_URLS = {
|
|
315
315
|
perplexity: "https://www.perplexity.ai/",
|
|
316
316
|
google: "https://www.google.com/",
|
|
317
|
+
chatgpt: "https://chatgpt.com/",
|
|
318
|
+
gemini: "https://gemini.google.com/app",
|
|
317
319
|
"semantic-scholar": "https://www.semanticscholar.org/",
|
|
318
320
|
semanticscholar: "https://www.semanticscholar.org/",
|
|
319
321
|
s2: "https://www.semanticscholar.org/",
|
package/docs/analysis.md
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
1
|
+
# Stealth Analysis — GreedySearch-pi vs CreepJS
|
|
2
|
+
|
|
3
|
+
> **Date:** 2026-07-04
|
|
4
|
+
> **Scope:** Headless Chrome 149 on Windows 10 with `--headless=new`
|
|
5
|
+
> **Test page:** <https://abrahamjuliot.github.io/creepjs/>
|
|
6
|
+
> **Comparison target:** stealth-browser-mcp (Python MCP server) — claims 0% headless/stealth on CreepJS
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Executive Summary
|
|
11
|
+
|
|
12
|
+
GreedySearch-pi applies extensive JavaScript-level stealth patches via `Page.addScriptToEvaluateOnNewDocument` in `extractors/common.mjs`. The current state:
|
|
13
|
+
|
|
14
|
+
| Test | Result |
|
|
15
|
+
|------|--------|
|
|
16
|
+
| Sannysoft Bot Detection | **20/20 clean** (all fingerprint scanner rows pass) |
|
|
17
|
+
| Intoli Headless Detection | **All checks pass** |
|
|
18
|
+
| CreepJS "like headless" | **19%** (fingerprint: `2eb544f2`) |
|
|
19
|
+
| CreepJS "headless" | **33%** (fingerprint: `6ed45504`) |
|
|
20
|
+
| CreepJS "stealth" | **20%** (fingerprint: `4b82ddf4`) |
|
|
21
|
+
|
|
22
|
+
A critical finding: **CreepJS scores are identical in headless and visible mode** — same hashes, same percentages. This means our JS patches successfully close the gap between headless and visible Chrome from CreepJS's perspective. The remaining signals are architectural (baked into the Chrome binary), not patchable from JavaScript.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 2. Full CreepJS Analysis Breakdown
|
|
27
|
+
|
|
28
|
+
The following data was extracted from a live CreepJS run at `https://abrahamjuliot.github.io/creepjs/?cb=1783160620710`.
|
|
29
|
+
|
|
30
|
+
### 2.1 Headless Section (hash: `87203106`)
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
56% like headless: de4e02aa
|
|
34
|
+
33% headless: 6ed45504
|
|
35
|
+
20% stealth: 4b82ddf4
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
#### "Like Headless" signals (56%)
|
|
39
|
+
|
|
40
|
+
| Signal | Value | Notes |
|
|
41
|
+
|--------|-------|-------|
|
|
42
|
+
| `noChrome` | `false` | ✅ `window.chrome` exists |
|
|
43
|
+
| `hasPermissionsBug` | `false` | ✅ Permissions API works |
|
|
44
|
+
| `noPlugins` | `false` | ✅ `navigator.plugins.length = 3` |
|
|
45
|
+
| `noMimeTypes` | `false` | ✅ `navigator.mimeTypes.length = 2` |
|
|
46
|
+
| `notificationIsDenied` | `false` | ✅ Notifications permission works |
|
|
47
|
+
| `hasKnownBgColor` | `true` | ✅ Preferred color scheme matches headed |
|
|
48
|
+
| `prefersLightColor` | `true` | ✅ No dark-mode inconsistency |
|
|
49
|
+
| `uaDataIsBlank` | `false` | ✅ `navigator.userAgentData` populated |
|
|
50
|
+
| `pdfIsDisabled` | `false` | ✅ `navigator.pdfViewerEnabled = true` |
|
|
51
|
+
| `noTaskbar` | **`true`** | ❌ Taskbar API unavailable (headless has no shell) |
|
|
52
|
+
| `hasVvpScreenRes` | **`true`** | ❌ Screen resolution spoofing detected via ScreenDetailed API |
|
|
53
|
+
| `hasSwiftShader` | **`true`** | ❌ WebGL uses SwiftShader software renderer |
|
|
54
|
+
| `noWebShare` | **`true`** | ❌ `navigator.share` missing |
|
|
55
|
+
| `noContentIndex` | **`true`** | ❌ `navigator.contentIndex` missing |
|
|
56
|
+
| `noContactsManager` | **`true`** | ❌ `navigator.contacts` missing |
|
|
57
|
+
| `noDownlinkMax` | **`true`** | ❌ `navigator.connection.downlinkMax` missing |
|
|
58
|
+
|
|
59
|
+
#### "Headless" signals (33%)
|
|
60
|
+
|
|
61
|
+
| Signal | Value | Notes |
|
|
62
|
+
|--------|-------|-------|
|
|
63
|
+
| `webDriverIsOn` | `false` | ✅ `navigator.webdriver` deleted |
|
|
64
|
+
| `hasHeadlessUA` | `false` | ✅ Main-thread UA does not contain "HeadlessChrome" |
|
|
65
|
+
| `hasHeadlessWorkerUA` | **`true`** | ❌ **Worker UA exposes `HeadlessChrome/150`** |
|
|
66
|
+
|
|
67
|
+
#### "Stealth" signals (20%)
|
|
68
|
+
|
|
69
|
+
| Signal | Value | Notes |
|
|
70
|
+
|--------|-------|-------|
|
|
71
|
+
| `hasIframeProxy` | `false` | ✅ No iframe proxy detected |
|
|
72
|
+
| `hasHighChromeIndex` | `false` | ✅ Chrome index looks normal |
|
|
73
|
+
| `hasBadChromeRuntime` | `false` | ✅ `chrome.runtime` looks normal |
|
|
74
|
+
| `hasToStringProxy` | `false` | ✅ `Function.prototype.toString` not detected as proxied |
|
|
75
|
+
| `hasBadWebGL` | **`true`** | ❌ WebGL patches detected (getParameter/readPixels overrides) |
|
|
76
|
+
|
|
77
|
+
### 2.2 Worker Scope (hash: `95cd472b`)
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
userAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
|
|
81
|
+
(KHTML, like Gecko) HeadlessChrome/150.0.0.0 Safari/537.36
|
|
82
|
+
platform: Linux x86_64
|
|
83
|
+
device: Linux (Linux x86_64) | Linux x86_64
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The Worker scope exposes **three leaks at once**:
|
|
87
|
+
|
|
88
|
+
1. **`HeadlessChrome/150.0.0.0`** — product name reveals headless mode
|
|
89
|
+
2. **`Linux x86_64`** — platform mismatch (main-thread reports `Windows`)
|
|
90
|
+
3. **Version 150 vs 149** — Worker UA version differs from our `--user-agent` flag
|
|
91
|
+
|
|
92
|
+
### 2.3 WebGL (hash: `3d883ba3`)
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
pixels: 50875bab × (fail)
|
|
96
|
+
params (78): eab0e944 × (fail)
|
|
97
|
+
exts (64): d3e8ebec × (fail)
|
|
98
|
+
gpu:
|
|
99
|
+
Google Inc.
|
|
100
|
+
Intel Iris OpenGL Engine ← our spoofed renderer
|
|
101
|
+
ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero)))
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
WebGL shows failures in **pixels, params, and extensions** — our `getParameter` spoofing and `readPixels` noise are detectable.
|
|
105
|
+
|
|
106
|
+
### 2.4 Screen (hash: `90135785`)
|
|
107
|
+
|
|
108
|
+
```
|
|
109
|
+
...screen: 1280 x 1280 ← not our spoofed 1920x1080
|
|
110
|
+
....avail: 1280 x 1280
|
|
111
|
+
viewport: 1280 1280 1280 1280 1280 1280
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
CreepJS detects the **real viewport** via `ScreenDetailed` API (`getScreenDetails()`), bypassing our `screen.width`/`height` spoofing.
|
|
115
|
+
|
|
116
|
+
### 2.5 Canvas 2D (hash: `437458e4`)
|
|
117
|
+
|
|
118
|
+
```
|
|
119
|
+
data: ecb280d2 × (fail)
|
|
120
|
+
text: 870c8e89
|
|
121
|
+
paint (GPU): 593da416
|
|
122
|
+
paint (CPU): 593da416
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Canvas `data` fails — CreepJS detects that `toDataURL` has been tampered with (our noise injection leaves detectable traces in the function's metadata).
|
|
126
|
+
|
|
127
|
+
### 2.6 Audio (hash: `9d6aa470`)
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
sum: 124.043...
|
|
131
|
+
values: dfc44fc6 × (fail)
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Audio `values` fail — CreepJS's `searchLies` detects that `AudioBuffer.prototype.getChannelData` has been patched.
|
|
135
|
+
|
|
136
|
+
### 2.7 Fonts (hash: `fe40d7c6`)
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
load (5/51): "DejaVu Sans", "Liberation Mono", "Noto Color Emoji",
|
|
140
|
+
KACSTOffice, OpenSymbol
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The font list contains **Linux-specific fonts** (`DejaVu Sans`, `Liberation Mono`, `Noto Color Emoji`) instead of Windows fonts (`Arial`, `Segoe UI`, `Calibri`). Headless Chrome on Windows uses a Linux font rendering stack.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## 3. Gap Classification & Mitigation Research
|
|
148
|
+
|
|
149
|
+
### 3.1 Immediately Fixable ✅
|
|
150
|
+
|
|
151
|
+
| Issue | Fix | Status |
|
|
152
|
+
|-------|-----|--------|
|
|
153
|
+
| `noDownlinkMax` | Add `downlinkMax: Infinity` to `navigator.connection` patch | ✅ Applied |
|
|
154
|
+
| `noWebShare` | Add `navigator.share` stub | ✅ Applied |
|
|
155
|
+
| `noContentIndex` | Add `navigator.contentIndex` stub | ✅ Applied |
|
|
156
|
+
|
|
157
|
+
### 3.2 Partially Fixable ⚠️
|
|
158
|
+
|
|
159
|
+
| Issue | Approach | Caveat |
|
|
160
|
+
|-------|----------|--------|
|
|
161
|
+
| `hasVvpScreenRes` | Intercept `ScreenDetailed.prototype` properties in stealth code | Cat-and-mouse — CreepJS can switch to other detection methods |
|
|
162
|
+
| `hasBadWebGL` | Refine `__markNative` to better hide our patches | CreepJS's `searchLies` has deep toString/descriptor checks |
|
|
163
|
+
| `noContactsManager` | Add `navigator.contacts` mock with `select()` method | Complex mock, easy to detect as fake |
|
|
164
|
+
| Canvas/Audio noise refinement | More sophisticated noise injection | Fundamental limitation — JS patches are detectable by design |
|
|
165
|
+
|
|
166
|
+
### 3.3 Requires Chromium Fork ❌
|
|
167
|
+
|
|
168
|
+
These issues stem from **compile-time baked properties** in the Chrome binary and **cannot be fixed from JavaScript**.
|
|
169
|
+
|
|
170
|
+
| Issue | Root Cause | Why JS Cannot Fix |
|
|
171
|
+
|-------|------------|-------------------|
|
|
172
|
+
| `hasHeadlessWorkerUA` | `HeadlessChrome` is set in C++ `content/child/user_agent.cc`. Workers read from compiled-in values, not `--user-agent` flag. | `Page.addScriptToEvaluateOnNewDocument` does NOT run in Worker/ServiceWorker scopes. Workers are separate V8 isolates. |
|
|
173
|
+
| `hasSwiftShader` | Headless Chrome selects SwiftShader as its GPU backend. No GPU passthrough in headless mode. | WebGL rendering originates in the GPU process, not the renderer process. JS can only patch the Blink API surface. |
|
|
174
|
+
| Linux fonts | Headless Chrome uses a Linux font config stack on all platforms. | Font enumeration comes from the system font manager, not JS-accessible API. |
|
|
175
|
+
| `hasVvpScreenRes` (deep) | `ScreenDetailed` API exposes real display info from the OS window manager. | API is backed by platform-level display info, not JS overridable properties. |
|
|
176
|
+
|
|
177
|
+
### 3.4 Research-Backed Verdicts
|
|
178
|
+
|
|
179
|
+
#### Worker UA (the single highest-impact fix)
|
|
180
|
+
|
|
181
|
+
**Research sources:**
|
|
182
|
+
|
|
183
|
+
- [IPASIS: Advanced Headless Chrome Detection](https://ipasis.com/blog/detecting-headless-chrome-workers-webgl) (Dec 2025): *"If a scraper injects a script to redefine navigator.userAgent in the main window, checking self.navigator.userAgent inside a newly spawned Worker often reveals the underlying Headless configuration."*
|
|
184
|
+
- [Browserbase: Why we forked Chromium](https://www.browserbase.com/blog/chromium-fork-for-ai-automation) (Nov 2025): *"Our patch updates the headless product name to Chrome and keeps the metadata and CDP reporting consistent with a normal Chrome build. Requests from workers, iframes, and service workers use the same values because the Network Service pulls from the same metadata."*
|
|
185
|
+
|
|
186
|
+
**Verdict:** The only proven mitigation is a Chromium fork that patches the C++ source (`content/child/user_agent.cc`) to replace `HeadlessChrome` with `Chrome` at compile time. Browserbase and CloakBrowser have both done this. No JavaScript-level workaround exists.
|
|
187
|
+
|
|
188
|
+
#### Chromium Forks That Solve These Issues
|
|
189
|
+
|
|
190
|
+
| Project | Approach | Open Source? |
|
|
191
|
+
|---------|----------|--------------|
|
|
192
|
+
| [Browserbase](https://www.browserbase.com/blog/chromium-fork-for-ai-automation) | Custom Chromium fork with patched UA, webdriver, and session management | ❌ Proprietary |
|
|
193
|
+
| [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) | C++ level patches to Chromium source | ✅ Open source |
|
|
194
|
+
| [ultrafunkamsterdam/undetected-chromedriver](https://deepwiki.com/ultrafunkamsterdam/undetected-chromedriver/4.3-headless-mode) | JS + config-level patches | ✅ Open source (JS only) |
|
|
195
|
+
|
|
196
|
+
#### The `--user-agent` Flag Limitation
|
|
197
|
+
|
|
198
|
+
Chrome's `--user-agent` flag modifies the **HTTP `User-Agent` header** and the **main-thread `navigator.userAgent`**, but it does NOT propagate to:
|
|
199
|
+
|
|
200
|
+
- Worker `navigator.userAgent`
|
|
201
|
+
- Service Worker `self.navigator.userAgent`
|
|
202
|
+
- Some internal Blink UA checks
|
|
203
|
+
|
|
204
|
+
This is documented in Chromium's source: the `UserAgent` in `content/child/user_agent.cc` has separate code paths for "product name" (compiled as `HeadlessChrome`) vs "user agent override" (from command line). Workers read from the product name, not the override.
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## 4. Practical Recommendations
|
|
209
|
+
|
|
210
|
+
### What we can do next (JS-level)
|
|
211
|
+
|
|
212
|
+
1. **Patch ScreenDetailed API** — intercept `getScreenDetails()` to return consistent values
|
|
213
|
+
2. **Add `navigator.contacts` stub** — low impact but easy
|
|
214
|
+
3. **Accept the ceiling** — Sannysoft 20/20 + Intoli pass means the practical anti-bot targets (Cloudflare, DataDome, Google) are well-covered
|
|
215
|
+
|
|
216
|
+
### What requires a Chromium fork
|
|
217
|
+
|
|
218
|
+
- Worker UA fix
|
|
219
|
+
- SwiftShader/GPU masking
|
|
220
|
+
- Font stack normalization
|
|
221
|
+
|
|
222
|
+
### Bottom line
|
|
223
|
+
|
|
224
|
+
Our JS patches achieve the **practical maximum** for headless Chrome stealth without modifying the browser binary:
|
|
225
|
+
|
|
226
|
+
- ✅ All JavaScript-accessible signals are normalized
|
|
227
|
+
- ✅ Headless and visible modes produce identical CreepJS fingerprints
|
|
228
|
+
- ✅ Sannysoft is 100% clean
|
|
229
|
+
- ✅ All engines (Perplexity, Bing, ChatGPT, Gemini, Google) work in headless mode
|
|
230
|
+
|
|
231
|
+
The remaining ~33% headless score is the **architectural ceiling** — Chrome's compiled-in "HeadlessChrome" product name, SwiftShader GPU, and Linux font stack are not reachable from JavaScript.
|
|
232
|
+
|
|
233
|
+
For reference: Browserbase's entire business model is built around patching these exact C++ signals, and they needed a dedicated build machine compiling Chromium in ~1 hour to iterate on the patches.
|