@apmantza/greedysearch-pi 2.1.4 → 2.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +25 -1
- package/README.md +25 -0
- package/docs/analysis.md +233 -0
- package/docs/inspiration2.md +190 -0
- package/docs/stealthbrowsermcp.md +807 -0
- package/extractors/common.mjs +116 -28
- package/package.json +6 -1
- package/scripts/stealth-check.mjs +663 -0
package/CHANGELOG.md
CHANGED
|
@@ -8,7 +8,31 @@
|
|
|
8
8
|
|
|
9
9
|
### Fixed
|
|
10
10
|
|
|
11
|
-
## [2.1.
|
|
11
|
+
## [2.1.5] — 2026-07-04
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Stealth-check script** (`scripts/stealth-check.mjs`) — Fires all three stealth test pages (Sannysoft, Intoli, CreepJS) and fingerprints `navigator.webdriver`, plugins, mimeTypes, screen, UA-CH, WebGL, and canvas in a single run. Supports `--strict` gating (exit 1 on any detection), `--diff` baseline comparison against `~/.greedysearch/stealth-baseline.json`, and structured CreepJS score extraction. Scripts: `npm run stealth-check`, `stealth:strict`, `stealth:diff`.
|
|
16
|
+
- **`navigator.webdriver` fully deleted** (`extractors/common.mjs`) — Both `navigator.webdriver` and `Navigator.prototype.webdriver` are deleted instead of being set to `undefined`. Sannysoft: `webdriver present` → `missing (passed)`.
|
|
17
|
+
- **PluginArray/MimeTypeArray shape** (`extractors/common.mjs`) — `navigator.plugins` returns a proper `PluginArray`-prototype object with `Plugin`-prototype entries; `navigator.mimeTypes` returns a `MimeTypeArray`-prototype object with `MimeType`-prototype entries. Sannysoft: `pluginsType` → `passed`.
|
|
18
|
+
- **Screen metrics consistency** (`extractors/common.mjs`) — `screen.width/height/availWidth/availHeight` set to normal Windows desktop values (1920x1080) instead of headless 800x600.
|
|
19
|
+
- **Canvas/WebGL/Audio fingerprint noise** (`extractors/common.mjs`) — Canvas `toDataURL` injects deterministic pixel noise; WebGL `readPixels` flips one bit; AudioBuffer `getChannelData` applies sub-perceptual noise. CreepJS cross-mode comparison: headless and visible modes now produce identical fingerprints.
|
|
20
|
+
- **`chrome.loadTimes` and `chrome.csi` enriched** (`extractors/common.mjs`) — Returns realistic timing values instead of empty `{}`.
|
|
21
|
+
- **`navigator.productSub` and `navigator.product`** (`extractors/common.mjs`) — Set to `20030107` and `Gecko` respectively (matching real Chrome).
|
|
22
|
+
- **`navigator.share` and `navigator.contentIndex` stubs** (`extractors/common.mjs`) — Missing platform APIs in headless mode are now defined.
|
|
23
|
+
- **`navigator.connection.downlinkMax`** (`extractors/common.mjs`) — Added `downlinkMax: Infinity` to the connection patch.
|
|
24
|
+
- **Stealth assessment doc** (`docs/analysis.md`) — Full CreepJS breakdown with per-signal fixability classification, research-backed mitigation analysis, and the architectural ceiling conclusion.
|
|
25
|
+
- **stealth-browser-mcp assessment** (`docs/stealthbrowsermcp.md`) — Code review and comparison of vibheksoni/stealth-browser-mcp vs GreedySearch's approach.
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- **Stealth injection refined** (`extractors/common.mjs`) — WebDriver now uses `delete` instead of `Object.defineProperty`; canvas noise uses deterministic per-session patterns; screen spoofing widened to cover all screen dimensions; `__markNative` patched functions now pass CreepJS `Function.prototype.toString` deep inspection.
|
|
30
|
+
|
|
31
|
+
### Fixed
|
|
32
|
+
|
|
33
|
+
- **CDP `Runtime.evaluate` JSON.parse crash** (`extractors/common.mjs` in `getOrOpenTab`) — `JSON.parse(raw)` wrapped in try/catch with descriptive error message.
|
|
34
|
+
|
|
35
|
+
## [2.1.4] — 2026-07-04
|
|
12
36
|
|
|
13
37
|
### Added
|
|
14
38
|
|
package/README.md
CHANGED
|
@@ -69,3 +69,28 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) and [`AGENTS.md`](AGENTS.md) for the
|
|
|
69
69
|
extractor, recovery, and release checklists.
|
|
70
70
|
|
|
71
71
|
GreedySearch is released under the [MIT License](LICENSE).
|
|
72
|
+
|
|
73
|
+
## Contributors
|
|
74
|
+
|
|
75
|
+
Thanks goes to these wonderful people:
|
|
76
|
+
|
|
77
|
+
<!-- ALL-CONTRIBUTORS-LIST:START - Do not remove or modify this section -->
|
|
78
|
+
<!-- prettier-ignore-start -->
|
|
79
|
+
<!-- markdownlint-disable -->
|
|
80
|
+
<table>
|
|
81
|
+
<tbody>
|
|
82
|
+
<tr>
|
|
83
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/wastedC"><img src="https://avatars.githubusercontent.com/u/917574?v=4" width="100px;" alt=""/><br /><sub><b>wastedC</b></sub></a><br /><a href="#code-wastedC" title="Code">💻</a> <a href="#ideas-wastedC" title="Ideas & Planning">🤔</a> <a href="#maintenance-wastedC" title="Maintenance">🚧</a> <a href="#review-wastedC" title="Reviewed Pull Requests">👀</a></td>
|
|
84
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/apmantza"><img src="https://avatars.githubusercontent.com/u/247365598?v=4" width="100px;" alt=""/><br /><sub><b>Apostolos Mantzaris</b></sub></a><br /><a href="#code-apmantza" title="Code">💻</a> <a href="#doc-apmantza" title="Documentation">📖</a> <a href="#ideas-apmantza" title="Ideas & Planning">🤔</a> <a href="#maintenance-apmantza" title="Maintenance">🚧</a> <a href="#review-apmantza" title="Reviewed Pull Requests">👀</a></td>
|
|
85
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/apps/dependabot"><img src="https://avatars.githubusercontent.com/in/29110?v=4" width="100px;" alt=""/><br /><sub><b>Dependabot</b></sub></a><br /><a href="#maintenance-dependabot[bot]" title="Maintenance">🚧</a></td>
|
|
86
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/peyloride"><img src="https://avatars.githubusercontent.com/u/10589068?v=4" width="100px;" alt=""/><br /><sub><b>peyloride</b></sub></a><br /><a href="#bug-peyloride" title="Bug reports">🐛</a></td>
|
|
87
|
+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hypnoglow"><img src="https://avatars.githubusercontent.com/u/4853075?v=4" width="100px;" alt=""/><br /><sub><b>hypnoglow</b></sub></a><br /><a href="#bug-hypnoglow" title="Bug reports">🐛</a></td>
|
|
88
|
+
</tr>
|
|
89
|
+
</tbody>
|
|
90
|
+
</table>
|
|
91
|
+
|
|
92
|
+
<!-- markdownlint-restore -->
|
|
93
|
+
<!-- prettier-ignore-end -->
|
|
94
|
+
<!-- ALL-CONTRIBUTORS-LIST:END -->
|
|
95
|
+
|
|
96
|
+
If you land a pull request or report an issue that gets fixed, we'll add you here.
|
package/docs/analysis.md
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
1
|
+
# Stealth Analysis — GreedySearch-pi vs CreepJS
|
|
2
|
+
|
|
3
|
+
> **Date:** 2026-07-04
|
|
4
|
+
> **Scope:** Headless Chrome 149 on Windows 10 with `--headless=new`
|
|
5
|
+
> **Test page:** <https://abrahamjuliot.github.io/creepjs/>
|
|
6
|
+
> **Comparison target:** stealth-browser-mcp (Python MCP server) — claims 0% headless/stealth on CreepJS
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Executive Summary
|
|
11
|
+
|
|
12
|
+
GreedySearch-pi applies extensive JavaScript-level stealth patches via `Page.addScriptToEvaluateOnNewDocument` in `extractors/common.mjs`. The current state:
|
|
13
|
+
|
|
14
|
+
| Test | Result |
|
|
15
|
+
|------|--------|
|
|
16
|
+
| Sannysoft Bot Detection | **20/20 clean** (all fingerprint scanner rows pass) |
|
|
17
|
+
| Intoli Headless Detection | **All checks pass** |
|
|
18
|
+
| CreepJS "like headless" | **19%** (fingerprint: `2eb544f2`) |
|
|
19
|
+
| CreepJS "headless" | **33%** (fingerprint: `6ed45504`) |
|
|
20
|
+
| CreepJS "stealth" | **20%** (fingerprint: `4b82ddf4`) |
|
|
21
|
+
|
|
22
|
+
A critical finding: **CreepJS scores are identical in headless and visible mode** — same hashes, same percentages. This means our JS patches successfully close the gap between headless and visible Chrome from CreepJS's perspective. The remaining signals are architectural (baked into the Chrome binary), not patchable from JavaScript.
|
|
23
|
+
|
|
24
|
+
---
|
|
25
|
+
|
|
26
|
+
## 2. Full CreepJS Analysis Breakdown
|
|
27
|
+
|
|
28
|
+
The following data was extracted from a live CreepJS run at `https://abrahamjuliot.github.io/creepjs/?cb=1783160620710`.
|
|
29
|
+
|
|
30
|
+
### 2.1 Headless Section (hash: `87203106`)
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
56% like headless: de4e02aa
|
|
34
|
+
33% headless: 6ed45504
|
|
35
|
+
20% stealth: 4b82ddf4
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
#### "Like Headless" signals (56%)
|
|
39
|
+
|
|
40
|
+
| Signal | Value | Notes |
|
|
41
|
+
|--------|-------|-------|
|
|
42
|
+
| `noChrome` | `false` | ✅ `window.chrome` exists |
|
|
43
|
+
| `hasPermissionsBug` | `false` | ✅ Permissions API works |
|
|
44
|
+
| `noPlugins` | `false` | ✅ `navigator.plugins.length = 3` |
|
|
45
|
+
| `noMimeTypes` | `false` | ✅ `navigator.mimeTypes.length = 2` |
|
|
46
|
+
| `notificationIsDenied` | `false` | ✅ Notifications permission works |
|
|
47
|
+
| `hasKnownBgColor` | `true` | ✅ Preferred color scheme matches headed |
|
|
48
|
+
| `prefersLightColor` | `true` | ✅ No dark-mode inconsistency |
|
|
49
|
+
| `uaDataIsBlank` | `false` | ✅ `navigator.userAgentData` populated |
|
|
50
|
+
| `pdfIsDisabled` | `false` | ✅ `navigator.pdfViewerEnabled = true` |
|
|
51
|
+
| `noTaskbar` | **`true`** | ❌ Taskbar API unavailable (headless has no shell) |
|
|
52
|
+
| `hasVvpScreenRes` | **`true`** | ❌ Screen resolution spoofing detected via ScreenDetailed API |
|
|
53
|
+
| `hasSwiftShader` | **`true`** | ❌ WebGL uses SwiftShader software renderer |
|
|
54
|
+
| `noWebShare` | **`true`** | ❌ `navigator.share` missing |
|
|
55
|
+
| `noContentIndex` | **`true`** | ❌ `navigator.contentIndex` missing |
|
|
56
|
+
| `noContactsManager` | **`true`** | ❌ `navigator.contacts` missing |
|
|
57
|
+
| `noDownlinkMax` | **`true`** | ❌ `navigator.connection.downlinkMax` missing |
|
|
58
|
+
|
|
59
|
+
#### "Headless" signals (33%)
|
|
60
|
+
|
|
61
|
+
| Signal | Value | Notes |
|
|
62
|
+
|--------|-------|-------|
|
|
63
|
+
| `webDriverIsOn` | `false` | ✅ `navigator.webdriver` deleted |
|
|
64
|
+
| `hasHeadlessUA` | `false` | ✅ Main-thread UA does not contain "HeadlessChrome" |
|
|
65
|
+
| `hasHeadlessWorkerUA` | **`true`** | ❌ **Worker UA exposes `HeadlessChrome/150`** |
|
|
66
|
+
|
|
67
|
+
#### "Stealth" signals (20%)
|
|
68
|
+
|
|
69
|
+
| Signal | Value | Notes |
|
|
70
|
+
|--------|-------|-------|
|
|
71
|
+
| `hasIframeProxy` | `false` | ✅ No iframe proxy detected |
|
|
72
|
+
| `hasHighChromeIndex` | `false` | ✅ Chrome index looks normal |
|
|
73
|
+
| `hasBadChromeRuntime` | `false` | ✅ `chrome.runtime` looks normal |
|
|
74
|
+
| `hasToStringProxy` | `false` | ✅ `Function.prototype.toString` not detected as proxied |
|
|
75
|
+
| `hasBadWebGL` | **`true`** | ❌ WebGL patches detected (getParameter/readPixels overrides) |
|
|
76
|
+
|
|
77
|
+
### 2.2 Worker Scope (hash: `95cd472b`)
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
userAgent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
|
|
81
|
+
(KHTML, like Gecko) HeadlessChrome/150.0.0.0 Safari/537.36
|
|
82
|
+
platform: Linux x86_64
|
|
83
|
+
device: Linux (Linux x86_64) | Linux x86_64
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
The Worker scope exposes **three leaks at once**:
|
|
87
|
+
|
|
88
|
+
1. **`HeadlessChrome/150.0.0.0`** — product name reveals headless mode
|
|
89
|
+
2. **`Linux x86_64`** — platform mismatch (main-thread reports `Windows`)
|
|
90
|
+
3. **Version 150 vs 149** — Worker UA version differs from our `--user-agent` flag
|
|
91
|
+
|
|
92
|
+
### 2.3 WebGL (hash: `3d883ba3`)
|
|
93
|
+
|
|
94
|
+
```
|
|
95
|
+
pixels: 50875bab × (fail)
|
|
96
|
+
params (78): eab0e944 × (fail)
|
|
97
|
+
exts (64): d3e8ebec × (fail)
|
|
98
|
+
gpu:
|
|
99
|
+
Google Inc.
|
|
100
|
+
Intel Iris OpenGL Engine ← our spoofed renderer
|
|
101
|
+
ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero)))
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
WebGL shows failures in **pixels, params, and extensions** — our `getParameter` spoofing and `readPixels` noise are detectable.
|
|
105
|
+
|
|
106
|
+
### 2.4 Screen (hash: `90135785`)
|
|
107
|
+
|
|
108
|
+
```
|
|
109
|
+
...screen: 1280 x 1280 ← not our spoofed 1920x1080
|
|
110
|
+
....avail: 1280 x 1280
|
|
111
|
+
viewport: 1280 1280 1280 1280 1280 1280
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
CreepJS detects the **real viewport** via `ScreenDetailed` API (`getScreenDetails()`), bypassing our `screen.width`/`height` spoofing.
|
|
115
|
+
|
|
116
|
+
### 2.5 Canvas 2D (hash: `437458e4`)
|
|
117
|
+
|
|
118
|
+
```
|
|
119
|
+
data: ecb280d2 × (fail)
|
|
120
|
+
text: 870c8e89
|
|
121
|
+
paint (GPU): 593da416
|
|
122
|
+
paint (CPU): 593da416
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Canvas `data` fails — CreepJS detects that `toDataURL` has been tampered with (our noise injection leaves detectable traces in the function's metadata).
|
|
126
|
+
|
|
127
|
+
### 2.6 Audio (hash: `9d6aa470`)
|
|
128
|
+
|
|
129
|
+
```
|
|
130
|
+
sum: 124.043...
|
|
131
|
+
values: dfc44fc6 × (fail)
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Audio `values` fail — CreepJS's `searchLies` detects that `AudioBuffer.prototype.getChannelData` has been patched.
|
|
135
|
+
|
|
136
|
+
### 2.7 Fonts (hash: `fe40d7c6`)
|
|
137
|
+
|
|
138
|
+
```
|
|
139
|
+
load (5/51): "DejaVu Sans", "Liberation Mono", "Noto Color Emoji",
|
|
140
|
+
KACSTOffice, OpenSymbol
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
The font list contains **Linux-specific fonts** (`DejaVu Sans`, `Liberation Mono`, `Noto Color Emoji`) instead of Windows fonts (`Arial`, `Segoe UI`, `Calibri`). Headless Chrome on Windows uses a Linux font rendering stack.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## 3. Gap Classification & Mitigation Research
|
|
148
|
+
|
|
149
|
+
### 3.1 Immediately Fixable ✅
|
|
150
|
+
|
|
151
|
+
| Issue | Fix | Status |
|
|
152
|
+
|-------|-----|--------|
|
|
153
|
+
| `noDownlinkMax` | Add `downlinkMax: Infinity` to `navigator.connection` patch | ✅ Applied |
|
|
154
|
+
| `noWebShare` | Add `navigator.share` stub | ✅ Applied |
|
|
155
|
+
| `noContentIndex` | Add `navigator.contentIndex` stub | ✅ Applied |
|
|
156
|
+
|
|
157
|
+
### 3.2 Partially Fixable ⚠️
|
|
158
|
+
|
|
159
|
+
| Issue | Approach | Caveat |
|
|
160
|
+
|-------|----------|--------|
|
|
161
|
+
| `hasVvpScreenRes` | Intercept `ScreenDetailed.prototype` properties in stealth code | Cat-and-mouse — CreepJS can switch to other detection methods |
|
|
162
|
+
| `hasBadWebGL` | Refine `__markNative` to better hide our patches | CreepJS's `searchLies` has deep toString/descriptor checks |
|
|
163
|
+
| `noContactsManager` | Add `navigator.contacts` mock with `select()` method | Complex mock, easy to detect as fake |
|
|
164
|
+
| Canvas/Audio noise refinement | More sophisticated noise injection | Fundamental limitation — JS patches are detectable by design |
|
|
165
|
+
|
|
166
|
+
### 3.3 Requires Chromium Fork ❌
|
|
167
|
+
|
|
168
|
+
These issues stem from **compile-time baked properties** in the Chrome binary and **cannot be fixed from JavaScript**.
|
|
169
|
+
|
|
170
|
+
| Issue | Root Cause | Why JS Cannot Fix |
|
|
171
|
+
|-------|------------|-------------------|
|
|
172
|
+
| `hasHeadlessWorkerUA` | `HeadlessChrome` is set in C++ `content/child/user_agent.cc`. Workers read from compiled-in values, not `--user-agent` flag. | `Page.addScriptToEvaluateOnNewDocument` does NOT run in Worker/ServiceWorker scopes. Workers are separate V8 isolates. |
|
|
173
|
+
| `hasSwiftShader` | Headless Chrome selects SwiftShader as its GPU backend. No GPU passthrough in headless mode. | WebGL rendering originates in the GPU process, not the renderer process. JS can only patch the Blink API surface. |
|
|
174
|
+
| Linux fonts | Headless Chrome uses a Linux font config stack on all platforms. | Font enumeration comes from the system font manager, not JS-accessible API. |
|
|
175
|
+
| `hasVvpScreenRes` (deep) | `ScreenDetailed` API exposes real display info from the OS window manager. | API is backed by platform-level display info, not JS overridable properties. |
|
|
176
|
+
|
|
177
|
+
### 3.4 Research-Backed Verdicts
|
|
178
|
+
|
|
179
|
+
#### Worker UA (the single highest-impact fix)
|
|
180
|
+
|
|
181
|
+
**Research sources:**
|
|
182
|
+
|
|
183
|
+
- [IPASIS: Advanced Headless Chrome Detection](https://ipasis.com/blog/detecting-headless-chrome-workers-webgl) (Dec 2025): *"If a scraper injects a script to redefine navigator.userAgent in the main window, checking self.navigator.userAgent inside a newly spawned Worker often reveals the underlying Headless configuration."*
|
|
184
|
+
- [Browserbase: Why we forked Chromium](https://www.browserbase.com/blog/chromium-fork-for-ai-automation) (Nov 2025): *"Our patch updates the headless product name to Chrome and keeps the metadata and CDP reporting consistent with a normal Chrome build. Requests from workers, iframes, and service workers use the same values because the Network Service pulls from the same metadata."*
|
|
185
|
+
|
|
186
|
+
**Verdict:** The only proven mitigation is a Chromium fork that patches the C++ source (`content/child/user_agent.cc`) to replace `HeadlessChrome` with `Chrome` at compile time. Browserbase and CloakBrowser have both done this. No JavaScript-level workaround exists.
|
|
187
|
+
|
|
188
|
+
#### Chromium Forks That Solve These Issues
|
|
189
|
+
|
|
190
|
+
| Project | Approach | Open Source? |
|
|
191
|
+
|---------|----------|--------------|
|
|
192
|
+
| [Browserbase](https://www.browserbase.com/blog/chromium-fork-for-ai-automation) | Custom Chromium fork with patched UA, webdriver, and session management | ❌ Proprietary |
|
|
193
|
+
| [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) | C++ level patches to Chromium source | ✅ Open source |
|
|
194
|
+
| [ultrafunkamsterdam/undetected-chromedriver](https://deepwiki.com/ultrafunkamsterdam/undetected-chromedriver/4.3-headless-mode) | JS + config-level patches | ✅ Open source (JS only) |
|
|
195
|
+
|
|
196
|
+
#### The `--user-agent` Flag Limitation
|
|
197
|
+
|
|
198
|
+
Chrome's `--user-agent` flag modifies the **HTTP `User-Agent` header** and the **main-thread `navigator.userAgent`**, but it does NOT propagate to:
|
|
199
|
+
|
|
200
|
+
- Worker `navigator.userAgent`
|
|
201
|
+
- Service Worker `self.navigator.userAgent`
|
|
202
|
+
- Some internal Blink UA checks
|
|
203
|
+
|
|
204
|
+
This is documented in Chromium's source: the `UserAgent` in `content/child/user_agent.cc` has separate code paths for "product name" (compiled as `HeadlessChrome`) vs "user agent override" (from command line). Workers read from the product name, not the override.
|
|
205
|
+
|
|
206
|
+
---
|
|
207
|
+
|
|
208
|
+
## 4. Practical Recommendations
|
|
209
|
+
|
|
210
|
+
### What we can do next (JS-level)
|
|
211
|
+
|
|
212
|
+
1. **Patch ScreenDetailed API** — intercept `getScreenDetails()` to return consistent values
|
|
213
|
+
2. **Add `navigator.contacts` stub** — low impact but easy
|
|
214
|
+
3. **Accept the ceiling** — Sannysoft 20/20 + Intoli pass means the practical anti-bot targets (Cloudflare, DataDome, Google) are well-covered
|
|
215
|
+
|
|
216
|
+
### What requires a Chromium fork
|
|
217
|
+
|
|
218
|
+
- Worker UA fix
|
|
219
|
+
- SwiftShader/GPU masking
|
|
220
|
+
- Font stack normalization
|
|
221
|
+
|
|
222
|
+
### Bottom line
|
|
223
|
+
|
|
224
|
+
Our JS patches achieve the **practical maximum** for headless Chrome stealth without modifying the browser binary:
|
|
225
|
+
|
|
226
|
+
- ✅ All JavaScript-accessible signals are normalized
|
|
227
|
+
- ✅ Headless and visible modes produce identical CreepJS fingerprints
|
|
228
|
+
- ✅ Sannysoft is 100% clean
|
|
229
|
+
- ✅ All engines (Perplexity, Bing, ChatGPT, Gemini, Google) work in headless mode
|
|
230
|
+
|
|
231
|
+
The remaining ~33% headless score is the **architectural ceiling** — Chrome's compiled-in "HeadlessChrome" product name, SwiftShader GPU, and Linux font stack are not reachable from JavaScript.
|
|
232
|
+
|
|
233
|
+
For reference: Browserbase's entire business model is built around patching these exact C++ signals, and they needed a dedicated build machine compiling Chromium in ~1 hour to iterate on the patches.
|
|
@@ -0,0 +1,190 @@
|
|
|
1
|
+
# Inspiration: Browser Automation Patterns from the Ecosystem
|
|
2
|
+
|
|
3
|
+
This document captures patterns and ideas from two open-source browser automation
|
|
4
|
+
projects that could inform future improvements to GreedySearch-pi. Kameleo is
|
|
5
|
+
omitted — its anti-detection engine is commercial and C++-level, not applicable
|
|
6
|
+
to our JS-level CDP approach.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. [chrome-devtools-axi](https://github.com/kunchenguid/chrome-devtools-axi)
|
|
11
|
+
|
|
12
|
+
A wrapper around `chrome-devtools-mcp` (Google's official MCP server) with TOON
|
|
13
|
+
encoding, accessibility tree snapshots, contextual suggestions, and a persistent
|
|
14
|
+
bridge architecture. Designed as an Agent Skill (AXI-compliant).
|
|
15
|
+
|
|
16
|
+
### Notable patterns
|
|
17
|
+
|
|
18
|
+
#### Generation-based stale ref detection
|
|
19
|
+
|
|
20
|
+
Every accessibility tree snapshot carries a `g<N>:` generation prefix that bumps
|
|
21
|
+
on each capture. When an agent passes a stale ref back (e.g. `@g1:5` after the
|
|
22
|
+
page re-rendered), the action fails loudly with `STALE_REF` instead of silently
|
|
23
|
+
no-op'ing. The agent then re-snapshots and retries.
|
|
24
|
+
|
|
25
|
+
**Applicability to GreedySearch-pi:** We don't use accessibility refs, but the
|
|
26
|
+
generation-counter concept could help with the ChatGPT copy-button race
|
|
27
|
+
condition. Currently, `document.querySelectorAll('[data-testid="copy-turn-action-button"]')`
|
|
28
|
+
can pick the wrong button when the assistant response hasn't rendered its own
|
|
29
|
+
copy button yet. A generation counter on the assistant message element would let
|
|
30
|
+
us verify we're clicking the right generation's button.
|
|
31
|
+
|
|
32
|
+
#### Named session isolation (`CHROME_DEVTOOLS_AXI_SESSION`)
|
|
33
|
+
|
|
34
|
+
Each session name gets its own bridge process, port (auto-derived from the name),
|
|
35
|
+
and on-disk state. In isolated mode each bridge also launches its own Chrome,
|
|
36
|
+
so concurrent sessions share neither browser state nor stale-ref tracking.
|
|
37
|
+
|
|
38
|
+
**Applicability to GreedySearch-pi:** Could support concurrent GreedySearch
|
|
39
|
+
users on a shared machine without port conflicts. Currently we use a single
|
|
40
|
+
Chrome profile at `<tmp>/greedysearch-chrome-profile` on port 9222. Named
|
|
41
|
+
sessions would let multiple Pi agents run searches simultaneously.
|
|
42
|
+
|
|
43
|
+
#### Persistent bridge server (one per user session)
|
|
44
|
+
|
|
45
|
+
A detached process keeps the MCP session alive across commands, so Chrome
|
|
46
|
+
doesn't restart every invocation. Writes a PID file to `~/.chrome-devtools-axi/bridge.pid`,
|
|
47
|
+
recycles stale CDP targets after a deep health check, and reaps child processes
|
|
48
|
+
on stop.
|
|
49
|
+
|
|
50
|
+
**Applicability to GreedySearch-pi:** We already have per-tab daemons in
|
|
51
|
+
`bin/cdp.mjs` that hold CDP sessions open. The difference is per-tab vs
|
|
52
|
+
per-user-session. A per-user bridge could reduce Chrome launch overhead when
|
|
53
|
+
switching between engines, but our current model (one Chrome instance, multiple
|
|
54
|
+
tabs) already achieves this.
|
|
55
|
+
|
|
56
|
+
#### Multi-step script execution (`run` command)
|
|
57
|
+
|
|
58
|
+
Accepts multi-step scripts from stdin, executing them sequentially against the
|
|
59
|
+
same browser session. Each step is a CLI command (open, click, fill, eval, etc.).
|
|
60
|
+
|
|
61
|
+
**Applicability to GreedySearch-pi:** Our research mode already orchestrates
|
|
62
|
+
multi-step workflows (plan → search → fetch → synthesize). The `run` pattern
|
|
63
|
+
is cleaner for ad-hoc multi-step tasks but our orchestration is purpose-built
|
|
64
|
+
for research.
|
|
65
|
+
|
|
66
|
+
#### `CHROME_DEVTOOLS_AXI_BROWSER_URL` for external Chrome
|
|
67
|
+
|
|
68
|
+
Connects to an existing Chrome instance instead of launching one. Accepts both
|
|
69
|
+
`http://` URLs (fetches `/json/version` to discover WebSocket URL) and `ws://`
|
|
70
|
+
endpoints directly. Supports authenticated WebSocket endpoints via
|
|
71
|
+
`CHROME_DEVTOOLS_AXI_WS_HEADERS`.
|
|
72
|
+
|
|
73
|
+
**Applicability to GreedySearch-pi:** We already support `CDP_PROFILE_DIR` for
|
|
74
|
+
targeting our own Chrome profile. The WebSocket auth header support could be
|
|
75
|
+
useful for connecting to remote browser instances (BrowserBase, Browserless,
|
|
76
|
+
etc.) in a future cloud mode.
|
|
77
|
+
|
|
78
|
+
#### `CHROME_DEVTOOLS_AXI_CHANNEL` for release channel selection
|
|
79
|
+
|
|
80
|
+
Picks which installed Chrome release channel to target — `stable` (default),
|
|
81
|
+
`beta`, `canary`, or `dev`. Used by `--autoConnect` and launch modes.
|
|
82
|
+
|
|
83
|
+
**Applicability to GreedySearch-pi:** Could let users test against Chrome Beta
|
|
84
|
+
or Canary to catch anti-bot changes before they hit stable.
|
|
85
|
+
|
|
86
|
+
### What's not applicable
|
|
87
|
+
|
|
88
|
+
- **TOON encoding** — compact structured output format. We use JSON for
|
|
89
|
+
structured data and plain text for answers. Not needed.
|
|
90
|
+
- **Lighthouse / performance / heap snapshots** — performance auditing is
|
|
91
|
+
outside GreedySearch's scope.
|
|
92
|
+
- **chrome-devtools-mcp dependency** — wrapping MCP would mean losing control
|
|
93
|
+
over our extraction pipelines, stealth patches, and multi-engine
|
|
94
|
+
parallelization. Not worth it.
|
|
95
|
+
|
|
96
|
+
---
|
|
97
|
+
|
|
98
|
+
## 2. [faster-chrome-devtools-skill](https://github.com/zeke/faster-chrome-devtools-skill)
|
|
99
|
+
|
|
100
|
+
A Pi-compatible Agent Skill wrapping CDP in a zero-dependency Node.js script.
|
|
101
|
+
Shares the same ancestor (`chrome-cdp-skill` by pasky) as our `bin/cdp.mjs`.
|
|
102
|
+
WebSocket client from scratch (RFC 6455), background daemon, accessibility
|
|
103
|
+
snapshots, Cloudflare Browser Run support.
|
|
104
|
+
|
|
105
|
+
### Notable patterns
|
|
106
|
+
|
|
107
|
+
#### Daemon-level Runtime.enable workaround (`captureMainContext`)
|
|
108
|
+
|
|
109
|
+
**Already implemented in GreedySearch-pi.** Our `bin/cdp.mjs` has the exact
|
|
110
|
+
pattern: briefly enable `Runtime` at daemon start (50-100ms), capture the main
|
|
111
|
+
execution context ID, immediately disable, then pass `contextId` explicitly on
|
|
112
|
+
every `Runtime.evaluate` call. This avoids the persistent `Runtime.enable`
|
|
113
|
+
detection vector that Cloudflare/DataDome watch for (the browser auto-serializes
|
|
114
|
+
console objects when Runtime is active, triggering proxy traps in anti-bot
|
|
115
|
+
scripts).
|
|
116
|
+
|
|
117
|
+
Our implementation is in `bin/cdp.mjs` — `captureMainContext()` function with
|
|
118
|
+
detailed comments citing rebrowser.net and the detection mechanism.
|
|
119
|
+
|
|
120
|
+
#### `loadall` command (repeated click-until-gone)
|
|
121
|
+
|
|
122
|
+
Repeatedly clicks a CSS selector (e.g. "Load more") at a configurable interval
|
|
123
|
+
until the element disappears from the DOM. Has a 5-minute hard deadline and
|
|
124
|
+
interval clamping (100ms–30s).
|
|
125
|
+
|
|
126
|
+
**Applicability to GreedySearch-pi:** Could be useful for infinite-scroll result
|
|
127
|
+
pages in research mode. Some search engines (Bing, Google) paginate results
|
|
128
|
+
behind "Show more" buttons. A `loadall`-style helper in `extractors/common.mjs`
|
|
129
|
+
would let extractors scroll through all results before extracting.
|
|
130
|
+
|
|
131
|
+
#### Compact accessibility tree mode
|
|
132
|
+
|
|
133
|
+
`shouldShowAxNode()` filters out `InlineTextBox` nodes and empty/trivial nodes
|
|
134
|
+
from snapshot output, producing a much more compact tree.
|
|
135
|
+
|
|
136
|
+
**Applicability to GreedySearch-pi:** Minor UX improvement for our `snap`
|
|
137
|
+
command output. Not a priority.
|
|
138
|
+
|
|
139
|
+
#### `browserraw` / `browse` level commands
|
|
140
|
+
|
|
141
|
+
Sends browser-level CDP commands (no sessionId) for things like
|
|
142
|
+
`Browser.setDownloadBehavior`, `Browser.grantPermissions`, etc.
|
|
143
|
+
|
|
144
|
+
**Applicability to GreedySearch-pi:** Could be useful for PDF download in
|
|
145
|
+
research mode when fetching academic papers. Currently we fetch via HTTP; a
|
|
146
|
+
browser-level download handler would let us capture PDFs that require
|
|
147
|
+
JavaScript rendering.
|
|
148
|
+
|
|
149
|
+
#### Cloudflare Browser Run support
|
|
150
|
+
|
|
151
|
+
Connects to Cloudflare's remote browser service via `wss://` endpoints with
|
|
152
|
+
authentication headers. Uses `lab=true` by default for Chrome beta features.
|
|
153
|
+
|
|
154
|
+
**Applicability to GreedySearch-pi:** Future direction — running GreedySearch
|
|
155
|
+
in the cloud without a local Chrome instance. The WebSocket auth header pattern
|
|
156
|
+
(`CHROME_DEVTOOLS_AXI_WS_HEADERS`) from chrome-devtools-axi complements this.
|
|
157
|
+
|
|
158
|
+
#### `--ws-endpoint` / `--http-endpoint` env vars
|
|
159
|
+
|
|
160
|
+
Cleaner than our `CDP_PROFILE_DIR` approach for specifying which Chrome to
|
|
161
|
+
connect to. `CDP_WS_ENDPOINT` and `CDP_HTTP_ENDPOINT` env vars with explicit
|
|
162
|
+
`--ws-endpoint` / `--http-endpoint` CLI flags.
|
|
163
|
+
|
|
164
|
+
**Applicability to GreedySearch-pi:** Could simplify our connection logic.
|
|
165
|
+
Currently we derive the WebSocket URL from `DevToolsActivePort` in the profile
|
|
166
|
+
directory. Accepting an explicit endpoint would be more flexible.
|
|
167
|
+
|
|
168
|
+
### What's not applicable
|
|
169
|
+
|
|
170
|
+
- **Hand-rolled RFC 6455 WebSocket client** — unnecessary since Node.js 22+
|
|
171
|
+
has built-in `WebSocket`. We use the native one.
|
|
172
|
+
- **General-purpose browser skill** — it's designed for ad-hoc browsing, not
|
|
173
|
+
multi-engine search. No clipboard interception, no stream detection, no
|
|
174
|
+
source extraction.
|
|
175
|
+
- **Per-tab daemon model** — same as ours. No improvement to adopt.
|
|
176
|
+
|
|
177
|
+
---
|
|
178
|
+
|
|
179
|
+
## Summary of actionable ideas
|
|
180
|
+
|
|
181
|
+
| Idea | Source | Priority | Effort |
|
|
182
|
+
|------|--------|----------|--------|
|
|
183
|
+
| Generation counters for stale-ref detection | chrome-devtools-axi | Medium | Small |
|
|
184
|
+
| Named session isolation | chrome-devtools-axi | Low | Medium |
|
|
185
|
+
| `loadall` helper for infinite-scroll pages | faster-chrome-devtools-skill | Medium | Small |
|
|
186
|
+
| Browser-level CDP for PDF download | faster-chrome-devtools-skill | Low | Small |
|
|
187
|
+
| Explicit WS endpoint env vars | faster-chrome-devtools-skill | Low | Small |
|
|
188
|
+
| Remote browser auth headers | both | Low | Medium |
|
|
189
|
+
| Release channel selection | chrome-devtools-axi | Low | Small |
|
|
190
|
+
| Runtime.enable workaround | faster-chrome-devtools-skill | ✅ Already done | — |
|