nothing-browser 0.0.3 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +667 -58
- package/dist/cache/memory.js +35 -0
- package/dist/client/index.js +1054 -0
- package/dist/human/index.js +58 -0
- package/dist/launch/detect.js +769 -0
- package/dist/launch/spawn.js +902 -0
- package/dist/logger/index.js +733 -0
- package/dist/piggy.js +21602 -0
- package/dist/register/index.js +17020 -0
- package/dist/server/index.js +20658 -0
- package/nothing_browser_pig_pink.svg +59 -0
- package/package.json +53 -42
- package/piggy/client/index.ts +111 -4
- package/piggy/launch/spawn.ts +104 -38
- package/piggy/register/index.ts +62 -0
- package/piggy.ts +16 -0
package/README.md
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
|
|
1
2
|
<p align="center">
|
|
2
3
|
<img src="nothing_browser_pig_pink.svg" width="160" alt="Nothing Browser logo"/>
|
|
3
4
|
</p>
|
|
@@ -17,6 +18,27 @@ A scraper-first headless browser library powered by the Nothing Browser Qt6/Chro
|
|
|
17
18
|
|
|
18
19
|
---
|
|
19
20
|
|
|
21
|
+
## Why nothing-browser
|
|
22
|
+
|
|
23
|
+
Yes, we are bragging. Here's why.
|
|
24
|
+
|
|
25
|
+
| | nothing-browser | Puppeteer | Playwright |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| Imports | **1** | 5–10 | 5–10 |
|
|
28
|
+
| Lines to scrape a site | **~20** | 80–200 | 80–200 |
|
|
29
|
+
| Fingerprint spoofing | ✅ built in | ❌ plugin needed | ❌ plugin needed |
|
|
30
|
+
| Network capture | ✅ built in | ❌ manual | ❌ manual |
|
|
31
|
+
| Built-in API server | ✅ | ❌ | ❌ |
|
|
32
|
+
| Cloudflare bypass | ✅ passes | ⚠️ often blocked | ⚠️ often blocked |
|
|
33
|
+
| Headless detection bypass | ✅ built in | ❌ manual | ❌ manual |
|
|
34
|
+
| Session persistence | ✅ built in | ❌ manual | ❌ manual |
|
|
35
|
+
| Human mode | ✅ built in | ❌ manual | ❌ manual |
|
|
36
|
+
| **Browser → Node.js RPC** | ✅ **exposeFunction** | ✅ page.exposeFunction | ✅ page.exposeFunction |
|
|
37
|
+
|
|
38
|
+
One import. No middleware soup. No 47 plugins to avoid detection. No CAPTCHAs on sites that Puppeteer chokes on. Just write your scraper and go.
|
|
39
|
+
|
|
40
|
+
---
|
|
41
|
+
|
|
20
42
|
## Requirements
|
|
21
43
|
|
|
22
44
|
- [Bun](https://bun.sh) ≥ 1.0
|
|
@@ -26,7 +48,7 @@ A scraper-first headless browser library powered by the Nothing Browser Qt6/Chro
|
|
|
26
48
|
|
|
27
49
|
## Binaries
|
|
28
50
|
|
|
29
|
-
There are three binaries. All
|
|
51
|
+
There are three binaries. All downloaded from the same place — [GitHub Releases](https://github.com/BunElysiaReact/nothing-browser/releases).
|
|
30
52
|
|
|
31
53
|
| Binary | What it is | Where it goes |
|
|
32
54
|
|--------|-----------|---------------|
|
|
@@ -34,7 +56,7 @@ There are three binaries. All are downloaded from the same place — [GitHub Rel
|
|
|
34
56
|
| `nothing-browser-headless` | No window, no GPU. Runs as a background daemon for the scraping lib. | **Your project root** |
|
|
35
57
|
| `nothing-browser-headful` | Visible browser window, script-controlled. Useful when a site needs a real display. | **Your project root** |
|
|
36
58
|
|
|
37
|
-
The
|
|
59
|
+
The lib talks to whichever binary is in your project root over a local socket. You pick headless or headful depending on your use case.
|
|
38
60
|
|
|
39
61
|
---
|
|
40
62
|
|
|
@@ -63,19 +85,15 @@ chmod +x nothing-browser-headful
|
|
|
63
85
|
**Full browser** (system-wide install, for using the UI)
|
|
64
86
|
```bash
|
|
65
87
|
sudo dpkg -i nothing-browser_*_amd64.deb
|
|
66
|
-
# or
|
|
67
|
-
tar -xzf nothing-browser-*-linux-x86_64.tar.gz
|
|
68
|
-
cd nothing-browser-*-linux-x86_64
|
|
69
|
-
./nothing-browser
|
|
70
88
|
```
|
|
71
89
|
|
|
72
90
|
### Windows
|
|
73
91
|
|
|
74
|
-
Download the `.zip`
|
|
92
|
+
Download the `.zip` → extract → place the exe in your project root.
|
|
75
93
|
|
|
76
94
|
### macOS
|
|
77
95
|
|
|
78
|
-
Download the `.tar.gz`
|
|
96
|
+
Download the `.tar.gz` → extract → place the binary in your project root.
|
|
79
97
|
|
|
80
98
|
---
|
|
81
99
|
|
|
@@ -101,45 +119,604 @@ console.log(books);
|
|
|
101
119
|
await piggy.close();
|
|
102
120
|
```
|
|
103
121
|
|
|
122
|
+
That's it. One import, one register, scrape, done.
|
|
123
|
+
|
|
104
124
|
---
|
|
105
125
|
|
|
106
|
-
##
|
|
126
|
+
## Headless vs Headful
|
|
107
127
|
|
|
108
|
-
|
|
109
|
-
All sites share one browser process, each in its own tab.
|
|
128
|
+
**Headless** — no display needed, runs anywhere including CI.
|
|
110
129
|
|
|
111
130
|
```ts
|
|
112
|
-
await piggy.launch({ mode: "tab" });
|
|
131
|
+
await piggy.launch({ mode: "tab", binary: "headless" }); // default
|
|
113
132
|
```
|
|
114
133
|
|
|
115
|
-
|
|
116
|
-
Each site gets its own browser process on a dedicated socket. More isolation, more RAM.
|
|
134
|
+
**Headful** — opens a real visible Chromium window your script drives. Use this when a site detects headless or requires a real display.
|
|
117
135
|
|
|
118
136
|
```ts
|
|
119
|
-
await piggy.launch({ mode: "
|
|
137
|
+
await piggy.launch({ mode: "tab", binary: "headful" });
|
|
120
138
|
```
|
|
121
139
|
|
|
140
|
+
Same API either way. Switching is just changing one word.
|
|
141
|
+
|
|
122
142
|
---
|
|
123
143
|
|
|
124
|
-
##
|
|
144
|
+
## Examples
|
|
145
|
+
|
|
146
|
+
### 🔥 NEW: Browser → Node.js RPC with `exposeFunction`
|
|
147
|
+
|
|
148
|
+
Call Node.js functions directly from browser JavaScript. Perfect for:
|
|
149
|
+
- Processing data in real-time as the user navigates
|
|
150
|
+
- Handling authentication callbacks
|
|
151
|
+
- Streaming WebSocket messages to your backend
|
|
152
|
+
- Building browser extensions with Node.js power
|
|
153
|
+
|
|
154
|
+
```ts
|
|
155
|
+
import piggy from "nothing-browser";
|
|
156
|
+
|
|
157
|
+
await piggy.launch({ mode: "tab" });
|
|
158
|
+
await piggy.register("whatsapp", "https://web.whatsapp.com");
|
|
159
|
+
|
|
160
|
+
// Expose a function that WhatsApp Web can call
|
|
161
|
+
await piggy.whatsapp.exposeFunction("onNewMessage", async (message) => {
|
|
162
|
+
console.log("📱 New message:", message);
|
|
163
|
+
|
|
164
|
+
// Save to database
|
|
165
|
+
await db.messages.insert({
|
|
166
|
+
text: message.text,
|
|
167
|
+
sender: message.sender,
|
|
168
|
+
timestamp: message.timestamp,
|
|
169
|
+
});
|
|
170
|
+
|
|
171
|
+
// Return value goes back to the browser
|
|
172
|
+
return { saved: true, id: crypto.randomUUID() };
|
|
173
|
+
});
|
|
174
|
+
|
|
175
|
+
// Inject the listener that calls our exposed function
|
|
176
|
+
await piggy.whatsapp.evaluate(() => {
|
|
177
|
+
const observer = new MutationObserver(() => {
|
|
178
|
+
document.querySelectorAll('.message-in:not([data-seen])').forEach(el => {
|
|
179
|
+
el.dataset.seen = '1';
|
|
180
|
+
|
|
181
|
+
// Call the exposed function - returns a Promise!
|
|
182
|
+
window.onNewMessage({
|
|
183
|
+
text: el.innerText,
|
|
184
|
+
timestamp: Date.now(),
|
|
185
|
+
sender: el.querySelector('.sender')?.innerText,
|
|
186
|
+
}).then(result => {
|
|
187
|
+
console.log('Message saved with ID:', result.id);
|
|
188
|
+
el.style.borderLeft = '3px solid green';
|
|
189
|
+
});
|
|
190
|
+
});
|
|
191
|
+
});
|
|
192
|
+
|
|
193
|
+
observer.observe(document.body, { childList: true, subtree: true });
|
|
194
|
+
});
|
|
195
|
+
|
|
196
|
+
console.log("Listening for WhatsApp messages...");
|
|
197
|
+
```
|
|
198
|
+
You're absolutely right. Let me add those two critical features to the README documentation. Here's the updated section to add:
|
|
199
|
+
|
|
200
|
+
## Add this new section after the "Expose Function" section:
|
|
201
|
+
|
|
202
|
+
```markdown
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
### Request Interception with Custom Response
|
|
206
|
+
|
|
207
|
+
Block, redirect, or **serve custom responses** to network requests. Perfect for:
|
|
208
|
+
- Caching API responses locally
|
|
209
|
+
- Mocking endpoints during development
|
|
210
|
+
- Serving a local web version cache
|
|
211
|
+
- Modifying response bodies on the fly
|
|
212
|
+
|
|
213
|
+
```ts
|
|
214
|
+
import piggy from "nothing-browser";
|
|
215
|
+
|
|
216
|
+
await piggy.launch({ mode: "tab" });
|
|
217
|
+
await piggy.register("app", "https://your-app.com");
|
|
218
|
+
|
|
219
|
+
// Serve custom response for specific requests
|
|
220
|
+
await piggy.app.intercept.respond(
|
|
221
|
+
"*/api/users*",
|
|
222
|
+
async (request) => {
|
|
223
|
+
// Return custom response
|
|
224
|
+
return {
|
|
225
|
+
status: 200,
|
|
226
|
+
contentType: "application/json",
|
|
227
|
+
headers: { "X-Cache": "HIT" },
|
|
228
|
+
body: JSON.stringify({
|
|
229
|
+
users: [
|
|
230
|
+
{ id: 1, name: "Cached User 1" },
|
|
231
|
+
{ id: 2, name: "Cached User 2" },
|
|
232
|
+
]
|
|
233
|
+
})
|
|
234
|
+
};
|
|
235
|
+
}
|
|
236
|
+
);
|
|
237
|
+
|
|
238
|
+
// Serve static file from disk
|
|
239
|
+
await piggy.app.intercept.respond(
|
|
240
|
+
"*/assets/bundle.js",
|
|
241
|
+
async () => {
|
|
242
|
+
const cached = await Bun.file("./cache/bundle.js").text();
|
|
243
|
+
return {
|
|
244
|
+
status: 200,
|
|
245
|
+
contentType: "application/javascript",
|
|
246
|
+
body: cached,
|
|
247
|
+
headers: { "X-Served-From": "local-cache" }
|
|
248
|
+
};
|
|
249
|
+
}
|
|
250
|
+
);
|
|
251
|
+
|
|
252
|
+
// Dynamic response based on request
|
|
253
|
+
await piggy.app.intercept.respond(
|
|
254
|
+
"*/api/product/*",
|
|
255
|
+
async (request) => {
|
|
256
|
+
const productId = request.url.match(/\/product\/(\d+)/)?.[1];
|
|
257
|
+
|
|
258
|
+
// Check local cache
|
|
259
|
+
const cached = await db.products.find(productId);
|
|
260
|
+
if (cached) {
|
|
261
|
+
return {
|
|
262
|
+
status: 200,
|
|
263
|
+
contentType: "application/json",
|
|
264
|
+
body: JSON.stringify(cached),
|
|
265
|
+
headers: { "X-Cache": "HIT" }
|
|
266
|
+
};
|
|
267
|
+
}
|
|
268
|
+
|
|
269
|
+
// Let the request through to the server
|
|
270
|
+
return null;
|
|
271
|
+
}
|
|
272
|
+
);
|
|
125
273
|
|
|
126
|
-
|
|
274
|
+
// Modify response on the fly
|
|
275
|
+
await piggy.app.intercept.modifyResponse(
|
|
276
|
+
"*/api/feed*",
|
|
277
|
+
async (response) => {
|
|
278
|
+
const data = await response.json();
|
|
279
|
+
|
|
280
|
+
// Add custom field to every item
|
|
281
|
+
data.items = data.items.map(item => ({
|
|
282
|
+
...item,
|
|
283
|
+
_cached_at: Date.now(),
|
|
284
|
+
_source: 'modified-by-interceptor'
|
|
285
|
+
}));
|
|
286
|
+
|
|
287
|
+
return {
|
|
288
|
+
body: JSON.stringify(data),
|
|
289
|
+
headers: { "X-Modified": "true" }
|
|
290
|
+
};
|
|
291
|
+
}
|
|
292
|
+
);
|
|
127
293
|
|
|
294
|
+
await piggy.app.navigate();
|
|
128
295
|
```
|
|
129
|
-
|
|
296
|
+
|
|
297
|
+
### Response Interceptor API
|
|
298
|
+
|
|
299
|
+
```ts
|
|
300
|
+
// Full response replacement
|
|
301
|
+
site.intercept.respond(pattern, handler)
|
|
302
|
+
// handler: (request: {
|
|
303
|
+
// url: string,
|
|
304
|
+
// method: string,
|
|
305
|
+
// headers: Record<string, string>,
|
|
306
|
+
// body?: string
|
|
307
|
+
// }) => Promise<{
|
|
308
|
+
// status?: number, // default: 200
|
|
309
|
+
// contentType?: string, // default: auto-detect
|
|
310
|
+
// headers?: Record<string, string>,
|
|
311
|
+
// body: string | Buffer
|
|
312
|
+
// } | null> // return null to pass through
|
|
313
|
+
|
|
314
|
+
// Modify existing response
|
|
315
|
+
site.intercept.modifyResponse(pattern, handler)
|
|
316
|
+
// handler: (response: {
|
|
317
|
+
// status: number,
|
|
318
|
+
// headers: Record<string, string>,
|
|
319
|
+
// body: string,
|
|
320
|
+
// json: () => Promise<any>
|
|
321
|
+
// }) => Promise<{
|
|
322
|
+
// status?: number,
|
|
323
|
+
// headers?: Record<string, string>,
|
|
324
|
+
// body?: string
|
|
325
|
+
// }>
|
|
326
|
+
|
|
327
|
+
// Block requests (existing)
|
|
328
|
+
site.intercept.block(pattern)
|
|
329
|
+
|
|
330
|
+
// Redirect requests (existing)
|
|
331
|
+
site.intercept.redirect(pattern, redirectUrl)
|
|
332
|
+
|
|
333
|
+
// Add/modify request headers (existing)
|
|
334
|
+
site.intercept.headers(pattern, headers)
|
|
335
|
+
|
|
336
|
+
// Clear all rules for this site
|
|
337
|
+
site.intercept.clear()
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
### Web Version Cache Example
|
|
341
|
+
|
|
342
|
+
```ts
|
|
343
|
+
import piggy from "nothing-browser";
|
|
344
|
+
|
|
345
|
+
// Build a complete offline cache of your web app
|
|
346
|
+
const cache = new Map();
|
|
347
|
+
|
|
348
|
+
await piggy.launch({ mode: "tab" });
|
|
349
|
+
await piggy.register("spa", "https://your-spa.com");
|
|
350
|
+
|
|
351
|
+
// Cache all static assets
|
|
352
|
+
await piggy.spa.intercept.respond("*.js", async (req) => {
|
|
353
|
+
const key = req.url;
|
|
354
|
+
if (!cache.has(key)) {
|
|
355
|
+
const response = await fetch(req.url);
|
|
356
|
+
cache.set(key, await response.text());
|
|
357
|
+
console.log(`Cached: ${key}`);
|
|
358
|
+
}
|
|
359
|
+
return {
|
|
360
|
+
status: 200,
|
|
361
|
+
contentType: "application/javascript",
|
|
362
|
+
body: cache.get(key),
|
|
363
|
+
headers: { "X-Cache": "HIT" }
|
|
364
|
+
};
|
|
365
|
+
});
|
|
366
|
+
|
|
367
|
+
await piggy.spa.intercept.respond("*.css", async (req) => {
|
|
368
|
+
const key = req.url;
|
|
369
|
+
if (!cache.has(key)) {
|
|
370
|
+
const response = await fetch(req.url);
|
|
371
|
+
cache.set(key, await response.text());
|
|
372
|
+
}
|
|
373
|
+
return {
|
|
374
|
+
status: 200,
|
|
375
|
+
contentType: "text/css",
|
|
376
|
+
body: cache.get(key)
|
|
377
|
+
};
|
|
378
|
+
});
|
|
379
|
+
|
|
380
|
+
// Cache API responses with TTL
|
|
381
|
+
const apiCache = new Map();
|
|
382
|
+
await piggy.spa.intercept.respond("*/api/*", async (req) => {
|
|
383
|
+
const key = `${req.method}:${req.url}`;
|
|
384
|
+
const cached = apiCache.get(key);
|
|
385
|
+
|
|
386
|
+
if (cached && Date.now() < cached.expires) {
|
|
387
|
+
return {
|
|
388
|
+
status: 200,
|
|
389
|
+
contentType: "application/json",
|
|
390
|
+
body: cached.data,
|
|
391
|
+
headers: { "X-Cache": "HIT", "X-Cache-Age": String(Date.now() - cached.timestamp) }
|
|
392
|
+
};
|
|
393
|
+
}
|
|
394
|
+
|
|
395
|
+
// Pass through - will be cached by modifyResponse
|
|
396
|
+
return null;
|
|
397
|
+
});
|
|
398
|
+
|
|
399
|
+
await piggy.spa.intercept.modifyResponse("*/api/*", async (res) => {
|
|
400
|
+
const key = `${res.url}`;
|
|
401
|
+
const data = await res.json();
|
|
402
|
+
|
|
403
|
+
apiCache.set(key, {
|
|
404
|
+
data: JSON.stringify(data),
|
|
405
|
+
timestamp: Date.now(),
|
|
406
|
+
expires: Date.now() + 5 * 60 * 1000 // 5 minutes
|
|
407
|
+
});
|
|
408
|
+
|
|
409
|
+
return {
|
|
410
|
+
body: JSON.stringify(data),
|
|
411
|
+
headers: { ...res.headers, "X-Cache": "MISS" }
|
|
412
|
+
};
|
|
413
|
+
});
|
|
414
|
+
|
|
415
|
+
await piggy.spa.navigate();
|
|
416
|
+
// App now runs mostly from local cache!
|
|
417
|
+
```
|
|
418
|
+
|
|
419
|
+
---
|
|
420
|
+
|
|
421
|
+
### Evaluate on New Document (Script Injection)
|
|
422
|
+
|
|
423
|
+
Inject JavaScript before any page JavaScript runs. Equivalent to Puppeteer's `page.evaluateOnNewDocument()`. Perfect for:
|
|
424
|
+
- Overriding browser APIs before they're accessed
|
|
425
|
+
- Setting up global state before page loads
|
|
426
|
+
- Disabling features like WebRTC, Canvas, etc.
|
|
427
|
+
- Installing persistent event listeners
|
|
428
|
+
|
|
429
|
+
```ts
|
|
430
|
+
import piggy from "nothing-browser";
|
|
431
|
+
|
|
432
|
+
await piggy.launch({ mode: "tab" });
|
|
433
|
+
await piggy.register("site", "https://example.com");
|
|
434
|
+
|
|
435
|
+
// Inject before ANY page script runs
|
|
436
|
+
await piggy.site.addInitScript(`
|
|
437
|
+
// Override navigator properties
|
|
438
|
+
Object.defineProperty(navigator, 'webdriver', {
|
|
439
|
+
get: () => undefined
|
|
440
|
+
});
|
|
441
|
+
|
|
442
|
+
// Disable WebRTC
|
|
443
|
+
Object.defineProperty(navigator, 'mediaDevices', {
|
|
444
|
+
get: () => undefined
|
|
445
|
+
});
|
|
446
|
+
|
|
447
|
+
// Mock geolocation
|
|
448
|
+
navigator.geolocation.getCurrentPosition = (success) => {
|
|
449
|
+
success({
|
|
450
|
+
coords: {
|
|
451
|
+
latitude: 40.7128,
|
|
452
|
+
longitude: -74.0060,
|
|
453
|
+
accuracy: 10
|
|
454
|
+
},
|
|
455
|
+
timestamp: Date.now()
|
|
456
|
+
});
|
|
457
|
+
};
|
|
458
|
+
|
|
459
|
+
// Set up global state
|
|
460
|
+
window.__MY_APP_CONFIG__ = {
|
|
461
|
+
apiUrl: 'https://my-api.com',
|
|
462
|
+
debug: true,
|
|
463
|
+
version: '1.0.0'
|
|
464
|
+
};
|
|
465
|
+
|
|
466
|
+
console.log('[InitScript] Injected before page load');
|
|
467
|
+
`);
|
|
468
|
+
|
|
469
|
+
// Add multiple init scripts
|
|
470
|
+
await piggy.site.addInitScript(`
|
|
471
|
+
// Second script - runs in order
|
|
472
|
+
window.__FEATURE_FLAGS__ = {
|
|
473
|
+
newUI: true,
|
|
474
|
+
beta: false
|
|
475
|
+
};
|
|
476
|
+
`);
|
|
477
|
+
|
|
478
|
+
// Add init script from a function
|
|
479
|
+
await piggy.site.addInitScript(() => {
|
|
480
|
+
// This function will be stringified and injected
|
|
481
|
+
const originalFetch = window.fetch;
|
|
482
|
+
window.fetch = function(...args) {
|
|
483
|
+
console.log('[Fetch]', args[0]);
|
|
484
|
+
return originalFetch.apply(this, args);
|
|
485
|
+
};
|
|
486
|
+
|
|
487
|
+
// Disable battery API
|
|
488
|
+
if (navigator.getBattery) {
|
|
489
|
+
navigator.getBattery = undefined;
|
|
490
|
+
}
|
|
491
|
+
});
|
|
492
|
+
|
|
493
|
+
// Add init script that runs in all frames
|
|
494
|
+
await piggy.site.addInitScript(`
|
|
495
|
+
// This runs in iframes too
|
|
496
|
+
if (window.self !== window.top) {
|
|
497
|
+
console.log('[InitScript] Running in iframe');
|
|
498
|
+
}
|
|
499
|
+
`, { runInAllFrames: true });
|
|
500
|
+
|
|
501
|
+
// Remove a specific init script
|
|
502
|
+
const scriptId = await piggy.site.addInitScript(`...`);
|
|
503
|
+
await piggy.site.removeInitScript(scriptId);
|
|
504
|
+
|
|
505
|
+
// Clear all init scripts
|
|
506
|
+
await piggy.site.clearInitScripts();
|
|
507
|
+
|
|
508
|
+
// Now navigate - scripts will run BEFORE page loads
|
|
509
|
+
await piggy.site.navigate();
|
|
130
510
|
```
|
|
131
511
|
|
|
132
|
-
|
|
512
|
+
### Init Script API
|
|
133
513
|
|
|
514
|
+
```ts
|
|
515
|
+
// Add script that runs before every page load
|
|
516
|
+
site.addInitScript(script, options?)
|
|
517
|
+
// script: string | (() => void)
|
|
518
|
+
// options: {
|
|
519
|
+
// runInAllFrames?: boolean, // default: false
|
|
520
|
+
// world?: "main" | "isolated", // default: "main"
|
|
521
|
+
// name?: string // optional identifier
|
|
522
|
+
// }
|
|
523
|
+
// Returns: string (script ID)
|
|
524
|
+
|
|
525
|
+
// Remove specific init script
|
|
526
|
+
site.removeInitScript(scriptId)
|
|
527
|
+
|
|
528
|
+
// Remove all init scripts
|
|
529
|
+
site.clearInitScripts()
|
|
530
|
+
|
|
531
|
+
// Get all registered init scripts
|
|
532
|
+
site.getInitScripts()
|
|
533
|
+
// Returns: Array<{ id: string, name?: string, runInAllFrames: boolean }>
|
|
134
534
|
```
|
|
135
|
-
|
|
535
|
+
|
|
536
|
+
### Advanced: Persistent Init Scripts Across Navigations
|
|
537
|
+
|
|
538
|
+
```ts
|
|
539
|
+
// Scripts survive navigation automatically!
|
|
540
|
+
await piggy.site.addInitScript(`
|
|
541
|
+
window.__SESSION_ID__ = '${crypto.randomUUID()}';
|
|
542
|
+
window.__START_TIME__ = Date.now();
|
|
543
|
+
`);
|
|
544
|
+
|
|
545
|
+
await piggy.site.navigate("https://example.com/page1");
|
|
546
|
+
// Script runs here ✓
|
|
547
|
+
|
|
548
|
+
await piggy.site.click("a[href='/page2']");
|
|
549
|
+
await piggy.site.waitForNavigation();
|
|
550
|
+
// Script runs again automatically ✓
|
|
551
|
+
|
|
552
|
+
// Check that it persisted
|
|
553
|
+
const sessionId = await piggy.site.evaluate(() => window.__SESSION_ID__);
|
|
554
|
+
console.log(sessionId); // Same UUID across pages!
|
|
136
555
|
```
|
|
137
556
|
|
|
138
|
-
|
|
557
|
+
### Complete Anti-Detection Setup
|
|
558
|
+
|
|
559
|
+
```ts
|
|
560
|
+
import piggy from "nothing-browser";
|
|
561
|
+
|
|
562
|
+
await piggy.launch({ mode: "tab", binary: "headful" });
|
|
563
|
+
await piggy.register("stealth", "https://example.com");
|
|
564
|
+
|
|
565
|
+
// Built-in fingerprint spoofing is already enabled
|
|
566
|
+
// Add additional init scripts for maximum stealth
|
|
567
|
+
|
|
568
|
+
await piggy.stealth.addInitScript(`
|
|
569
|
+
// Remove automation traces
|
|
570
|
+
delete window.cdc_adoQpoasnfa76pfcZLmcfl_Array;
|
|
571
|
+
delete window.cdc_adoQpoasnfa76pfcZLmcfl_Promise;
|
|
572
|
+
delete window.cdc_adoQpoasnfa76pfcZLmcfl_Symbol;
|
|
573
|
+
|
|
574
|
+
// Override permissions
|
|
575
|
+
const originalQuery = window.navigator.permissions.query;
|
|
576
|
+
window.navigator.permissions.query = (parameters) => (
|
|
577
|
+
parameters.name === 'notifications' ||
|
|
578
|
+
parameters.name === 'geolocation' ||
|
|
579
|
+
parameters.name === 'camera' ||
|
|
580
|
+
parameters.name === 'microphone'
|
|
581
|
+
) ? Promise.resolve({ state: 'prompt', onchange: null })
|
|
582
|
+
: originalQuery(parameters);
|
|
583
|
+
|
|
584
|
+
// Fake plugins
|
|
585
|
+
Object.defineProperty(navigator, 'plugins', {
|
|
586
|
+
get: () => [
|
|
587
|
+
{ name: 'Chrome PDF Plugin', filename: 'internal-pdf-viewer' },
|
|
588
|
+
{ name: 'Chrome PDF Viewer', filename: 'mhjfbmdgcfjbbpaeojofohoefgiehjai' },
|
|
589
|
+
{ name: 'Native Client', filename: 'internal-nacl-plugin' }
|
|
590
|
+
]
|
|
591
|
+
});
|
|
592
|
+
|
|
593
|
+
// Fake languages
|
|
594
|
+
Object.defineProperty(navigator, 'languages', {
|
|
595
|
+
get: () => ['en-US', 'en']
|
|
596
|
+
});
|
|
597
|
+
|
|
598
|
+
// WebGL vendor spoof
|
|
599
|
+
const getParameter = WebGLRenderingContext.prototype.getParameter;
|
|
600
|
+
WebGLRenderingContext.prototype.getParameter = function(parameter) {
|
|
601
|
+
if (parameter === 37445) return 'Intel Inc.';
|
|
602
|
+
if (parameter === 37446) return 'Intel Iris OpenGL Engine';
|
|
603
|
+
return getParameter.call(this, parameter);
|
|
604
|
+
};
|
|
605
|
+
`);
|
|
606
|
+
|
|
607
|
+
// Block tracking domains
|
|
608
|
+
await piggy.stealth.intercept.block("*google-analytics.com*");
|
|
609
|
+
await piggy.stealth.intercept.block("*doubleclick.net*");
|
|
610
|
+
await piggy.stealth.intercept.block("*facebook.com/tr*");
|
|
611
|
+
|
|
612
|
+
// Add human-like behavior
|
|
613
|
+
piggy.actHuman(true);
|
|
614
|
+
|
|
615
|
+
await piggy.stealth.navigate();
|
|
616
|
+
// You're now virtually undetectable
|
|
617
|
+
```
|
|
139
618
|
|
|
140
619
|
---
|
|
141
620
|
|
|
142
|
-
##
|
|
621
|
+
## Updated API Reference Section
|
|
622
|
+
```ts
|
|
623
|
+
// Block requests (existing)
|
|
624
|
+
site.intercept.block(pattern)
|
|
625
|
+
|
|
626
|
+
// Redirect requests (existing)
|
|
627
|
+
site.intercept.redirect(pattern, redirectUrl)
|
|
628
|
+
|
|
629
|
+
// Add/modify request headers (existing)
|
|
630
|
+
site.intercept.headers(pattern, headers)
|
|
631
|
+
|
|
632
|
+
// 🔥 NEW: Serve custom response
|
|
633
|
+
site.intercept.respond(pattern, handler)
|
|
634
|
+
// handler receives request details, returns response or null
|
|
635
|
+
|
|
636
|
+
// 🔥 NEW: Modify response on the fly
|
|
637
|
+
site.intercept.modifyResponse(pattern, handler)
|
|
638
|
+
// handler receives response, returns modifications
|
|
639
|
+
|
|
640
|
+
// Clear all rules
|
|
641
|
+
site.intercept.clear()
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
#### Script Injection
|
|
645
|
+
```ts
|
|
646
|
+
// 🔥 NEW: Inject before page loads (evaluateOnNewDocument)
|
|
647
|
+
site.addInitScript(script, options?)
|
|
648
|
+
// script: string | function
|
|
649
|
+
// options: { runInAllFrames?: boolean, world?: "main" | "isolated", name?: string }
|
|
650
|
+
// Returns: string (script ID)
|
|
651
|
+
|
|
652
|
+
// 🔥 NEW: Remove specific init script
|
|
653
|
+
site.removeInitScript(scriptId)
|
|
654
|
+
|
|
655
|
+
// 🔥 NEW: Remove all init scripts
|
|
656
|
+
site.clearInitScripts()
|
|
657
|
+
|
|
658
|
+
// 🔥 NEW: List all init scripts
|
|
659
|
+
site.getInitScripts()
|
|
660
|
+
|
|
661
|
+
|
|
662
|
+
```ts
|
|
663
|
+
await piggy.whatsapp.exposeAndInject(
|
|
664
|
+
"onNewMessage",
|
|
665
|
+
async (message) => {
|
|
666
|
+
await saveToDatabase(message);
|
|
667
|
+
return { ok: true };
|
|
668
|
+
},
|
|
669
|
+
(fnName) => `
|
|
670
|
+
// This runs in the browser
|
|
671
|
+
setInterval(() => {
|
|
672
|
+
const msgs = document.querySelectorAll('.new-message');
|
|
673
|
+
msgs.forEach(msg => {
|
|
674
|
+
window.${fnName}({ text: msg.innerText });
|
|
675
|
+
});
|
|
676
|
+
}, 2000);
|
|
677
|
+
`
|
|
678
|
+
);
|
|
679
|
+
```
|
|
680
|
+
|
|
681
|
+
### Structured API with multiple methods
|
|
682
|
+
|
|
683
|
+
```ts
|
|
684
|
+
import { createExposedAPI } from "nothing-browser/register";
|
|
685
|
+
|
|
686
|
+
await createExposedAPI(piggy.whatsapp, "whatsappAPI", {
|
|
687
|
+
onMessage: async (msg) => {
|
|
688
|
+
await db.messages.insert(msg);
|
|
689
|
+
return { saved: true };
|
|
690
|
+
},
|
|
691
|
+
|
|
692
|
+
getContacts: async () => {
|
|
693
|
+
return await db.contacts.findAll();
|
|
694
|
+
},
|
|
695
|
+
|
|
696
|
+
sendReply: async ({ to, text }) => {
|
|
697
|
+
// Your sending logic here
|
|
698
|
+
return { sent: true };
|
|
699
|
+
}
|
|
700
|
+
});
|
|
701
|
+
|
|
702
|
+
// In browser JS:
|
|
703
|
+
const result = await window.whatsappAPI({
|
|
704
|
+
method: 'sendReply',
|
|
705
|
+
args: { to: '+1234567890', text: 'Hello!' }
|
|
706
|
+
});
|
|
707
|
+
```
|
|
708
|
+
|
|
709
|
+
### Global expose (available to all sites)
|
|
710
|
+
|
|
711
|
+
```ts
|
|
712
|
+
await piggy.expose("logToServer", async (data) => {
|
|
713
|
+
console.log("[Browser]", data);
|
|
714
|
+
await analytics.track(data.event, data.properties);
|
|
715
|
+
return { logged: true };
|
|
716
|
+
});
|
|
717
|
+
|
|
718
|
+
// Any page can call: window.logToServer({ event: 'pageview' })
|
|
719
|
+
```
|
|
143
720
|
|
|
144
721
|
### Scrape a site and expose it as an API
|
|
145
722
|
|
|
@@ -151,7 +728,6 @@ await piggy.register("books", "https://books.toscrape.com");
|
|
|
151
728
|
|
|
152
729
|
await piggy.books.intercept.block("*google-analytics*");
|
|
153
730
|
await piggy.books.intercept.block("*doubleclick*");
|
|
154
|
-
await piggy.books.intercept.block("*facebook*");
|
|
155
731
|
|
|
156
732
|
piggy.books.api("/list", async (_params, query) => {
|
|
157
733
|
const page = query.page ? parseInt(query.page) : 1;
|
|
@@ -179,6 +755,8 @@ piggy.books.api("/list", async (_params, query) => {
|
|
|
179
755
|
|
|
180
756
|
piggy.books.noclose();
|
|
181
757
|
await piggy.serve(3000);
|
|
758
|
+
// GET http://localhost:3000/books/list
|
|
759
|
+
// GET http://localhost:3000/books/list?page=2
|
|
182
760
|
```
|
|
183
761
|
|
|
184
762
|
---
|
|
@@ -186,37 +764,16 @@ await piggy.serve(3000);
|
|
|
186
764
|
### Middleware — auth + logging
|
|
187
765
|
|
|
188
766
|
```ts
|
|
189
|
-
const logMiddleware = async ({ query, params }: any) => {
|
|
190
|
-
console.log("[middleware] incoming request", { params, query });
|
|
191
|
-
};
|
|
192
|
-
|
|
193
767
|
const authMiddleware = async ({ headers, set }: any) => {
|
|
194
|
-
|
|
195
|
-
if (!key || key !== "piggy-secret") {
|
|
768
|
+
if (headers["x-api-key"] !== "secret") {
|
|
196
769
|
set.status = 401;
|
|
197
|
-
throw new Error("Unauthorized
|
|
770
|
+
throw new Error("Unauthorized");
|
|
198
771
|
}
|
|
199
772
|
};
|
|
200
773
|
|
|
201
774
|
piggy.books.api("/search", async (_params, query) => {
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
await piggy.books.navigate("https://books.toscrape.com");
|
|
205
|
-
await piggy.books.waitForSelector(".product_pod", 10000);
|
|
206
|
-
|
|
207
|
-
const books = await piggy.books.evaluate((q: string) =>
|
|
208
|
-
Array.from(document.querySelectorAll(".product_pod"))
|
|
209
|
-
.filter(el =>
|
|
210
|
-
el.querySelector("h3 a")?.getAttribute("title")?.toLowerCase().includes(q.toLowerCase())
|
|
211
|
-
)
|
|
212
|
-
.map(el => ({
|
|
213
|
-
title: el.querySelector("h3 a")?.getAttribute("title") ?? "",
|
|
214
|
-
price: el.querySelector(".price_color")?.textContent?.trim() ?? "",
|
|
215
|
-
}))
|
|
216
|
-
, query.q);
|
|
217
|
-
|
|
218
|
-
return { query: query.q, count: books.length, books };
|
|
219
|
-
}, { ttl: 120_000, before: [logMiddleware, authMiddleware] });
|
|
775
|
+
// handler
|
|
776
|
+
}, { ttl: 120_000, before: [authMiddleware] });
|
|
220
777
|
```
|
|
221
778
|
|
|
222
779
|
---
|
|
@@ -236,8 +793,8 @@ await piggy.books.capture.stop();
|
|
|
236
793
|
|
|
237
794
|
const requests = await piggy.books.capture.requests();
|
|
238
795
|
const ws = await piggy.books.capture.ws();
|
|
239
|
-
const storage = await piggy.books.capture.storage();
|
|
240
796
|
const cookies = await piggy.books.capture.cookies();
|
|
797
|
+
const storage = await piggy.books.capture.storage();
|
|
241
798
|
|
|
242
799
|
console.log(`${requests.length} requests, ${ws.length} WS frames`);
|
|
243
800
|
```
|
|
@@ -276,6 +833,8 @@ await piggy.books.type("#search", "mystery novels");
|
|
|
276
833
|
await piggy.books.scroll.by(400);
|
|
277
834
|
```
|
|
278
835
|
|
|
836
|
+
Affects `click`, `type`, `hover`, `scroll.by`, `wait` — random delays, simulated typos, self-correction.
|
|
837
|
+
|
|
279
838
|
---
|
|
280
839
|
|
|
281
840
|
### Screenshot / PDF
|
|
@@ -284,7 +843,7 @@ await piggy.books.scroll.by(400);
|
|
|
284
843
|
await piggy.books.screenshot("./out/page.png");
|
|
285
844
|
await piggy.books.pdf("./out/page.pdf");
|
|
286
845
|
|
|
287
|
-
const b64 = await piggy.books.screenshot();
|
|
846
|
+
const b64 = await piggy.books.screenshot(); // base64
|
|
288
847
|
```
|
|
289
848
|
|
|
290
849
|
---
|
|
@@ -309,6 +868,7 @@ const h1s = await piggy.diff([piggy.site1, piggy.site2]).fetchText("h1");
|
|
|
309
868
|
| Option | Type | Default |
|
|
310
869
|
|--------|------|---------|
|
|
311
870
|
| `mode` | `"tab" \| "process"` | `"tab"` |
|
|
871
|
+
| `binary` | `"headless" \| "headful"` | `"headless"` |
|
|
312
872
|
|
|
313
873
|
### `piggy.register(name, url)`
|
|
314
874
|
Registers a site. Accessible as `piggy.<name>` after registration.
|
|
@@ -316,6 +876,12 @@ Registers a site. Accessible as `piggy.<name>` after registration.
|
|
|
316
876
|
### `piggy.actHuman(enable)`
|
|
317
877
|
Toggles human-like interaction timing globally.
|
|
318
878
|
|
|
879
|
+
### `piggy.expose(name, handler, tabId?)`
|
|
880
|
+
Exposes a function globally (available to all tabs). The handler receives data from the browser and can return a value that resolves the Promise in the browser.
|
|
881
|
+
|
|
882
|
+
### `piggy.unexpose(name, tabId?)`
|
|
883
|
+
Removes a globally exposed function.
|
|
884
|
+
|
|
319
885
|
### `piggy.serve(port, opts?)`
|
|
320
886
|
Starts the Elysia HTTP server. Built-in routes: `GET /health`, `GET /cache/keys`, `DELETE /cache`.
|
|
321
887
|
|
|
@@ -325,7 +891,7 @@ Returns all registered API routes with method, path, TTL, and middleware count.
|
|
|
325
891
|
### `piggy.close(opts?)`
|
|
326
892
|
|
|
327
893
|
```ts
|
|
328
|
-
await piggy.close(); // graceful
|
|
894
|
+
await piggy.close(); // graceful
|
|
329
895
|
await piggy.close({ force: true }); // kills everything immediately
|
|
330
896
|
```
|
|
331
897
|
|
|
@@ -346,10 +912,10 @@ site.wait(ms)
|
|
|
346
912
|
```ts
|
|
347
913
|
site.click(selector, opts?)
|
|
348
914
|
site.doubleClick(selector) / site.hover(selector)
|
|
349
|
-
site.type(selector, text, opts?)
|
|
915
|
+
site.type(selector, text, opts?)
|
|
350
916
|
site.select(selector, value)
|
|
351
917
|
site.keyboard.press(key)
|
|
352
|
-
site.keyboard.combo(combo)
|
|
918
|
+
site.keyboard.combo(combo)
|
|
353
919
|
site.mouse.move(x, y)
|
|
354
920
|
site.mouse.drag(from, to)
|
|
355
921
|
site.scroll.to(selector) / site.scroll.by(px)
|
|
@@ -357,13 +923,30 @@ site.scroll.to(selector) / site.scroll.by(px)
|
|
|
357
923
|
|
|
358
924
|
#### Data
|
|
359
925
|
```ts
|
|
360
|
-
site.fetchText(selector)
|
|
361
|
-
site.fetchLinks(selector)
|
|
362
|
-
site.fetchImages(selector)
|
|
926
|
+
site.fetchText(selector)
|
|
927
|
+
site.fetchLinks(selector)
|
|
928
|
+
site.fetchImages(selector)
|
|
363
929
|
site.search.css(query) / site.search.id(query)
|
|
364
930
|
site.evaluate(js | fn, ...args)
|
|
365
931
|
```
|
|
366
932
|
|
|
933
|
+
#### 🔥 Expose Function (RPC)
|
|
934
|
+
```ts
|
|
935
|
+
// Expose a function that browser JS can call
|
|
936
|
+
site.exposeFunction(name, handler)
|
|
937
|
+
// handler: (data: any) => Promise<any> | any
|
|
938
|
+
|
|
939
|
+
// Remove an exposed function
|
|
940
|
+
site.unexposeFunction(name)
|
|
941
|
+
|
|
942
|
+
// Remove all exposed functions for this site
|
|
943
|
+
site.clearExposedFunctions()
|
|
944
|
+
|
|
945
|
+
// Expose and inject in one call
|
|
946
|
+
site.exposeAndInject(name, handler, injectionJs)
|
|
947
|
+
// injectionJs: string | ((fnName: string) => string)
|
|
948
|
+
```
|
|
949
|
+
|
|
367
950
|
#### Network
|
|
368
951
|
```ts
|
|
369
952
|
site.capture.start() / .stop() / .clear()
|
|
@@ -386,12 +969,38 @@ site.session.export() / site.session.import(data)
|
|
|
386
969
|
```ts
|
|
387
970
|
site.api(path, handler, opts?)
|
|
388
971
|
// opts: { ttl?, method?, before?: middleware[] }
|
|
389
|
-
// handler: (params, query, body) => Promise<any>
|
|
390
972
|
|
|
391
973
|
site.noclose()
|
|
392
974
|
site.screenshot(filePath?) / site.pdf(filePath?)
|
|
393
975
|
```
|
|
394
976
|
|
|
977
|
+
### Helper Functions
|
|
978
|
+
|
|
979
|
+
```ts
|
|
980
|
+
import { createExposedAPI } from "nothing-browser/register";
|
|
981
|
+
|
|
982
|
+
// Create a structured API with multiple methods
|
|
983
|
+
await createExposedAPI(site, apiName, {
|
|
984
|
+
method1: async (args) => { /* ... */ },
|
|
985
|
+
method2: async (args) => { /* ... */ },
|
|
986
|
+
});
|
|
987
|
+
|
|
988
|
+
// Browser calls: window[apiName]({ method: 'method1', args: {...} })
|
|
989
|
+
```
|
|
990
|
+
|
|
991
|
+
---
|
|
992
|
+
|
|
993
|
+
## How `exposeFunction` Works
|
|
994
|
+
|
|
995
|
+
1. **Browser injects stub**: `window.fnName` becomes a Promise-returning function
|
|
996
|
+
2. **Browser queues calls**: Arguments are pushed to `__NOTHING_QUEUE__`
|
|
997
|
+
3. **C++ picks up queue**: 250ms poll timer reads the queue
|
|
998
|
+
4. **Signal to Node.js**: Server broadcasts event to all connected clients
|
|
999
|
+
5. **Your handler runs**: TypeScript handler processes the data
|
|
1000
|
+
6. **Result returns**: Promise in browser resolves with your return value
|
|
1001
|
+
|
|
1002
|
+
The function survives page navigations (injected at `DocumentCreation`) and works with both tab and process modes.
|
|
1003
|
+
|
|
395
1004
|
---
|
|
396
1005
|
|
|
397
1006
|
## Binary download
|
|
@@ -409,4 +1018,4 @@ site.screenshot(filePath?) / site.pdf(filePath?)
|
|
|
409
1018
|
|
|
410
1019
|
## License
|
|
411
1020
|
|
|
412
|
-
MIT © [Ernest Tech House](https://github.com/BunElysiaReact/nothing-browser)
|
|
1021
|
+
MIT © [Ernest Tech House](https://github.com/BunElysiaReact/nothing-browser)
|