amv-playwright 1.82.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- amv_playwright-1.82.0.dist-info/METADATA +938 -0
- amv_playwright-1.82.0.dist-info/RECORD +25 -0
- amv_playwright-1.82.0.dist-info/WHEEL +5 -0
- amv_playwright-1.82.0.dist-info/entry_points.txt +5 -0
- amv_playwright-1.82.0.dist-info/licenses/LICENSE +21 -0
- amv_playwright-1.82.0.dist-info/top_level.txt +1 -0
- termux_playwright/__init__.py +172 -0
- termux_playwright/__main__.py +107 -0
- termux_playwright/adapter.py +65 -0
- termux_playwright/browser.py +1223 -0
- termux_playwright/cli_protocol.py +60 -0
- termux_playwright/control/__init__.py +5 -0
- termux_playwright/control/component.py +275 -0
- termux_playwright/control/status.py +20 -0
- termux_playwright/exceptions.py +34 -0
- termux_playwright/installer.py +528 -0
- termux_playwright/mobile.py +223 -0
- termux_playwright/patcher.py +216 -0
- termux_playwright/physics.py +347 -0
- termux_playwright/platform.py +336 -0
- termux_playwright/py.typed +1 -0
- termux_playwright/reaper.py +710 -0
- termux_playwright/stealth.py +331 -0
- termux_playwright/tunnel.py +278 -0
- termux_playwright/waf.py +121 -0
|
@@ -0,0 +1,938 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: amv-playwright
|
|
3
|
+
Version: 1.82.0
|
|
4
|
+
Summary: Production-grade Playwright & Chromium browser automation and stealth runtime optimizer for Android Termux (Dual-Engine Python & Node.js)
|
|
5
|
+
Home-page: https://github.com/uno-km/termux-playwright
|
|
6
|
+
Author: uno-km (AMEVA Foundation)
|
|
7
|
+
Author-email: "uno-km (AMEVA Foundation)" <hosequelbo@gmail.com>
|
|
8
|
+
License: MIT
|
|
9
|
+
Project-URL: Homepage, https://github.com/uno-km/amv-playwright
|
|
10
|
+
Project-URL: Documentation, https://uno-km.github.io/amv-playwright/
|
|
11
|
+
Project-URL: Repository, https://github.com/uno-km/amv-playwright
|
|
12
|
+
Project-URL: Bug Tracker, https://github.com/uno-km/amv-playwright/issues
|
|
13
|
+
Keywords: playwright,termux,android,chromium,automation,web-scraping
|
|
14
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
15
|
+
Classifier: Intended Audience :: Developers
|
|
16
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.8
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.9
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
22
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
23
|
+
Classifier: Operating System :: POSIX :: Linux
|
|
24
|
+
Classifier: Operating System :: Android
|
|
25
|
+
Requires-Python: >=3.8
|
|
26
|
+
Description-Content-Type: text/markdown
|
|
27
|
+
License-File: LICENSE
|
|
28
|
+
Requires-Dist: ameva-component-sdk<2.0,>=0.1.0
|
|
29
|
+
Dynamic: author
|
|
30
|
+
Dynamic: home-page
|
|
31
|
+
Dynamic: license-file
|
|
32
|
+
Dynamic: requires-python
|
|
33
|
+
|
|
34
|
+
# π± AMV-Playwright
|
|
35
|
+
|
|
36
|
+
> **λ§μ΄κ·Έλ μ΄μ
μλ΄**: μ΄ ν¨ν€μ§λ `termux-playwright`μμ `amv-playwright`λ‘ μ΄λ¦μ΄ λ°λμμ΅λλ€. κΈ°μ‘΄ `termux-playwright` λ°°ν¬λ³Έ(PyPI/npm)μ νΈνμ μν΄ λΉλΆκ° μ μ§λλ deprecated μνμ΄λ©° λ μ΄μ κ°±μ λμ§ μμ΅λλ€. `pip install amv-playwright` / `npm install amv-playwright`λ‘ μ ννμμμ€.
|
|
37
|
+
>
|
|
38
|
+
> Android **Termux νκ²½** λλ **μλλ‘μ΄λ Linux 컀λ(루ν
μμ΄ Linux νκ²½μ΄ κ΅¬λλλ λͺ¨λ λλ°μ΄μ€)** μμ λμν©λλ€ β Termux μ μ©μ΄ μλλλ€.
|
|
39
|
+
|
|
40
|
+
[](https://pypi.org/project/amv-playwright/)
|
|
41
|
+
[](https://www.npmjs.com/package/amv-playwright)
|
|
42
|
+
[](https://pepy.tech/projects/amv-playwright)
|
|
43
|
+
[](https://opencollective.com/ameva-fund)
|
|
44
|
+
[](https://github.com/sponsors/uno-km)
|
|
45
|
+
[](LICENSE)
|
|
46
|
+
[](https://www.python.org/)
|
|
47
|
+
[](https://nodejs.org/)
|
|
48
|
+
[-green.svg)](https://termux.dev/)
|
|
49
|
+
[](tests/)
|
|
50
|
+
|
|
51
|
+
> **Run genuine Chromium browser automation (Headless & full JavaScript SPA rendering) directly on Android devices inside Termux without PRoot or root privileges.**
|
|
52
|
+
> **Dual Engine Support: Native Python & Node.js / JavaScript.**
|
|
53
|
+
|
|
54
|
+
Transform any spare Android smartphone into a 24/7 autonomous web scraping and data harvesting node.
|
|
55
|
+
|
|
56
|
+
[π Korean Deep-Dive Engineering Documentation](docs/blog_post.md)
|
|
57
|
+
|
|
58
|
+
---
|
|
59
|
+
|
|
60
|
+
## What's New in v1.82.0 β Platform Detection Standardization & TermuxWakeLock Interface
|
|
61
|
+
|
|
62
|
+
- Refactored TermuxWakeLock interface for long-running headless Chromium automation.
|
|
63
|
+
- Standardized is_termux platform detection across Android Bionic environments.
|
|
64
|
+
- Removed redundant pip self-upgrade antipatterns and enforced SPDX Apache-2.0 headers.
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## β‘ Quick Start (1-Click Installation)
|
|
69
|
+
|
|
70
|
+
### π Python:
|
|
71
|
+
```bash
|
|
72
|
+
pip install amv-playwright && amv-playwright-install
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### β Node.js / JavaScript:
|
|
76
|
+
```bash
|
|
77
|
+
npm install amv-playwright && npx amv-playwright install
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
### πͺ Universal Shell Bootstrapper (Zero-Friction):
|
|
81
|
+
```bash
|
|
82
|
+
curl -sL https://raw.githubusercontent.com/uno-km/amv-playwright/main/install.sh | bash
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
> [!TIP]
|
|
86
|
+
> **π‘ Pro-Tip for Flaky Network Mirrors:**
|
|
87
|
+
> If `pkg install` ever stalls or reports HTTP mirror errors on a fresh Termux install, simply switch to an optimal mirror by running `termux-change-repo` and `pkg update -y` manually before retrying.
|
|
88
|
+
|
|
89
|
+
> **π₯ What the automated installer provisions behind the scenes:**
|
|
90
|
+
> 1. Provisions native Termux packages (`x11-repo`, `chromium`, `nodejs`, `python-greenlet`, `procps`, `termux-api`) with zero 1.2GB Clang build bloat.
|
|
91
|
+
> 2. Downloads and injects the official architecture-specific Playwright wheel as platform-agnostic `none-any.whl`.
|
|
92
|
+
> 3. Atomically applies the `coreBundle.js` platform verification bypass patch.
|
|
93
|
+
> 4. Runs a comprehensive 7-phase `amv-playwright-doctor` diagnostic health check.
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## ποΈ 1. Complete Dependency Architecture Matrix (`pkg` vs `pip` vs `Patch`)
|
|
98
|
+
|
|
99
|
+
| Layer | Package / Component | Provider | Type | Prerequisite | Key Responsibility |
|
|
100
|
+
| :--- | :--- | :---: | :---: | :--- | :--- |
|
|
101
|
+
| **0. Language Runtime** | `python` (3.8+) | **`pkg`** | C Binary | Termux Base | Python script and crawler execution engine |
|
|
102
|
+
| **1. Native Browser** | `chromium` | **`pkg`** | C++ Binary | Termux X11/GUI | Real native Chromium browser controlled via CDP |
|
|
103
|
+
| **2. Driver RPC Server** | `nodejs` | **`pkg`** | C++ Binary | Android Bionic | Node.js RPC bridge connecting Python and Chromium |
|
|
104
|
+
| **3. Async C-Extension** | `python-greenlet` | **`pkg`** | C Binary | `python` | Precompiled async coroutine loop (avoids 1.2GB Clang compile) |
|
|
105
|
+
| **4. Power Management** | `termux-api` | **`pkg`** | C Binary | Android API | Prevents CPU sleep when screen is off (`termux-wake-lock`) |
|
|
106
|
+
| **5. Pure Python (A)** | `typing-extensions` | **`pip`** | Pure Python | `python` | Backported type hinting compatibility across Python versions |
|
|
107
|
+
| **6. Pure Python (B)** | `pyee` | **`pip`** | Pure Python | `python` | High-performance event emitter for browser events |
|
|
108
|
+
| **7. Runtime Optimizer** | `amv-playwright` | **`pip`** | Pure Python | `pyee`, `typing-ext` | Android runtime tuning, installer, and session zombie reaper |
|
|
109
|
+
| **8. Upstream Core Wheel**| `playwright` (aarch64/x86_64) | **`pip (bypass)`** | Wheel Packaging | `python-greenlet` | Official PyPI wheel injected via `none-any` platform bypass |
|
|
110
|
+
| **9. Core JS Engine Patch** | `coreBundle.js` Patch | **Internal** | JS Byte Injection | `playwright` | Spoofs `process.platform = 'linux'` in driver RPC bundle |
|
|
111
|
+
|
|
112
|
+
---
|
|
113
|
+
|
|
114
|
+
## π 2. Installation Lifecycle Flowchart
|
|
115
|
+
|
|
116
|
+
```mermaid
|
|
117
|
+
flowchart TD
|
|
118
|
+
subgraph S0["[Phase 0] Environment Baseline"]
|
|
119
|
+
A["pkg update && pkg upgrade"]
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
subgraph S1["[Phase 1] Native OS Binaries (pkg)"]
|
|
123
|
+
B1["pkg install -y chromium"]
|
|
124
|
+
B2["pkg install -y nodejs"]
|
|
125
|
+
B3["pkg install -y python python-greenlet"]
|
|
126
|
+
B4["pkg install -y termux-api"]
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
subgraph S2["[Phase 2] Lightweight Python Dependencies (pip)"]
|
|
130
|
+
C1["pip install pyee typing-extensions"]
|
|
131
|
+
C2["pip install amv-playwright<br/>(Instant install without C compiler)"]
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
subgraph S3["[Phase 3] Playwright Wheel Bypass Injection (Installer)"]
|
|
135
|
+
D1["amv-playwright-install"]
|
|
136
|
+
D2["Fetch verified architecture wheel from PyPI"]
|
|
137
|
+
D3["Rename to none-any.whl & pip inject"]
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
subgraph S4["[Phase 4] Core JS Platform Bypass Patch (Patcher)"]
|
|
141
|
+
E1["Locate coreBundle.js"]
|
|
142
|
+
E2["Inject process.platform = 'linux'"]
|
|
143
|
+
end
|
|
144
|
+
|
|
145
|
+
subgraph S5["[Phase 5] Diagnostic Health Verification (Doctor)"]
|
|
146
|
+
F["amv-playwright-doctor (7/7 Checks Passed)"]
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
A --> B1 & B2 & B3 & B4
|
|
150
|
+
B1 & B2 & B3 --> C1 --> C2
|
|
151
|
+
C2 --> D1 --> D2 --> D3
|
|
152
|
+
D3 --> E1 --> E2
|
|
153
|
+
E2 --> F
|
|
154
|
+
|
|
155
|
+
classDef pkgNode fill:#2E7D32,stroke:#1B5E20,color:#fff,font-weight:bold;
|
|
156
|
+
classDef pipNode fill:#1565C0,stroke:#0D47A1,color:#fff,font-weight:bold;
|
|
157
|
+
classDef patchNode fill:#E65100,stroke:#BF360C,color:#fff,font-weight:bold;
|
|
158
|
+
classDef verifyNode fill:#6A1B9A,stroke:#4A148C,color:#fff,font-weight:bold;
|
|
159
|
+
|
|
160
|
+
class B1,B2,B3,B4 pkgNode;
|
|
161
|
+
class C1,C2,D3 pipNode;
|
|
162
|
+
class D1,D2,E1,E2 patchNode;
|
|
163
|
+
class F verifyNode;
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
---
|
|
167
|
+
|
|
168
|
+
## β‘ 3. Fail-Safe 5-Step Manual Installation Guide
|
|
169
|
+
|
|
170
|
+
### π’ Step 1: Native System Package Provisioning (`pkg`)
|
|
171
|
+
Install pre-compiled native binaries to avoid triggering heavy in-place compilation:
|
|
172
|
+
```bash
|
|
173
|
+
pkg update -y
|
|
174
|
+
pkg install -y x11-repo chromium nodejs python python-greenlet procps termux-api
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### π΅ Step 2: Python Tooling & Pure Packages (`pip`)
|
|
178
|
+
Install pure-Python dependencies cleanly:
|
|
179
|
+
```bash
|
|
180
|
+
pip install --upgrade pip setuptools
|
|
181
|
+
pip install pyee typing-extensions amv-playwright
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
> [!NOTE]
|
|
185
|
+
> **Virtual Environment Best Practice:** If you use a virtual environment, always create it with `--system-site-packages` to allow access to the native `python-greenlet` binary:
|
|
186
|
+
> ```bash
|
|
187
|
+
> python -m venv --system-site-packages venv
|
|
188
|
+
> source venv/bin/activate
|
|
189
|
+
> ```
|
|
190
|
+
|
|
191
|
+
### π Step 3~4: Automated Wheel Bypass & Core JS Patching
|
|
192
|
+
Download the architecture wheel, apply the platform verification bypass, and patch `coreBundle.js`:
|
|
193
|
+
```bash
|
|
194
|
+
amv-playwright-install
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### π£ Step 5: Diagnostic Verification (`doctor`)
|
|
198
|
+
Verify system readiness across all 7 health indicators:
|
|
199
|
+
```bash
|
|
200
|
+
amv-playwright-doctor
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
> [!TIP]
|
|
204
|
+
> **π‘ Key Engineering Design Principles:**
|
|
205
|
+
> 1. **Greenlet Ownership Isolation:** Pre-compiled `python-greenlet` MUST be installed via `pkg` to prevent `pip` from invoking `clang` compilation failure on Android Bionic.
|
|
206
|
+
> 2. **Slim `setup.py` Metadata:** `amv-playwright` specifies pure-Python dependencies to enable instant 1-second installation on mobile devices.
|
|
207
|
+
> 3. **Deterministic Order:** `pkg` $\rightarrow$ `pip` $\rightarrow$ `installer (wheel + patch)` $\rightarrow$ `doctor` maintains a 100% fail-safe deployment.
|
|
208
|
+
|
|
209
|
+
---
|
|
210
|
+
|
|
211
|
+
## π‘ Mobile Engineering Pro-Tips (pip, npm, venv, & Virtualization)
|
|
212
|
+
|
|
213
|
+
### π 1. Python Virtual Environments (`venv`) & Pip Optimization
|
|
214
|
+
* **The `--system-site-packages` Rule:** Never run bare `python -m venv .venv` on Termux. Standard venvs isolate C-extensions, forcing pip to attempt building `greenlet` from source using 1.2GB Clang. Always pass `--system-site-packages`:
|
|
215
|
+
```bash
|
|
216
|
+
python -m venv --system-site-packages .venv
|
|
217
|
+
source .venv/bin/activate
|
|
218
|
+
```
|
|
219
|
+
* **Pip Cache Acceleration:** To conserve mobile internal storage (eMMC), disable temporary wheel caching during fast script installs:
|
|
220
|
+
```bash
|
|
221
|
+
pip install --no-cache-dir amv-playwright
|
|
222
|
+
```
|
|
223
|
+
* **Poetry / Pipenv Configuration:** If using modern dependency managers in Termux, tell them to inherit system packages:
|
|
224
|
+
```toml
|
|
225
|
+
# pyproject.toml / poetry config
|
|
226
|
+
[virtualenvs]
|
|
227
|
+
system-site-packages = true
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
### β 2. Node.js & npm Storage & Speed Optimization
|
|
233
|
+
* **Flash Wear Reduction & Speed:** In Termux, `node_modules` can create thousands of small inodes that slow down mobile storage. Speed up installs with:
|
|
234
|
+
```bash
|
|
235
|
+
npm install --no-fund --no-audit --prefer-offline
|
|
236
|
+
```
|
|
237
|
+
* **pnpm Hardlink Deduplication (Saves ~70% Storage):** Use `pnpm` to share package binaries across projects without duplicating files on mobile flash storage:
|
|
238
|
+
```bash
|
|
239
|
+
pkg install -y pnpm
|
|
240
|
+
pnpm add amv-playwright
|
|
241
|
+
```
|
|
242
|
+
* **V8 Heap Constraint on Mobile:** Default Node.js allocates up to 1.4GB heap. On 2GB~4GB RAM phones, always limit V8 heap size to prevent Android Low Memory Killer (LMK) execution:
|
|
243
|
+
```bash
|
|
244
|
+
node --max-old-space-size=256 app.js
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
### π‘οΈ 3. Native Bionic vs PRoot / Virtualization (Why Native Wins)
|
|
250
|
+
* **Zero Virtualization Overhead:** Do **NOT** install `proot-distro` (Ubuntu/Debian) just to run Playwright. PRoot intercepts every system call via `ptrace`, causing 3x~5x CPU latency, 60% higher RAM consumption, and broken `/dev/shm` shared memory.
|
|
251
|
+
* **Native Speed:** `amv-playwright` orchestrates Termux's native Android Bionic-compiled Chromium and Node.js directly, delivering full ARM64 hardware performance with zero root required.
|
|
252
|
+
|
|
253
|
+
---
|
|
254
|
+
|
|
255
|
+
### π 4. 24/7 Autonomous Background Scraping (Daemon Guide)
|
|
256
|
+
Android aggressively suspends background apps and kills child processes when the screen turns off.
|
|
257
|
+
|
|
258
|
+
#### β Node.js Production Daemon with PM2:
|
|
259
|
+
```bash
|
|
260
|
+
# Install PM2 process manager
|
|
261
|
+
npm install -g pm2
|
|
262
|
+
|
|
263
|
+
# Launch scraper with 256MB memory cap and automatic crash recovery
|
|
264
|
+
pm2 start app.js --name "mobile-scraper" --node-args="--max-old-space-size=256 --expose-gc"
|
|
265
|
+
|
|
266
|
+
# Keep alive on reboot / background
|
|
267
|
+
pm2 save
|
|
268
|
+
pm2 monit
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
#### π Python Production Daemon with Tmux & WakeLock:
|
|
272
|
+
```bash
|
|
273
|
+
# Keep CPU awake and detach terminal session
|
|
274
|
+
termux-wake-lock
|
|
275
|
+
pkg install -y tmux
|
|
276
|
+
tmux new -s scraper 'python crawler.py'
|
|
277
|
+
# Detach with: Ctrl+B, then D
|
|
278
|
+
# Re-attach anytime with: tmux attach -t scraper
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
---
|
|
282
|
+
|
|
283
|
+
### β‘ 5. Android 14+ Phantom Process Killer Bypass
|
|
284
|
+
Android 12~14 limits background child processes to 32. Heavy browsers spawn multiple renderer and utility processes.
|
|
285
|
+
* **In-App Single-Process Mode:** Enable `single_process=True` (Python) or `singleProcess: true` (Node.js) to collapse Chromium into a single lightweight process that stays permanently under the Android 32-process limit.
|
|
286
|
+
* **ADB Command (Optional Permanent Bypass):**
|
|
287
|
+
```bash
|
|
288
|
+
adb shell "/system/bin/device_config put activity_manager max_phantom_processes 2147483647"
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
---
|
|
292
|
+
|
|
293
|
+
## π Usage Examples
|
|
294
|
+
|
|
295
|
+
### π Direct Socket DNS Bypass-Tunnel Architecture (v1.81.0)
|
|
296
|
+
|
|
297
|
+
> [!IMPORTANT]
|
|
298
|
+
> **Overcoming the Android VPN DNS Blackhole (Tailscale, WireGuard, AdGuard):**
|
|
299
|
+
> On non-root Android 14~16 devices operating under active VPN tunnels (such as Tailscale or WireGuard), standard OS `getaddrinfo()` system queries pass through Android's system `dnsproxyd`. When MagicDNS or local VPN resolvers encounter public domain requests from unprivileged Termux sandbox processes, DNS resolution stalls or deadlocks on internal gateway interfaces (`100.100.100.100`), causing browser navigations (`page.goto`) to exceed timeouts and fail.
|
|
300
|
+
>
|
|
301
|
+
> `amv-playwright` v1.81.0 introduces a **Zero-Dependency RFC 1035 Direct Socket UDP Resolver** coupled with an **In-Process Loopback HTTP CONNECT Proxy** (`127.0.0.1:<port>`). By resolving public DNS records directly via low-level UDP sockets (defaulting to Google `8.8.8.8` or Cloudflare `1.1.1.1` or any custom upstream DNS) and establishing transparent TCP tunnels upstream, browser automation remains 100% immune to OS VPN deadlocks.
|
|
302
|
+
|
|
303
|
+
#### π Live Benchmark on Real Hardware (Samsung Galaxy A35 5G Β· Android 16):
|
|
304
|
+
| Configuration | Tailscale VPN State | DNS Query Mode | Navigation Result | Dynamic DOM Extraction | Total Latency |
|
|
305
|
+
| :--- | :---: | :---: | :---: | :---: | :---: |
|
|
306
|
+
| **Standard Chromium** | Active (MagicDNS) | Android `dnsproxyd` | β **TIMEOUT (8000ms exceeded)** | Failed (Exit Code 1) | 12.91s |
|
|
307
|
+
| **v1.81.0 Bypass Tunnel** | Active (MagicDNS) | Direct UDP Socket (`8.8.8.8`) | π **HTTP 200 OK** | **435 chars extracted** | **7.34s** |
|
|
308
|
+
|
|
309
|
+
#### π Python SDK: Fluent `BrowserBuilder`
|
|
310
|
+
```python
|
|
311
|
+
import asyncio
|
|
312
|
+
from termux_playwright import BrowserBuilder
|
|
313
|
+
|
|
314
|
+
async def main():
|
|
315
|
+
# Fluent builder pattern with automatic VPN bypass tunnel
|
|
316
|
+
browser = await (BrowserBuilder()
|
|
317
|
+
.headless(True)
|
|
318
|
+
.bypass_tunnel(True, dns="8.8.8.8") # Circumvent Android VPN DNS deadlock
|
|
319
|
+
.launch())
|
|
320
|
+
|
|
321
|
+
page = await browser.new_page()
|
|
322
|
+
await page.goto("https://www.naver.com", timeout=30000)
|
|
323
|
+
print("Page Title:", await page.title())
|
|
324
|
+
await browser.close()
|
|
325
|
+
|
|
326
|
+
if __name__ == "__main__":
|
|
327
|
+
asyncio.run(main())
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
#### β Node.js / JavaScript SDK: Fluent `BrowserBuilder`
|
|
331
|
+
```javascript
|
|
332
|
+
const { BrowserBuilder } = require('amv-playwright');
|
|
333
|
+
|
|
334
|
+
async function main() {
|
|
335
|
+
const browser = await new BrowserBuilder()
|
|
336
|
+
.headless(true)
|
|
337
|
+
.bypassTunnel(true, '8.8.8.8') // Direct RFC 1035 UDP resolver via in-process CONNECT proxy
|
|
338
|
+
.launch();
|
|
339
|
+
|
|
340
|
+
const page = await browser.newPage();
|
|
341
|
+
await page.goto('https://www.naver.com', { timeout: 30000 });
|
|
342
|
+
console.log('Page Title:', await page.title());
|
|
343
|
+
await browser.close();
|
|
344
|
+
}
|
|
345
|
+
|
|
346
|
+
main().catch(console.error);
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
#### π₯οΈ Multi-Platform CLI Crawl Command
|
|
350
|
+
```bash
|
|
351
|
+
# Automated headless crawl with direct socket DNS bypass tunnel
|
|
352
|
+
amv-playwright crawl https://www.naver.com --bypass-tunnel
|
|
353
|
+
|
|
354
|
+
# Specify a custom upstream DNS server (e.g. Cloudflare DNS)
|
|
355
|
+
amv-playwright crawl https://news.ycombinator.com --bypass-tunnel --dns 1.1.1.1
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
---
|
|
359
|
+
|
|
360
|
+
### Python Asynchronous API (`examples/basic_crawler.py`)
|
|
361
|
+
```python
|
|
362
|
+
import asyncio
|
|
363
|
+
from termux_playwright import async_playwright_termux, launch
|
|
364
|
+
|
|
365
|
+
async def main():
|
|
366
|
+
# async_playwright_termux configures memory caps and ensures child process cleanup
|
|
367
|
+
async with async_playwright_termux() as p:
|
|
368
|
+
# Automatically detects Termux binaries, injects eMMC zero-wear flags and --no-sandbox
|
|
369
|
+
browser = await launch(p, headless=True)
|
|
370
|
+
page = await browser.new_page()
|
|
371
|
+
|
|
372
|
+
await page.goto("https://news.ycombinator.com", timeout=60000)
|
|
373
|
+
print(f"Page Title: {await page.title()}")
|
|
374
|
+
|
|
375
|
+
await browser.close()
|
|
376
|
+
|
|
377
|
+
if __name__ == "__main__":
|
|
378
|
+
asyncio.run(main())
|
|
379
|
+
```
|
|
380
|
+
|
|
381
|
+
### β Node.js / JavaScript API (`examples/crawler.js`)
|
|
382
|
+
```javascript
|
|
383
|
+
const { launch, setupStealthContext, blockHeavyResources } = require('amv-playwright');
|
|
384
|
+
|
|
385
|
+
async function main() {
|
|
386
|
+
// Automatically provisions session ledger, eMMC RAM cache, and WakeLock
|
|
387
|
+
const browser = await launch({
|
|
388
|
+
headless: true,
|
|
389
|
+
stealth: true,
|
|
390
|
+
lowMemoryMode: true,
|
|
391
|
+
wakeLock: true
|
|
392
|
+
});
|
|
393
|
+
|
|
394
|
+
try {
|
|
395
|
+
const context = await setupStealthContext(browser, {
|
|
396
|
+
locale: 'en-US',
|
|
397
|
+
timezoneId: 'America/New_York'
|
|
398
|
+
});
|
|
399
|
+
const page = await context.newPage();
|
|
400
|
+
|
|
401
|
+
// Abort images and media to save mobile data & CPU
|
|
402
|
+
await blockHeavyResources(page, { images: true, media: true, fonts: true });
|
|
403
|
+
|
|
404
|
+
await page.goto('https://news.ycombinator.com', { timeout: 45000, waitUntil: 'domcontentloaded' });
|
|
405
|
+
console.log('Page Title:', await page.title());
|
|
406
|
+
} finally {
|
|
407
|
+
await browser.close();
|
|
408
|
+
}
|
|
409
|
+
}
|
|
410
|
+
|
|
411
|
+
main().catch(console.error);
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
### π 24/7 Unattended Crawling with WakeLock & Context Recycling (`examples/advanced_crawler.py`)
|
|
415
|
+
```python
|
|
416
|
+
import asyncio
|
|
417
|
+
from termux_playwright import async_playwright_termux, launch, TermuxWakeLock
|
|
418
|
+
|
|
419
|
+
async def run_247_crawler():
|
|
420
|
+
# Acquire Termux WakeLock to prevent Android CPU sleep when phone screen is off
|
|
421
|
+
with TermuxWakeLock(fail_silently=True):
|
|
422
|
+
async with async_playwright_termux() as p:
|
|
423
|
+
browser = await launch(
|
|
424
|
+
p,
|
|
425
|
+
headless=True,
|
|
426
|
+
low_memory_mode=False, # Set True for <= 2GB RAM devices
|
|
427
|
+
jitless=True, # Adhere to Android 10+ W^X SELinux policy
|
|
428
|
+
)
|
|
429
|
+
|
|
430
|
+
# Best Practice: Periodically recycle contexts to clear Node.js RPC buffers
|
|
431
|
+
context = await browser.new_context()
|
|
432
|
+
page = await context.new_page()
|
|
433
|
+
|
|
434
|
+
await page.goto("https://github.com", timeout=45000)
|
|
435
|
+
print("Fetched:", await page.title())
|
|
436
|
+
|
|
437
|
+
await context.close()
|
|
438
|
+
await browser.close()
|
|
439
|
+
|
|
440
|
+
if __name__ == "__main__":
|
|
441
|
+
asyncio.run(run_247_crawler())
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
### ποΈ Customizing Chromium Arguments (`args=[...]`)
|
|
445
|
+
You can pass custom browser arguments directly to `launch()`. Key-value options (e.g. `--window-size`, `--disk-cache-dir`) automatically override default parameters cleanly:
|
|
446
|
+
|
|
447
|
+
```python
|
|
448
|
+
browser = await launch(
|
|
449
|
+
p,
|
|
450
|
+
headless=True,
|
|
451
|
+
args=[
|
|
452
|
+
"--window-size=1920,1080", # Custom viewport resolution
|
|
453
|
+
"--disk-cache-dir=/tmp/my_browser_cache", # Custom cache directory
|
|
454
|
+
"--media-cache-size=20", # Media cache size in MB
|
|
455
|
+
"--user-agent=MyCustomBot/1.0", # Custom HTTP User-Agent
|
|
456
|
+
]
|
|
457
|
+
)
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
### π₯· Stealth Mode & Anti-Bot Evasion (`setup_stealth_context`)
|
|
461
|
+
Bypass Cloudflare, DataDome, CreepJS, and advanced fingerprinting engines with prototype-safe navigator masking, Sub-pixel Canvas 2D LSB noise injection, AudioContext micro-frequency deviation, and WebGL driver spoofing:
|
|
462
|
+
|
|
463
|
+
```python
|
|
464
|
+
import asyncio
|
|
465
|
+
from termux_playwright import (
|
|
466
|
+
async_playwright_termux,
|
|
467
|
+
launch,
|
|
468
|
+
setup_stealth_context,
|
|
469
|
+
HumanMouse,
|
|
470
|
+
HumanKeyboard,
|
|
471
|
+
CellularIpRotator,
|
|
472
|
+
TurnstileEvaluator,
|
|
473
|
+
)
|
|
474
|
+
|
|
475
|
+
async def main():
|
|
476
|
+
async with async_playwright_termux() as p:
|
|
477
|
+
browser = await launch(p, headless=True, stealth=True, single_process=True)
|
|
478
|
+
|
|
479
|
+
# Configure stealth context with granular noise and fingerprint toggles
|
|
480
|
+
context = await setup_stealth_context(
|
|
481
|
+
browser,
|
|
482
|
+
user_agent="Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Mobile Safari/537.36",
|
|
483
|
+
enable_canvas_noise=True, # 1-bit LSB micro-noise on 10x10 top-left pixels
|
|
484
|
+
enable_audio_noise=True, # Micro frequency variance in AudioBuffer
|
|
485
|
+
enable_webgl_mask=True, # UNMASKED_VENDOR/RENDERER spoofing
|
|
486
|
+
enable_webdriver_mask=True, # Prototype-safe navigator.webdriver deletion
|
|
487
|
+
extra_headers={"Accept-Language": "en-US,en;q=0.9"},
|
|
488
|
+
)
|
|
489
|
+
|
|
490
|
+
page = await context.new_page()
|
|
491
|
+
await page.goto("https://bot.sannysoft.com", timeout=60000)
|
|
492
|
+
print("Page Title:", await page.title())
|
|
493
|
+
|
|
494
|
+
# Human interaction: Non-linear BΓ©zier trajectory & Gaussian typing
|
|
495
|
+
mouse = HumanMouse(page)
|
|
496
|
+
await mouse.click((200, 300), steps=30, jitter=True, overshoot=True)
|
|
497
|
+
await HumanKeyboard.type_text(page, "Automated data verification", selector="input[type='text']")
|
|
498
|
+
|
|
499
|
+
# Solve Cloudflare Turnstile if present
|
|
500
|
+
if await TurnstileEvaluator.detect_challenge(page):
|
|
501
|
+
await TurnstileEvaluator.solve_turnstile(page, human_mouse=mouse)
|
|
502
|
+
|
|
503
|
+
await browser.close()
|
|
504
|
+
|
|
505
|
+
if __name__ == "__main__":
|
|
506
|
+
asyncio.run(main())
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
#### ποΈ Next-Gen Feature Toggle & Parameter Matrix
|
|
510
|
+
|
|
511
|
+
| Module | Class / Function | Config Parameter | Type / Default | Description |
|
|
512
|
+
| :--- | :--- | :--- | :---: | :--- |
|
|
513
|
+
| **`stealth`** | `setup_stealth_context` | `enable_canvas_noise` | `bool = True` | Injects 1-bit LSB noise into Canvas 2D to randomize canvas fingerprint hashes without visual distortion. |
|
|
514
|
+
| **`stealth`** | `setup_stealth_context` | `enable_audio_noise` | `bool = True` | Injects \((Math.random() - 0.5) \cdot 10^{-7}\) noise into `AudioBuffer` and `AnalyserNode`. |
|
|
515
|
+
| **`stealth`** | `setup_stealth_context` | `enable_webgl_mask` | `bool = True` | Spoofs WebGL unmasked vendor and renderer to standard ANGLE/Intel. |
|
|
516
|
+
| **`stealth`** | `setup_stealth_context` | `canvas_noise_seed` | `int? = None` | Optional seed for deterministic Canvas noise testing. |
|
|
517
|
+
| **`physics`** | `HumanMouse.click` | `jitter`, `overshoot` | `bool = True` | Simulates human hand muscle tremor (sub-pixel jitter) and target overshooting/correction. |
|
|
518
|
+
| **`physics`** | `HumanKeyboard.type_text` | `mean_delay`, `std_dev`| `0.12s, 0.035s`| Generates Gaussian-distributed typing intervals between keystrokes. |
|
|
519
|
+
| **`mobile`** | `CellularIpRotator` | `mode` | `'auto'` | Selects `termux_native` (device-internal) or `pc_adb_bridge` (PC USB/Wi-Fi ADB tethering). |
|
|
520
|
+
| **`mobile`** | `CellularIpRotator` | `verify_ip_change` | `bool = True` | Polls multiple public IP endpoints to guarantee a fresh residential IP after toggle. |
|
|
521
|
+
| **`waf`** | `TurnstileEvaluator` | `solve_turnstile` | `timeout=12.0s`| Detects Turnstile iframe and clicks verification checkbox with natural BΓ©zier mouse physics. |
|
|
522
|
+
|
|
523
|
+
---
|
|
524
|
+
|
|
525
|
+
### π± Dual-Mode Cellular IP Rotator (`termux_playwright.mobile`)
|
|
526
|
+
Rotate your mobile carrier (LTE/5G) residential IP within 2~3 seconds via Android airplane mode toggling:
|
|
527
|
+
|
|
528
|
+
```python
|
|
529
|
+
import asyncio
|
|
530
|
+
from termux_playwright import CellularIpRotator, RotationMode
|
|
531
|
+
|
|
532
|
+
async def rotate_ip_demo():
|
|
533
|
+
# Auto-detects whether running inside Termux or on PC via ADB bridge
|
|
534
|
+
rotator = CellularIpRotator(mode=RotationMode.AUTO)
|
|
535
|
+
|
|
536
|
+
current_ip = await rotator.get_public_ip()
|
|
537
|
+
print(f"Current Public IP: {current_ip}")
|
|
538
|
+
|
|
539
|
+
result = await rotator.rotate_ip(verify_ip_change=True)
|
|
540
|
+
print(f"Rotation Result: Success={result['success']}, New IP={result['new_ip']}, Time={result['elapsed_seconds']}s")
|
|
541
|
+
|
|
542
|
+
asyncio.run(rotate_ip_demo())
|
|
543
|
+
```
|
|
544
|
+
|
|
545
|
+
---
|
|
546
|
+
|
|
547
|
+
### β Node.js / TypeScript Next-Gen API:
|
|
548
|
+
```javascript
|
|
549
|
+
const {
|
|
550
|
+
launch,
|
|
551
|
+
setupStealthContext,
|
|
552
|
+
HumanMouse,
|
|
553
|
+
HumanKeyboard,
|
|
554
|
+
CellularIpRotator,
|
|
555
|
+
TurnstileEvaluator
|
|
556
|
+
} = require('amv-playwright');
|
|
557
|
+
|
|
558
|
+
async function main() {
|
|
559
|
+
const browser = await launch(null, { headless: true, stealth: true });
|
|
560
|
+
const context = await setupStealthContext(browser, {
|
|
561
|
+
enableCanvasNoise: true,
|
|
562
|
+
enableAudioNoise: true,
|
|
563
|
+
enableWebglMask: true
|
|
564
|
+
});
|
|
565
|
+
const page = await context.newPage();
|
|
566
|
+
await page.goto('https://bot.sannysoft.com');
|
|
567
|
+
|
|
568
|
+
const mouse = new HumanMouse(page);
|
|
569
|
+
await mouse.click([200, 300], { steps: 25, jitter: true });
|
|
570
|
+
await HumanKeyboard.typeText(page, 'Search query', { selector: 'input[name="q"]' });
|
|
571
|
+
|
|
572
|
+
await browser.close();
|
|
573
|
+
}
|
|
574
|
+
main();
|
|
575
|
+
```
|
|
576
|
+
|
|
577
|
+
---
|
|
578
|
+
|
|
579
|
+
## π° Standalone Fortress Mode vs. Cooperative Multi-Tasking Mode
|
|
580
|
+
|
|
581
|
+
`amv-playwright` provides two distinct execution profiles designed for different concurrency and isolation requirements:
|
|
582
|
+
|
|
583
|
+
```python
|
|
584
|
+
# π€ 1. Default Mode: Cooperative Multi-Tasking
|
|
585
|
+
# Non-blocking async event loop delegation; ideal for concurrent crawlers, bots, and background daemons.
|
|
586
|
+
browser = await launch(p, headless=True)
|
|
587
|
+
|
|
588
|
+
# π° 2. Standalone Fortress Mode
|
|
589
|
+
# 100% clean-room ephemeral profile, anti-throttling flags, max CPU priority, auto-wakelock, auto-purged on exit.
|
|
590
|
+
browser = await launch(p, headless=True, standalone_mode=True, wake_lock=True)
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
### βοΈ Execution Modes & Trade-Off Matrix
|
|
594
|
+
|
|
595
|
+
| Feature / Dimension | π€ Cooperative Multi-Tasking (Default) | π° Standalone Fortress (`standalone_mode=True`) |
|
|
596
|
+
| :--- | :--- | :--- |
|
|
597
|
+
| **Philosophy & Intent** | Cooperative multitasking alongside bots & daemons | Exclusive solo stage with 100% zero interference |
|
|
598
|
+
| **Profile Isolation** | Standard shared profile directory | **100% Isolated Ephemeral Profile (`/tmp/tp_solo_UUID`)** created on launch & completely purged on exit |
|
|
599
|
+
| **Event Loop Cleanups** | Non-blocking async worker thread (`asyncio.to_thread`) | Non-blocking async worker thread + Instant profile wipe |
|
|
600
|
+
| **CPU & Timer Priority** | Standard OS/Chromium power-saving scheduling | **Anti-Throttling Enabled** (`--disable-background-timer-throttling`, `--disable-renderer-backgrounding`) |
|
|
601
|
+
| **WakeLock Integration** | Manual `with TermuxWakeLock():` context | **Seamlessly coupled to browser lifecycle via `wake_lock=True`** |
|
|
602
|
+
| **Disk/Storage Impact** | Zero additional disk churn | Ephemeral profile in `/tmp` (Wiped 100% on close) |
|
|
603
|
+
| **Best Used For** | 24/7 background scrapers, parallel tabs, Telegram bots | **High-priority solo crawling, banking/auth sessions, benchmarks** |
|
|
604
|
+
|
|
605
|
+
---
|
|
606
|
+
|
|
607
|
+
## π©Ή Runtime Self-Healing Engine
|
|
608
|
+
|
|
609
|
+
If Playwright is updated in the future (e.g. `pip install --upgrade playwright`), the upstream package overwrites `coreBundle.js` with its unpatched version.
|
|
610
|
+
|
|
611
|
+
`amv-playwright` detects this automatically in 0.001s upon `launch()` / `launch_sync()` and **auto-applies the platform patch on the fly**, guaranteeing 100% zero-friction operation without throwing cryptic `Unsupported platform: android` errors.
|
|
612
|
+
|
|
613
|
+
---
|
|
614
|
+
|
|
615
|
+
## π Repository Structure
|
|
616
|
+
|
|
617
|
+
```
|
|
618
|
+
amv-playwright/
|
|
619
|
+
βββ docs/ # Technical documentation & guides
|
|
620
|
+
β βββ blog_post.md # Complete Korean engineering writeup
|
|
621
|
+
β βββ PHANTOM_PROCESS_KILLER_GUIDE.md # Step-by-step Phantom Killer ADB guide
|
|
622
|
+
βββ examples/ # Ready-to-run crawling demos
|
|
623
|
+
β βββ basic_crawler.py # Basic asynchronous scraping demo
|
|
624
|
+
β βββ advanced_crawler.py # 24/7 unattended crawler with WakeLock
|
|
625
|
+
βββ lib/ # Node.js / JavaScript Dual Engine
|
|
626
|
+
β βββ index.js # Node.js public exports
|
|
627
|
+
β βββ index.d.ts # Full TypeScript definitions
|
|
628
|
+
β βββ browser.js # Node Chromium launcher
|
|
629
|
+
β βββ stealth.js # Canvas 2D LSB & Audio noise engine
|
|
630
|
+
β βββ physics.js # Cubic BΓ©zier & Gaussian typing
|
|
631
|
+
β βββ mobile.js # Dual-Mode Cellular IP rotator
|
|
632
|
+
β βββ waf.js # Cloudflare Turnstile auto-solver
|
|
633
|
+
β βββ reaper.js # Node process reaper & WakeLock
|
|
634
|
+
β βββ platform.js # Node platform & storage checks
|
|
635
|
+
βββ termux_playwright/ # Python Dual Engine package
|
|
636
|
+
β βββ __init__.py
|
|
637
|
+
β βββ browser.py # Android-hardened browser launcher & V8 args
|
|
638
|
+
β βββ stealth.py # Canvas 2D LSB & Audio noise engine
|
|
639
|
+
β βββ physics.py # Cubic BΓ©zier & Gaussian typing
|
|
640
|
+
β βββ mobile.py # Dual-Mode Cellular IP rotator
|
|
641
|
+
β βββ waf.py # Cloudflare Turnstile auto-solver
|
|
642
|
+
β βββ exceptions.py # Typed exception hierarchy
|
|
643
|
+
β βββ installer.py # PyPI wheel bypass and dependency engine
|
|
644
|
+
β βββ patcher.py # Atomic JS coreBundle platform patcher
|
|
645
|
+
β βββ platform.py # Architecture and storage inspection
|
|
646
|
+
β βββ reaper.py # Session-scoped process reaper & WakeLock
|
|
647
|
+
βββ tests/ # Python pytest test suite (100 tests)
|
|
648
|
+
β βββ test_browser.py
|
|
649
|
+
β βββ test_installer.py
|
|
650
|
+
β βββ test_nextgen_mobile.py
|
|
651
|
+
β βββ test_nextgen_physics.py
|
|
652
|
+
β βββ test_nextgen_stealth.py
|
|
653
|
+
β βββ test_nextgen_waf.py
|
|
654
|
+
β βββ test_patcher.py
|
|
655
|
+
β βββ test_platform.py
|
|
656
|
+
β βββ test_reaper.py
|
|
657
|
+
βββ tests_js/ # Node.js test suite (26 tests)
|
|
658
|
+
β βββ test_core.test.js
|
|
659
|
+
β βββ test_nextgen_mobile.test.js
|
|
660
|
+
β βββ test_nextgen_physics.test.js
|
|
661
|
+
β βββ test_nextgen_stealth.test.js
|
|
662
|
+
β βββ test_nextgen_waf.test.js
|
|
663
|
+
βββ CHANGELOG.md # Version release history
|
|
664
|
+
βββ LICENSE # MIT License
|
|
665
|
+
βββ package.json # npm package definition
|
|
666
|
+
βββ pyproject.toml # Build configuration
|
|
667
|
+
βββ README.md # Project documentation
|
|
668
|
+
βββ setup.py # Setuptools distribution definition
|
|
669
|
+
```
|
|
670
|
+
|
|
671
|
+
---
|
|
672
|
+
|
|
673
|
+
## π‘οΈ Reliability & Security Architecture
|
|
674
|
+
|
|
675
|
+
1. **Session-Scoped Process Reaper**: Injects `--termux-session-id={uuid}` to deterministically reap orphaned Chromium processes without collateral damage to other browser instances.
|
|
676
|
+
2. **Flash Memory (eMMC) Protection**: Injects `--disk-cache-dir=/dev/null` and `--disable-application-cache` to eliminate flash wear during intensive 24/7 crawling.
|
|
677
|
+
3. **Android 10+ W^X Policy Compliance**: Automatically injects `--js-flags=--jitless` on Android 10+ (SDK $\ge 29$) to adhere to SELinux executable memory policies.
|
|
678
|
+
4. **Thread-Safe Concurrency**: All process tracking collections are guarded by `threading.RLock()` with snapshot-and-clear concurrency.
|
|
679
|
+
|
|
680
|
+
---
|
|
681
|
+
|
|
682
|
+
## βοΈ Resource Limits, Memory Tuning & Troubleshooting
|
|
683
|
+
|
|
684
|
+
Smartphone hardware differs significantly from servers: low-power CPUs, constrained RAM (1GB~4GB), and aggressive OS Doze/LMK (Low Memory Killer) daemons. Here is how to tune resources and prevent crashes:
|
|
685
|
+
|
|
686
|
+
### 1. πΎ Storage Exhaustion (`StorageExhaustionError`, Baseline: 150MB~300MB)
|
|
687
|
+
* **Root Cause:** Chromium creates temporary browser profiles under `/data/data/com.termux/files/usr/tmp`. Loading modern Single-Page Applications (SPAs) generates IndexedDB databases, font caches, and DOM snapshots. When free space is exhausted, the Android kernel locks I/O with `ENOSPC`, crashing Chromium.
|
|
688
|
+
* **Resolution & Tuning:**
|
|
689
|
+
```bash
|
|
690
|
+
# 1. Clean package and temp caches (Recommended)
|
|
691
|
+
pkg clean && rm -rf $TMPDIR/*
|
|
692
|
+
|
|
693
|
+
# 2. Adjust threshold via environment variable (Default: Browser 150MB, Installer 300MB)
|
|
694
|
+
export TERMUX_PLAYWRIGHT_MIN_STORAGE_MB=100
|
|
695
|
+
```
|
|
696
|
+
|
|
697
|
+
---
|
|
698
|
+
|
|
699
|
+
### 2. β‘ V8 JavaScript Heap OOM (`Page crashed!`, Default: 256MB / Low-Memory: 128MB)
|
|
700
|
+
* **Root Cause:** Visiting complex SPA sites with `low_memory_mode=True` (128MB cap) can trigger `FatalProcessOutOfMemory` when DOM trees or JS bundles exceed the heap ceiling, causing SIGABRT renderer termination.
|
|
701
|
+
* **Resolution & Tuning:**
|
|
702
|
+
* **Low-end devices ($\le$ 2GB RAM):** Keep `low_memory_mode=True` and block unnecessary assets (images, fonts).
|
|
703
|
+
* **Standard devices ($\ge$ 3GB RAM):** Keep default `low_memory_mode=False` (allocates 256MB V8 heap).
|
|
704
|
+
* **Customize V8 Heap via Environment Variable:**
|
|
705
|
+
```bash
|
|
706
|
+
export TERMUX_PLAYWRIGHT_V8_MEMORY_MB=512
|
|
707
|
+
```
|
|
708
|
+
|
|
709
|
+
---
|
|
710
|
+
|
|
711
|
+
### 3. π₯οΈ Node.js RPC Buffer Accumulation (`Connection closed`, Default: 512MB)
|
|
712
|
+
* **Root Cause:** Running an uninterrupted browser instance for days across thousands of pages causes Chrome DevTools Protocol (CDP) message queues and event listeners to accumulate in Node.js heap.
|
|
713
|
+
* **Best Practice (Cyclic Context Recycling):**
|
|
714
|
+
```python
|
|
715
|
+
# Recycle context every 100~200 pages to completely purge Node.js RPC buffers
|
|
716
|
+
for batch in chunked(urls, 100):
|
|
717
|
+
context = await browser.new_context()
|
|
718
|
+
page = await context.new_page()
|
|
719
|
+
for url in batch:
|
|
720
|
+
await page.goto(url)
|
|
721
|
+
await context.close() # Flushes all RPC buffers and temporary heap
|
|
722
|
+
```
|
|
723
|
+
* Expand Node.js Memory Cap:
|
|
724
|
+
```bash
|
|
725
|
+
export TERMUX_PLAYWRIGHT_NODE_MEMORY_MB=768
|
|
726
|
+
```
|
|
727
|
+
|
|
728
|
+
---
|
|
729
|
+
|
|
730
|
+
### 4. β‘ JavaScript JIT Execution vs Android 10+ W^X Security Policy (`--jitless`)
|
|
731
|
+
|
|
732
|
+
Chromium's V8 JavaScript engine has two execution tiers:
|
|
733
|
+
1. **Ignition (Bytecode Interpreter):** Interprets JS bytecode sequentially without JIT compilation. Safe, low memory, but slower.
|
|
734
|
+
2. **TurboFan & Maglev (JIT Compiler):** Dynamically compiles JavaScript directly into ARM64 native machine code in RAM for 5x~20x faster execution.
|
|
735
|
+
|
|
736
|
+
#### π‘οΈ Play Store Chrome vs Termux Chromium (`mmap(RWX)` & W^X Policy):
|
|
737
|
+
* **Official Google Chrome App:** A signed APK with OS entitlements utilizing system WebView/V8 memory channels.
|
|
738
|
+
* **Termux Chromium:** Runs as an unprivileged user-space Linux process inside the Termux sandbox. When Chromium's V8 JIT compiler attempts `mmap(..., PROT_READ | PROT_WRITE | PROT_EXEC)` to allocate dynamic executable machine code in RAM, Android 10+'s SELinux kernel blocks it as a security violation and **instantly terminates Chromium with `SIGSEGV` / `SELinux violation` (Exit 139)**.
|
|
739
|
+
* Therefore, on standard non-root Android 10+ devices, **`--jitless` is an essential survival shield** that forces Chromium to run on the Ignition interpreter, preventing instant crashes.
|
|
740
|
+
|
|
741
|
+
#### βοΈ The Fundamental Trade-off Matrix:
|
|
742
|
+
| Execution Mode | How to Enable in Code | JavaScript Speed | Stability on Android 10+ | Recommendation |
|
|
743
|
+
| :--- | :--- | :--- | :--- | :--- |
|
|
744
|
+
| **Interpreter (`--jitless`)** | `launch(p)` or `jitless=True` | Standard (Slower for heavy JS) | π **100% Rock-Solid (Zero Crashes)** | **Default & Recommended for 24/7 Scraping** |
|
|
745
|
+
| **Full V8 JIT (TurboFan)** | `launch(p, jitless=False)` | β‘ 5x~20x Faster | π₯ Instant Crash on unrooted Android 10+ | Android 9 or Rooted devices ONLY |
|
|
746
|
+
|
|
747
|
+
#### π± Automatic Version Adaptation:
|
|
748
|
+
* **Android 9 or Older (e.g. Android 8.0/8.1 Oreo):** Does NOT have the W^X restriction. Our launcher **automatically leaves JIT enabled** for full-speed execution.
|
|
749
|
+
* **Android 10+ (API $\ge 29$):** Automatically injects `--js-flags=--jitless`.
|
|
750
|
+
* **Explicit Parameter Control:**
|
|
751
|
+
```python
|
|
752
|
+
# Auto-detected by default (jitless=None)
|
|
753
|
+
browser = await launch(p, jitless=True) # Force interpreter mode (Rock-solid stability)
|
|
754
|
+
browser = await launch(p, jitless=False) # Force full JIT (Requires Android 9 or rooted device)
|
|
755
|
+
```
|
|
756
|
+
|
|
757
|
+
#### π How to Speed Up Heavy SPA Crawling under `--jitless`:
|
|
758
|
+
Because complex Single-Page Applications (SPAs like Naver, YouTube, Twitter) execute megabytes of JS, running without JIT on low-power mobile CPUs can take 20~40 seconds to complete full rendering. Use our built-in 1-line accelerator and best practices:
|
|
759
|
+
|
|
760
|
+
```python
|
|
761
|
+
from termux_playwright import async_playwright_termux, launch, block_heavy_resources
|
|
762
|
+
|
|
763
|
+
async with async_playwright_termux() as p:
|
|
764
|
+
browser = await launch(p, headless=True)
|
|
765
|
+
page = await browser.new_page()
|
|
766
|
+
|
|
767
|
+
# β‘ 1-Line Built-in Accelerator: Block heavy images/fonts/media (3x~5x speed boost)
|
|
768
|
+
await block_heavy_resources(page)
|
|
769
|
+
|
|
770
|
+
# π Best Practice: Extract data immediately once DOM is ready (60s timeout)
|
|
771
|
+
await page.goto("https://www.naver.com", timeout=60000, wait_until="domcontentloaded")
|
|
772
|
+
```
|
|
773
|
+
|
|
774
|
+
> [!WARNING]
|
|
775
|
+
> **β οΈ Unlocking Full JIT on Android 10+:**
|
|
776
|
+
> Running full V8 JIT without `--jitless` requires either an Android 9 or older device, or a rooted device with permissive SELinux (`setenforce 0`). **Rooting or disabling SELinux is strictly NOT recommended** due to severe device security and integrity risks.
|
|
777
|
+
|
|
778
|
+
---
|
|
779
|
+
|
|
780
|
+
## π 24/7 Unattended Background Operation & Android Deep Sleep Prevention
|
|
781
|
+
|
|
782
|
+
When your smartphone screen is turned off or left idle, Android OS aggressively triggers **Doze Mode** and puts the CPU into **Deep Sleep**, which suspends all background scripts and network connections.
|
|
783
|
+
|
|
784
|
+
To keep your Termux crawlers running continuously 24/7, use the following battle-tested setup:
|
|
785
|
+
|
|
786
|
+
### Method 1: Termux CLI Commands (Recommended & Fail-Safe)
|
|
787
|
+
Acquire the CPU wake lock directly in your terminal before launching long-running crawling tasks:
|
|
788
|
+
|
|
789
|
+
```bash
|
|
790
|
+
# 1. Prevent Android CPU from entering Deep Sleep
|
|
791
|
+
termux-wake-lock
|
|
792
|
+
|
|
793
|
+
# 2. Run your crawler in the background (using tmux, nohup, or background job)
|
|
794
|
+
nohup python examples/advanced_crawler.py > crawler.log 2>&1 &
|
|
795
|
+
|
|
796
|
+
# 3. Release the lock when you are finished
|
|
797
|
+
termux-wake-unlock
|
|
798
|
+
```
|
|
799
|
+
|
|
800
|
+
### Method 2: Android OS Battery Optimization & Phantom Process Killer Exemption
|
|
801
|
+
For uninterrupted multi-day execution, configure your smartphone OS settings:
|
|
802
|
+
|
|
803
|
+
1. **Android App Battery Settings:**
|
|
804
|
+
* Open Android **Settings** $\rightarrow$ **Apps** $\rightarrow$ **Termux**.
|
|
805
|
+
* Select **Battery** (or App Battery Usage).
|
|
806
|
+
* Set to **Unrestricted** (or "Don't Optimize").
|
|
807
|
+
* Enable **"Allow background activity"**.
|
|
808
|
+
|
|
809
|
+
2. **Android 12 / 13 / 14 Phantom Process Killer Exemption:**
|
|
810
|
+
* Android 12+ kills background child processes if an app spawns more than 32 sub-processes.
|
|
811
|
+
* To prevent Chromium renderer processes from being killed by the OS, either pass `single_process=True` in `launch()`, or disable the limit via ADB.
|
|
812
|
+
* π **Detailed Step-by-Step ADB Guide:** See [docs/PHANTOM_PROCESS_KILLER_GUIDE.md](docs/PHANTOM_PROCESS_KILLER_GUIDE.md) for full instructions (USB Debugging, Wireless Debugging without PC, and reboot persistence).
|
|
813
|
+
```bash
|
|
814
|
+
adb shell "/system/bin/device_config put activity_manager max_phantom_processes 2147483647"
|
|
815
|
+
adb shell "/system/bin/device_config set_sync_disabled_for_tests persistent"
|
|
816
|
+
```
|
|
817
|
+
|
|
818
|
+
3. **Termux:API Companion APK Installation:**
|
|
819
|
+
* WakeLock management requires both the command-line package (`pkg install termux-api`) and the companion Android app.
|
|
820
|
+
* Install **Termux:API** from [F-Droid](https://f-droid.org/packages/com.termux.api/) so the system can communicate with the Android power management subsystem.
|
|
821
|
+
|
|
822
|
+
### Method 3: Python Context Manager (Optional Helper)
|
|
823
|
+
```python
|
|
824
|
+
from termux_playwright import TermuxWakeLock, async_playwright_termux, launch
|
|
825
|
+
|
|
826
|
+
# Acquire CPU wake lock for the duration of the crawler block
|
|
827
|
+
with TermuxWakeLock(fail_silently=True):
|
|
828
|
+
async with async_playwright_termux() as p:
|
|
829
|
+
browser = await launch(p)
|
|
830
|
+
# Your 24/7 crawling logic here...
|
|
831
|
+
```
|
|
832
|
+
|
|
833
|
+
---
|
|
834
|
+
|
|
835
|
+
## π€ AI Coding Agents & LLM Quick-Reference (`llms.txt`)
|
|
836
|
+
|
|
837
|
+
This repository conforms to the standard [llms.txt](llms.txt) specification for AI coding assistants (Claude, Cursor, Copilot, ChatGPT, Gemini).
|
|
838
|
+
|
|
839
|
+
### π§ AI Agent Decision Matrix: Choosing the Right `launch()` Flags
|
|
840
|
+
|
|
841
|
+
| Scenario / Goal | Recommended `launch()` Options | Technical Rationale |
|
|
842
|
+
| :--- | :--- | :--- |
|
|
843
|
+
| **Standard 24/7 Web Crawling (Default)** | `launch(p, headless=True)` | Multi-process Chromium; full multi-core performance and individual tab crash isolation. |
|
|
844
|
+
| **Android 14+ without ADB Unlock** | `launch(p, headless=True, single_process=True)` | Collapses all tabs into 1 process to bypass Android 14's 32-process Phantom Killer. |
|
|
845
|
+
| **Anti-Bot / Cloudflare Evasion** | `launch(p, headless=True, stealth=True)` + `setup_stealth_context()` | Injects anti-bot evasion scripts, masks `navigator.webdriver`, and syncs real Chromium version. |
|
|
846
|
+
| **High-Priority Solo Jobs / Banking** | `launch(p, headless=True, standalone_mode=True, wake_lock=True)` | 100% clean-room ephemeral profile auto-purged on exit + anti-throttling flags and max CPU priority. |
|
|
847
|
+
| **Low-End Android Phones (<= 2GB RAM)** | `launch(p, headless=True, low_memory_mode=True)` + `block_heavy_resources()` | Caps V8 heap at 128MB, limits renderers to 1, and drops images/fonts to prevent LMK kills. |
|
|
848
|
+
|
|
849
|
+
### π― Canonical Golden Pattern for LLMs:
|
|
850
|
+
```python
|
|
851
|
+
import asyncio
|
|
852
|
+
from termux_playwright import async_playwright_termux, launch
|
|
853
|
+
|
|
854
|
+
async def main():
|
|
855
|
+
async with async_playwright_termux() as p:
|
|
856
|
+
# ALWAYS use termux_playwright.launch(), NEVER playwright.chromium.launch()
|
|
857
|
+
browser = await launch(p, headless=True)
|
|
858
|
+
page = await browser.new_page()
|
|
859
|
+
await page.goto("https://example.com", timeout=45000)
|
|
860
|
+
print(await page.title())
|
|
861
|
+
await browser.close()
|
|
862
|
+
|
|
863
|
+
if __name__ == "__main__":
|
|
864
|
+
asyncio.run(main())
|
|
865
|
+
```
|
|
866
|
+
|
|
867
|
+
### π« Rules for AI-Generated Code:
|
|
868
|
+
1. **Never** invoke standard `playwright.chromium.launch()` directly on Termux without `termux_playwright.launch()`.
|
|
869
|
+
2. **Always** use `async_playwright_termux()` or `sync_playwright_termux()` to guarantee process reaper tracking.
|
|
870
|
+
3. Use `low_memory_mode=True` when generating code for low-spec mobile devices ($\le$ 2GB RAM).
|
|
871
|
+
4. For Android 14+ devices without ADB access, pass `single_process=True` to prevent the OS Phantom Killer.
|
|
872
|
+
5. For Cloudflare-protected sites, pass `stealth=True` and configure pages via `setup_stealth_context()`.
|
|
873
|
+
|
|
874
|
+
---
|
|
875
|
+
|
|
876
|
+
## π Version History & Changelog
|
|
877
|
+
|
|
878
|
+
### π’ [Current] v1.81.0 β Direct Socket DNS Bypass-Tunnel (2026-09-06)
|
|
879
|
+
* **Direct Socket RFC 1035 UDP DNS Resolver**: Zero-dependency pure socket DNS client in Python (`termux_playwright.tunnel`) and Node.js (`lib/tunnel.js`) to completely circumvent Android 14~16 VPN (Tailscale, WireGuard) `dnsproxyd` deadlocks.
|
|
880
|
+
* **In-Process HTTP CONNECT Proxy**: High-performance loopback proxy (`127.0.0.1:<dynamic_port>`) establishing transparent TCP tunnels upstream via direct socket resolution.
|
|
881
|
+
* **Fluent `BrowserBuilder` SDK**:
|
|
882
|
+
- Python: `BrowserBuilder().headless(True).bypass_tunnel(True, dns="8.8.8.8").launch()`
|
|
883
|
+
- Node.js: `new BrowserBuilder().headless(true).bypassTunnel(true, '8.8.8.8').launch()`
|
|
884
|
+
* **Unified CLI Crawl Subcommand**: `amv-playwright crawl <url> [--bypass-tunnel] [--dns <ip>]` with headless execution and dynamic DOM extraction.
|
|
885
|
+
* **100% Full Verification**: Verified on real hardware (Samsung Galaxy A35 Android 16) with active Tailscale VPN (7.34s crawl vs 12.91s standard timeout) and 142 automated tests passed (111 Python + 31 Node.js).
|
|
886
|
+
|
|
887
|
+
### π΅ [Previous] v1.80.4 β Enterprise Hardening & Adapter Contract Sync
|
|
888
|
+
* **File-Backed Persistent Session Ledger:** Added `$TMPDIR/.tp_ledger/` to guarantee 100% automatic orphan Chromium discovery and reaping even across hard kernel crashes (`SIGKILL` / Android LMK).
|
|
889
|
+
* **Stat-Driven Dynamic Chromium Version Detection:** Real-time `mtime` checking automatically syncs Client Hints headers across live `pkg upgrade chromium` updates.
|
|
890
|
+
* **Prototype-Safe Anti-Bot Stealth:** Prototype deletion (`delete Object.getPrototypeOf(navigator).webdriver`) with native `permissions.query` and `window.chrome.runtime` mocks to bypass Cloudflare Turnstile & DataDome.
|
|
891
|
+
* **Android 14+ Single-Process Option:** Added `single_process=True` to merge all tabs into 1 process for devices with locked Phantom Process Killer (32-process limit).
|
|
892
|
+
* **Virtualenv Guidance:** Clear diagnostic guidance for `--system-site-packages` requirement.
|
|
893
|
+
* **Storage Auto-Purge Rescue:** Automatic pre-flight cleanup of unowned ephemeral profiles on storage exhaustion.
|
|
894
|
+
|
|
895
|
+
### π΅ [Previous] v1.61.1 β Doctor Diagnostics & Dev-Shm (2026-08-18)
|
|
896
|
+
* **Diagnostic Tooling:** Added `amv-playwright-doctor`, `amv-playwright-install`, `amv-playwright-patch`, and `amv-playwright-reap` CLI commands.
|
|
897
|
+
* **eMMC Protection & Memory Optimization:** `/dev/shm` RAM disk cache and Node.js V8 512MB heap limits.
|
|
898
|
+
* **Standalone Fortress Mode & WakeLock:** Clean-room ephemeral profiles (`tp_solo_*`) and Android CPU wake lock integration.
|
|
899
|
+
|
|
900
|
+
> [!TIP]
|
|
901
|
+
> **Full Version Archive:** For earlier release notes and in-depth changelogs, explore the complete [docs/version/](docs/version) directory:
|
|
902
|
+
> * [v1.80.0 Release Notes](docs/version/v1.80.0.md) (Universal Android 15 & Next-Gen Mobile Automation)
|
|
903
|
+
> * [v1.61.2 Release Notes](docs/version/v1.61.2.md)
|
|
904
|
+
> * [v1.61.1 Release Notes](docs/version/v1.61.1.md)
|
|
905
|
+
> * [v1.61.0 Release Notes](docs/version/v1.61.0.md)
|
|
906
|
+
> * [v1.60.0 Release Notes](docs/version/v1.60.0.md)
|
|
907
|
+
|
|
908
|
+
---
|
|
909
|
+
|
|
910
|
+
## π The AMEVA Mobile AI & Automation Ecosystem
|
|
911
|
+
|
|
912
|
+
* **π¨ [Termux-Diffusion](https://github.com/uno-km/termux-diffusion)** ([PyPI](https://pypi.org/project/termux-diffusion/) | [npm](https://www.npmjs.com/package/termux-diffusion) | [π Official Docs](https://uno-km.github.io/termux-diffusion/)): Production-grade on-device Stable Diffusion AI image generation pipeline for Android Termux & Samsung Galaxy without root or PRoot.
|
|
913
|
+
* **Python:** `pip install termux-diffusion && termux-diffusion-install`
|
|
914
|
+
* **Node.js:** `npm install -g termux-diffusion && npx termux-diffusion install`
|
|
915
|
+
|
|
916
|
+
---
|
|
917
|
+
|
|
918
|
+
## βοΈ Disclaimer
|
|
919
|
+
|
|
920
|
+
> *This project is an independent, community-driven open-source library engineered for the Android Termux ecosystem. It is not directly affiliated with, maintained by, or officially endorsed by the core Termux project or Google LLC.*
|
|
921
|
+
|
|
922
|
+
---
|
|
923
|
+
|
|
924
|
+
## π License
|
|
925
|
+
|
|
926
|
+
This project is licensed under the terms of the [MIT License](LICENSE).
|
|
927
|
+
|
|
928
|
+
|
|
929
|
+
|
|
930
|
+
---
|
|
931
|
+
|
|
932
|
+
## π Sponsorship & Community Backing
|
|
933
|
+
|
|
934
|
+
AMEVA is an independent open-source public good governed under the **AMEVA Open-Source Foundation (AOSF)**. All sponsorship funds are 100% publicly audited and dedicated to physical ARM64 testbeds and CI/CD GPU runners.
|
|
935
|
+
|
|
936
|
+
- **Open Collective (Non-Profit 501(c)(6))**: [https://opencollective.com/ameva-fund](https://opencollective.com/ameva-fund)
|
|
937
|
+
- **GitHub Sponsors**: [https://github.com/sponsors/uno-km](https://github.com/sponsors/uno-km)
|
|
938
|
+
- **Official Foundation Portal**: [https://uno-km.vercel.app/docs/foundation/sponsorship.html](https://uno-km.vercel.app/docs/foundation/sponsorship.html)
|