ja4proxy 0.1.0__tar.gz → 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/PKG-INFO +691 -688
  2. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/README.md +6 -3
  3. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/__init__.py +1 -1
  4. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/certs.py +24 -0
  5. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/cli.py +8 -1
  6. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/engine/bridge.py +25 -18
  7. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/parser.py +15 -0
  8. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/profiles.py +127 -0
  9. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/server.py +344 -195
  10. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/PKG-INFO +691 -688
  11. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/pyproject.toml +1 -1
  12. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/setup.cfg +4 -4
  13. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/tests/test_live_proxy.py +100 -18
  14. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/LICENSE +0 -0
  15. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy/engine/loader.py +0 -0
  16. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/SOURCES.txt +0 -0
  17. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/dependency_links.txt +0 -0
  18. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/entry_points.txt +0 -0
  19. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/requires.txt +0 -0
  20. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/ja4proxy.egg-info/top_level.txt +0 -0
  21. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/tests/test_certs.py +0 -0
  22. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/tests/test_engine.py +0 -0
  23. {ja4proxy-0.1.0 → ja4proxy-0.2.0}/tests/test_parser.py +0 -0
@@ -1,688 +1,691 @@
1
- Metadata-Version: 2.4
2
- Name: ja4proxy
3
- Version: 0.1.0
4
- Summary: High-performance universal proxy server with browser TLS (JA4) and HTTP/2 impersonation
5
- Author-email: cipherwire <dev@cipherwire.link>
6
- License-Expression: MIT
7
- Requires-Python: >=3.10
8
- Description-Content-Type: text/markdown
9
- License-File: LICENSE
10
- Requires-Dist: cryptography>=41.0.0
11
- Requires-Dist: click>=8.0.0
12
- Dynamic: license-file
13
-
14
- <div align="center">
15
-
16
- ```text
17
- _ __ __
18
- (_)___ _/ // / ____ _________ _ ____ __
19
- / / __ `/ // /_/ __ \/ ___/ __ \| |/_/ / / /
20
- / / /_/ /__ __/ /_/ / / / /_/ /> </ /_/ /
21
- __/ /\__,_/ /_/ / .___/_/ \____/_/|_|\__, /
22
- /___/ /_/ /____/
23
- ```
24
-
25
- ### High-performance universal forward proxy with authentic browser TLS (JA4/JA3) and HTTP/2 framing
26
-
27
- <p>
28
- <img src="https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-2563eb" alt="Platforms">
29
- <img src="https://img.shields.io/badge/engine-uTLS%20%2B%20fhttp-4285F4" alt="Engine">
30
- <img src="https://img.shields.io/badge/protocols-HTTP%20%7C%20HTTPS%20%7C%20SOCKS5-00599C" alt="Protocols">
31
- <img src="https://img.shields.io/badge/fingerprint-JA4%20%2F%20JA3%20%2F%20Akamai%20H2-6366f1" alt="Fingerprint">
32
- <img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT">
33
- <img src="https://img.shields.io/badge/status-production-22c55e" alt="Status">
34
- </p>
35
-
36
- </div>
37
-
38
- ---
39
-
40
- ## Table of Contents
41
-
42
- - [The Fingerprinting Problem](#the-fingerprinting-problem)
43
- - [How ja4proxy Works](#how-ja4proxy-works)
44
- - [Architecture & Protocol Pipeline](#architecture--protocol-pipeline)
45
- - [Comparative Analysis](#comparative-analysis)
46
- - [Key Specifications](#key-specifications)
47
- - [Installation](#installation)
48
- - [Quick Start](#quick-start)
49
- - [Multi-Language Client Integration](#multi-language-client-integration)
50
- - [Python (requests, httpx, aiohttp)](#python-requests-httpx-aiohttp)
51
- - [Node.js (axios, undici, fetch)](#nodejs-axios-undici-fetch)
52
- - [Go (net/http)](#go-nethttp)
53
- - [Rust (reqwest)](#rust-reqwest)
54
- - [Scrapy Framework](#scrapy-framework)
55
- - [Dynamic Per-Request Directives](#dynamic-per-request-directives)
56
- - [Via Proxy Credentials (Tag Routing)](#via-proxy-credentials-tag-routing)
57
- - [Via Custom Request Headers](#via-custom-request-headers)
58
- - [Residential Proxy Chaining](#residential-proxy-chaining)
59
- - [Supported Browser Presets](#supported-browser-presets)
60
- - [Production Deployment](#production-deployment)
61
- - [Docker & Docker Compose](#docker--docker-compose)
62
- - [Systemd Service Unit](#systemd-service-unit)
63
- - [System-Wide CA Trust Installation](#system-wide-ca-trust-installation)
64
- - [Security & Engineering Boundaries](#security--engineering-boundaries)
65
- - [CLI Reference](#cli-reference)
66
- - [Automated Verification Suite](#automated-verification-suite)
67
- - [License](#license)
68
-
69
- ---
70
-
71
- ## The Fingerprinting Problem
72
-
73
- Modern anti-bot mitigation platforms (Cloudflare Turnstile / Bot Management, Akamai Edge Workers, Imperva, Datadome, Kasada) no longer rely solely on IP reputation. Instead, automated scrapers and bots are blocked at Layer 7 by inspecting client-side protocol artifacts during the connection handshake:
74
-
75
- ```text
76
- Layer 7 Defense Hierarchy
77
- ├── 1. TLS ClientHello Fingerprinting (JA3 / JA4)
78
- │ ├── Cipher suites order & availability
79
- │ ├── Extension list & GREASE (RFC 8701) injection
80
- │ ├── Elliptic curve groups & signature algorithms
81
- │ └── ALPN (Application-Layer Protocol Negotiation)
82
- ├── 2. HTTP/2 Transport Fingerprinting (Akamai HTTP2 Hash)
83
- │ ├── SETTINGS frame parameters (HEADER_TABLE_SIZE, MAX_CONCURRENT_STREAMS)
84
- │ ├── WINDOW_UPDATE increments
85
- │ ├── Pseudo-header serialization order (:method, :authority, :scheme, :path)
86
- │ └── Priority trees and frame dependency graphs
87
- └── 3. Layer 7 HTTP Header Consistency
88
- ├── Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform
89
- └── User-Agent string alignment with actual cryptographic capabilities
90
- ```
91
-
92
- Standard networking stacks (OpenSSL in Python `urllib3`, Go `crypto/tls`, Node.js `tls`) expose hard-coded, static handshakes that immediately flag requests as non-browser traffic.
93
-
94
- ### Fingerprint Breakdown: OpenSSL vs. ja4proxy (Chrome 133)
95
-
96
- | Component | Standard Python / Node / Go | ja4proxy (Chrome 133 Profile) |
97
- | :--- | :--- | :--- |
98
- | **JA4 Signature** | `t13d030800_...` (Flagged as Bot) | `t13d1516h2_8daaf6152771_d8a2da3f94cd` (Identical to Chrome 133) |
99
- | **GREASE Support** | None (RFC 8701 absent) | Fully randomized GREASE cipher and extension values |
100
- | **HTTP Version** | Often forced to HTTP/1.1 | Native HTTP/2 (RFC 7540) with authentic browser stream frames |
101
- | **Pseudo-Header Order** | Variable or alphabetical | `:method`, `:authority`, `:scheme`, `:path` |
102
- | **H2 SETTINGS** | Default OpenSSL/Go parameters | `1:65536, 2:0, 4:6291456, 6:262144` matching Chromium |
103
-
104
- ---
105
-
106
- ## How ja4proxy Works
107
-
108
- `ja4proxy` is an in-process, high-concurrency forward proxy. Rather than forwarding raw TCP sockets (which passes through your runtime's detectable OpenSSL handshake), `ja4proxy` terminates client requests locally and re-originates the upstream connection using an in-process uTLS and fhttp engine written in Go.
109
-
110
- ```text
111
- Any Existing Client Application
112
- (cURL / requests / Scrapy / Playwright / Node / Go / Rust)
113
- │
114
- HTTP CONNECT / Plain HTTP / SOCKS5 binary
115
- │
116
- ▼
117
- ┌────────────────────────────────────────────────────────────────────────┐
118
- │ ja4proxy Runtime (127.0.0.1:8888) │
119
- │ │
120
- │ [Ingress Sniffer] -> Auto-detects protocol by magic byte: │
121
- │ - 0x05 (SOCKS5 RFC 1928) │
122
- │ - CONNECT host:443 (HTTPS Tunnel) │
123
- │ - GET / POST / http://... (HTTP Gateway) │
124
- │ │
125
- │ [Directive Parser] -> Extracts profile, session, upstream proxy: │
126
- │ - Via credential tags: user-profile_chrome133 │
127
- │ - Via custom headers: X-JA4-Profile │
128
- │ │
129
- │ [SSRF Guard] -> Validates target IP against private/loopback CIDRs │
130
- │ │
131
- │ [TLS Interceptor] -> Generates dynamic in-memory SAN leaf certs │
132
- │ │
133
- │ [Native Zero-GIL FFI] -> Dispatches request into Go uTLS engine │
134
- └───────────────────────────────────┬────────────────────────────────────┘
135
- │
136
- Outbound: Real Chrome 133 JA4 + HTTP/2 Multiplexing
137
- │
138
- ▼
139
- Upstream Target (WAF / Protected Host)
140
- ```
141
-
142
- ---
143
-
144
- ## Architecture & Protocol Pipeline
145
-
146
- ### 1. Single-Port Protocol Sniffing
147
- `ja4proxy` binds to a single port and inspects the initial byte stream using non-destructive socket inspection:
148
- * Byte `0x05`: Dispatches to the binary SOCKS5 state machine (RFC 1928), negotiating `NO AUTH` or `USERNAME/PASSWORD` before taking target host and port.
149
- * ASCII `CONNECT`: Intercepts HTTPS tunnels, performs an SSL handshake using dynamic per-hostname leaf certificates signed on-the-fly by the local CA, and decrypts request bytes.
150
- * ASCII `GET / POST / HEAD`: Handles standard forward proxy requests or gateway URLs (`http://127.0.0.1:8888/https://example.com`).
151
-
152
- ### 2. Zero-GIL Native C-Shared FFI
153
- Python manages client I/O asynchronously via `asyncio`. When dispatching outbound HTTP/TLS requests, `ja4proxy` calls into `libutls` through `ctypes`. The Python Global Interpreter Lock (GIL) is released during socket creation, cryptographic handshakes, and body streaming.
154
-
155
- ### 3. Memory Hygiene & Free Routine
156
- Native C allocations returned by the Go runtime are wrapped in a structured memory protocol. Once response headers and body buffers are consumed in Python, `freeMemory` is explicitly invoked over the C pointer, preventing memory leaks across long-running proxy processes.
157
-
158
- ---
159
-
160
- ## Comparative Analysis
161
-
162
- | Feature | ja4proxy | curl-impersonate | tls-client (Python) | Puppeteer / Playwright | Squid / Privoxy |
163
- | :--- | :--- | :--- | :--- | :--- | :--- |
164
- | **Drop-in for Any Language** | Yes (via standard proxy env) | CLI / C only | Python only | Requires browser process | Yes |
165
- | **Zero Code Changes** | Yes (`HTTP_PROXY=...`) | No (must replace curl binary) | No (rewrite to custom client) | No (heavy refactor) | Yes |
166
- | **JA4 / JA3 Fingerprint** | Authentic Browser Preset | Authentic Browser Preset | Authentic Browser Preset | Authentic Browser | Leaks OS OpenSSL |
167
- | **HTTP/2 SETTINGS Impersonation** | Yes | Yes | Yes | Yes | No |
168
- | **RAM per Worker** | ~18 MB | ~15 MB | ~20 MB | ~150 - 300 MB | ~30 MB |
169
- | **SOCKS5 Ingress** | Yes (Auto-detected) | No | No | No | Separate port |
170
- | **Upstream Proxy Chaining** | Dynamic per-request | Static flags | Per-client | Per-browser context | Configuration file |
171
-
172
- ---
173
-
174
- ## Key Specifications
175
-
176
- - **Listen Protocols**: HTTP Gateway, HTTP CONNECT, SOCKS5 (RFC 1928) on a single port.
177
- - **TLS Engine**: uTLS with randomized GREASE (RFC 8701) and custom extension order permutations.
178
- - **HTTP Engine**: fhttp with browser-matched HTTP/2 SETTINGS framing and pseudo-header ordering.
179
- - **Certificate Authority**: 2048-bit RSA Root CA (`ca.pem`) with automatic in-memory leaf certificate generation and Subject Alternative Name (SAN) caching.
180
- - **Security Boundary**: Pre-flight connection-time SSRF blocking for RFC 1918 private subnets, loopback, and cloud metadata (`169.254.169.254`).
181
- - **Decompression Sync**: Automatic header reconciliation (`Content-Encoding`, `Content-Length`) to prevent client decoder desync.
182
-
183
- ---
184
-
185
- ## Installation
186
-
187
- ### Via Source
188
-
189
- ```bash
190
- git clone https://github.com/cipherwire/ja4proxy.git
191
- cd ja4proxy
192
- pip install .
193
- ```
194
-
195
- ### Via Docker
196
-
197
- ```bash
198
- docker build -t ja4proxy .
199
- docker run -d -p 8888:8888 --name ja4proxy ja4proxy
200
- ```
201
-
202
- ---
203
-
204
- ## Quick Start
205
-
206
- ### 1. Launch the Server
207
-
208
- ```bash
209
- ja4proxy start --host 127.0.0.1 --port 8888 --profile chrome_133
210
- ```
211
-
212
- Output:
213
- ```text
214
- [*] ja4proxy v0.1.0 starting on http://127.0.0.1:8888
215
- [*] Default Profile: chrome_133
216
- [*] Protocols Supported: HTTP CONNECT, HTTP Gateway, SOCKS5 (Auto-detect)
217
- [*] CA Certificate: C:\Users\AnonNp\.ja4proxy\ca.pem
218
- [*] Ready for HTTP and SOCKS5 clients
219
- ```
220
-
221
- ### 2. Export the Root CA
222
-
223
- Export the root certificate so your HTTP client can verify intercept certificates without disabling SSL verification:
224
-
225
- ```bash
226
- ja4proxy ca --export ./ca.pem
227
- ```
228
-
229
- ### 3. Verify TLS Handshake Against tls.peet.ws
230
-
231
- #### Using HTTP CONNECT:
232
- ```bash
233
- curl -x http://127.0.0.1:8888 --cacert ./ca.pem https://tls.peet.ws/api/all
234
- ```
235
-
236
- #### Using SOCKS5 (Identical Port):
237
- ```bash
238
- curl -x socks5h://127.0.0.1:8888 --cacert ./ca.pem https://tls.peet.ws/api/all
239
- ```
240
-
241
- #### Using Gateway Mode (No CA certificate required):
242
- ```bash
243
- curl http://127.0.0.1:8888/https://tls.peet.ws/api/all
244
- ```
245
-
246
- Verified output from `tls.peet.ws`:
247
- ```json
248
- {
249
- "http_version": "h2",
250
- "tls": {
251
- "ja4": "t13d1516h2_8daaf6152771_d8a2da3f94cd",
252
- "ja3": "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0"
253
- }
254
- }
255
- ```
256
-
257
- ---
258
-
259
- ## Multi-Language Client Integration
260
-
261
- ### Python (requests, httpx, aiohttp)
262
-
263
- #### `requests`
264
- ```python
265
- import os
266
- import requests
267
-
268
- ca_cert = os.path.expanduser("~/.ja4proxy/ca.pem")
269
- proxies = {
270
- "http": "http://127.0.0.1:8888",
271
- "https": "http://127.0.0.1:8888",
272
- }
273
-
274
- # Standard requests call immediately adopts Chrome 133 JA4 fingerprint
275
- res = requests.get("https://tls.peet.ws/api/all", proxies=proxies, verify=ca_cert)
276
- print("JA4:", res.json()["tls"]["ja4"])
277
- ```
278
-
279
- #### `httpx` (Async with SOCKS5)
280
- ```python
281
- import asyncio
282
- import os
283
- import httpx
284
-
285
- async def main():
286
- ca_cert = os.path.expanduser("~/.ja4proxy/ca.pem")
287
- async with httpx.AsyncClient(
288
- proxy="socks5://127.0.0.1:8888",
289
- verify=ca_cert,
290
- ) as client:
291
- res = await client.get("https://tls.peet.ws/api/all")
292
- print("JA4:", res.json()["tls"]["ja4"])
293
-
294
- asyncio.run(main())
295
- ```
296
-
297
- ---
298
-
299
- ### Node.js (axios, undici, fetch)
300
-
301
- #### `axios`
302
- ```javascript
303
- const axios = require('axios');
304
- const fs = require('fs');
305
- const https = require('https');
306
-
307
- const ca = fs.readFileSync('./ca.pem');
308
- const httpsAgent = new https.Agent({ ca });
309
-
310
- axios.get('https://tls.peet.ws/api/all', {
311
- proxy: { host: '127.0.0.1', port: 8888 },
312
- httpsAgent: httpsAgent
313
- }).then(res => {
314
- console.log('JA4:', res.data.tls.ja4);
315
- });
316
- ```
317
-
318
- #### Native Node `fetch` (with `undici`)
319
- ```javascript
320
- import { fetch, ProxyAgent } from 'undici';
321
- import fs from 'fs';
322
-
323
- const client = new ProxyAgent({
324
- uri: 'http://127.0.0.1:8888',
325
- requestTls: { ca: fs.readFileSync('./ca.pem') }
326
- });
327
-
328
- const res = await fetch('https://tls.peet.ws/api/all', { dispatcher: client });
329
- const json = await res.json();
330
- console.log('JA4:', json.tls.ja4);
331
- ```
332
-
333
- ---
334
-
335
- ### Go (net/http)
336
-
337
- ```go
338
- package main
339
-
340
- import (
341
- "crypto/x509"
342
- "fmt"
343
- "io"
344
- "net/http"
345
- "net/url"
346
- "os"
347
- )
348
-
349
- func main() {
350
- caCert, _ := os.ReadFile("ca.pem")
351
- caCertPool := x509.NewCertPool()
352
- caCertPool.AppendCertsFromPEM(caCert)
353
-
354
- proxyURL, _ := url.Parse("http://127.0.0.1:8888")
355
- transport := &http.Transport{
356
- Proxy: http.ProxyURL(proxyURL),
357
- TLSClientConfig: &tls.Config{
358
- RootCAs: caCertPool,
359
- },
360
- }
361
-
362
- client := &http.Client{Transport: transport}
363
- resp, err := client.Get("https://tls.peet.ws/api/all")
364
- if err != nil {
365
- panic(err)
366
- }
367
- defer resp.Body.Close()
368
-
369
- body, _ := io.ReadAll(resp.Body)
370
- fmt.Println(string(body))
371
- }
372
- ```
373
-
374
- ---
375
-
376
- ### Rust (reqwest)
377
-
378
- ```rust
379
- use reqwest::{Certificate, Client, Proxy};
380
- use std::fs;
381
-
382
- #[tokio::main]
383
- async fn main() -> Result<(), Box<dyn std::error::Error>> {
384
- let cert_bytes = fs::read("ca.pem")?;
385
- let cert = Certificate::from_pem(&cert_bytes)?;
386
-
387
- let proxy = Proxy::all("http://127.0.0.1:8888")?;
388
- let client = Client::builder()
389
- .proxy(proxy)
390
- .add_root_certificate(cert)
391
- .build()?;
392
-
393
- let res = client.get("https://tls.peet.ws/api/all").send().await?;
394
- println!("Response: {}", res.text().await?);
395
- Ok(())
396
- }
397
- ```
398
-
399
- ---
400
-
401
- ### Scrapy Framework
402
-
403
- In your Scrapy project's `settings.py`:
404
-
405
- ```python
406
- # Enable standard HTTP proxy middleware
407
- DOWNLOADER_MIDDLEWARES = {
408
- 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
409
- }
410
-
411
- # Route all traffic through ja4proxy with Safari 18 profile
412
- HTTP_PROXY = "http://user-profile_safari18:nopass@127.0.0.1:8888"
413
- REQUESTS_CA_BUNDLE = "/path/to/ca.pem"
414
- ```
415
-
416
- ---
417
-
418
- ## Dynamic Per-Request Directives
419
-
420
- `ja4proxy` allows you to customize the browser profile, session pool, or upstream routing on a granular per-request basis without restarting the server.
421
-
422
- ### Via Proxy Credentials (Tag Routing)
423
-
424
- Pass directives directly inside the proxy username string formatted as `tag_value`:
425
-
426
- ```text
427
- Format:
428
- http://[user]-profile_<profile>-session_<id>-proxy_<upstream_url>:[password]@127.0.0.1:8888
429
- ```
430
-
431
- Examples:
432
-
433
- 1. **Switch to Safari 18**:
434
- ```bash
435
- curl -x "http://user-profile_safari18:pass@127.0.0.1:8888" --cacert ./ca.pem https://tls.peet.ws/api/all
436
- ```
437
-
438
- 2. **Maintain a Sticky Session Pool**:
439
- ```bash
440
- curl -x "http://user-profile_chrome133-session_worker12:pass@127.0.0.1:8888" --cacert ./ca.pem https://target.com
441
- ```
442
-
443
- 3. **Chain Out to an Upstream SOCKS5 Proxy**:
444
- ```bash
445
- curl -x "http://user-profile_chrome133-proxy_socks5://resuser:respass@residential.net:1080:pass@127.0.0.1:8888" --cacert ./ca.pem https://target.com
446
- ```
447
-
448
- ### Via Custom Request Headers
449
-
450
- When connecting via plain HTTP gateway or custom client hooks, pass directives as standard headers:
451
-
452
- | Header | Type | Description | Example |
453
- | :--- | :--- | :--- | :--- |
454
- | `X-JA4-Profile` | String | Target browser preset | `chrome_133`, `safari_18_0`, `firefox_133` |
455
- | `X-JA4-Session` | String | Persistent session identifier | `client_worker_01` |
456
- | `X-JA4-Upstream-Proxy` | URL | Forwarding proxy endpoint | `http://user:pass@10.0.0.1:8080` |
457
- | `X-JA4-Timeout` | Integer | Connection timeout (seconds) | `30` |
458
-
459
- ---
460
-
461
- ## Residential Proxy Chaining
462
-
463
- `ja4proxy` acts as a TLS and protocol transformer. You can position `ja4proxy` locally on your server or laptop, while routing the actual egress IP through your existing residential or datacenter proxy network:
464
-
465
- ```text
466
- Client -> ja4proxy (Applies Chrome 133 JA4 + HTTP/2) -> Upstream Residential IP -> Cloudflare
467
- ```
468
-
469
- To configure global upstream forwarding at startup:
470
-
471
- ```bash
472
- ja4proxy start --port 8888 --upstream-proxy "http://username:password@residential-provider.com:8000"
473
- ```
474
-
475
- Or dynamically per request:
476
-
477
- ```python
478
- import requests
479
-
480
- proxies = {
481
- # Dynamically inject residential upstream via tag syntax
482
- "https": "http://user-proxy_http://resuser:respass@gate.io:8000:pass@127.0.0.1:8888"
483
- }
484
-
485
- res = requests.get("https://api.ipify.org?format=json", proxies=proxies, verify="ca.pem")
486
- print("Egress IP:", res.json()["ip"])
487
- ```
488
-
489
- ---
490
-
491
- ## Supported Browser Presets
492
-
493
- Run `ja4proxy profiles` to list all supported profiles:
494
-
495
- | Profile Identifier | Engine | Platform | Key Protocol Features |
496
- | :--- | :--- | :--- | :--- |
497
- | `chrome_133` (Default) | Chromium 133 | Windows / macOS | GREASE, ALPN h2, PSK, Post-Quantum Kyber key exchange |
498
- | `chrome_131` | Chromium 131 | Windows | GREASE, ALPN h2, ChaCha20, AES-GCM |
499
- | `chrome_124` | Chromium 124 | Windows | GREASE, ALPN h2, TLS 1.3 |
500
- | `chrome_120` | Chromium 120 | Windows | Standard Chromium TLS 1.3 profile |
501
- | `safari_16_0` | WebKit | macOS 13 | WebKit TLS 1.3 profile, Safari extension order & H2 framing |
502
- | `safari_15_6_1` | WebKit | macOS 12 | WebKit TLS 1.3 legacy profile |
503
- | `safari_ios_16_0` | WebKit iOS | iOS 16 | Mobile WebKit TLS profile |
504
- | `firefox_133` | Gecko | Windows / Linux | Mozilla extension permutation, no GREASE, specific ALPN order |
505
- | `firefox_132` | Gecko | Windows | Mozilla TLS 1.3 profile |
506
- | `firefox_120` | Gecko | Linux | Standard Firefox TLS 1.3 profile |
507
- | `opera_90` | Chromium / Opera | Windows | Opera Desktop fingerprint |
508
- | `opera_89` | Chromium / Opera | Windows | Legacy Opera Desktop profile |
509
-
510
- ---
511
-
512
- ## Production Deployment
513
-
514
- ### Docker & Docker Compose
515
-
516
- #### `docker-compose.yml`
517
- ```yaml
518
- version: '3.8'
519
-
520
- services:
521
- ja4proxy:
522
- image: ja4proxy:latest
523
- build:
524
- context: .
525
- dockerfile: Dockerfile
526
- container_name: ja4proxy
527
- restart: unless-stopped
528
- ports:
529
- - "8888:8888"
530
- volumes:
531
- - ja4proxy_certs:/root/.ja4proxy
532
- environment:
533
- - JA4PROXY_PORT=8888
534
- - JA4PROXY_PROFILE=chrome_133
535
- healthcheck:
536
- test: ["CMD", "curl", "-f", "http://127.0.0.1:8888/https://httpbin.org/status/200"]
537
- interval: 30s
538
- timeout: 5s
539
- retries: 3
540
-
541
- volumes:
542
- ja4proxy_certs:
543
- ```
544
-
545
- Launch with:
546
- ```bash
547
- docker compose up -d
548
- ```
549
-
550
- ### Systemd Service Unit
551
-
552
- Create `/etc/systemd/system/ja4proxy.service`:
553
-
554
- ```ini
555
- [Unit]
556
- Description=ja4proxy TLS & HTTP/2 Impersonation Proxy
557
- After=network.target
558
-
559
- [Service]
560
- Type=simple
561
- User=proxyuser
562
- WorkingDirectory=/opt/ja4proxy
563
- ExecStart=/usr/local/bin/ja4proxy start --host 0.0.0.0 --port 8888 --profile chrome_133
564
- Restart=always
565
- RestartSec=3
566
- LimitNOFILE=65535
567
-
568
- [Install]
569
- WantedBy=multi-user.target
570
- ```
571
-
572
- Enable and start:
573
- ```bash
574
- sudo systemctl daemon-reload
575
- sudo systemctl enable --now ja4proxy
576
- ```
577
-
578
- ### System-Wide CA Trust Installation
579
-
580
- To use `ja4proxy` transparently without passing `--cacert` or `verify=...` in your scripts:
581
-
582
- #### Debian / Ubuntu
583
- ```bash
584
- ja4proxy ca --export /tmp/ja4proxy-ca.crt
585
- sudo cp /tmp/ja4proxy-ca.crt /usr/local/share/ca-certificates/
586
- sudo update-ca-certificates
587
- ```
588
-
589
- #### macOS
590
- ```bash
591
- ja4proxy ca --export /tmp/ja4proxy-ca.crt
592
- sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/ja4proxy-ca.crt
593
- ```
594
-
595
- #### Windows (PowerShell Administrator)
596
- ```powershell
597
- ja4proxy ca --export "$env:TEMP\ja4proxy-ca.crt"
598
- certutil -addstore -f "Root" "$env:TEMP\ja4proxy-ca.crt"
599
- ```
600
-
601
- ---
602
-
603
- ## Security & Engineering Boundaries
604
-
605
- 1. **Pre-Flight SSRF Protection**:
606
- `ja4proxy` resolves hostnames prior to outbound dispatch and drops requests resolving to private address spaces (RFC 1918 `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), loopback (`127.0.0.0/8`, `::1`), and cloud metadata (`169.254.169.254`). Internal forwarding can be enabled with `--allow-private-networks`.
607
-
608
- 2. **Decompression & Content-Length Sync**:
609
- The native Go engine decodes gzip/brotli/deflate responses transparently. `ja4proxy` strips stale `Content-Encoding` and `Transfer-Encoding` headers and recalculates accurate `Content-Length` before serving downstream, preventing double-decompression exceptions.
610
-
611
- 3. **Header Sanitization**:
612
- Incoming client hop-by-hop headers (`Proxy-Connection`, `Keep-Alive`, `Te`) and automated OpenSSL/Python identifiers are stripped. Canonical browser headers (`User-Agent`, `Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`) matching the selected profile are automatically synthesized when missing.
613
-
614
- 4. **Zero-GIL Execution**:
615
- Native C FFI functions execute outside Python's GIL. Concurrency scales across available CPU cores and network threads without interpreter bottlenecks.
616
-
617
- ---
618
-
619
- ## CLI Reference
620
-
621
- ```text
622
- Usage: ja4proxy [OPTIONS] COMMAND [ARGS]...
623
-
624
- ja4proxy: High-performance universal forward proxy server with browser TLS (JA4)
625
- and HTTP/2 impersonation.
626
-
627
- Options:
628
- --version Show the version and exit.
629
- --help Show this message and exit.
630
-
631
- Commands:
632
- ca Inspects or exports the local Root CA certificate.
633
- profiles Lists all supported browser profiles for TLS and HTTP/2.
634
- start Starts the ja4proxy server.
635
- test Performs a direct live handshake against tls.peet.ws.
636
- ```
637
-
638
- ### `ja4proxy start` Flags
639
-
640
- | Option | Default | Description |
641
- | :--- | :--- | :--- |
642
- | `--host` | `127.0.0.1` | Network interface to bind |
643
- | `--port` | `8888` | Port to listen for HTTP, HTTPS CONNECT, and SOCKS5 |
644
- | `--profile` | `chrome_133` | Default browser profile |
645
- | `--upstream-proxy` | None | Global upstream forwarding proxy |
646
- | `--allow-private-networks` | False | Permits forwarding to private/internal IPs |
647
- | `--ca-dir` | `~/.ja4proxy` | Custom directory for CA certificates |
648
-
649
- ---
650
-
651
- ## Automated Verification Suite
652
-
653
- Run the full automated test suite covering CA generation, SOCKS5 state machines, chunked request decoding, SSRF guards, and live upstream verification against `tls.peet.ws` and `httpbin.org`:
654
-
655
- ```bash
656
- python -m unittest discover -s tests -p "test_*.py" -v
657
- ```
658
-
659
- All 16 unit and live tests execute with zero warnings and zero regressions:
660
- ```text
661
- test_ca_creation (test_certs.TestCertificateAuthority) ... ok
662
- test_leaf_cert_generation (test_certs.TestCertificateAuthority) ... ok
663
- test_leaf_cert_ip_address (test_certs.TestCertificateAuthority) ... ok
664
- test_live_handshake_chrome (test_engine.TestNativeEngine) ... ok
665
- test_live_post_httpbin (test_engine.TestNativeEngine) ... ok
666
- test_bad_gateway_mapping_status_502 (test_live_proxy.TestLiveProxy) ... ok
667
- test_chunked_request_upload (test_live_proxy.TestLiveProxy) ... ok
668
- test_gateway_mode_gzip (test_live_proxy.TestLiveProxy) ... ok
669
- test_socks5_ingress_tls_peet (test_live_proxy.TestLiveProxy) ... ok
670
- test_ssrf_protection_blocked (test_live_proxy.TestLiveProxy) ... ok
671
- test_ssrf_restricted_addresses (test_live_proxy.TestLiveProxy) ... ok
672
- test_extract_directives_from_headers (test_parser.TestParser) ... ok
673
- test_extract_directives_priority_header_over_auth (test_parser.TestParser) ... ok
674
- test_parse_credentials_empty (test_parser.TestParser) ... ok
675
- test_parse_credentials_proxy_with_hyphens (test_parser.TestParser) ... ok
676
- test_parse_credentials_tags (test_parser.TestParser) ... ok
677
-
678
- ----------------------------------------------------------------------
679
- Ran 16 tests in 5.588s
680
-
681
- OK
682
- ```
683
-
684
- ---
685
-
686
- ## License
687
-
688
- [MIT](file:///D:/what%20to%20make/LICENSE) License. Designed and engineered by `cipherwire` / `hainahola`.
1
+ Metadata-Version: 2.4
2
+ Name: ja4proxy
3
+ Version: 0.2.0
4
+ Summary: High-performance universal proxy server with browser TLS (JA4) and HTTP/2 impersonation
5
+ Author-email: cipherwire <dev@cipherwire.link>
6
+ License-Expression: MIT
7
+ Requires-Python: >=3.10
8
+ Description-Content-Type: text/markdown
9
+ License-File: LICENSE
10
+ Requires-Dist: cryptography>=41.0.0
11
+ Requires-Dist: click>=8.0.0
12
+ Dynamic: license-file
13
+
14
+ <div align="center">
15
+
16
+ ```text
17
+ _ __ __
18
+ (_)___ _/ // / ____ _________ _ ____ __
19
+ / / __ `/ // /_/ __ \/ ___/ __ \| |/_/ / / /
20
+ / / /_/ /__ __/ /_/ / / / /_/ /> </ /_/ /
21
+ __/ /\__,_/ /_/ / .___/_/ \____/_/|_|\__, /
22
+ /___/ /_/ /____/
23
+ ```
24
+
25
+ ### High-performance universal forward proxy with authentic browser TLS (JA4/JA3) and HTTP/2 framing
26
+
27
+ <p>
28
+ <img src="https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-2563eb" alt="Platforms">
29
+ <img src="https://img.shields.io/badge/engine-uTLS%20%2B%20fhttp-4285F4" alt="Engine">
30
+ <img src="https://img.shields.io/badge/protocols-HTTP%20%7C%20HTTPS%20%7C%20SOCKS5-00599C" alt="Protocols">
31
+ <img src="https://img.shields.io/badge/fingerprint-JA4%20%2F%20JA3%20%2F%20Akamai%20H2-6366f1" alt="Fingerprint">
32
+ <img src="https://img.shields.io/badge/license-MIT-green" alt="License: MIT">
33
+ <img src="https://img.shields.io/badge/status-production-22c55e" alt="Status">
34
+ </p>
35
+
36
+ </div>
37
+
38
+ ---
39
+
40
+ ## Table of Contents
41
+
42
+ - [The Fingerprinting Problem](#the-fingerprinting-problem)
43
+ - [How ja4proxy Works](#how-ja4proxy-works)
44
+ - [Architecture & Protocol Pipeline](#architecture--protocol-pipeline)
45
+ - [Comparative Analysis](#comparative-analysis)
46
+ - [Key Specifications](#key-specifications)
47
+ - [Installation](#installation)
48
+ - [Quick Start](#quick-start)
49
+ - [Multi-Language Client Integration](#multi-language-client-integration)
50
+ - [Python (requests, httpx, aiohttp)](#python-requests-httpx-aiohttp)
51
+ - [Node.js (axios, undici, fetch)](#nodejs-axios-undici-fetch)
52
+ - [Go (net/http)](#go-nethttp)
53
+ - [Rust (reqwest)](#rust-reqwest)
54
+ - [Scrapy Framework](#scrapy-framework)
55
+ - [Dynamic Per-Request Directives](#dynamic-per-request-directives)
56
+ - [Via Proxy Credentials (Tag Routing)](#via-proxy-credentials-tag-routing)
57
+ - [Via Custom Request Headers](#via-custom-request-headers)
58
+ - [Residential Proxy Chaining](#residential-proxy-chaining)
59
+ - [Supported Browser Presets](#supported-browser-presets)
60
+ - [Production Deployment](#production-deployment)
61
+ - [Docker & Docker Compose](#docker--docker-compose)
62
+ - [Systemd Service Unit](#systemd-service-unit)
63
+ - [System-Wide CA Trust Installation](#system-wide-ca-trust-installation)
64
+ - [Security & Engineering Boundaries](#security--engineering-boundaries)
65
+ - [CLI Reference](#cli-reference)
66
+ - [Automated Verification Suite](#automated-verification-suite)
67
+ - [License](#license)
68
+
69
+ ---
70
+
71
+ ## The Fingerprinting Problem
72
+
73
+ Modern anti-bot mitigation platforms (Cloudflare Turnstile / Bot Management, Akamai Edge Workers, Imperva, Datadome, Kasada) no longer rely solely on IP reputation. Instead, automated scrapers and bots are blocked at Layer 7 by inspecting client-side protocol artifacts during the connection handshake:
74
+
75
+ ```text
76
+ Layer 7 Defense Hierarchy
77
+ ├── 1. TLS ClientHello Fingerprinting (JA3 / JA4)
78
+ │ ├── Cipher suites order & availability
79
+ │ ├── Extension list & GREASE (RFC 8701) injection
80
+ │ ├── Elliptic curve groups & signature algorithms
81
+ │ └── ALPN (Application-Layer Protocol Negotiation)
82
+ ├── 2. HTTP/2 Transport Fingerprinting (Akamai HTTP2 Hash)
83
+ │ ├── SETTINGS frame parameters (HEADER_TABLE_SIZE, MAX_CONCURRENT_STREAMS)
84
+ │ ├── WINDOW_UPDATE increments
85
+ │ ├── Pseudo-header serialization order (:method, :authority, :scheme, :path)
86
+ │ └── Priority trees and frame dependency graphs
87
+ └── 3. Layer 7 HTTP Header Consistency
88
+ ├── Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform
89
+ └── User-Agent string alignment with actual cryptographic capabilities
90
+ ```
91
+
92
+ Standard networking stacks (OpenSSL in Python `urllib3`, Go `crypto/tls`, Node.js `tls`) expose hard-coded, static handshakes that immediately flag requests as non-browser traffic.
93
+
94
+ ### Fingerprint Breakdown: OpenSSL vs. ja4proxy (Chrome 133)
95
+
96
+ | Component | Standard Python / Node / Go | ja4proxy (Chrome 133 Profile) |
97
+ | :--- | :--- | :--- |
98
+ | **JA4 Signature** | `t13d030800_...` (Flagged as Bot) | `t13d1516h2_8daaf6152771_d8a2da3f94cd` (Identical to Chrome 133) |
99
+ | **GREASE Support** | None (RFC 8701 absent) | Fully randomized GREASE cipher and extension values |
100
+ | **HTTP Version** | Often forced to HTTP/1.1 | Native HTTP/2 (RFC 7540) with authentic browser stream frames |
101
+ | **Pseudo-Header Order** | Variable or alphabetical | `:method`, `:authority`, `:scheme`, `:path` |
102
+ | **H2 SETTINGS** | Default OpenSSL/Go parameters | `1:65536, 2:0, 4:6291456, 6:262144` matching Chromium |
103
+
104
+ ---
105
+
106
+ ## How ja4proxy Works
107
+
108
+ `ja4proxy` is an in-process, high-concurrency forward proxy. Rather than forwarding raw TCP sockets (which passes through your runtime's detectable OpenSSL handshake), `ja4proxy` terminates client requests locally and re-originates the upstream connection using an in-process uTLS and fhttp engine written in Go.
109
+
110
+ ```text
111
+ Any Existing Client Application
112
+ (cURL / requests / Scrapy / Playwright / Node / Go / Rust)
113
+ │
114
+ HTTP CONNECT / Plain HTTP / SOCKS5 binary
115
+ │
116
+ ▼
117
+ ┌────────────────────────────────────────────────────────────────────────┐
118
+ │ ja4proxy Runtime (127.0.0.1:8888) │
119
+ │ │
120
+ │ [Ingress Sniffer] -> Auto-detects protocol by magic byte: │
121
+ │ - 0x05 (SOCKS5 RFC 1928) │
122
+ │ - CONNECT host:443 (HTTPS Tunnel) │
123
+ │ - GET / POST / http://... (HTTP Gateway) │
124
+ │ │
125
+ │ [Directive Parser] -> Extracts profile, session, upstream proxy: │
126
+ │ - Via credential tags: user-profile_chrome133 │
127
+ │ - Via custom headers: X-JA4-Profile │
128
+ │ │
129
+ │ [SSRF Guard] -> Validates target IP against private/loopback CIDRs │
130
+ │ │
131
+ │ [TLS Interceptor] -> Generates dynamic in-memory SAN leaf certs │
132
+ │ │
133
+ │ [Native Zero-GIL FFI] -> Dispatches request into Go uTLS engine │
134
+ └───────────────────────────────────┬────────────────────────────────────┘
135
+ │
136
+ Outbound: Real Chrome 133 JA4 + HTTP/2 Multiplexing
137
+ │
138
+ ▼
139
+ Upstream Target (WAF / Protected Host)
140
+ ```
141
+
142
+ ---
143
+
144
+ ## Architecture & Protocol Pipeline
145
+
146
+ ### 1. Single-Port Protocol Sniffing
147
+ `ja4proxy` binds to a single port and inspects the initial byte stream using non-destructive socket inspection:
148
+ * Byte `0x05`: Dispatches to the binary SOCKS5 state machine (RFC 1928), negotiating `NO AUTH` or `USERNAME/PASSWORD` before taking target host and port.
149
+ * ASCII `CONNECT`: Intercepts HTTPS tunnels, performs an SSL handshake using dynamic per-hostname leaf certificates signed on-the-fly by the local CA, and decrypts request bytes.
150
+ * ASCII `GET / POST / HEAD`: Handles standard forward proxy requests or gateway URLs (`http://127.0.0.1:8888/https://example.com`).
151
+
152
+ ### 2. Zero-GIL Native C-Shared FFI
153
+ Python manages client I/O asynchronously via `asyncio`. When dispatching outbound HTTP/TLS requests, `ja4proxy` calls into `libutls` through `ctypes`. The Python Global Interpreter Lock (GIL) is released during socket creation, cryptographic handshakes, and body streaming.
154
+
155
+ ### 3. Memory Hygiene & Free Routine
156
+ Native C allocations returned by the Go runtime are wrapped in a structured memory protocol. Once response headers and body buffers are consumed in Python, `freeMemory` is explicitly invoked over the C pointer, preventing memory leaks across long-running proxy processes.
157
+
158
+ ---
159
+
160
+ ## Comparative Analysis
161
+
162
+ | Feature | ja4proxy | curl-impersonate | tls-client (Python) | Puppeteer / Playwright | Squid / Privoxy |
163
+ | :--- | :--- | :--- | :--- | :--- | :--- |
164
+ | **Drop-in for Any Language** | Yes (via standard proxy env) | CLI / C only | Python only | Requires browser process | Yes |
165
+ | **Zero Code Changes** | Yes (`HTTP_PROXY=...`) | No (must replace curl binary) | No (rewrite to custom client) | No (heavy refactor) | Yes |
166
+ | **JA4 / JA3 Fingerprint** | Authentic Browser Preset | Authentic Browser Preset | Authentic Browser Preset | Authentic Browser | Leaks OS OpenSSL |
167
+ | **HTTP/2 SETTINGS Impersonation** | Yes | Yes | Yes | Yes | No |
168
+ | **RAM per Worker** | ~18 MB | ~15 MB | ~20 MB | ~150 - 300 MB | ~30 MB |
169
+ | **SOCKS5 Ingress** | Yes (Auto-detected) | No | No | No | Separate port |
170
+ | **Upstream Proxy Chaining** | Dynamic per-request | Static flags | Per-client | Per-browser context | Configuration file |
171
+
172
+ ---
173
+
174
+ ## Key Specifications
175
+
176
+ - **Listen Protocols**: HTTP Gateway, HTTP CONNECT, SOCKS5 (RFC 1928) on a single port.
177
+ - **TLS Engine**: uTLS with randomized GREASE (RFC 8701) and custom extension order permutations.
178
+ - **HTTP Engine**: fhttp with browser-matched HTTP/2 SETTINGS framing and pseudo-header ordering.
179
+ - **Certificate Authority**: 2048-bit RSA Root CA (`ca.pem`) with automatic in-memory leaf certificate generation and Subject Alternative Name (SAN) caching.
180
+ - **Security Boundary**: Pre-flight connection-time SSRF blocking for RFC 1918 private subnets, loopback, and cloud metadata (`169.254.169.254`).
181
+ - **Decompression Sync**: Automatic header reconciliation (`Content-Encoding`, `Content-Length`) to prevent client decoder desync.
182
+
183
+ ---
184
+
185
+ ## Installation
186
+
187
+ ### Via Source
188
+
189
+ ```bash
190
+ git clone https://github.com/cipherwire/ja4proxy.git
191
+ cd ja4proxy
192
+ pip install .
193
+ ```
194
+
195
+ ### Via Docker
196
+
197
+ ```bash
198
+ docker build -t ja4proxy .
199
+ docker run -d -p 8888:8888 --name ja4proxy ja4proxy
200
+ ```
201
+
202
+ ---
203
+
204
+ ## Quick Start
205
+
206
+ ### 1. Launch the Server
207
+
208
+ ```bash
209
+ ja4proxy start --host 127.0.0.1 --port 8888 --profile chrome_133
210
+ ```
211
+
212
+ Output:
213
+ ```text
214
+ [*] ja4proxy v0.1.0 starting on http://127.0.0.1:8888
215
+ [*] Default Profile: chrome_133
216
+ [*] Protocols Supported: HTTP CONNECT, HTTP Gateway, SOCKS5 (Auto-detect)
217
+ [*] CA Certificate: C:\Users\AnonNp\.ja4proxy\ca.pem
218
+ [*] Ready for HTTP and SOCKS5 clients
219
+ ```
220
+
221
+ ### 2. Export the Root CA
222
+
223
+ Export the root certificate so your HTTP client can verify intercept certificates without disabling SSL verification:
224
+
225
+ ```bash
226
+ ja4proxy ca --export ./ca.pem
227
+ ```
228
+
229
+ ### 3. Verify TLS Handshake Against tls.peet.ws
230
+
231
+ #### Using HTTP CONNECT:
232
+ ```bash
233
+ curl -x http://127.0.0.1:8888 --cacert ./ca.pem https://tls.peet.ws/api/all
234
+ ```
235
+
236
+ #### Using SOCKS5 (Identical Port):
237
+ ```bash
238
+ curl -x socks5h://127.0.0.1:8888 --cacert ./ca.pem https://tls.peet.ws/api/all
239
+ ```
240
+
241
+ #### Using Gateway Mode (No CA certificate required):
242
+ ```bash
243
+ curl http://127.0.0.1:8888/https://tls.peet.ws/api/all
244
+ ```
245
+
246
+ Verified output from `tls.peet.ws`:
247
+ ```json
248
+ {
249
+ "http_version": "h2",
250
+ "tls": {
251
+ "ja4": "t13d1516h2_8daaf6152771_d8a2da3f94cd",
252
+ "ja3": "771,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,0-23-65281-10-11-35-16-5-13-18-51-45-43-27-21,29-23-24,0"
253
+ }
254
+ }
255
+ ```
256
+
257
+ ---
258
+
259
+ ## Multi-Language Client Integration
260
+
261
+ ### Python (requests, httpx, aiohttp)
262
+
263
+ #### `requests`
264
+ ```python
265
+ import os
266
+ import requests
267
+
268
+ ca_cert = os.path.expanduser("~/.ja4proxy/ca.pem")
269
+ proxies = {
270
+ "http": "http://127.0.0.1:8888",
271
+ "https": "http://127.0.0.1:8888",
272
+ }
273
+
274
+ # Standard requests call immediately adopts Chrome 133 JA4 fingerprint
275
+ res = requests.get("https://tls.peet.ws/api/all", proxies=proxies, verify=ca_cert)
276
+ print("JA4:", res.json()["tls"]["ja4"])
277
+ ```
278
+
279
+ #### `httpx` (Async with SOCKS5)
280
+ ```python
281
+ import asyncio
282
+ import os
283
+ import httpx
284
+
285
+ async def main():
286
+ ca_cert = os.path.expanduser("~/.ja4proxy/ca.pem")
287
+ async with httpx.AsyncClient(
288
+ proxy="socks5://127.0.0.1:8888",
289
+ verify=ca_cert,
290
+ ) as client:
291
+ res = await client.get("https://tls.peet.ws/api/all")
292
+ print("JA4:", res.json()["tls"]["ja4"])
293
+
294
+ asyncio.run(main())
295
+ ```
296
+
297
+ ---
298
+
299
+ ### Node.js (axios, undici, fetch)
300
+
301
+ #### `axios`
302
+ ```javascript
303
+ const axios = require('axios');
304
+ const fs = require('fs');
305
+ const https = require('https');
306
+
307
+ const ca = fs.readFileSync('./ca.pem');
308
+ const httpsAgent = new https.Agent({ ca });
309
+
310
+ axios.get('https://tls.peet.ws/api/all', {
311
+ proxy: { host: '127.0.0.1', port: 8888 },
312
+ httpsAgent: httpsAgent
313
+ }).then(res => {
314
+ console.log('JA4:', res.data.tls.ja4);
315
+ });
316
+ ```
317
+
318
+ #### Native Node `fetch` (with `undici`)
319
+ ```javascript
320
+ import { fetch, ProxyAgent } from 'undici';
321
+ import fs from 'fs';
322
+
323
+ const client = new ProxyAgent({
324
+ uri: 'http://127.0.0.1:8888',
325
+ requestTls: { ca: fs.readFileSync('./ca.pem') }
326
+ });
327
+
328
+ const res = await fetch('https://tls.peet.ws/api/all', { dispatcher: client });
329
+ const json = await res.json();
330
+ console.log('JA4:', json.tls.ja4);
331
+ ```
332
+
333
+ ---
334
+
335
+ ### Go (net/http)
336
+
337
+ ```go
338
+ package main
339
+
340
+ import (
341
+ "crypto/x509"
342
+ "fmt"
343
+ "io"
344
+ "net/http"
345
+ "net/url"
346
+ "os"
347
+ )
348
+
349
+ func main() {
350
+ caCert, _ := os.ReadFile("ca.pem")
351
+ caCertPool := x509.NewCertPool()
352
+ caCertPool.AppendCertsFromPEM(caCert)
353
+
354
+ proxyURL, _ := url.Parse("http://127.0.0.1:8888")
355
+ transport := &http.Transport{
356
+ Proxy: http.ProxyURL(proxyURL),
357
+ TLSClientConfig: &tls.Config{
358
+ RootCAs: caCertPool,
359
+ },
360
+ }
361
+
362
+ client := &http.Client{Transport: transport}
363
+ resp, err := client.Get("https://tls.peet.ws/api/all")
364
+ if err != nil {
365
+ panic(err)
366
+ }
367
+ defer resp.Body.Close()
368
+
369
+ body, _ := io.ReadAll(resp.Body)
370
+ fmt.Println(string(body))
371
+ }
372
+ ```
373
+
374
+ ---
375
+
376
+ ### Rust (reqwest)
377
+
378
+ ```rust
379
+ use reqwest::{Certificate, Client, Proxy};
380
+ use std::fs;
381
+
382
+ #[tokio::main]
383
+ async fn main() -> Result<(), Box<dyn std::error::Error>> {
384
+ let cert_bytes = fs::read("ca.pem")?;
385
+ let cert = Certificate::from_pem(&cert_bytes)?;
386
+
387
+ let proxy = Proxy::all("http://127.0.0.1:8888")?;
388
+ let client = Client::builder()
389
+ .proxy(proxy)
390
+ .add_root_certificate(cert)
391
+ .build()?;
392
+
393
+ let res = client.get("https://tls.peet.ws/api/all").send().await?;
394
+ println!("Response: {}", res.text().await?);
395
+ Ok(())
396
+ }
397
+ ```
398
+
399
+ ---
400
+
401
+ ### Scrapy Framework
402
+
403
+ In your Scrapy project's `settings.py`:
404
+
405
+ ```python
406
+ # Enable standard HTTP proxy middleware
407
+ DOWNLOADER_MIDDLEWARES = {
408
+ 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
409
+ }
410
+
411
+ # Route all traffic through ja4proxy with Safari 18 profile
412
+ HTTP_PROXY = "http://user-profile_safari18:nopass@127.0.0.1:8888"
413
+ REQUESTS_CA_BUNDLE = "/path/to/ca.pem"
414
+ ```
415
+
416
+ ---
417
+
418
+ ## Dynamic Per-Request Directives
419
+
420
+ `ja4proxy` allows you to customize the browser profile, session pool, or upstream routing on a granular per-request basis without restarting the server.
421
+
422
+ ### Via Proxy Credentials (Tag Routing)
423
+
424
+ Pass directives directly inside the proxy username string formatted as `tag_value`:
425
+
426
+ ```text
427
+ Format:
428
+ http://[user]-profile_<profile>-session_<id>-proxy_<upstream_url>:[password]@127.0.0.1:8888
429
+ ```
430
+
431
+ Examples:
432
+
433
+ 1. **Switch to Safari 18**:
434
+ ```bash
435
+ curl -x "http://user-profile_safari18:pass@127.0.0.1:8888" --cacert ./ca.pem https://tls.peet.ws/api/all
436
+ ```
437
+
438
+ 2. **Maintain a Sticky Session Pool**:
439
+ ```bash
440
+ curl -x "http://user-profile_chrome133-session_worker12:pass@127.0.0.1:8888" --cacert ./ca.pem https://target.com
441
+ ```
442
+
443
+ 3. **Chain Out to an Upstream SOCKS5 Proxy**:
444
+ ```bash
445
+ curl -x "http://user-profile_chrome133-proxy_socks5://resuser:respass@residential.net:1080:pass@127.0.0.1:8888" --cacert ./ca.pem https://target.com
446
+ ```
447
+
448
+ ### Via Custom Request Headers
449
+
450
+ When connecting via plain HTTP gateway or custom client hooks, pass directives as standard headers:
451
+
452
+ | Header | Type | Description | Example |
453
+ | :--- | :--- | :--- | :--- |
454
+ | `X-JA4-Profile` | String | Target browser preset | `chrome_133`, `safari_16_0`, `firefox_133` |
455
+ | `X-JA4-Session` | String | Persistent session identifier | `client_worker_01` |
456
+ | `X-JA4-Upstream-Proxy` | URL | Forwarding proxy endpoint | `http://user:pass@10.0.0.1:8080` |
457
+ | `X-JA4-Timeout` | Integer | Connection timeout (seconds) | `30` |
458
+
459
+ ---
460
+
461
+ ## Residential Proxy Chaining
462
+
463
+ `ja4proxy` acts as a TLS and protocol transformer. You can position `ja4proxy` locally on your server or laptop, while routing the actual egress IP through your existing residential or datacenter proxy network:
464
+
465
+ ```text
466
+ Client -> ja4proxy (Applies Chrome 133 JA4 + HTTP/2) -> Upstream Residential IP -> Cloudflare
467
+ ```
468
+
469
+ To configure global upstream forwarding at startup:
470
+
471
+ ```bash
472
+ ja4proxy start --port 8888 --upstream-proxy "http://username:password@residential-provider.com:8000"
473
+ ```
474
+
475
+ Or dynamically per request:
476
+
477
+ ```python
478
+ import requests
479
+
480
+ proxies = {
481
+ # Dynamically inject residential upstream via tag syntax
482
+ "https": "http://user-proxy_http://resuser:respass@gate.io:8000:pass@127.0.0.1:8888"
483
+ }
484
+
485
+ res = requests.get("https://api.ipify.org?format=json", proxies=proxies, verify="ca.pem")
486
+ print("Egress IP:", res.json()["ip"])
487
+ ```
488
+
489
+ ---
490
+
491
+ ## Supported Browser Presets
492
+
493
+ Run `ja4proxy profiles` to list all supported profiles:
494
+
495
+ | Profile Identifier | Engine | Platform | Key Protocol Features |
496
+ | :--- | :--- | :--- | :--- |
497
+ | `chrome_133` (Default) | Chromium 133 | Windows / macOS | GREASE, ALPN h2, PSK, Post-Quantum Kyber key exchange |
498
+ | `chrome_131` | Chromium 131 | Windows | GREASE, ALPN h2, ChaCha20, AES-GCM |
499
+ | `chrome_124` | Chromium 124 | Windows | GREASE, ALPN h2, TLS 1.3 |
500
+ | `chrome_120` | Chromium 120 | Windows | Standard Chromium TLS 1.3 profile |
501
+ | `safari_16_0` | WebKit | macOS 13 | WebKit TLS 1.3 profile, Safari extension order & H2 framing |
502
+ | `safari_15_6_1` | WebKit | macOS 12 | WebKit TLS 1.3 legacy profile |
503
+ | `safari_ios_16_0` | WebKit iOS | iOS 16 | Mobile WebKit TLS profile |
504
+ | `firefox_133` | Gecko | Windows / Linux | Mozilla extension permutation, no GREASE, specific ALPN order |
505
+ | `firefox_132` | Gecko | Windows | Mozilla TLS 1.3 profile |
506
+ | `firefox_120` | Gecko | Linux | Standard Firefox TLS 1.3 profile |
507
+ | `opera_90` | Chromium / Opera | Windows | Opera Desktop fingerprint |
508
+ | `opera_89` | Chromium / Opera | Windows | Legacy Opera Desktop profile |
509
+
510
+ ---
511
+
512
+ ## Production Deployment
513
+
514
+ ### Docker & Docker Compose
515
+
516
+ #### `docker-compose.yml`
517
+ ```yaml
518
+ version: '3.8'
519
+
520
+ services:
521
+ ja4proxy:
522
+ image: ja4proxy:latest
523
+ build:
524
+ context: .
525
+ dockerfile: Dockerfile
526
+ container_name: ja4proxy
527
+ restart: unless-stopped
528
+ ports:
529
+ - "8888:8888"
530
+ volumes:
531
+ - ja4proxy_certs:/root/.ja4proxy
532
+ environment:
533
+ - JA4PROXY_PORT=8888
534
+ - JA4PROXY_PROFILE=chrome_133
535
+ healthcheck:
536
+ test: ["CMD", "curl", "-f", "http://127.0.0.1:8888/https://httpbin.org/status/200"]
537
+ interval: 30s
538
+ timeout: 5s
539
+ retries: 3
540
+
541
+ volumes:
542
+ ja4proxy_certs:
543
+ ```
544
+
545
+ Launch with:
546
+ ```bash
547
+ docker compose up -d
548
+ ```
549
+
550
+ ### Systemd Service Unit
551
+
552
+ Create `/etc/systemd/system/ja4proxy.service`:
553
+
554
+ ```ini
555
+ [Unit]
556
+ Description=ja4proxy TLS & HTTP/2 Impersonation Proxy
557
+ After=network.target
558
+
559
+ [Service]
560
+ Type=simple
561
+ User=proxyuser
562
+ WorkingDirectory=/opt/ja4proxy
563
+ ExecStart=/usr/local/bin/ja4proxy start --host 0.0.0.0 --port 8888 --profile chrome_133
564
+ Restart=always
565
+ RestartSec=3
566
+ LimitNOFILE=65535
567
+
568
+ [Install]
569
+ WantedBy=multi-user.target
570
+ ```
571
+
572
+ Enable and start:
573
+ ```bash
574
+ sudo systemctl daemon-reload
575
+ sudo systemctl enable --now ja4proxy
576
+ ```
577
+
578
+ ### System-Wide CA Trust Installation
579
+
580
+ To use `ja4proxy` transparently without passing `--cacert` or `verify=...` in your scripts:
581
+
582
+ #### Debian / Ubuntu
583
+ ```bash
584
+ ja4proxy ca --export /tmp/ja4proxy-ca.crt
585
+ sudo cp /tmp/ja4proxy-ca.crt /usr/local/share/ca-certificates/
586
+ sudo update-ca-certificates
587
+ ```
588
+
589
+ #### macOS
590
+ ```bash
591
+ ja4proxy ca --export /tmp/ja4proxy-ca.crt
592
+ sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /tmp/ja4proxy-ca.crt
593
+ ```
594
+
595
+ #### Windows (PowerShell Administrator)
596
+ ```powershell
597
+ ja4proxy ca --export "$env:TEMP\ja4proxy-ca.crt"
598
+ certutil -addstore -f "Root" "$env:TEMP\ja4proxy-ca.crt"
599
+ ```
600
+
601
+ ---
602
+
603
+ ## Security & Engineering Boundaries
604
+
605
+ 1. **Pre-Flight SSRF Protection**:
606
+ `ja4proxy` resolves hostnames prior to outbound dispatch and drops requests resolving to private address spaces (RFC 1918 `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), loopback (`127.0.0.0/8`, `::1`), and cloud metadata (`169.254.169.254`). Internal forwarding can be enabled with `--allow-private-networks`.
607
+
608
+ 2. **Decompression & Content-Length Sync**:
609
+ The native Go engine decodes gzip/brotli/deflate responses transparently. `ja4proxy` strips stale `Content-Encoding` and `Transfer-Encoding` headers and recalculates accurate `Content-Length` before serving downstream, preventing double-decompression exceptions.
610
+
611
+ 3. **Header Sanitization**:
612
+ Incoming client hop-by-hop headers (`Proxy-Connection`, `Keep-Alive`, `Te`) and automated OpenSSL/Python identifiers are stripped. Canonical browser headers (`User-Agent`, `Sec-CH-UA`, `Sec-CH-UA-Mobile`, `Sec-CH-UA-Platform`) matching the selected profile are automatically synthesized when missing.
613
+
614
+ 4. **Zero-GIL Execution**:
615
+ Native C FFI functions execute outside Python's GIL. Concurrency scales across available CPU cores and network threads without interpreter bottlenecks.
616
+
617
+ ---
618
+
619
+ ## CLI Reference
620
+
621
+ ```text
622
+ Usage: ja4proxy [OPTIONS] COMMAND [ARGS]...
623
+
624
+ ja4proxy: High-performance universal forward proxy server with browser TLS (JA4)
625
+ and HTTP/2 impersonation.
626
+
627
+ Options:
628
+ --version Show the version and exit.
629
+ --help Show this message and exit.
630
+
631
+ Commands:
632
+ ca Inspects or exports the local Root CA certificate.
633
+ profiles Lists all supported browser profiles for TLS and HTTP/2.
634
+ start Starts the ja4proxy server.
635
+ test Performs a direct live handshake against tls.peet.ws.
636
+ ```
637
+
638
+ ### `ja4proxy start` Flags
639
+
640
+ | Option | Default | Description |
641
+ | :--- | :--- | :--- |
642
+ | `--host` | `127.0.0.1` | Network interface to bind |
643
+ | `--port` | `8888` | Port to listen for HTTP, HTTPS CONNECT, and SOCKS5 |
644
+ | `--profile` | `chrome_133` | Default browser profile |
645
+ | `--upstream-proxy` | None | Global upstream forwarding proxy |
646
+ | `--auth` | None | Require username:password authentication for clients |
647
+ | `--allow-private-networks` | False | Permits forwarding to private/internal IPs |
648
+ | `--ca-dir` | `~/.ja4proxy` | Custom directory for CA certificates |
649
+
650
+ ---
651
+
652
+ ## Automated Verification Suite
653
+
654
+ Run the full automated test suite covering CA generation, SOCKS5 state machines, chunked request decoding, SSRF guards, and live upstream verification against `tls.peet.ws` and `httpbin.org`:
655
+
656
+ ```bash
657
+ python -m unittest discover -s tests -p "test_*.py" -v
658
+ ```
659
+
660
+ All 18 unit and live tests execute with zero warnings and zero regressions:
661
+ ```text
662
+ test_ca_creation (test_certs.TestCertificateAuthority) ... ok
663
+ test_leaf_cert_generation (test_certs.TestCertificateAuthority) ... ok
664
+ test_leaf_cert_ip_address (test_certs.TestCertificateAuthority) ... ok
665
+ test_live_handshake_chrome (test_engine.TestNativeEngine) ... ok
666
+ test_live_post_httpbin (test_engine.TestNativeEngine) ... ok
667
+ test_bad_gateway_mapping_status_502 (test_live_proxy.TestLiveProxy) ... ok
668
+ test_chunked_request_upload (test_live_proxy.TestLiveProxy) ... ok
669
+ test_client_proxy_authentication (test_live_proxy.TestLiveProxy) ... ok
670
+ test_gateway_mode_gzip (test_live_proxy.TestLiveProxy) ... ok
671
+ test_safari_ios_headers (test_live_proxy.TestLiveProxy) ... ok
672
+ test_socks5_ingress_tls_peet (test_live_proxy.TestLiveProxy) ... ok
673
+ test_ssrf_protection_blocked (test_live_proxy.TestLiveProxy) ... ok
674
+ test_ssrf_restricted_addresses (test_live_proxy.TestLiveProxy) ... ok
675
+ test_extract_directives_from_headers (test_parser.TestParser) ... ok
676
+ test_extract_directives_priority_header_over_auth (test_parser.TestParser) ... ok
677
+ test_parse_credentials_empty (test_parser.TestParser) ... ok
678
+ test_parse_credentials_proxy_with_hyphens (test_parser.TestParser) ... ok
679
+ test_parse_credentials_tags (test_parser.TestParser) ... ok
680
+
681
+ ----------------------------------------------------------------------
682
+ Ran 18 tests in 6.164s
683
+
684
+ OK
685
+ ```
686
+
687
+ ---
688
+
689
+ ## License
690
+
691
+ [MIT](file:///D:/what%20to%20make/LICENSE) License. Designed and engineered by `cipherwire` / `hainahola`.