@rester159/blacktip 0.1.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/CHANGELOG.md +190 -0
  2. package/README.md +95 -0
  3. package/dist/behavioral/parsers.d.ts +89 -0
  4. package/dist/behavioral/parsers.d.ts.map +1 -0
  5. package/dist/behavioral/parsers.js +223 -0
  6. package/dist/behavioral/parsers.js.map +1 -0
  7. package/dist/blacktip.d.ts +86 -0
  8. package/dist/blacktip.d.ts.map +1 -1
  9. package/dist/blacktip.js +193 -0
  10. package/dist/blacktip.js.map +1 -1
  11. package/dist/browser-core.d.ts.map +1 -1
  12. package/dist/browser-core.js +125 -33
  13. package/dist/browser-core.js.map +1 -1
  14. package/dist/diagnostics.d.ts +150 -0
  15. package/dist/diagnostics.d.ts.map +1 -0
  16. package/dist/diagnostics.js +389 -0
  17. package/dist/diagnostics.js.map +1 -0
  18. package/dist/identity-pool.d.ts +160 -0
  19. package/dist/identity-pool.d.ts.map +1 -0
  20. package/dist/identity-pool.js +288 -0
  21. package/dist/identity-pool.js.map +1 -0
  22. package/dist/index.d.ts +7 -0
  23. package/dist/index.d.ts.map +1 -1
  24. package/dist/index.js +8 -0
  25. package/dist/index.js.map +1 -1
  26. package/dist/tls-side-channel.d.ts +82 -0
  27. package/dist/tls-side-channel.d.ts.map +1 -0
  28. package/dist/tls-side-channel.js +241 -0
  29. package/dist/tls-side-channel.js.map +1 -0
  30. package/dist/types.d.ts +26 -0
  31. package/dist/types.d.ts.map +1 -1
  32. package/dist/types.js.map +1 -1
  33. package/docs/akamai-bypass.md +257 -0
  34. package/docs/anti-bot-validation.md +84 -0
  35. package/docs/calibration-validation.md +93 -0
  36. package/docs/identity-pool.md +176 -0
  37. package/docs/tls-side-channel.md +83 -0
  38. package/native/tls-client/go.mod +21 -0
  39. package/native/tls-client/go.sum +36 -0
  40. package/native/tls-client/main.go +216 -0
  41. package/package.json +8 -2
  42. package/scripts/fit-cmu-keystroke.mjs +186 -0
@@ -0,0 +1,84 @@
1
+ # Anti-bot validation scoreboard
2
+
3
+ This document records BlackTip's results against commercial anti-bot vendors on real, in-the-wild targets. Every entry is generated by `bt.testAgainstAntiBot(url)`, which both detects challenge/block pages AND captures vendor signals (cookies, scripts) on a passing page so we can prove the target is actually protected — not a false negative on an unprotected URL.
4
+
5
+ Methodology and reproduction recipe at the bottom.
6
+
7
+ ## Live scoreboard — 2026-04-09 (BlackTip 0.2.0)
8
+
9
+ | Target | Vendors detected on page | Block? | Vendor signals on success | Notes |
10
+ |---|---|---|---|---|
11
+ | **vinted.com** | DataDome | pass | datadome cookie, cloudflare script | Real catalog renders with prices, brands, listings |
12
+ | **bestbuy.com** | Akamai | pass | akamai cookie + script | Earlier classification as PerimeterX was wrong — BestBuy is Akamai |
13
+ | **walmart.com** | Akamai + PerimeterX | pass | akamai cookie + script, perimeterx cookie + script | Walmart runs both vendors simultaneously; BlackTip slides past both |
14
+ | **crunchbase.com** | Cloudflare | pass | cloudflare script | Real homepage content visible |
15
+ | **ticketmaster.com** | none detected on homepage | pass | none on `/` | TM only arms PerimeterX on event/checkout pages, not the marketing homepage |
16
+ | **opentable.com** (Gjelina deep link, restref=76651) | Akamai | pass | akamai cookie + script | Real time slots render; Reservation at Gjelina, Apr 11 2026 |
17
+ | **chatgpt.com** | Cloudflare | pass | cloudflare script + cf_clearance cookie | **Cloudflare managed challenge auto-passed**; cf_clearance issued silently |
18
+ | **twitch.tv** | Kasada | pass | kasada script signal | **Kasada validated** — first published BlackTip pass against a real Kasada-armed target |
19
+ | **canadagoose.com** | (no live signals on homepage) | pass | none | Was historically Kasada but the homepage no longer surfaces Kasada cookies/scripts; may be lazy-loaded on cart/checkout |
20
+ | **hyatt.com** | (no live signals on landing page) | pass | none on `/loyalty/en-US` | RT (Akamai mPulse RUM) present, no Bot Manager indicators |
21
+ | **footlocker.com** | (no live signals on homepage) | pass | none | Same as Canada Goose — Kasada may arm only on PDP/cart |
22
+ | **datadome.co/bot-tester** | DataDome (bypassed via marketing redirect) | pass | none captured | Redirected to /signup/ with marketing content; DataDome did not arm a challenge |
23
+ | **antoinevastel.com/bots-vue.html** | none | pass | none | Demo page no longer arms a probe — author moved to a marketing homepage |
24
+ | **nowsecure.nl** (Cloudflare bot fight, nodriver author's benchmark) | none | pass | none on this barebones page | Passes regression — earlier validation in v0.1.0 |
25
+
26
+ ### Cloudflare managed challenge — silent auto-pass evidence
27
+
28
+ The `cf_clearance` cookie is set ONLY after Cloudflare's managed challenge accepts a request as human. Across the v0.3.0 validation run, BlackTip earned `cf_clearance` cookies on multiple Cloudflare-protected domains without ever surfacing a "Just a moment..." interstitial to the user:
29
+
30
+ - `.vinted.com` (cf_clearance httpOnly)
31
+ - `.crunchbase.com` (cf_clearance httpOnly)
32
+ - `.chatgpt.com` (cf_clearance httpOnly + cf_bm + cfuvid)
33
+
34
+ These cookies are not visible to `document.cookie` because they are httpOnly — the v0.2.0 detector missed them. v0.3.0 fixed the detector to read via the BlackTip cookies API, surfacing this evidence. **A `cf_clearance` cookie is the strongest possible proof of a Cloudflare managed-challenge pass on a real Chrome session.**
35
+
36
+ **Eight commercial-detector targets, eight passes.** The most load-bearing validations are Walmart (Akamai + PerimeterX simultaneously) and OpenTable (Akamai's full Bot Manager on a high-value booking endpoint that previously hard-blocked v0.1.0).
37
+
38
+ ## What "passing" means here
39
+
40
+ `bt.testAgainstAntiBot(url)` returns `passed: true` when:
41
+ 1. The page title does not match a known vendor block pattern (Akamai "Access Denied", Cloudflare "Just a moment...", PerimeterX "Press & Hold", DataDome captcha-delivery interstitial, Imperva incident page, etc.)
42
+ 2. The body text does not contain vendor block markers
43
+ 3. Real page content is rendered (verifiable in `bodyPreview`)
44
+
45
+ The `vendorSignals` field separately reports what vendor cookies and scripts are present even on a passing page. If a target shows `vendorSignals: []`, either the vendor doesn't arm protection on that URL (Ticketmaster homepage), or BlackTip's signal patterns missed something — note both honestly here.
46
+
47
+ ## Vendors recognised
48
+
49
+ | Vendor | Block-page tells | Cookie tells | Script tells |
50
+ |---|---|---|---|
51
+ | Akamai Bot Manager | title `Access Denied`, body `errors.edgesuite.net`, `Reference #...` | `_abck`, `bm_sz`, `ak_bmsc`, `bm_sv` | `akam/`, `ak.bmpsdk`, `akamaihd.net/sensor` |
52
+ | DataDome | body `geo.captcha-delivery.com`, `datado.me` | `datadome`, `dd_s`, `dd_cookie_test_` | `js.datadome.co`, `datado.me` |
53
+ | Cloudflare Bot Fight / Turnstile | title `Just a moment...`, body `cf-browser-verification`, `Sorry, you have been blocked` | `cf_clearance`, `__cf_bm`, `__cflb` | `challenges.cloudflare.com`, `cdn-cgi/challenge-platform` |
54
+ | HUMAN / PerimeterX | title `Access to this page has been denied`, body `Press & Hold` | `_px*`, `_pxhd` | `perimeterx`, `px-cdn`, `px-captcha`, `human-security` |
55
+ | Imperva / Incapsula | body `Request unsuccessful. Incapsula incident ID` | `visid_incap_`, `incap_ses_` | (mostly server-side) |
56
+ | Kasada | body `kpsdk` | (none captured) | `x-kpsdk-` headers, `ips.js` |
57
+ | Arkose Labs / FunCaptcha | body `client-api.arkoselabs.com`, `funcaptcha` | (none captured) | `client-api.arkoselabs.com` |
58
+
59
+ ## How to reproduce
60
+
61
+ ```bash
62
+ # Terminal 1
63
+ cd /path/to/blacktip
64
+ node dist/cli.js serve
65
+
66
+ # Terminal 2 — single target
67
+ node dist/cli.js send "return await bt.testAgainstAntiBot('https://www.vinted.com/')" --pretty
68
+
69
+ # Or batch:
70
+ node dist/cli.js batch '[
71
+ "return await bt.testAgainstAntiBot(\"https://www.vinted.com/\")",
72
+ "return await bt.testAgainstAntiBot(\"https://www.bestbuy.com/\")",
73
+ "return await bt.testAgainstAntiBot(\"https://www.walmart.com/\")",
74
+ "return await bt.testAgainstAntiBot(\"https://www.crunchbase.com/\")",
75
+ "return await bt.testAgainstAntiBot(\"https://www.opentable.com/booking/restref/availability?rid=76651&restref=76651&partySize=2&dateTime=2026-04-11T19%3A00\")"
76
+ ]'
77
+ ```
78
+
79
+ ## Caveats
80
+
81
+ - **One-shot probes.** This scoreboard records single navigations from a residential IP. Anti-bot vendors profile sessions over time and across requests; a target that passes a one-shot probe may still flag a multi-step automation flow where the behavioral signature looks too clean. The deep validation is the OpenTable booking flow itself, where we drove the form to completion through Akamai's Bot Manager.
82
+ - **Vendor classification can be wrong.** Targets like Walmart run multiple vendors in parallel. The `vendorSignals` field is the source of truth; the "expected vendor" column above is informational.
83
+ - **IP matters.** Run from a residential network or known-clean residential proxy. A datacenter IP will fail every entry on this scoreboard regardless of how good BlackTip's stealth is. Use `bt.checkIpReputation()` first.
84
+ - **The scoreboard goes stale.** Vendors update their detection logic constantly. Re-run on each release and update this doc. If a target moves from pass to fail, file it against the next BlackTip version, capture the failing fingerprint, and dig in.
@@ -0,0 +1,93 @@
1
+ # Behavioral calibration validation (v0.3.0)
2
+
3
+ This document records the result of fitting BlackTip's behavioral profile against the real CMU Keystroke Dynamics dataset (Killourhy & Maxion 2009) and validating the fit against held-out subjects.
4
+
5
+ ## TL;DR
6
+
7
+ The calibrated profile measurably beats BlackTip's canonical `HUMAN_PROFILE` on a held-out subject set:
8
+
9
+ | Metric | Canonical KS distance | Calibrated KS distance | Improvement |
10
+ |---|---|---|---|
11
+ | **Hold time** | 0.4297 | 0.2018 | **53% closer to real humans** |
12
+ | **Flight time** | 0.4811 | 0.4152 | 13.7% closer to real humans |
13
+
14
+ This is the first time the BlackTip behavioral pipeline has been validated end-to-end against a real public dataset. Up through v0.2.0, the engine's parameters were sane defaults; v0.3.0 makes them empirically grounded.
15
+
16
+ ## Methodology
17
+
18
+ 1. **Dataset**: CMU Keystroke Dynamics (`DSL-StrongPasswordData.csv`) — 51 subjects each typing the fixed phrase `.tie5Roanl` 50 times across 8 sessions, for 20,400 total phrase reps.
19
+ 2. **Split**: deterministic 80/20 by subject. 40 subjects (16,000 phrases) → training. 11 subjects (4,400 phrases) → held-out test.
20
+ 3. **Fit**: training set → `fitTypingDynamics()` → empirical hold-time and flight-time distributions plus per-digraph latencies.
21
+ 4. **Compare**: synthesized 5,000 samples from each of (a) BlackTip's canonical `HUMAN_PROFILE` ranges and (b) the fitted `[p5, p95]` ranges. Computed Kolmogorov–Smirnov distance (max empirical CDF gap) against the held-out test set.
22
+ 5. **Report**: lower KS distance → closer to real human distribution.
23
+
24
+ The KS test is the standard non-parametric goodness-of-fit measure. It does not assume any particular distribution shape, which matters here because keystroke timings are right-skewed log-normal-ish, not Gaussian. The improvement ratio is `1 - calibrated / canonical`.
25
+
26
+ ## Fitted parameters
27
+
28
+ ```
29
+ Hold time:
30
+ mean = 90.3 ms
31
+ p5 = 48.3 ms
32
+ p50 = 85.8 ms
33
+ p95 = 148.8 ms
34
+
35
+ Flight time:
36
+ mean = 151.4 ms
37
+ p5 = 0.0 ms (some adjacent keystrokes overlap — concurrent press/release)
38
+ p50 = 91.3 ms
39
+ p95 = 513.5 ms
40
+
41
+ Digraphs fit: 6 (the unique a–z transitions in the phrase)
42
+ ```
43
+
44
+ The fitted profile is saved to `data/cmu-keystroke/calibrated-profile.json` and ready to load:
45
+
46
+ ```typescript
47
+ import calibrated from './data/cmu-keystroke/calibrated-profile.json' with { type: 'json' };
48
+ import { BlackTip } from '@rester159/blacktip';
49
+
50
+ const bt = new BlackTip({
51
+ behaviorProfile: calibrated.profileConfig,
52
+ // ... rest of your config
53
+ });
54
+ ```
55
+
56
+ ## Why this matters
57
+
58
+ Behavioral biometrics services (BioCatch, NuData, SecuredTouch) profile users on dimensions like:
59
+
60
+ - Hold time mean and variance per key
61
+ - Flight time distributions per digraph
62
+ - Tap pressure (mobile only — n/a here)
63
+ - Mouse curvature, click dwell, scroll deceleration
64
+
65
+ A bot that types with uniform 100 ms holds and flat flight times stands out instantly because real humans have right-skewed log-normal distributions with subject-specific clustering. BlackTip's canonical `HUMAN_PROFILE` was already in the right ballpark, but the canonical hold-time range `[50, 200]` was 53% farther from the real distribution than the empirically-fitted `[48, 149]`. The fitted range is tighter and centered correctly, so BlackTip's keystroke output now sits inside the real human distribution rather than scattered across a too-wide canonical range.
66
+
67
+ ## Reproducing the result
68
+
69
+ ```bash
70
+ cd /path/to/blacktip
71
+ mkdir -p data/cmu-keystroke
72
+ curl -fsSL -o data/cmu-keystroke/DSL-StrongPasswordData.csv \
73
+ https://www.cs.cmu.edu/~keystroke/DSL-StrongPasswordData.csv
74
+ npm run build
75
+ node scripts/fit-cmu-keystroke.mjs
76
+ ```
77
+
78
+ The script writes its output to `data/cmu-keystroke/calibrated-profile.json` and prints the validation table to stdout. Re-runs are deterministic (the train/test split is sorted, not random) so the numbers match this document byte-for-byte.
79
+
80
+ ## What this does NOT prove
81
+
82
+ - The KS test compares marginal distributions, not joint ones. A profile that matches the marginals perfectly could still have unrealistic correlation structure (e.g. correct hold times but uncorrelated with flight times). A real biometrics test against a commercial service would catch this; we don't have one.
83
+ - The CMU dataset is 50 reps of one fixed phrase from each of 51 American English typists. The fitted profile generalises best to American English long-form typing; non-Latin scripts and very short fields may need a different calibration.
84
+ - Flight time fit improvement (13.7%) is much smaller than hold time (53%). The CMU phrase is short and contains transitions that aren't representative of free-text typing — the held-out flights span a wide range that the canonical `[80, 150]` and fitted `[0, 514]` are both bad fits for. A larger free-text dataset (e.g. Buffalo or GREYC) would likely produce a better flight fit. Future work.
85
+
86
+ ## Future calibration sources
87
+
88
+ Once a parser exists for each, the same pipeline applies:
89
+
90
+ - **Balabit Mouse Dynamics Challenge** — for `fitMouseDynamics()`. Parser exists in `parseBalabitMouseCsv()`; needs an actual fit run against the real dataset.
91
+ - **GREYC-NISLAB** — free-text keystroke dynamics from 110 subjects. Better representative coverage than CMU's fixed phrase.
92
+ - **Buffalo Free-Text** — multi-session keystroke data across 148 subjects. The canonical reference for keystroke behavioral biometrics literature.
93
+ - Your own telemetry — `parseGenericTelemetryJson()` accepts the normalized `MouseMovement` / `TypingSession` shapes directly. Bring your own data.
@@ -0,0 +1,176 @@
1
+ # IdentityPool — long-running session and identity rotation (v0.4.0)
2
+
3
+ `IdentityPool` is BlackTip's answer to the question "how do I rotate across many identities cleanly without my whole flow looking like one bot retried under different IPs?" An identity is the union of everything that makes a session look like one specific human: cookies, localStorage, proxy, device profile, behavior profile, locale, timezone. The pool persists to a JSON file so identities survive restarts, and each identity has a per-domain burn list so an identity blocked on opentable.com is still eligible for amazon.com.
4
+
5
+ ## When you need this
6
+
7
+ Most BlackTip flows do not need an IdentityPool. A single launch with the right device profile and a residential connection covers the common case. The pool earns its keep when:
8
+
9
+ 1. You're running many flows against the same target and need to look like many different users (price scraping, market research, multi-account ops on services where multi-account is allowed).
10
+ 2. You want resilience: when identity A gets blocked on opentable.com, you want identity B to take over without manual intervention.
11
+ 3. You want session persistence across process restarts so a logged-in identity from yesterday is still logged in today.
12
+ 4. You want a feedback loop: when an identity gets burned, the proxy bound to it should be marked dirty in `ProxyPool` so it isn't reused for the same target until the ban window decays.
13
+
14
+ ## Composition
15
+
16
+ `IdentityPool` does not reinvent persistence or proxy selection. It composes:
17
+
18
+ - **`SnapshotManager`** for cookies + localStorage + sessionStorage. The pool calls `captureSnapshot(bt, identity)` after a successful flow to save state.
19
+ - **`ProxyPool`** for proxy selection and ban tracking. New identities draw a proxy from the pool at creation time. When an identity is burned per-domain, the pool reports a ban on that proxy/domain pair so future selections skip it.
20
+ - **`BlackTipConfig`** is produced by `pool.applyToConfig(identity)` and passed to `new BlackTip(config)`.
21
+
22
+ ## Quick start
23
+
24
+ ```typescript
25
+ import { BlackTip, IdentityPool, ProxyPool, ProxyProviders } from '@rester159/blacktip';
26
+
27
+ // 1. Build a ProxyPool from whatever provider you use.
28
+ const proxyPool = new ProxyPool([
29
+ ProxyProviders.brightData('your-customer-id', 'your-password', 'residential'),
30
+ ProxyProviders.oxylabs('your-username', 'your-password'),
31
+ ]);
32
+
33
+ // 2. Build the IdentityPool, backed by a JSON file on disk.
34
+ const pool = new IdentityPool({
35
+ storePath: './.blacktip/identities.json',
36
+ proxyPool,
37
+ rotationPolicy: {
38
+ maxUses: 50, // burn after 50 uses
39
+ maxAgeMs: 7 * 24 * 60 * 60 * 1000, // burn after 7 days idle
40
+ },
41
+ });
42
+
43
+ // 3. First time only: seed the pool with N identities. Subsequent runs
44
+ // load from the store file.
45
+ if (pool.size() === 0) {
46
+ for (let i = 0; i < 5; i++) {
47
+ pool.add({
48
+ deviceProfile: i % 2 === 0 ? 'desktop-windows' : 'desktop-macos',
49
+ label: `identity-${i + 1}`,
50
+ locale: 'en-US',
51
+ timezone: 'America/New_York',
52
+ });
53
+ }
54
+ }
55
+
56
+ // 4. For each flow: acquire, launch, run, capture, release.
57
+ const identity = pool.acquire('opentable.com');
58
+ if (!identity) throw new Error('No eligible identity for opentable.com — pool exhausted');
59
+
60
+ const config = pool.applyToConfig(identity, { logLevel: 'info', timeout: 15_000 });
61
+ const bt = new BlackTip(config);
62
+ await bt.launch();
63
+
64
+ // Restore the identity's prior session (cookies, storage). No-op if first use.
65
+ await pool.restoreSnapshot(bt, identity);
66
+
67
+ try {
68
+ await bt.navigate('https://www.opentable.com/');
69
+ await bt.waitForStable();
70
+ // ... rest of the flow
71
+
72
+ // On success, save the updated session state back into the identity.
73
+ await pool.captureSnapshot(bt, identity);
74
+ } catch (err) {
75
+ // On failure, mark this identity burned for this domain. The proxy
76
+ // gets banned in ProxyPool too, so the next identity drawn from the
77
+ // pool won't reuse the same proxy on this target.
78
+ pool.markBurned(identity.id, err instanceof Error ? err.message : String(err), 'opentable.com');
79
+ } finally {
80
+ await bt.close();
81
+ }
82
+ ```
83
+
84
+ ## API
85
+
86
+ ### `new IdentityPool(options)`
87
+
88
+ ```typescript
89
+ {
90
+ storePath: string; // required — JSON file path
91
+ proxyPool?: ProxyPool; // optional — for proxy binding & feedback
92
+ rotationPolicy?: {
93
+ maxUses?: number; // default: Infinity
94
+ maxAgeMs?: number; // default: Infinity
95
+ preferLeastRecentlyUsed?: boolean; // default: true
96
+ };
97
+ }
98
+ ```
99
+
100
+ ### `add(init)` → `Identity`
101
+
102
+ Create a new identity. `deviceProfile` is required. If a `proxyPool` was supplied to the IdentityPool and `proxy` is omitted, the pool draws one. Auto-saves to disk.
103
+
104
+ ### `acquire(domain?)` → `Identity | null`
105
+
106
+ Pick an identity for use. Skips identities burned on the requested domain. Applies rotation policy: identities exceeding `maxUses` or `maxAgeMs` are auto-burned. Returns null if no eligible identity exists.
107
+
108
+ ### `markBurned(id, reason, domain?)` → `boolean`
109
+
110
+ Mark an identity burned. With `domain`, only burns for that domain (per-domain burn list). Without `domain`, fully burns the identity. Per-domain burns also report a proxy ban back to `ProxyPool` if one is wired up.
111
+
112
+ ### `clearBurn(id, domain?)` → `boolean`
113
+
114
+ Manually unban. Useful when you know the burn was a transient issue.
115
+
116
+ ### `applyToConfig(identity, baseConfig?)` → `BlackTipConfig`
117
+
118
+ Build a `BlackTipConfig` from an identity. Sets `deviceProfile`, `behaviorProfile`, `locale`, `timezone`, and `proxy` (URL-formatted via `proxyToUrl`). Other base config fields pass through unchanged.
119
+
120
+ ### `restoreSnapshot(bt, identity)` → `Promise<void>`
121
+
122
+ After `bt.launch()`, apply the identity's saved cookies + localStorage to the running browser. No-op if the identity has no snapshot yet.
123
+
124
+ ### `captureSnapshot(bt, identity)` → `Promise<void>`
125
+
126
+ Save the current BlackTip session state into the identity. Call after a successful flow so the next acquire of this identity starts from a known-good logged-in state.
127
+
128
+ ### `list()`, `available()`, `size()`, `remove(id)`
129
+
130
+ Standard inspection. `available()` returns identities not fully burned (per-domain burns don't count).
131
+
132
+ ## Persistence format
133
+
134
+ The store file is plain JSON with a schema version. Sample:
135
+
136
+ ```json
137
+ {
138
+ "version": 1,
139
+ "savedAt": "2026-04-10T22:30:00.000Z",
140
+ "identities": [
141
+ {
142
+ "id": "1f2e3d4c-...",
143
+ "label": "identity-1",
144
+ "createdAt": "2026-04-10T20:00:00.000Z",
145
+ "lastUsedAt": "2026-04-10T22:25:00.000Z",
146
+ "useCount": 12,
147
+ "burnedAt": null,
148
+ "burnedReason": null,
149
+ "burnedDomains": ["sears.com"],
150
+ "snapshot": { /* SessionSnapshot */ },
151
+ "proxy": { "id": "brightdata-residential", "...": "..." },
152
+ "behaviorProfile": "human",
153
+ "deviceProfile": "desktop-windows",
154
+ "locale": "en-US",
155
+ "timezone": "America/New_York"
156
+ }
157
+ ]
158
+ }
159
+ ```
160
+
161
+ The schema is versioned so future migrations are explicit. Don't hand-edit the file while a process is reading it — use the API.
162
+
163
+ ## IP reputation gate
164
+
165
+ v0.4.0 also adds `BlackTipConfig.requireResidentialIp`. When set, BlackTip runs `bt.checkIpReputation()` immediately after `launch()` and either warns or throws based on the verdict:
166
+
167
+ ```typescript
168
+ new BlackTip({
169
+ // 'throw': refuse to launch if egress IP is on a known datacenter ASN.
170
+ // 'warn': log a warning but allow the launch.
171
+ // false / unset: no check.
172
+ requireResidentialIp: 'throw',
173
+ });
174
+ ```
175
+
176
+ Use `'throw'` in production / CI where a flagged IP would burn a real account. Use `'warn'` for local dev. Combine with `IdentityPool` and `ProxyPool` to ensure every launch goes through a residential exit before touching the target.
@@ -0,0 +1,83 @@
1
+ # TLS side-channel (v0.3.0)
2
+
3
+ The v0.3.0 answer to "an edge gates the very first request before BlackTip's browser even has a session." BlackTip ships a Go-based daemon built on `bogdanfinn/tls-client` that performs HTTP requests with a real Chrome TLS ClientHello, real H2 frame settings, and real H2 frame order. You use it to make gating requests the browser can't make through itself, then inject the resulting cookies into the browser session before navigating.
4
+
5
+ ## When you need this
6
+
7
+ Most BlackTip flows do not need the TLS side-channel. The browser's own TLS via `channel: 'chrome'` is real Chrome and passes every detector we've validated against. The side-channel is for the cases where it isn't enough:
8
+
9
+ 1. **First-request edge gating.** Some Akamai-protected sites refuse to serve a session cookie to a navigation that doesn't already have one. You hit them via the side-channel first (which goes through `bm_s` → sensor data POST → `bm_sv` issuance), then inject the resulting cookies into the browser and navigate normally.
10
+ 2. **Cross-platform User-Agent spoofing.** You're running on Linux but want the target to see Windows. The browser's TLS comes from the host OS Chrome, so you can't fake the platform without a TLS rewriter. The side-channel can. v0.2.0's L016 fix removed the broken UA-override path; this is the supported alternative.
11
+ 3. **API-level operations.** You want to call a JSON API the site exposes, but the API edge enforces the same TLS profile as the browser. Use the side-channel to call the API without paying browser-render overhead per request.
12
+ 4. **Pre-warming for proxy rotation.** You're rotating residential proxies and need to "warm" each new IP with a Chrome-TLS handshake before the browser session uses it.
13
+
14
+ ## Build the daemon
15
+
16
+ The daemon is a small Go program in `native/tls-client/`. Go is the only build dependency. Install Go from https://go.dev/dl/ then:
17
+
18
+ ```bash
19
+ cd native/tls-client
20
+ go build -o blacktip-tls . # Linux / macOS
21
+ go build -o blacktip-tls.exe . # Windows
22
+ ```
23
+
24
+ The build produces a single ~14 MB statically-linked binary. BlackTip resolves it via `native/tls-client/blacktip-tls[.exe]` automatically; override with `BLACKTIP_TLS_BIN=/abs/path/to/binary` if you want to ship it elsewhere.
25
+
26
+ ## Usage
27
+
28
+ ```typescript
29
+ import { BlackTip } from '@rester159/blacktip';
30
+
31
+ const bt = new BlackTip({ logLevel: 'info' });
32
+ await bt.launch();
33
+
34
+ // 1. Make the gating request through the side-channel.
35
+ const resp = await bt.fetchWithTls({
36
+ url: 'https://www.opentable.com/',
37
+ headers: {
38
+ 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36',
39
+ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
40
+ 'Accept-Language': 'en-US,en;q=0.9',
41
+ },
42
+ });
43
+ console.log('TLS fetch status:', resp.status);
44
+ console.log('Earned cookies:', resp.cookies.map(c => c.name));
45
+
46
+ // 2. Inject the cookies into the browser session.
47
+ const injected = await bt.injectTlsCookies(resp, 'https://www.opentable.com/');
48
+ console.log('Injected', injected, 'cookies');
49
+
50
+ // 3. Navigate normally — the browser carries the side-channel-earned tokens.
51
+ await bt.navigate('https://www.opentable.com/');
52
+ await bt.waitForStable();
53
+ // ... rest of flow
54
+
55
+ await bt.close(); // Closes the TLS daemon too
56
+ ```
57
+
58
+ ## Architecture
59
+
60
+ - **Wire protocol**: newline-delimited JSON over the daemon's stdin/stdout. Each request has a string `id`; responses match by id, so multiple in-flight `fetch()` calls don't interleave.
61
+ - **Daemon lifecycle**: spawned lazily on first `fetchWithTls()` call, kept alive until `bt.close()`. No subprocess startup cost per request.
62
+ - **Concurrency**: the daemon handles each request in its own Go goroutine. The Node side dispatches them in parallel and reassembles by id.
63
+ - **Profile selection**: defaults to `chrome_133`. Override per-request via `bt.fetchWithTls({ url, profile: 'chrome_124' })`. Available profiles are whatever `bogdanfinn/tls-client/profiles` ships — at the time of writing: `chrome_120`, `chrome_124`, `chrome_131`, `chrome_133`, `firefox_120`, `safari_ios_16_0`.
64
+ - **Body encoding**: bodies are base64 on the wire to avoid newline / Unicode issues with the line-delimited protocol. The wrapper handles encoding/decoding transparently.
65
+
66
+ ## Validated TLS fingerprint
67
+
68
+ Run via the integration test (`tests/tls-side-channel.integration.test.ts`):
69
+
70
+ ```
71
+ JA4: t13d1516h2_8daaf6152771_d8a2da3f94cd
72
+ First cipher: TLS_GREASE (0x5A5A)
73
+ HTTP/2 Akamai fingerprint: 1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p
74
+ ```
75
+
76
+ This is byte-for-byte identical to real Chrome 133. The leading `t13d` JA4 prefix confirms TLS 1.3 with Chrome's count signature; the GREASE first cipher confirms proper GREASE rotation; the H2 fingerprint matches Chrome's frame settings and frame order (`m,a,s,p` = method/authority/scheme/path).
77
+
78
+ ## Caveats
79
+
80
+ - **Not a Chrome browser.** The side-channel is HTTP-only — no JavaScript execution, no DOM, no Cookie JAR persistence beyond what you inject manually. You use it to acquire tokens, not to drive a flow.
81
+ - **Akamai sensor data is not bypassed.** A first request to an Akamai-protected URL through the side-channel returns 403 with `bm_s`/`bm_ss`/`bm_so` set — the same place a browser would be at after one request. Akamai expects a sensor data POST to follow before serving 200s. The side-channel collects the session cookies; the sensor POST is not yet automated. You can either (a) inject the bm_s cookies into the browser and let the browser do the sensor POST (this is what the OpenTable validation flow does), or (b) implement the sensor POST yourself.
82
+ - **Always set the User-Agent header.** The Go HTTP client defaults to `Go-http-client/2.0` if you don't override it. That's a textbook automation tell. The wrapper auto-sets `Accept-Language` to `en-US,en;q=0.9` if missing, but UA is your job.
83
+ - **Linux/macOS support is built but not validated in CI.** The `go build` command produces native binaries for whatever platform you run it on; the Node-side `resolveBinaryPath()` picks the right name based on `process.platform`. If you ship to a server, build the daemon in the same OS the server runs.
@@ -0,0 +1,21 @@
1
+ module github.com/rester159/blacktip/native/tls-client
2
+
3
+ go 1.26.2
4
+
5
+ require (
6
+ github.com/andybalholm/brotli v1.2.0 // indirect
7
+ github.com/bdandy/go-errors v1.2.2 // indirect
8
+ github.com/bdandy/go-socks4 v1.2.3 // indirect
9
+ github.com/bogdanfinn/fhttp v0.6.8 // indirect
10
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
11
+ github.com/bogdanfinn/tls-client v1.14.0 // indirect
12
+ github.com/bogdanfinn/utls v1.7.7-barnius // indirect
13
+ github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
14
+ github.com/klauspost/compress v1.18.2 // indirect
15
+ github.com/quic-go/qpack v0.6.0 // indirect
16
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
17
+ golang.org/x/crypto v0.46.0 // indirect
18
+ golang.org/x/net v0.48.0 // indirect
19
+ golang.org/x/sys v0.39.0 // indirect
20
+ golang.org/x/text v0.32.0 // indirect
21
+ )
@@ -0,0 +1,36 @@
1
+ github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
2
+ github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
3
+ github.com/bdandy/go-errors v1.2.2 h1:WdFv/oukjTJCLa79UfkGmwX7ZxONAihKu4V0mLIs11Q=
4
+ github.com/bdandy/go-errors v1.2.2/go.mod h1:NkYHl4Fey9oRRdbB1CoC6e84tuqQHiqrOcZpqFEkBxM=
5
+ github.com/bdandy/go-socks4 v1.2.3 h1:Q6Y2heY1GRjCtHbmlKfnwrKVU/k81LS8mRGLRlmDlic=
6
+ github.com/bdandy/go-socks4 v1.2.3/go.mod h1:98kiVFgpdogR8aIGLWLvjDVZ8XcKPsSI/ypGrO+bqHI=
7
+ github.com/bogdanfinn/fhttp v0.6.8 h1:LiQyHOY3i0QoxxNB7nq27/nGNNbtPj0fuBPozhR7Ws4=
8
+ github.com/bogdanfinn/fhttp v0.6.8/go.mod h1:A+EKDzMx2hb4IUbMx4TlkoHnaJEiLl8r/1Ss1Y+5e5M=
9
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls h1:tV6eDEiRbRCcepALSzxR94JUVD3N3ACIiRLgyc2Ep8s=
10
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls/go.mod h1:aHph9B9H9yPOt5xnhWKSOum27DJAqpiHzwX+gjvaXcg=
11
+ github.com/bogdanfinn/tls-client v1.14.0 h1:vyk7Cn4BIvLAGVuMfb0tP22OqogfO1lYamquQNEZU1A=
12
+ github.com/bogdanfinn/tls-client v1.14.0/go.mod h1:LsU6mXVn8MOFDwTkyRfI7V1BZM1p0wf2ZfZsICW/1fM=
13
+ github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5ZlrCR7fOLU=
14
+ github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
15
+ github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
16
+ github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
17
+ github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
18
+ github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
19
+ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
20
+ github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
21
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
22
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
23
+ golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
24
+ golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
25
+ golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
26
+ golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
27
+ golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
28
+ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
29
+ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
30
+ golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
31
+ golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
32
+ golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
33
+ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
34
+ golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
35
+ golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
36
+ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=