@rester159/blacktip 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -1
- package/README.md +4 -0
- package/dist/akamai-sensor.d.ts +128 -0
- package/dist/akamai-sensor.d.ts.map +1 -0
- package/dist/akamai-sensor.js +190 -0
- package/dist/akamai-sensor.js.map +1 -0
- package/dist/blacktip.d.ts +34 -0
- package/dist/blacktip.d.ts.map +1 -1
- package/dist/blacktip.js +35 -0
- package/dist/blacktip.js.map +1 -1
- package/dist/browser-core.d.ts +10 -0
- package/dist/browser-core.d.ts.map +1 -1
- package/dist/browser-core.js +49 -0
- package/dist/browser-core.js.map +1 -1
- package/dist/index.d.ts +4 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +4 -0
- package/dist/index.js.map +1 -1
- package/dist/tls-rewriter.d.ts +74 -0
- package/dist/tls-rewriter.d.ts.map +1 -0
- package/dist/tls-rewriter.js +203 -0
- package/dist/tls-rewriter.js.map +1 -0
- package/dist/tls-side-channel.d.ts +13 -4
- package/dist/tls-side-channel.d.ts.map +1 -1
- package/dist/tls-side-channel.js +9 -2
- package/dist/tls-side-channel.js.map +1 -1
- package/dist/types.d.ts +31 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/docs/akamai-sensor.md +183 -0
- package/docs/tls-rewriting.md +121 -0
- package/package.json +1 -1
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* TLS rewriting via CDP Fetch interception.
|
|
3
|
+
*
|
|
4
|
+
* The v0.5.0 answer to "every wire request should present a real Chrome
|
|
5
|
+
* TLS fingerprint, not just the gating ones." Without a TCP-level MITM
|
|
6
|
+
* proxy (and the OS-specific cert installation hell that entails), we
|
|
7
|
+
* use Chrome DevTools Protocol's `Fetch.enable` to pause every HTTP
|
|
8
|
+
* request the browser issues, hand it to the Go `bogdanfinn/tls-client`
|
|
9
|
+
* daemon for upstream execution, and fulfill the response back through
|
|
10
|
+
* CDP. The browser never opens an upstream TCP connection — all its
|
|
11
|
+
* HTTP is fulfilled by us.
|
|
12
|
+
*
|
|
13
|
+
* Patchright/Playwright expose this as `context.route('**', handler)`,
|
|
14
|
+
* which is the high-level wrapper around CDP Fetch. We use the route
|
|
15
|
+
* handler so we don't need to manage CDP sessions ourselves.
|
|
16
|
+
*
|
|
17
|
+
* What this fixes vs the v0.3.0 side-channel (`bt.fetchWithTls`):
|
|
18
|
+
* - The side-channel only handled gating requests the caller made
|
|
19
|
+
* explicitly. Page subresources, XHR, fetch() from page JS, all
|
|
20
|
+
* went through Chrome's own TLS — meaning the host OS's Chrome
|
|
21
|
+
* fingerprint reached the wire. With this rewriter installed, every
|
|
22
|
+
* subresource also goes through Go.
|
|
23
|
+
* - Cross-platform UA spoofing is restored. The daemon controls every
|
|
24
|
+
* header on the wire; spoof to your heart's content.
|
|
25
|
+
*
|
|
26
|
+
* Known limitations:
|
|
27
|
+
* - WebSocket upgrades can't be intercepted by Fetch.enable. They
|
|
28
|
+
* bypass the rewriter and present Chrome's native TLS. The rewriter
|
|
29
|
+
* logs WS leaks for awareness.
|
|
30
|
+
* - Streaming responses are buffered fully. Bad for video, fine for
|
|
31
|
+
* HTML/JSON/typical web pages.
|
|
32
|
+
* - HTTP/3 (QUIC) requests bypass Fetch.enable entirely because Chrome
|
|
33
|
+
* short-circuits them. We launch with `--disable-quic` so this never
|
|
34
|
+
* fires in practice.
|
|
35
|
+
*/
|
|
36
|
+
/**
|
|
37
|
+
* Headers that Chrome's request lifecycle manages itself and that we
|
|
38
|
+
* MUST NOT pass through verbatim — re-sending them through the daemon
|
|
39
|
+
* either breaks the request (Content-Length) or duplicates state
|
|
40
|
+
* (Cookie, which the daemon will set automatically from the upstream's
|
|
41
|
+
* Set-Cookie response, while the browser's own cookie jar handles the
|
|
42
|
+
* reverse direction). The browser cookie jar IS the source of truth;
|
|
43
|
+
* we forward Cookie verbatim and let Set-Cookie come back via fulfill.
|
|
44
|
+
*/
|
|
45
|
+
const STRIP_REQUEST_HEADERS = new Set([
|
|
46
|
+
'host',
|
|
47
|
+
'content-length',
|
|
48
|
+
'connection',
|
|
49
|
+
'keep-alive',
|
|
50
|
+
'transfer-encoding',
|
|
51
|
+
'proxy-authorization',
|
|
52
|
+
'proxy-connection',
|
|
53
|
+
'upgrade',
|
|
54
|
+
'expect',
|
|
55
|
+
]);
|
|
56
|
+
/**
|
|
57
|
+
* Headers Chrome's response lifecycle re-computes and that we should
|
|
58
|
+
* NOT pass back via fulfill — letting them through breaks framing.
|
|
59
|
+
*/
|
|
60
|
+
const STRIP_RESPONSE_HEADERS = new Set([
|
|
61
|
+
'content-length',
|
|
62
|
+
'content-encoding', // upstream returns gzip; we hand fulfill the decoded body
|
|
63
|
+
'transfer-encoding',
|
|
64
|
+
'connection',
|
|
65
|
+
'keep-alive',
|
|
66
|
+
]);
|
|
67
|
+
/**
|
|
68
|
+
* Install the TLS rewriter on a Playwright BrowserContext. After this
|
|
69
|
+
* call, every HTTP/HTTPS request the browser issues is intercepted and
|
|
70
|
+
* forwarded through the TLS daemon. Returns a `stats()` accessor and an
|
|
71
|
+
* `uninstall()` callback.
|
|
72
|
+
*/
|
|
73
|
+
export async function installTlsRewriter(context, options) {
|
|
74
|
+
const { channel, logger } = options;
|
|
75
|
+
const profile = options.profile ?? 'chrome_133';
|
|
76
|
+
const perRequestTimeoutMs = options.perRequestTimeoutMs ?? 30_000;
|
|
77
|
+
let intercepted = 0;
|
|
78
|
+
let fulfilled = 0;
|
|
79
|
+
let fellThrough = 0;
|
|
80
|
+
let webSocketLeaks = 0;
|
|
81
|
+
let totalDurationMs = 0;
|
|
82
|
+
const handler = async (route, request) => {
|
|
83
|
+
intercepted++;
|
|
84
|
+
const url = request.url();
|
|
85
|
+
const method = request.method();
|
|
86
|
+
// WebSocket upgrades come through `route` but Fetch can't intercept
|
|
87
|
+
// the upgrade itself — Chrome handles WS frames at a layer we can't
|
|
88
|
+
// see from here. Let them pass through and log a warning.
|
|
89
|
+
if (request.isNavigationRequest() && (url.startsWith('ws://') || url.startsWith('wss://'))) {
|
|
90
|
+
webSocketLeaks++;
|
|
91
|
+
logger.warn('TLS rewriter: WebSocket leak — upgrade bypasses the rewriter', { url });
|
|
92
|
+
await route.continue();
|
|
93
|
+
return;
|
|
94
|
+
}
|
|
95
|
+
const upgradeHeader = request.headers()['upgrade'];
|
|
96
|
+
if (upgradeHeader && upgradeHeader.toLowerCase() === 'websocket') {
|
|
97
|
+
webSocketLeaks++;
|
|
98
|
+
logger.warn('TLS rewriter: WebSocket leak — upgrade bypasses the rewriter', { url });
|
|
99
|
+
await route.continue();
|
|
100
|
+
return;
|
|
101
|
+
}
|
|
102
|
+
// Build the daemon request from the browser-side request.
|
|
103
|
+
const reqHeaders = {};
|
|
104
|
+
for (const [key, value] of Object.entries(request.headers())) {
|
|
105
|
+
if (STRIP_REQUEST_HEADERS.has(key.toLowerCase()))
|
|
106
|
+
continue;
|
|
107
|
+
reqHeaders[key] = value;
|
|
108
|
+
}
|
|
109
|
+
const postBuffer = request.postDataBuffer();
|
|
110
|
+
const body = postBuffer ?? undefined;
|
|
111
|
+
try {
|
|
112
|
+
const resp = await Promise.race([
|
|
113
|
+
channel.fetch({
|
|
114
|
+
url,
|
|
115
|
+
method,
|
|
116
|
+
headers: reqHeaders,
|
|
117
|
+
body,
|
|
118
|
+
profile,
|
|
119
|
+
timeoutMs: perRequestTimeoutMs,
|
|
120
|
+
}),
|
|
121
|
+
new Promise((_resolve, reject) => {
|
|
122
|
+
setTimeout(() => reject(new Error(`TLS rewriter: per-request timeout ${perRequestTimeoutMs}ms`)), perRequestTimeoutMs + 1000).unref();
|
|
123
|
+
}),
|
|
124
|
+
]);
|
|
125
|
+
totalDurationMs += resp.durationMs;
|
|
126
|
+
// Flatten response headers. Multi-valued headers (e.g. Set-Cookie)
|
|
127
|
+
// need special handling: Playwright's fulfill takes a single string
|
|
128
|
+
// per key, but lets you pass an array via the headers parameter
|
|
129
|
+
// since 1.50 — we use the comma-join fallback for older versions.
|
|
130
|
+
// For Set-Cookie specifically, we use the multi-value extension
|
|
131
|
+
// because cookies must not be merged.
|
|
132
|
+
const respHeaders = {};
|
|
133
|
+
for (const [key, values] of Object.entries(resp.headers)) {
|
|
134
|
+
if (STRIP_RESPONSE_HEADERS.has(key.toLowerCase()))
|
|
135
|
+
continue;
|
|
136
|
+
if (values.length === 1) {
|
|
137
|
+
respHeaders[key] = values[0];
|
|
138
|
+
}
|
|
139
|
+
else {
|
|
140
|
+
// For Set-Cookie, joining with comma is wrong (cookies have
|
|
141
|
+
// their own commas in Expires). Playwright's `fulfill` accepts
|
|
142
|
+
// multiValueHeaders via the headers field as Record<string, string>
|
|
143
|
+
// by joining with `\n` for some headers. Safest fallback: pass
|
|
144
|
+
// each Set-Cookie as a separate header by using the array form
|
|
145
|
+
// if available, otherwise the last cookie wins.
|
|
146
|
+
//
|
|
147
|
+
// Since we can't pass arrays directly to Playwright's fulfill
|
|
148
|
+
// headers, we encode the multi-value as `\n`-separated for
|
|
149
|
+
// Set-Cookie (Chrome accepts this) and comma-join for everything
|
|
150
|
+
// else.
|
|
151
|
+
if (key.toLowerCase() === 'set-cookie') {
|
|
152
|
+
respHeaders[key] = values.join('\n');
|
|
153
|
+
}
|
|
154
|
+
else {
|
|
155
|
+
respHeaders[key] = values.join(', ');
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
await route.fulfill({
|
|
160
|
+
status: resp.status,
|
|
161
|
+
headers: respHeaders,
|
|
162
|
+
body: resp.bodyBuffer,
|
|
163
|
+
});
|
|
164
|
+
fulfilled++;
|
|
165
|
+
}
|
|
166
|
+
catch (err) {
|
|
167
|
+
fellThrough++;
|
|
168
|
+
logger.warn('TLS rewriter: daemon path failed, falling through to native fetch', {
|
|
169
|
+
url,
|
|
170
|
+
error: err instanceof Error ? err.message : String(err),
|
|
171
|
+
});
|
|
172
|
+
// Fall through to the browser's native request. Less stealthy but
|
|
173
|
+
// doesn't break the page.
|
|
174
|
+
try {
|
|
175
|
+
await route.continue();
|
|
176
|
+
}
|
|
177
|
+
catch {
|
|
178
|
+
// Route may already be fulfilled/aborted in race conditions; ignore.
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
};
|
|
182
|
+
// Install the route handler for all URLs. Patchright/Playwright accept
|
|
183
|
+
// a glob ('**/*') or RegExp; we use the glob for clarity.
|
|
184
|
+
await context.route('**/*', handler);
|
|
185
|
+
return {
|
|
186
|
+
stats: () => ({
|
|
187
|
+
intercepted,
|
|
188
|
+
fulfilled,
|
|
189
|
+
fellThrough,
|
|
190
|
+
webSocketLeaks,
|
|
191
|
+
avgDurationMs: fulfilled > 0 ? Math.round(totalDurationMs / fulfilled) : 0,
|
|
192
|
+
}),
|
|
193
|
+
uninstall: async () => {
|
|
194
|
+
try {
|
|
195
|
+
await context.unroute('**/*', handler);
|
|
196
|
+
}
|
|
197
|
+
catch {
|
|
198
|
+
// Context may be closing; ignore.
|
|
199
|
+
}
|
|
200
|
+
},
|
|
201
|
+
};
|
|
202
|
+
}
|
|
203
|
+
//# sourceMappingURL=tls-rewriter.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tls-rewriter.js","sourceRoot":"","sources":["../src/tls-rewriter.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AAMH;;;;;;;;GAQG;AACH,MAAM,qBAAqB,GAAG,IAAI,GAAG,CAAC;IACpC,MAAM;IACN,gBAAgB;IAChB,YAAY;IACZ,YAAY;IACZ,mBAAmB;IACnB,qBAAqB;IACrB,kBAAkB;IAClB,SAAS;IACT,QAAQ;CACT,CAAC,CAAC;AAEH;;;GAGG;AACH,MAAM,sBAAsB,GAAG,IAAI,GAAG,CAAC;IACrC,gBAAgB;IAChB,kBAAkB,EAAE,0DAA0D;IAC9E,mBAAmB;IACnB,YAAY;IACZ,YAAY;CACb,CAAC,CAAC;AA6BH;;;;;GAKG;AACH,MAAM,CAAC,KAAK,UAAU,kBAAkB,CACtC,OAAuB,EACvB,OAA2B;IAK3B,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;IACpC,MAAM,OAAO,GAAG,OAAO,CAAC,OAAO,IAAI,YAAY,CAAC;IAChD,MAAM,mBAAmB,GAAG,OAAO,CAAC,mBAAmB,IAAI,MAAM,CAAC;IAElE,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,IAAI,SAAS,GAAG,CAAC,CAAC;IAClB,IAAI,WAAW,GAAG,CAAC,CAAC;IACpB,IAAI,cAAc,GAAG,CAAC,CAAC;IACvB,IAAI,eAAe,GAAG,CAAC,CAAC;IAExB,MAAM,OAAO,GAAG,KAAK,EAAE,KAAY,EAAE,OAA0B,EAAiB,EAAE;QAChF,WAAW,EAAE,CAAC;QACd,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,EAAE,CAAC;QAC1B,MAAM,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC;QAEhC,oEAAoE;QACpE,oEAAoE;QACpE,0DAA0D;QAC1D,IAAI,OAAO,CAAC,mBAAmB,EAAE,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,OAAO,CAAC,IAAI,GAAG,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,EAAE,CAAC;YAC3F,cAAc,EAAE,CAAC;YACjB,MAAM,CAAC,IAAI,CAAC,8DAA8D,EAAE,EAAE,GAAG,EAAE,CAAC,CAAC;YACrF,MAAM,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvB,OAAO;QACT,CAAC;QACD,MAAM,aAAa,GAAG,OAAO,CAAC,OAAO,EAAE,CAAC,SAAS,CAAC,CAAC;QACnD,IAAI,aAAa,IAAI,aAAa,CAAC,WAAW,EAAE,KAAK,WAAW,EAAE,CAAC;YACjE,cAAc,EAAE,CAAC;YACjB,MAAM,CAAC,IAAI,CAAC,8DAA8D,EAAE,EAAE,GAAG,EAAE,CAAC,CAAC;YACrF,MAAM,KAAK,CAAC,QAAQ,EAAE,CAAC;YACvB,OAAO;QACT,CAAC;QAED,0DAA0D;QAC1D,MAAM,UAAU,GAA2B,EAAE,CAAC;QAC9C,KAAK,MAAM,CAAC,GAAG,EAAE,KAAK,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,OAAO,CAAC,OAAO,EAAE,CAAC,EAAE,CAAC;YAC7D,IAAI,qBAAqB,CAAC,GAAG,CAAC,GAAG,CAAC,WAAW,EAAE,CAAC;gBAAE,SAAS;YAC3D,UAAU,CAAC,GAAG,CAAC,GAAG,KAAK,CAAC;QAC1B,CAAC;QAED,MAAM,UAAU,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;QAC5C,MAAM,IAAI,GAAG,UAAU,IAAI,SAAS,CAAC;QAErC,IAAI,CAAC;YACH,MAAM,IAAI,GAAG,MAAM,OAAO,CAAC,IAAI,CAAC;gBAC9B,OAAO,CAAC,KAAK,CAAC;oBACZ,GAAG;oBACH,MAAM;oBACN,OAAO,EAAE,UAAU;oBACnB,IAAI;oBACJ,OAAO;oBACP,SAAS,EAAE,mBAAmB;iBAC/B,CAAC;gBACF,IAAI,OAAO,CAAQ,CAAC,QAAQ,EAAE,MAAM,EAAE,EAAE;oBACtC,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,qCAAqC,mBAAmB,IAAI,CAAC,CAAC,EAAE,mBAAmB,GAAG,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC;gBACxI,CAAC,CAAC;aACH,CAAC,CAAC;YAEH,eAAe,IAAI,IAAI,CAAC,UAAU,CAAC;YAEnC,mEAAmE;YACnE,oEAAoE;YACpE,gEAAgE;YAChE,kEAAkE;YAClE,gEAAgE;YAChE,sCAAsC;YACtC,MAAM,WAAW,GAA2B,EAAE,CAAC;YAC/C,KAAK,MAAM,CAAC,GAAG,EAAE,MAAM,CAAC,IAAI,MAAM,CAAC,OAAO,CAAC,IAAI,CAAC,OAAO,CAAC,EAAE,CAAC;gBACzD,IAAI,sBAAsB,CAAC,GAAG,CAAC,GAAG,CAAC,WAAW,EAAE,CAAC;oBAAE,SAAS;gBAC5D,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;oBACxB,WAAW,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,CAAC,CAAE,CAAC;gBAChC,CAAC;qBAAM,CAAC;oBACN,4DAA4D;oBAC5D,+DAA+D;oBAC/D,oEAAoE;oBACpE,+DAA+D;oBAC/D,+DAA+D;oBAC/D,gDAAgD;oBAChD,EAAE;oBACF,8DAA8D;oBAC9D,2DAA2D;oBAC3D,iEAAiE;oBACjE,QAAQ;oBACR,IAAI,GAAG,CAAC,WAAW,EAAE,KAAK,YAAY,EAAE,CAAC;wBACvC,WAAW,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;oBACvC,CAAC;yBAAM,CAAC;wBACN,WAAW,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;oBACvC,CAAC;gBACH,CAAC;YACH,CAAC;YAED,MAAM,KAAK,CAAC,OAAO,CAAC;gBAClB,MAAM,EAAE,IAAI,CAAC,MAAM;gBACnB,OAAO,EAAE,WAAW;gBACpB,IAAI,EAAE,IAAI,CAAC,UAAU;aACtB,CAAC,CAAC;YACH,SAAS,EAAE,CAAC;QACd,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,WAAW,EAAE,CAAC;YACd,MAAM,CAAC,IAAI,CAAC,mEAAmE,EAAE;gBAC/E,GAAG;gBACH,KAAK,EAAE,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC;aACxD,CAAC,CAAC;YACH,kEAAkE;YAClE,0BAA0B;YAC1B,IAAI,CAAC;gBACH,MAAM,KAAK,CAAC,QAAQ,EAAE,CAAC;YACzB,CAAC;YAAC,MAAM,CAAC;gBACP,qEAAqE;YACvE,CAAC;QACH,CAAC;IACH,CAAC,CAAC;IAEF,uEAAuE;IACvE,0DAA0D;IAC1D,MAAM,OAAO,CAAC,KAAK,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAErC,OAAO;QACL,KAAK,EAAE,GAAqB,EAAE,CAAC,CAAC;YAC9B,WAAW;YACX,SAAS;YACT,WAAW;YACX,cAAc;YACd,aAAa,EAAE,SAAS,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,eAAe,GAAG,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC;SAC3E,CAAC;QACF,SAAS,EAAE,KAAK,IAAmB,EAAE;YACnC,IAAI,CAAC;gBACH,MAAM,OAAO,CAAC,OAAO,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;YACzC,CAAC;YAAC,MAAM,CAAC;gBACP,kCAAkC;YACpC,CAAC;QACH,CAAC;KACF,CAAC;AACJ,CAAC"}
|
|
@@ -30,18 +30,27 @@ export interface TlsRequest {
|
|
|
30
30
|
url: string;
|
|
31
31
|
method?: string;
|
|
32
32
|
headers?: Record<string, string>;
|
|
33
|
-
/** Request body as a UTF-8 string
|
|
34
|
-
|
|
33
|
+
/** Request body as a UTF-8 string OR raw Buffer. Buffer is required for
|
|
34
|
+
* binary uploads (multipart, octet-stream); string is fine for form
|
|
35
|
+
* bodies and JSON. Encoded as base64 on the wire. */
|
|
36
|
+
body?: string | Buffer;
|
|
35
37
|
timeoutMs?: number;
|
|
36
38
|
/** Chrome / Firefox / Safari profile name; defaults to chrome_133. */
|
|
37
39
|
profile?: string;
|
|
38
40
|
}
|
|
39
41
|
export interface TlsResponse {
|
|
40
42
|
status: number;
|
|
41
|
-
/** Headers from the upstream response
|
|
43
|
+
/** Headers from the upstream response. Multi-valued — Set-Cookie commonly
|
|
44
|
+
* has multiple entries. Header keys preserve the casing the upstream sent. */
|
|
42
45
|
headers: Record<string, string[]>;
|
|
43
|
-
/** Response body as a UTF-8 string
|
|
46
|
+
/** Response body as a UTF-8 string. May be garbage for binary content;
|
|
47
|
+
* use `bodyBuffer` for that. Kept as the primary body field for callers
|
|
48
|
+
* that just want JSON / HTML. */
|
|
44
49
|
body: string;
|
|
50
|
+
/** Response body as raw bytes. Use this when the upstream returns binary
|
|
51
|
+
* data (images, fonts, video, anything non-UTF-8). The TLS rewriting
|
|
52
|
+
* route handler always uses this so subresources don't get mangled. */
|
|
53
|
+
bodyBuffer: Buffer;
|
|
45
54
|
finalUrl: string;
|
|
46
55
|
durationMs: number;
|
|
47
56
|
/** Cookies parsed from `Set-Cookie` headers, ready to inject via `bt.setCookies()`. */
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"tls-side-channel.d.ts","sourceRoot":"","sources":["../src/tls-side-channel.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AAUH,MAAM,WAAW,UAAU;IACzB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC
|
|
1
|
+
{"version":3,"file":"tls-side-channel.d.ts","sourceRoot":"","sources":["../src/tls-side-channel.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AAUH,MAAM,WAAW,UAAU;IACzB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC;;0DAEsD;IACtD,IAAI,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;IACvB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,sEAAsE;IACtE,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,WAAW;IAC1B,MAAM,EAAE,MAAM,CAAC;IACf;mFAC+E;IAC/E,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,EAAE,CAAC,CAAC;IAClC;;sCAEkC;IAClC,IAAI,EAAE,MAAM,CAAC;IACb;;4EAEwE;IACxE,UAAU,EAAE,MAAM,CAAC;IACnB,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,uFAAuF;IACvF,OAAO,EAAE,YAAY,EAAE,CAAC;CACzB;AAED,MAAM,WAAW,YAAY;IAC3B,IAAI,EAAE,MAAM,CAAC;IACb,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,OAAO,CAAC;IAClB,MAAM,EAAE,OAAO,CAAC;IAChB,QAAQ,CAAC,EAAE,QAAQ,GAAG,KAAK,GAAG,MAAM,CAAC;IACrC,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAoGD,qBAAa,cAAc;IACzB,OAAO,CAAC,IAAI,CAAiC;IAC7C,OAAO,CAAC,EAAE,CAAoB;IAC9B,OAAO,CAAC,OAAO,CAAqC;IACpD,OAAO,CAAC,MAAM,CAAK;IACnB,OAAO,CAAC,MAAM,CAAS;IAEvB,OAAO;IAYP;;;OAGG;WACU,KAAK,IAAI,OAAO,CAAC,cAAc,CAAC;IAU7C,OAAO,CAAC,UAAU;IAgDlB;;;;OAIG;IACG,KAAK,CAAC,GAAG,EAAE,UAAU,GAAG,OAAO,CAAC,WAAW,CAAC;IA4BlD,wDAAwD;IAClD,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;CAc7B"}
|
package/dist/tls-side-channel.js
CHANGED
|
@@ -168,7 +168,8 @@ export class TlsSideChannel {
|
|
|
168
168
|
const finalUrl = parsed.finalUrl ?? '';
|
|
169
169
|
const headers = parsed.headers ?? {};
|
|
170
170
|
const bodyB64 = parsed.body ?? '';
|
|
171
|
-
const
|
|
171
|
+
const bodyBuffer = Buffer.from(bodyB64, 'base64');
|
|
172
|
+
const body = bodyBuffer.toString('utf-8');
|
|
172
173
|
// Parse Set-Cookie headers. The header key may be `Set-Cookie`
|
|
173
174
|
// or `set-cookie` depending on the daemon's Go HTTP version.
|
|
174
175
|
const cookies = [];
|
|
@@ -187,6 +188,7 @@ export class TlsSideChannel {
|
|
|
187
188
|
status: parsed.status ?? 0,
|
|
188
189
|
headers,
|
|
189
190
|
body,
|
|
191
|
+
bodyBuffer,
|
|
190
192
|
finalUrl,
|
|
191
193
|
durationMs: parsed.durationMs,
|
|
192
194
|
cookies,
|
|
@@ -201,12 +203,17 @@ export class TlsSideChannel {
|
|
|
201
203
|
if (this.closed)
|
|
202
204
|
throw new Error('TLS daemon is closed');
|
|
203
205
|
const id = `r${this.nextId++}`;
|
|
206
|
+
let bodyB64 = '';
|
|
207
|
+
if (req.body != null) {
|
|
208
|
+
const buf = Buffer.isBuffer(req.body) ? req.body : Buffer.from(req.body, 'utf-8');
|
|
209
|
+
bodyB64 = buf.toString('base64');
|
|
210
|
+
}
|
|
204
211
|
const wire = {
|
|
205
212
|
id,
|
|
206
213
|
url: req.url,
|
|
207
214
|
method: req.method ?? 'GET',
|
|
208
215
|
headers: req.headers ?? {},
|
|
209
|
-
body:
|
|
216
|
+
body: bodyB64,
|
|
210
217
|
timeoutMs: req.timeoutMs ?? 15000,
|
|
211
218
|
profile: req.profile ?? 'chrome_133',
|
|
212
219
|
};
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"tls-side-channel.js","sourceRoot":"","sources":["../src/tls-side-channel.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AAEH,OAAO,EAAE,KAAK,EAAuC,MAAM,oBAAoB,CAAC;AAChF,OAAO,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AACrC,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAC1C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,eAAe,EAAuC,MAAM,eAAe,CAAC;
|
|
1
|
+
{"version":3,"file":"tls-side-channel.js","sourceRoot":"","sources":["../src/tls-side-channel.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;GA2BG;AAEH,OAAO,EAAE,KAAK,EAAuC,MAAM,oBAAoB,CAAC;AAChF,OAAO,EAAE,UAAU,EAAE,MAAM,SAAS,CAAC;AACrC,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,WAAW,CAAC;AAC1C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAAE,eAAe,EAAuC,MAAM,eAAe,CAAC;AA0DrF,wCAAwC;AAExC;;;;;GAKG;AACH,SAAS,iBAAiB;IACxB,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,gBAAgB,CAAC;IACzC,IAAI,GAAG,IAAI,UAAU,CAAC,GAAG,CAAC;QAAE,OAAO,GAAG,CAAC;IAEvC,qEAAqE;IACrE,4CAA4C;IAC5C,MAAM,IAAI,GAAG,OAAO,CAAC,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC;IACrD,MAAM,GAAG,GAAG,OAAO,CAAC,QAAQ,KAAK,OAAO,CAAC,CAAC,CAAC,kBAAkB,CAAC,CAAC,CAAC,cAAc,CAAC;IAC/E,MAAM,UAAU,GAAG;QACjB,IAAI,CAAC,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,YAAY,EAAE,GAAG,CAAC;QAC7C,IAAI,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,QAAQ,EAAE,YAAY,EAAE,GAAG,CAAC,EAAE,0BAA0B;KAChF,CAAC;IACF,KAAK,MAAM,CAAC,IAAI,UAAU,EAAE,CAAC;QAC3B,IAAI,UAAU,CAAC,CAAC,CAAC;YAAE,OAAO,CAAC,CAAC;IAC9B,CAAC;IACD,MAAM,IAAI,KAAK,CACb,wDAAwD;QACtD,yCAAyC,GAAG,MAAM;QAClD,8CAA8C,CACjD,CAAC;AACJ,CAAC;AAED,uBAAuB;AACvB,EAAE;AACF,mEAAmE;AACnE,sEAAsE;AACtE,sEAAsE;AACtE,qEAAqE;AACrE,mEAAmE;AACnE,mDAAmD;AAEnD,SAAS,WAAW,CAAC,SAAiB,EAAE,aAAqB;IAC3D,4EAA4E;IAC5E,MAAM,KAAK,GAAG,SAAS,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;IACxD,MAAM,KAAK,GAAG,KAAK,CAAC,CAAC,CAAC,CAAC;IACvB,IAAI,CAAC,KAAK;QAAE,OAAO,IAAI,CAAC;IACxB,MAAM,EAAE,GAAG,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;IAC9B,IAAI,EAAE,GAAG,CAAC;QAAE,OAAO,IAAI,CAAC;IACxB,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,IAAI,EAAE,CAAC;IACvC,MAAM,KAAK,GAAG,KAAK,CAAC,KAAK,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACzC,IAAI,CAAC,IAAI;QAAE,OAAO,IAAI,CAAC;IAEvB,IAAI,MAAM,GAAG,aAAa,CAAC;IAC3B,IAAI,IAAI,GAAG,GAAG,CAAC;IACf,IAAI,QAAQ,GAAG,KAAK,CAAC;IACrB,IAAI,MAAM,GAAG,KAAK,CAAC;IACnB,IAAI,QAAkC,CAAC;IACvC,IAAI,OAA2B,CAAC;IAEhC,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACtC,MAAM,IAAI,GAAG,KAAK,CAAC,CAAC,CAAE,CAAC;QACvB,MAAM,EAAE,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC;QAC7B,MAAM,GAAG,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC;QACrE,MAAM,GAAG,GAAG,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QACpD,IAAI,GAAG,KAAK,QAAQ;YAAE,MAAM,GAAG,GAAG,CAAC,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;aACjD,IAAI,GAAG,KAAK,MAAM;YAAE,IAAI,GAAG,GAAG,CAAC;aAC/B,IAAI,GAAG,KAAK,UAAU;YAAE,QAAQ,GAAG,IAAI,CAAC;aACxC,IAAI,GAAG,KAAK,QAAQ;YAAE,MAAM,GAAG,IAAI,CAAC;aACpC,IAAI,GAAG,KAAK,UAAU,EAAE,CAAC;YAC5B,MAAM,CAAC,GAAG,GAAG,CAAC,WAAW,EAAE,CAAC;YAC5B,IAAI,CAAC,KAAK,QAAQ;gBAAE,QAAQ,GAAG,QAAQ,CAAC;iBACnC,IAAI,CAAC,KAAK,KAAK;gBAAE,QAAQ,GAAG,KAAK,CAAC;iBAClC,IAAI,CAAC,KAAK,MAAM;gBAAE,QAAQ,GAAG,MAAM,CAAC;QAC3C,CAAC;aAAM,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YAC7B,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;YAC1B,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC;gBAAE,OAAO,GAAG,CAAC,GAAG,IAAI,CAAC;QAC3C,CAAC;IACH,CAAC;IAED,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,QAAQ,EAAE,MAAM,EAAE,QAAQ,EAAE,OAAO,EAAE,CAAC;AAC5E,CAAC;AASD,MAAM,OAAO,cAAc;IACjB,IAAI,CAAiC;IACrC,EAAE,CAAoB;IACtB,OAAO,GAAG,IAAI,GAAG,EAA0B,CAAC;IAC5C,MAAM,GAAG,CAAC,CAAC;IACX,MAAM,GAAG,KAAK,CAAC;IAEvB,YAAoB,IAAoC;QACtD,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC;QACjB,IAAI,CAAC,EAAE,GAAG,eAAe,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC,CAAC;QAClD,IAAI,CAAC,EAAE,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC;QACpD,IAAI,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,EAAE;YACvB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;YACnB,MAAM,GAAG,GAAG,IAAI,KAAK,CAAC,+BAA+B,IAAI,EAAE,CAAC,CAAC;YAC7D,KAAK,MAAM,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE;gBAAE,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;YACrD,IAAI,CAAC,OAAO,CAAC,KAAK,EAAE,CAAC;QACvB,CAAC,CAAC,CAAC;IACL,CAAC;IAED;;;OAGG;IACH,MAAM,CAAC,KAAK,CAAC,KAAK;QAChB,MAAM,GAAG,GAAG,iBAAiB,EAAE,CAAC;QAChC,MAAM,IAAI,GAAG,KAAK,CAAC,GAAG,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC,CAAC;QACjE,0DAA0D;QAC1D,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC,MAAM,EAAE,CAAC,KAAa,EAAE,EAAE;YACvC,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,kBAAkB,KAAK,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;QAC7D,CAAC,CAAC,CAAC;QACH,OAAO,IAAI,cAAc,CAAC,IAAI,CAAC,CAAC;IAClC,CAAC;IAEO,UAAU,CAAC,IAAY;QAC7B,IAAI,CAAC,IAAI,CAAC,IAAI,EAAE;YAAE,OAAO;QACzB,IAAI,MAAsB,CAAC;QAC3B,IAAI,CAAC;YACH,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAmB,CAAC;QAC9C,CAAC;QAAC,MAAM,CAAC;YACP,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,oCAAoC,IAAI,IAAI,CAAC,CAAC;YACnE,OAAO;QACT,CAAC;QACD,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QAC5C,IAAI,CAAC,OAAO;YAAE,OAAO;QACrB,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;QAE/B,IAAI,CAAC,MAAM,CAAC,EAAE,EAAE,CAAC;YACf,OAAO,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,KAAK,IAAI,sBAAsB,CAAC,CAAC,CAAC;YAClE,OAAO;QACT,CAAC;QAED,MAAM,QAAQ,GAAG,MAAM,CAAC,QAAQ,IAAI,EAAE,CAAC;QACvC,MAAM,OAAO,GAAG,MAAM,CAAC,OAAO,IAAI,EAAE,CAAC;QACrC,MAAM,OAAO,GAAG,MAAM,CAAC,IAAI,IAAI,EAAE,CAAC;QAClC,MAAM,UAAU,GAAG,MAAM,CAAC,IAAI,CAAC,OAAO,EAAE,QAAQ,CAAC,CAAC;QAClD,MAAM,IAAI,GAAG,UAAU,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QAE1C,+DAA+D;QAC/D,6DAA6D;QAC7D,MAAM,OAAO,GAAmB,EAAE,CAAC;QACnC,MAAM,aAAa,GAAG,OAAO,CAAC,YAAY,CAAC,IAAI,OAAO,CAAC,YAAY,CAAC,IAAI,EAAE,CAAC;QAC3E,IAAI,aAAa,GAAG,EAAE,CAAC;QACvB,IAAI,CAAC;YACH,aAAa,GAAG,IAAI,GAAG,CAAC,QAAQ,CAAC,CAAC,QAAQ,CAAC;QAC7C,CAAC;QAAC,MAAM,CAAC,CAAC,iBAAiB,CAAC,CAAC;QAC7B,KAAK,MAAM,EAAE,IAAI,aAAa,EAAE,CAAC;YAC/B,MAAM,CAAC,GAAG,WAAW,CAAC,EAAE,EAAE,aAAa,CAAC,CAAC;YACzC,IAAI,CAAC;gBAAE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;QACzB,CAAC;QAED,OAAO,CAAC,OAAO,CAAC;YACd,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,CAAC;YAC1B,OAAO;YACP,IAAI;YACJ,UAAU;YACV,QAAQ;YACR,UAAU,EAAE,MAAM,CAAC,UAAU;YAC7B,OAAO;SACR,CAAC,CAAC;IACL,CAAC;IAED;;;;OAIG;IACH,KAAK,CAAC,KAAK,CAAC,GAAe;QACzB,IAAI,IAAI,CAAC,MAAM;YAAE,MAAM,IAAI,KAAK,CAAC,sBAAsB,CAAC,CAAC;QACzD,MAAM,EAAE,GAAG,IAAI,IAAI,CAAC,MAAM,EAAE,EAAE,CAAC;QAC/B,IAAI,OAAO,GAAG,EAAE,CAAC;QACjB,IAAI,GAAG,CAAC,IAAI,IAAI,IAAI,EAAE,CAAC;YACrB,MAAM,GAAG,GAAG,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,EAAE,OAAO,CAAC,CAAC;YAClF,OAAO,GAAG,GAAG,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAC;QACnC,CAAC;QACD,MAAM,IAAI,GAAG;YACX,EAAE;YACF,GAAG,EAAE,GAAG,CAAC,GAAG;YACZ,MAAM,EAAE,GAAG,CAAC,MAAM,IAAI,KAAK;YAC3B,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,EAAE;YAC1B,IAAI,EAAE,OAAO;YACb,SAAS,EAAE,GAAG,CAAC,SAAS,IAAI,KAAK;YACjC,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,YAAY;SACrC,CAAC;QACF,OAAO,IAAI,OAAO,CAAc,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YAClD,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,EAAE,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC,CAAC;YAC1C,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,IAAI,EAAE,CAAC,GAAG,EAAE,EAAE;gBACzD,IAAI,GAAG,EAAE,CAAC;oBACR,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,CAAC,CAAC;oBACxB,MAAM,CAAC,GAAG,CAAC,CAAC;gBACd,CAAC;YACH,CAAC,CAAC,CAAC;QACL,CAAC,CAAC,CAAC;IACL,CAAC;IAED,wDAAwD;IACxD,KAAK,CAAC,KAAK;QACT,IAAI,IAAI,CAAC,MAAM;YAAE,OAAO;QACxB,IAAI,CAAC,MAAM,GAAG,IAAI,CAAC;QACnB,IAAI,CAAC,EAAE,CAAC,KAAK,EAAE,CAAC;QAChB,OAAO,IAAI,OAAO,CAAO,CAAC,OAAO,EAAE,EAAE;YACnC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,EAAE,CAAC,OAAO,EAAE,CAAC,CAAC;YACxC,IAAI,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,EAAE,CAAC;YACtB,2DAA2D;YAC3D,UAAU,CAAC,GAAG,EAAE;gBACd,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM;oBAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;gBACjD,OAAO,EAAE,CAAC;YACZ,CAAC,EAAE,IAAI,CAAC,CAAC,KAAK,EAAE,CAAC;QACnB,CAAC,CAAC,CAAC;IACL,CAAC;CACF"}
|
package/dist/types.d.ts
CHANGED
|
@@ -40,6 +40,37 @@ export interface BlackTipConfig {
|
|
|
40
40
|
* unset for offline / air-gapped use.
|
|
41
41
|
*/
|
|
42
42
|
requireResidentialIp?: boolean | 'warn' | 'throw';
|
|
43
|
+
/**
|
|
44
|
+
* TLS rewriting (v0.5.0). When set to `'all'`, every browser request is
|
|
45
|
+
* intercepted via Chrome DevTools Protocol's `Fetch.enable` and forwarded
|
|
46
|
+
* through the Go-based `bogdanfinn/tls-client` daemon, which makes the
|
|
47
|
+
* upstream call with a real Chrome TLS ClientHello, real H2 frame
|
|
48
|
+
* settings, and real H2 frame order. The browser never opens an upstream
|
|
49
|
+
* TCP connection — all its HTTP is fulfilled by the daemon.
|
|
50
|
+
*
|
|
51
|
+
* This restores cross-platform UA spoofing (run on Linux, present as
|
|
52
|
+
* Windows or macOS — the daemon controls every header on the wire) and
|
|
53
|
+
* gives total fingerprint control without the cert-installation hell of
|
|
54
|
+
* a TCP-level MITM proxy.
|
|
55
|
+
*
|
|
56
|
+
* Tradeoffs:
|
|
57
|
+
* - WebSocket upgrades cannot be intercepted via Fetch and leak
|
|
58
|
+
* Chrome's native TLS. Mitigation: BlackTip auto-disables QUIC/HTTP3
|
|
59
|
+
* (`--disable-quic`) and the rewriter logs WS leaks for awareness.
|
|
60
|
+
* - Streaming/large response bodies are fully buffered (Fetch.fulfill
|
|
61
|
+
* takes a complete body). Bad for video; fine for HTML pages.
|
|
62
|
+
* - 5–10ms round-trip overhead per request. ~250–500ms added on a
|
|
63
|
+
* typical page with 50 subresources.
|
|
64
|
+
*
|
|
65
|
+
* Requires the Go daemon binary at `native/tls-client/blacktip-tls[.exe]`
|
|
66
|
+
* — see `docs/tls-side-channel.md` for build instructions. If the daemon
|
|
67
|
+
* binary is missing, launch will throw rather than silently falling back.
|
|
68
|
+
*
|
|
69
|
+
* Set to `'off'` (default) for normal operation, `'all'` for full
|
|
70
|
+
* rewriting. Future versions may add `'selective:<domain-glob>'` for
|
|
71
|
+
* per-domain rewriting.
|
|
72
|
+
*/
|
|
73
|
+
tlsRewriting?: 'off' | 'all';
|
|
43
74
|
}
|
|
44
75
|
export type LogLevel = 'debug' | 'info' | 'warn' | 'error';
|
|
45
76
|
export interface ProfileConfig {
|
package/dist/types.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,cAAc;IAC7B,eAAe,CAAC,EAAE,MAAM,GAAG,aAAa,CAAC;IACzC,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gBAAgB,CAAC,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IACrD,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;;;;;;OASG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;;;;;;;;OAaG;IACH,oBAAoB,CAAC,EAAE,OAAO,GAAG,MAAM,GAAG,OAAO,CAAC;
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAEA,MAAM,WAAW,cAAc;IAC7B,eAAe,CAAC,EAAE,MAAM,GAAG,aAAa,CAAC;IACzC,QAAQ,CAAC,EAAE,QAAQ,CAAC;IACpB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,gBAAgB,CAAC,EAAE;QAAE,KAAK,EAAE,MAAM,CAAC;QAAC,MAAM,EAAE,MAAM,CAAA;KAAE,CAAC;IACrD,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;;;;;;OASG;IACH,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB;;;;;;;;;;;;;OAaG;IACH,oBAAoB,CAAC,EAAE,OAAO,GAAG,MAAM,GAAG,OAAO,CAAC;IAClD;;;;;;;;;;;;;;;;;;;;;;;;;;;;;OA6BG;IACH,YAAY,CAAC,EAAE,KAAK,GAAG,KAAK,CAAC;CAC9B;AAED,MAAM,MAAM,QAAQ,GAAG,OAAO,GAAG,MAAM,GAAG,MAAM,GAAG,OAAO,CAAC;AAI3D,MAAM,WAAW,aAAa;IAC5B,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAChC,qBAAqB,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACxC,aAAa,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAChC,kBAAkB,EAAE,QAAQ,GAAG,QAAQ,CAAC;IACxC,YAAY,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC/B,UAAU,EAAE,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IAC7B,WAAW,EAAE,MAAM,CAAC;IACpB,gBAAgB,EAAE,SAAS,GAAG,MAAM,CAAC;IACrC,cAAc,EAAE,MAAM,CAAC;CACxB;AAID,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,mBAAmB,EAAE,MAAM,CAAC;IAC5B,YAAY,EAAE,MAAM,CAAC;IACrB,cAAc,EAAE,MAAM,CAAC;IACvB,WAAW,EAAE,MAAM,CAAC;IACpB,YAAY,EAAE,MAAM,CAAC;IACrB,gBAAgB,EAAE,MAAM,CAAC;IACzB,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,MAAM,CAAC;IACpB,aAAa,EAAE,MAAM,CAAC;IACtB,SAAS,EAAE,MAAM,EAAE,CAAC;IACpB,OAAO,EAAE,UAAU,EAAE,CAAC;IACtB,KAAK,EAAE,MAAM,EAAE,CAAC;CACjB;AAED,MAAM,WAAW,UAAU;IACzB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,MAAM,CAAC;IACjB,SAAS,EAAE;QAAE,IAAI,EAAE,MAAM,CAAC;QAAC,QAAQ,EAAE,MAAM,CAAC;QAAC,WAAW,EAAE,MAAM,CAAA;KAAE,EAAE,CAAC;CACtE;AAID,MAAM,WAAW,YAAY;IAC3B,OAAO,EAAE,OAAO,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,MAAM,CAAC;CACpB;AAED,MAAM,WAAW,cAAc;IAC7B,OAAO,EAAE,OAAO,CAAC;IACjB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,gBAAgB;IAC/B,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,KAAK,GAAG,MAAM,CAAC;IACvB,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,UAAU;IACzB,OAAO,EAAE,OAAO,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAID,MAAM,WAAW,eAAe;IAC9B,SAAS,CAAC,EAAE,MAAM,GAAG,kBAAkB,GAAG,aAAa,CAAC;IACxD,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED;;;;;GAKG;AACH,MAAM,MAAM,gBAAgB,GAAG,KAAK,GAAG,QAAQ,GAAG,MAAM,CAAC;AAEzD,MAAM,WAAW,YAAY;IAC3B,MAAM,CAAC,EAAE,MAAM,GAAG,OAAO,GAAG,QAAQ,CAAC;IACrC,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,gBAAgB,CAAC;CAC/B;AAED,MAAM,WAAW,WAAW;IAC1B,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,UAAU,CAAC,EAAE,OAAO,CAAC;IACrB,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,gBAAgB,CAAC;CAC/B;AAED,MAAM,WAAW,aAAa;IAC5B,SAAS,CAAC,EAAE,IAAI,GAAG,MAAM,GAAG,MAAM,GAAG,OAAO,CAAC;IAC7C,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,MAAM,CAAC,EAAE,OAAO,CAAC;CAClB;AAED,MAAM,WAAW,YAAY;IAC3B,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,aAAa;IAC5B,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,eAAe;IAC9B,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,iBAAiB;IAChC,MAAM,CAAC,EAAE,MAAM,GAAG,OAAO,GAAG,UAAU,CAAC;IACvC,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,kBAAkB;IACjC,QAAQ,CAAC,EAAE,OAAO,CAAC;CACpB;AAED,MAAM,WAAW,cAAc;IAC7B,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;IAClB,MAAM,CAAC,EAAE,OAAO,CAAC;CAClB;AAED,MAAM,WAAW,wBAAwB;IACvC,SAAS,CAAC,EAAE,MAAM,GAAG,kBAAkB,GAAG,aAAa,CAAC;IACxD,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,MAAM,WAAW,iBAAiB;IAChC,QAAQ,CAAC,EAAE,OAAO,CAAC;IACnB,MAAM,CAAC,EAAE,KAAK,GAAG,MAAM,CAAC;IACxB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,IAAI,CAAC,EAAE,MAAM,CAAC;CACf;AAED,MAAM,WAAW,kBAAkB;IACjC,MAAM,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;CAC1B;AAID,MAAM,WAAW,OAAO;IACtB,KAAK,EAAE,MAAM,CAAC;IACd,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,OAAO,CAAC;CACjB;AAED,MAAM,WAAW,SAAS;IACxB,IAAI,EAAE,MAAM,CAAC;IACb,GAAG,EAAE,MAAM,CAAC;IACZ,KAAK,EAAE,MAAM,CAAC;CACf;AAID,MAAM,WAAW,WAAW;IAC1B,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,SAAS,GAAG,SAAS,CAAC;IAC/B,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;IAChB,UAAU,CAAC,EAAE,kBAAkB,CAAC;IAChC,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAED,MAAM,WAAW,kBAAkB;IACjC,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,cAAc,CAAC,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,MAAM,CAAC;IAClB,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,EAAE,MAAM,CAAC;IAChB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,MAAM,CAAC;IACf,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,MAAM,CAAC;IAClB,MAAM,EAAE,MAAM,CAAC;IACf,MAAM,EAAE,MAAM,CAAC;IACf,OAAO,EAAE,MAAM,CAAC;IAChB,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,aAAa,CAAC;IACxB,KAAK,EAAE,MAAM,CAAC;CACf;AAED,MAAM,MAAM,aAAa,GAAG,UAAU,GAAG,MAAM,GAAG,QAAQ,GAAG,aAAa,GAAG,QAAQ,GAAG,eAAe,CAAC;AAExG,MAAM,WAAW,cAAc;IAC7B,SAAS,EAAE,MAAM,CAAC;IAClB,QAAQ,EAAE,MAAM,CAAC;IACjB,GAAG,EAAE,MAAM,CAAC;IACZ,MAAM,EAAE,QAAQ,GAAG,QAAQ,GAAG,UAAU,CAAC;CAC1C;AAED,MAAM,WAAW,QAAQ;IACvB,SAAS,EAAE,MAAM,CAAC;IAClB,KAAK,EAAE,QAAQ,CAAC;IAChB,OAAO,EAAE,MAAM,CAAC;IAChB,IAAI,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CAChC;AAID,eAAO,MAAM,UAAU;;;;;;;;;;;CAWb,CAAC;AAEX,MAAM,MAAM,SAAS,GAAG,OAAO,UAAU,CAAC,MAAM,OAAO,UAAU,CAAC,CAAC;AAInE,MAAM,WAAW,KAAK;IACpB,CAAC,EAAE,MAAM,CAAC;IACV,CAAC,EAAE,MAAM,CAAC;CACX;AAED,MAAM,WAAW,WAAW;IAC1B,CAAC,EAAE,MAAM,CAAC;IACV,CAAC,EAAE,MAAM,CAAC;IACV,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,WAAW,aAAa;IAC5B,QAAQ,EAAE,MAAM,CAAC;IACjB,WAAW,EAAE,WAAW,CAAC;IACzB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,OAAO,CAAC;IACnB,UAAU,CAAC,EAAE,MAAM,CAAC;CACrB"}
|
package/dist/types.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.js","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,sBAAsB;
|
|
1
|
+
{"version":3,"file":"types.js","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA,sBAAsB;AAoTtB,oBAAoB;AAEpB,MAAM,CAAC,MAAM,UAAU,GAAG;IACxB,OAAO,EAAE,SAAS;IAClB,iBAAiB,EAAE,mBAAmB;IACtC,iBAAiB,EAAE,mBAAmB;IACtC,QAAQ,EAAE,UAAU;IACpB,aAAa,EAAE,eAAe;IAC9B,eAAe,EAAE,iBAAiB;IAClC,eAAe,EAAE,iBAAiB;IAClC,WAAW,EAAE,aAAa;IAC1B,eAAe,EAAE,iBAAiB;IAClC,YAAY,EAAE,cAAc;CACpB,CAAC"}
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
# Akamai sensor challenge solver (v0.5.0)
|
|
2
|
+
|
|
3
|
+
`bt.solveAkamaiChallenge(url)` is the v0.5.0 answer to "I want to call Akamai-protected APIs from a sessionless TLS daemon, but the first request always 403s because Akamai gates everything behind a sensor data POST." Solve the challenge once in a real browser, get back the validated cookies plus a recommended header set, then replay arbitrary requests via `bt.fetchWithTls()` (or any other HTTP client) for as long as the cookies stay valid. **Empirically validated against OpenTable Akamai Bot Manager: 5/5 replay calls return 200 with real content. ~600ms per replay vs ~4s per browser launch.**
|
|
4
|
+
|
|
5
|
+
## Why this isn't a pure-Go solver
|
|
6
|
+
|
|
7
|
+
Reverse-engineering Akamai's `bm.js` to generate sensor data without a browser is intentionally hostile work and the maintenance economics are bad:
|
|
8
|
+
|
|
9
|
+
1. **bm.js is heavily obfuscated.** OpenTable's current sensor JS is 26 KB of hex-encoded string array references with no recognizable function names. References to `gyroscope`, `hardwareConcurrency`, `selenium`, `Chrome`, `vendor`, `ShockwaveFlash` are scattered through it — clearly the sensor collector — but extracting them requires symbolic execution, not just regex.
|
|
10
|
+
2. **Sensor data is encrypted with a runtime-derived key** that lives inside the obfuscated code. You can't just capture the POST body Chrome sends and replay it — the key changes per session.
|
|
11
|
+
3. **Akamai rotates the obfuscation monthly.** A pure-Go reimplementation would be a 1–2 week reverse engineering project, and the result would rot in ~6 weeks. Bad ROI.
|
|
12
|
+
|
|
13
|
+
What works instead, and what BlackTip ships in v0.5.0: launch a real BlackTip browser, navigate to the URL, let Akamai's bm.js execute naturally (real Chrome runs the JS, generates the sensor payload, POSTs it back), and capture the validated cookies. The caller then injects those cookies into thousands of sessionless TLS-daemon API calls until they expire.
|
|
14
|
+
|
|
15
|
+
**This is NOT "no browser needed for Akamai." It IS "amortize browser cost across many subsequent API calls instead of paying it per request."** For most use cases that's the same thing — you pay one browser session per hour and run hundreds of API calls in between.
|
|
16
|
+
|
|
17
|
+
## Quick start
|
|
18
|
+
|
|
19
|
+
```typescript
|
|
20
|
+
import { BlackTip } from '@rester159/blacktip';
|
|
21
|
+
|
|
22
|
+
const bt = new BlackTip({ logLevel: 'info' });
|
|
23
|
+
await bt.launch();
|
|
24
|
+
|
|
25
|
+
// 1. Solve the Akamai challenge in the browser. ~15s.
|
|
26
|
+
const solved = await bt.solveAkamaiChallenge(
|
|
27
|
+
'https://www.opentable.com/booking/restref/availability?rid=76651&restref=76651&partySize=2&dateTime=2026-04-11T19:00',
|
|
28
|
+
);
|
|
29
|
+
|
|
30
|
+
console.log('Validated:', solved.validated);
|
|
31
|
+
console.log('Akamai cookies:', solved.cookies.map(c => c.name));
|
|
32
|
+
// → [ 'bm_ss', 'bm_so', 'bm_mi', 'bm_sz', 'ak_bmsc', 'bm_s', 'bm_sv', '_abck' ]
|
|
33
|
+
|
|
34
|
+
// 2. Replay arbitrary requests via the TLS daemon. ~600ms each, no browser.
|
|
35
|
+
for (let i = 0; i < 100; i++) {
|
|
36
|
+
const resp = await bt.fetchWithTls({
|
|
37
|
+
url: 'https://www.opentable.com/api/some-endpoint',
|
|
38
|
+
headers: solved.recommendedHeaders, // Cookie + Sec-Ch-Ua + Sec-Fetch-* baked in
|
|
39
|
+
});
|
|
40
|
+
console.log('Call', i, '→', resp.status);
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
await bt.close();
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
## Result shape
|
|
47
|
+
|
|
48
|
+
```typescript
|
|
49
|
+
interface AkamaiChallengeResult {
|
|
50
|
+
/**
|
|
51
|
+
* True when EITHER:
|
|
52
|
+
* - _abck reached validated state (~0~), OR
|
|
53
|
+
* - The page rendered without an Akamai block (sensor not enforced).
|
|
54
|
+
*
|
|
55
|
+
* Akamai's sensor validation is only enforced when other signals
|
|
56
|
+
* (TLS, IP, behavior) look suspicious. For real-Chrome sessions on
|
|
57
|
+
* residential connections, Akamai often admits the request without
|
|
58
|
+
* ever requiring the JS-layer sensor POST.
|
|
59
|
+
*/
|
|
60
|
+
validated: boolean;
|
|
61
|
+
|
|
62
|
+
/**
|
|
63
|
+
* Actual sensor validation state:
|
|
64
|
+
* 0 → validated as human (gold standard)
|
|
65
|
+
* -1 → sensor not enforced (page admitted without it)
|
|
66
|
+
* 1+ → flagged as bot
|
|
67
|
+
* null → no _abck cookie set (target may not be Akamai-protected)
|
|
68
|
+
*/
|
|
69
|
+
abckState: -1 | 0 | 1 | null;
|
|
70
|
+
|
|
71
|
+
/** The full _abck cookie value at the end of the wait window. */
|
|
72
|
+
abckValue: string | null;
|
|
73
|
+
|
|
74
|
+
/** Whether the rendered page is the Akamai Access Denied block page. */
|
|
75
|
+
blocked: boolean;
|
|
76
|
+
|
|
77
|
+
/** All Akamai-related cookies, ready to inject into other sessions. */
|
|
78
|
+
cookies: Array<{ name: string; value: string; domain: string; path: string }>;
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* Pre-built header set for replay calls. Includes Cookie, User-Agent,
|
|
82
|
+
* Accept, Accept-Language, Sec-Ch-Ua, Sec-Ch-Ua-Mobile, Sec-Ch-Ua-Platform,
|
|
83
|
+
* Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site, Sec-Fetch-User,
|
|
84
|
+
* Upgrade-Insecure-Requests. Pass directly to `bt.fetchWithTls()`.
|
|
85
|
+
*
|
|
86
|
+
* Replays without these headers will 403 even with valid cookies —
|
|
87
|
+
* Akamai validates the full request shape, not just the cookie jar.
|
|
88
|
+
*/
|
|
89
|
+
recommendedHeaders: Record<string, string>;
|
|
90
|
+
|
|
91
|
+
finalUrl: string;
|
|
92
|
+
title: string;
|
|
93
|
+
durationMs: number;
|
|
94
|
+
notes: string[];
|
|
95
|
+
}
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
## The cost amortization story
|
|
99
|
+
|
|
100
|
+
OpenTable's Gjelina booking endpoint, measured on a residential connection:
|
|
101
|
+
|
|
102
|
+
| Approach | Cost per call | 100 calls |
|
|
103
|
+
|---|---|---|
|
|
104
|
+
| Browser launch + navigate per call | ~4s | ~400s |
|
|
105
|
+
| `solveAkamaiChallenge` once + 100 daemon replays | 15s + 100×0.6s = 75s | **75s** |
|
|
106
|
+
| **Speedup** | | **~5.3x** |
|
|
107
|
+
|
|
108
|
+
Larger N gets better. At 1,000 calls, it's 15s + 600s = 615s vs 4,000s — almost 7x. The crossover is at ~5 calls (below that, browser-per-call is faster because the solve overhead dominates).
|
|
109
|
+
|
|
110
|
+
## What "validated" actually means
|
|
111
|
+
|
|
112
|
+
Akamai's sensor validation has three observable states encoded in the second `~`-delimited field of the `_abck` cookie:
|
|
113
|
+
|
|
114
|
+
| `_abck` state | Meaning | What you can do |
|
|
115
|
+
|---|---|---|
|
|
116
|
+
| `~0~` | Sensor data validated as human | Maximum trust — replay anything |
|
|
117
|
+
| `~-1~` | Sensor not enforced (Akamai admitted on other signals) | Replay safely — same as `~0~` for most APIs |
|
|
118
|
+
| `~1~` (or higher) | Sensor flagged as bot | Session burned — solve again with a different identity |
|
|
119
|
+
|
|
120
|
+
The interesting case is `~-1~`. On every empirical test against OpenTable from a residential connection, Akamai admitted the request without ever requiring sensor validation — `_abck` stayed at `~-1~` but the page rendered fine and the cookies worked for daemon replays. This is consistent with Akamai's own marketing: the sensor JS is one layer of a multi-factor decision, and high-confidence requests (good TLS, good IP, real Chrome behavior) get admitted without it.
|
|
121
|
+
|
|
122
|
+
That's why `validated` is `true` for both `~0~` and `~-1~` outcomes — both unlock the replay path.
|
|
123
|
+
|
|
124
|
+
## Replay headers — why all of them matter
|
|
125
|
+
|
|
126
|
+
The first thing I tried after solving was naive: solve in browser, copy cookies, pass them through `Cookie:` header to the daemon. **It 403'd.** Then I added the full Chrome header set: `Sec-Ch-Ua`, `Sec-Ch-Ua-Mobile`, `Sec-Ch-Ua-Platform`, `Sec-Fetch-Dest`, `Sec-Fetch-Mode`, `Sec-Fetch-Site`, `Sec-Fetch-User`, `Upgrade-Insecure-Requests`. **It returned 200.**
|
|
127
|
+
|
|
128
|
+
Akamai is validating the full request shape, not just the cookie jar. Without the Sec-Fetch-* headers, the request looks like a programmatic fetch and Akamai blocks it even with valid cookies. With them, the request looks like a navigation from a real Chrome and Akamai admits it.
|
|
129
|
+
|
|
130
|
+
The `recommendedHeaders` field on the solver result includes all of these. Don't strip them; pass the whole object to `bt.fetchWithTls({ url, headers: solved.recommendedHeaders })`.
|
|
131
|
+
|
|
132
|
+
## Combining with IdentityPool
|
|
133
|
+
|
|
134
|
+
If you're running long-lived flows with identity rotation, the natural pattern is to attach the solved Akamai cookies to an IdentityPool snapshot:
|
|
135
|
+
|
|
136
|
+
```typescript
|
|
137
|
+
import { BlackTip, IdentityPool } from '@rester159/blacktip';
|
|
138
|
+
|
|
139
|
+
const pool = new IdentityPool({ storePath: './.bt/identities.json' });
|
|
140
|
+
const identity = pool.acquire('opentable.com')!;
|
|
141
|
+
|
|
142
|
+
const config = pool.applyToConfig(identity);
|
|
143
|
+
const bt = new BlackTip(config);
|
|
144
|
+
await bt.launch();
|
|
145
|
+
await pool.restoreSnapshot(bt, identity);
|
|
146
|
+
|
|
147
|
+
// Solve Akamai once for this identity
|
|
148
|
+
const solved = await bt.solveAkamaiChallenge('https://www.opentable.com/booking/...');
|
|
149
|
+
if (!solved.validated) {
|
|
150
|
+
pool.markBurned(identity.id, 'Akamai blocked', 'opentable.com');
|
|
151
|
+
return;
|
|
152
|
+
}
|
|
153
|
+
|
|
154
|
+
// Save the post-solve session state into the identity for next time
|
|
155
|
+
await pool.captureSnapshot(bt, identity);
|
|
156
|
+
|
|
157
|
+
// Run N daemon replays. When _abck eventually expires, re-solve.
|
|
158
|
+
for (let i = 0; i < 100; i++) {
|
|
159
|
+
const resp = await bt.fetchWithTls({
|
|
160
|
+
url: 'https://www.opentable.com/api/...',
|
|
161
|
+
headers: solved.recommendedHeaders,
|
|
162
|
+
});
|
|
163
|
+
// ...
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
await bt.close();
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Now your identity is durable: cookies + storage + solved Akamai state, all persisted, ready to resume tomorrow without re-solving.
|
|
170
|
+
|
|
171
|
+
## Limitations
|
|
172
|
+
|
|
173
|
+
- **Still requires a browser to solve.** This is the whole point of the architecture decision documented above. If you need pure-Go API access without ever launching a browser, you're going to write a lot of obfuscation reverse-engineering code that breaks every 6 weeks. v0.5.0 doesn't ship that path.
|
|
174
|
+
- **Cookies expire.** Akamai's session window is typically ~1 hour for the validated state. After that, replays start returning 403 again and you need to re-solve. Re-solving from the same browser context is fast (~3s on subsequent calls because the browser already has the prior state).
|
|
175
|
+
- **`_abck` flagged state means session burned.** If `abckState === 1`, the cookies are useless — Akamai marked you as a bot and even the browser session won't recover. You need a fresh BlackTip launch with a different identity (different proxy, fresh user data dir, possibly different device profile).
|
|
176
|
+
- **Per-domain.** This solver is tested against OpenTable. The pattern works against any Akamai Bot Manager target but the specific URL format and timing may vary. Adjust `dwellMsBeforePolling` and `timeoutMs` per-target.
|
|
177
|
+
|
|
178
|
+
## See also
|
|
179
|
+
|
|
180
|
+
- `docs/tls-side-channel.md` — the underlying `bt.fetchWithTls()` daemon
|
|
181
|
+
- `docs/tls-rewriting.md` — the v0.5.0 full-rewriting mode (TLS rewriter intercepts every browser request)
|
|
182
|
+
- `docs/identity-pool.md` — long-running session and identity rotation
|
|
183
|
+
- `docs/akamai-bypass.md` — the v0.2.0 plan that documents Akamai's detection layer stack and the L016 fix that opened the door
|