tunnelfetch 1.2.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -278,6 +278,32 @@ it saves do not pay it back — see [What this cannot do](#what-this-cannot-do-a
278
278
  are validated as HTTP tokens, a decoder that throws fails the body closed rather than truncating it,
279
279
  and an unregistered coding is still refused.
280
280
 
281
+ ### The Chrome identity, in one import
282
+
283
+ ```js
284
+ import { Client } from 'tunnelfetch';
285
+ import { chrome } from 'tunnelfetch/profile/chrome';
286
+
287
+ const client = new Client({ profile: chrome, connect, proxy, decoders: { br, zstd } });
288
+ ```
289
+
290
+ The subpath carries the two primitives this runtime has no native path for — ML-KEM-768 for the
291
+ `X25519MLKEM768` key exchange and ChaCha20-Poly1305 for the record layer, both compiled to
292
+ freestanding WASM and both with known-answer tests in this repository. **Importing it is the
293
+ opt-in:** a bundler pulls them in only for code on this path, so the default identity carries none
294
+ of it.
295
+
296
+ `br` and `zstd` stay yours. They are not cryptography and there is no single right implementation,
297
+ so the profile keeps refusing until you supply them — a Chrome that advertises `br` and cannot read
298
+ it is worse than one that says so.
299
+
300
+ Verified end to end, not merely constructed:
301
+
302
+ | Origin | | TLS | Group | HTTP |
303
+ |---|---|---|---|---|
304
+ | `blog.cloudflare.com` | 200 | 1.3 | `0x11ec` X25519MLKEM768 | h2 |
305
+ | `www.shopify.com` | 200 | 1.3 | `0x11ec` X25519MLKEM768 | h2 |
306
+
281
307
  ### HTTP/2 — access, not speed
282
308
 
283
309
  The client offers `h2` and `http/1.1` in ALPN by default and speaks whichever the server selects.
@@ -344,16 +370,19 @@ trades a fingerprint mismatch for a broken handshake, which is worse and fails s
344
370
 
345
371
  | curl sends | This package | Why |
346
372
  |---|---|---|
347
- | 30 cipher suites, incl. RSA key exchange and CBC | 6 AEAD suites | Not implemented, by design. A server selecting `TLS_RSA_WITH_AES_256_CBC_SHA` would get a dead connection |
348
- | `X25519MLKEM768` group and a 1216-byte key share | not offered | ML-KEM is not implemented |
373
+ | 30 cipher suites, incl. ChaCha20, RSA key exchange and CBC | 6 AEAD suites, or 7 with ChaCha20 injected | RSA-kx and CBC are refused by design a server selecting `TLS_RSA_WITH_AES_256_CBC_SHA` would get a dead connection. `TLS_CHACHA20_POLY1305_SHA256` is injectable via `ciphers: { chacha20 }`: WebCrypto has no ChaCha20 here, so an implementation is supplied rather than a `node:crypto` dependency taken |
374
+ | `X25519MLKEM768` group and a 1216-byte key share | offered only when injected | ML-KEM is not a WebCrypto primitive; supply it as `groups: { x25519mlkem768 }` (what `profiles.chrome` requires) and the group and its 1216-byte hybrid key share go on the wire |
349
375
  | SHA-1 signature schemes | not offered | Refused deliberately |
350
376
  | `encrypt_then_mac` | not sent | Applies only to CBC suites, which are not offered |
351
377
  | `post_handshake_auth` | not sent | Invites a post-handshake `CertificateRequest`, which is not implemented |
352
378
  | — | `status_request` | curl does not ask for a stapled OCSP response; this package must, because a staple is its only revocation signal |
353
379
 
354
- Closing that gap means implementing ML-KEM, RSA key exchange and CBC suites — a different project,
355
- and the last two are things this package refuses on purpose. `tls.ciphers` and `tls.groups` will let
356
- you offer them anyway; the handshake will then fail if a server picks one, and that is yours to own.
380
+ ChaCha20-Poly1305 and X25519MLKEM768 are reachable by **injection**: an implementation of each is
381
+ supplied through `ciphers` / `groups`, which is precisely what `profiles.chrome` requires, and
382
+ neither is offered unless one is a suite or group advertised but not performable is a dead
383
+ connection if a server takes it. RSA key exchange and CBC suites stay refused on purpose;
384
+ `tls.ciphers` and `tls.groups` will let you offer them anyway, and the handshake will then fail if a
385
+ server picks one, which is yours to own.
357
386
 
358
387
  A test asserts this delta is exactly the list above, so gaining one of these capabilities without
359
388
  updating the table fails the build.
@@ -381,7 +410,9 @@ res.tunnelfetch.httpVersion; // '2' if the server chose h2, '1.1' otherwise
381
410
  | `connect` | — | Socket factory. On Workers, the `connect` export of `cloudflare:sockets`. Required for anything the platform's `fetch` cannot serve. |
382
411
  | `proxy` | `null` | URL string or object. `http:`, `https:`, `socks5:`, `socks5h:`. |
383
412
  | `trust` | `{mode:'system'}` | Certificate policy; see below. |
384
- | `tls` | `{}` | Handshake options (`alpn`, `groups`, `ciphers`, `offerGroups`). |
413
+ | `tls` | `{}` | Handshake options (`alpn`, `groups`, `ciphers`, `offerGroups`). Here `groups`/`ciphers` are number lists — the suite and group ids to offer, in preference order. |
414
+ | `ciphers` | `{}` | Injected AEAD implementations by capability name: `{ chacha20 }` (a `seal`/`open` pair). WebCrypto has no ChaCha20 here, so `TLS_CHACHA20_POLY1305_SHA256` is offered only when this is supplied. Required by `profiles.chrome`. |
415
+ | `groups` | `{}` | Injected key-exchange implementations by capability name: `{ x25519mlkem768 }` (ML-KEM-768 `keygen`/`encapsulate`/`decapsulate`). ML-KEM is not a WebCrypto primitive, so the post-quantum hybrid group is offered only when this is supplied. Required by `profiles.chrome`. |
385
416
  | `timeouts` | see below | `connectMs`, `handshakeMs`, `headersMs`, `idleMs`, `totalMs`. |
386
417
  | `cookies` | `false` | Enable a per-Client cookie jar. |
387
418
  | `maxRedirects` | `20` | |
@@ -493,9 +524,15 @@ Not implemented, and not planned:
493
524
  since 2020. Note the distinction: a server that *also* supports 1.0/1.1 is fine, because we will
494
525
  negotiate 1.2 or 1.3 with it. Only a server that supports *nothing else* is out of reach.
495
526
  - **RSA key transport.** No forward secrecy.
496
- - **ChaCha20-Poly1305.** It buys nothing: a server can only pick a suite we offered, TLS 1.3
497
- mandates AES-128-GCM, and AES-GCM is universal in TLS 1.2 deployments. WebCrypto has no
498
- ChaCha20, so offering it would mean a `node:crypto` dependency for zero compatibility gain.
527
+ - **ChaCha20-Poly1305, unless injected.** WebCrypto has no ChaCha20 on this runtime, so it is not
528
+ built in taking a `node:crypto` dependency would cost the package its "web platform only"
529
+ property. It is *injectable*, though: pass `ciphers: { chacha20 }` (a `seal`/`open` pair, e.g. a
530
+ WASM build) and `TLS_CHACHA20_POLY1305_SHA256` is offered — in curl's captured position, second
531
+ after AES-256-GCM — and used. Not built in because a server can only pick a suite we offered, TLS
532
+ 1.3 mandates AES-128-GCM, and AES-GCM is universal in TLS 1.2; the reason to add it is matching a
533
+ browser that offers it (`profiles.chrome`), not compatibility. The post-quantum
534
+ **X25519MLKEM768** group is injectable the same way — `groups: { x25519mlkem768 }` — for the same
535
+ reason: ML-KEM is not a WebCrypto primitive here.
499
536
  - **Client certificates (mTLS), 0-RTT, renegotiation.** A `HelloRequest` is refused rather than
500
537
  honoured. 0-RTT is a decision rather than an omission: early data can be replayed, so offering
501
538
  it would let an attacker who captured a POST replay it. (Session resumption itself *is*
@@ -565,25 +602,28 @@ Fetching a size-controlled origin through a proxy, warm, medians over seven-plus
565
602
  isolate, gzip on the wire. The last column is the same numbers as a rate, which is the form worth
566
603
  carrying around:
567
604
 
568
- | | New connection | Each further request, same connection | Marginal rate |
569
- | --- | --- | --- | --- |
570
- | 1 KB body | 11 ms | 2–3 ms | — |
571
- | 16 KB body | 11 ms | 3 ms | — |
572
- | 64 KB body | 11 ms | 2–4 ms | — |
573
- | 256 KB body | 7 ms | 3.4 ms | ~13 ms/MB |
574
- | 1 MB body | 11 ms | 6–12 ms | ~9 ms/MB |
575
- | 4 MB body | 31 ms | 21–35 ms | ~7 ms/MB |
576
- | **Same 16 KB page over HTTP/2** | 12 ms | 2.2 ms | — |
577
- | **First request in a fresh isolate** | 46 ms | — | — |
578
- | **…after `warmup({ iterations: 5 })`** | 16 ms | — | — |
579
-
580
- One model fits every body-size row to within its spread:
581
-
582
- > **≈ 9.5 ms to open a connection + 2 ms per request + 5–8 ms per MB of body**
583
-
584
- The connection term recovered independently from each row lands between 5 and 11 ms, agreeing with
585
- the 9–12 ms measured for a new connection by other means. Most of it is the TLS handshake and
586
- certificate chain validation; almost none of it is parsing (see below).
605
+ | | New connection (first request included) | Each further request, same connection |
606
+ | --- | --- | --- |
607
+ | 1 KB body | 6.4 ms | 1.6 ms |
608
+ | 16 KB body | 6.5 ms | 1.7 ms |
609
+ | 64 KB body | 6.7 ms | 1.9 ms |
610
+ | 256 KB body | 7.4 ms | 2.6 ms |
611
+ | 1 MB body | 10.4 ms | 5.6 ms |
612
+ | 4 MB body | 22.4 ms | 17.6 ms |
613
+ | **First request in a fresh isolate** | 46 ms | — |
614
+ | **…after `warmup({ iterations: 5 })`** | 16 ms | — |
615
+
616
+ Re-measured for 1.4.0 against a size-controlled origin through a proxy, ten rounds per size, HTTP/2
617
+ negotiated, gzip on the wire. The rows are not independent measurements — they are one model, fitted
618
+ to the sweep and then checked back against it:
619
+
620
+ > **≈ 6.4 ms to open a connection + 1.6 ms per further request + ~4 ms per MB of body**
621
+
622
+ Each sweep request fetches five pages — one on a fresh connection and four reusing it so the two
623
+ terms were separated by varying the reuse count rather than assumed: two pages against ten gives
624
+ 1.63 ms per further request, and the connection term falls out of the remainder. Fitted on 1 KB and
625
+ 4 MB, the model predicts 32.9 ms for the 1 MB row against 35 measured, and 92.9 for the 4 MB row
626
+ against 94.
587
627
 
588
628
  The ranges are real, not imprecision: absolute CPU on this platform varies by up to ~1.5× between
589
629
  isolates and runs — the same sweep repeated lands on faster and slower machines — so the values are
@@ -609,6 +649,31 @@ excess above the warm floor without `warmup()`, 15 ms with it at five iterations
609
649
  1.1 ms per request respectively, amortised over an isolate's early life. Warming costs 10 ms of
610
650
  startup at one iteration and 22 ms at five, against a 1 s budget, and does not lower the warm floor.
611
651
 
652
+ ### What the optional switches cost
653
+
654
+ Everything above is the default identity: curl's fingerprint, gzip and deflate, no post-quantum, no
655
+ GREASE. Each switch below is off unless asked for, and the table is what asking costs. Measured on
656
+ the edge the same way as the rest — differencing two work counts, minimum of samples.
657
+
658
+ | Switch | Cost | Paid |
659
+ |---|---|---|
660
+ | `grease: true` | not measurable | a handful of extra bytes in one hello |
661
+ | `tls.extensionOrder: 'shuffle'` | not measurable | shuffling ~11 items, once per handshake |
662
+ | `headerOrder` | not measurable — the ordered list is *faster* than the platform `Headers` (1.6 µs against 3.8 µs) | per request |
663
+ | `groups: { x25519mlkem768 }` | **+0.15 ms** with the bundled WASM, **+1.35 ms** with a pure-JS ML-KEM | per **connection**, not per request — amortised across every request that reuses it |
664
+ | `ciphers: { chacha20 }` | **+2.0 ms/MB**, and only if the server *selects* it | per byte. Servers with AES hardware generally prefer AES-GCM, so the usual cost is zero and the offer is what matters |
665
+ | `decoders: { br }` | **+4.4 ms/MB** | per byte, whenever an origin serves brotli. Harder compression is worse, not better: quality 11 is 16% smaller on the wire and 46% dearer to decode |
666
+ | `profile: chrome` | the sum of the three above | |
667
+
668
+ Two defaults moved in 1.4.0 and neither is visible in the table above them: matching curl's cipher
669
+ order means AES-256-GCM is negotiated where AES-128-GCM used to be, measured at **+4%** per MB
670
+ (1.50 against 1.45 ms/MB — hardware AES makes the extra rounds cheap), and the ordered header list
671
+ replaced the platform `Headers`, which is slightly *cheaper*. Both are inside the ±1.5× spread the
672
+ figures above already carry.
673
+
674
+ Bundle: importing `tunnelfetch/profile/chrome` adds **~22 KB gzipped** for the two WASM primitives.
675
+ Nothing else imports them, so a caller on the default identity carries none of it.
676
+
612
677
  ### What that costs in dollars
613
678
 
614
679
  Workers Standard bills $5/month including 10 million requests and 30 million CPU milliseconds, then
package/README.zh-CN.md CHANGED
@@ -219,6 +219,26 @@ const client = new Client({ connect, proxy, decoders: { br: brotli } });
219
219
 
220
220
  brotli 本身落在原生 inflate 的 1.9 倍。这个差价就是这个编码的成本,而它省下的线上字节买不回来——见[这个包做不到什么](#这个包做不到什么为什么)。解码器名字会按 HTTP token 校验;解码器抛错时响应体 fail closed,而不是被截断;未注册的编码依然会被拒绝。
221
221
 
222
+ ### 一行 import 得到 Chrome 身份
223
+
224
+ ```js
225
+ import { Client } from 'tunnelfetch';
226
+ import { chrome } from 'tunnelfetch/profile/chrome';
227
+
228
+ const client = new Client({ profile: chrome, connect, proxy, decoders: { br, zstd } });
229
+ ```
230
+
231
+ 这个子路径带着运行时没有原生实现的两个原语——`X25519MLKEM768` 密钥交换要的 ML-KEM-768,以及记录层要的 ChaCha20-Poly1305,都编译成无依赖的 WASM,并且都在本仓库里有已知答案测试。**导入即 opt-in**:打包器只为走这条路径的代码拉入它们,默认身份一个字节都不背。
232
+
233
+ `br` 和 `zstd` 仍然要你自己带。它们不是密码学、没有唯一正解,所以 profile 会一直拒绝到你提供为止——**一个声称支持 br 却读不了的 Chrome,比一个老实说不支持的更糟**。
234
+
235
+ 不是「能构造出来」,是端到端验证过:
236
+
237
+ | 源站 | | TLS | 群 | HTTP |
238
+ |---|---|---|---|---|
239
+ | `blog.cloudflare.com` | 200 | 1.3 | `0x11ec` X25519MLKEM768 | h2 |
240
+ | `www.shopify.com` | 200 | 1.3 | `0x11ec` X25519MLKEM768 | h2 |
241
+
222
242
  ### HTTP/2 — 要的是访问,不是速度
223
243
 
224
244
  客户端默认在 ALPN 中同时报出 `h2` 与 `http/1.1`,服务器选中哪个就说哪个。没有单独的 API:
@@ -410,17 +430,22 @@ CertificateError [CERT_PIN_MISMATCH]: no certificate in the chain matches any co
410
430
  通过代理抓取一个尺寸可控的源站,热态,同一 isolate 上 7 轮以上取中位数,传输走 gzip。最后一列是同样的数字
411
431
  换算成速率,那是更值得随身记住的形式:
412
432
 
413
- | | 新连接 | 同一连接上的后续请求 | 边际速率 |
414
- | --- | --- | --- | --- |
415
- | 1 KB body | 11 ms | 2–3 ms | — |
416
- | 16 KB body | 11 ms | 3 ms | — |
417
- | 64 KB body | 11 ms | 2–4 ms | — |
418
- | 256 KB body | 7 ms | 3.4 ms | ~13 ms/MB |
419
- | 1 MB body | 11 ms | 6–12 ms | ~9 ms/MB |
420
- | 4 MB body | 31 ms | 21–35 ms | ~7 ms/MB |
421
- | **同一个 16 KB 页面走 HTTP/2** | 12 ms | 2.2 ms | — |
422
- | **全新 isolate 的第一个请求** | 46 ms | | — |
423
- | **……调用 `warmup({ iterations: 5 })` 之后** | 16 ms | — | — |
433
+ | | 新连接(含首个请求) | 同连接每次后续请求 |
434
+ | --- | --- | --- |
435
+ | 1 KB body | 6.4 ms | 1.6 ms |
436
+ | 16 KB body | 6.5 ms | 1.7 ms |
437
+ | 64 KB body | 6.7 ms | 1.9 ms |
438
+ | 256 KB body | 7.4 ms | 2.6 ms |
439
+ | 1 MB body | 10.4 ms | 5.6 ms |
440
+ | 4 MB body | 22.4 ms | 17.6 ms |
441
+ | **全新 isolate 的第一个请求** | 46 ms | — |
442
+ | **…执行 `warmup({ iterations: 5 })` 之后** | 16 ms | — |
443
+
444
+ 为 1.4.0 重新测量:经代理打一个尺寸可控的源站,每个尺寸十轮,协商 HTTP/2,线上走 gzip。这些行**不是各自独立的测量**,而是一个模型拟合出来再回代验证的:
445
+
446
+ > **≈ 开一条连接 6.4 ms + 每次后续请求 1.6 ms + 每 MB body 约 4 ms**
447
+
448
+ 每次扫描请求会抓五个页面——一个走新连接、四个复用它——所以两项是靠**改变复用次数**分离出来的,不是假设的:两页对十页得出每次后续请求 1.63 ms,连接项由余数得到。用 1 KB 和 4 MB 拟合,模型预测 1 MB 那行 32.9 ms(实测 35)、4 MB 那行 92.9 ms(实测 94)。
424
449
 
425
450
  有一个模型能把每一个尺寸行都拟合到它的散布之内:
426
451
 
@@ -517,6 +542,24 @@ inflate 本身(约 2 ms/MB)加上把 body 物化成 JS 字符串(约 1.7 m
517
542
  导入这个包是免费的。121 个内置锚是以主题 DN 哈希为索引的 base64 字符串,只有链实际落到的那一个会被解码,所以
518
543
  380 KB 打包(gzip 后 133 KB)的启动时间保持在约 2 ms,而一个导入了但没使用本包的请求是 0 ms。
519
544
 
545
+ ### 可选开关各自的成本
546
+
547
+ 上面所有数字都是默认身份:curl 指纹、gzip 与 deflate、无后量子、无 GREASE。下面每个开关默认都关着,这张表是「打开它要付什么」。测法与其余部分一致——差分两个工作量、取样本最小值。
548
+
549
+ | 开关 | 成本 | 何时付 |
550
+ |---|---|---|
551
+ | `grease: true` | 测不出来 | 一次 hello 里多几个字节 |
552
+ | `tls.extensionOrder: 'shuffle'` | 测不出来 | 每次握手洗牌约 11 个元素 |
553
+ | `headerOrder` | 测不出来——有序列表比平台的 `Headers` **更快**(1.6 µs 对 3.8 µs) | 每请求 |
554
+ | `groups: { x25519mlkem768 }` | 用内置 WASM **+0.15 ms**,用纯 JS 的 ML-KEM **+1.35 ms** | **每连接**,不是每请求——复用该连接的所有请求共同摊薄 |
555
+ | `ciphers: { chacha20 }` | **+2.0 ms/MB**,而且只有服务器**选中**它时才付 | 每字节。有 AES 硬件加速的服务器通常偏好 AES-GCM,所以实际成本往往是零,真正起作用的是"出现在 offer 里" |
556
+ | `decoders: { br }` | **+4.4 ms/MB** | 每字节,只要源站发 brotli。压得越狠越亏:quality 11 线上小 16%,解码贵 46% |
557
+ | `profile: chrome` | 上面三项之和 | |
558
+
559
+ 1.4.0 里有两个默认值变了,而它们都没有出现在上面那张表里:套件顺序对齐 curl 之后,原本协商 AES-128-GCM 的地方现在协商 AES-256-GCM,实测**贵 4%**(1.50 对 1.45 ms/MB——硬件 AES 让多出来的轮次很便宜);有序 header 列表取代了平台 `Headers`,反而**略微更便宜**。两者都落在上面数字本来就带的 ±1.5× 波动之内。
560
+
561
+ 体积:`import 'tunnelfetch/profile/chrome'` 会为两个 WASM 原语增加约 **22 KB(gzip)**。没有其它地方引用它们,所以走默认身份的使用者一个字节都不背。
562
+
520
563
  ### 计划限制
521
564
 
522
565
  付费计划上 30 秒的默认 CPU 上限不是约束点——那大约是一次调用里 3000 条连接或 1 GB 的 body,而"同时等待响应头
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tunnelfetch",
3
- "version": "1.2.0",
3
+ "version": "1.4.0",
4
4
  "description": "A fetch-shaped HTTP client that can route through HTTP CONNECT / HTTPS / SOCKS5 proxies on runtimes with only raw TCP, such as Cloudflare Workers. Implements TLS in userland because the runtime cannot verify a tunnelled peer.",
5
5
  "keywords": [
6
6
  "fetch",
@@ -51,6 +51,10 @@
51
51
  "./roots": {
52
52
  "types": "./types/trust/roots.d.ts",
53
53
  "default": "./src/trust/roots.js"
54
+ },
55
+ "./profile/chrome": {
56
+ "types": "./types/profile/chrome.d.ts",
57
+ "default": "./src/profile/chrome.js"
54
58
  }
55
59
  },
56
60
  "types": "./types/index.d.ts",
@@ -0,0 +1,159 @@
1
+ // Request header order and case — the part of a fingerprint that needs no TLS inspection at all.
2
+ //
3
+ // The platform's `Headers` is the wrong wire representation and always was: it iterates
4
+ // lexicographically sorted and lowercased. So a request built through it goes out as
5
+ // `accept, accept-encoding, connection, host, referer, user-agent` whatever the caller wrote —
6
+ // alphabetical order, with `user-agent` last, which no real client does. curl sends:
7
+ //
8
+ // Host, User-Agent, Accept, Accept-Encoding, <the caller's headers, in order>,
9
+ // Content-Length, Content-Type
10
+ //
11
+ // captured off the wire from curl 8.21.0. Note where the framing headers go: last, AFTER the
12
+ // caller's. That is why the default order below carries a `'*'` marker rather than being a plain
13
+ // ranking — "everything else" belongs in the middle, not at the end.
14
+ //
15
+ // Case matters too, and only on HTTP/1.1: real clients send `Host:` and `X-Custom:`, not `host:`
16
+ // and `x-custom:`. HTTP/2 is the opposite — RFC 9113 s8.2.1 REQUIRES lowercase field names, and a
17
+ // server must treat an uppercase one as malformed. So this preserves the case it was given and the
18
+ // h2 path lowercases at the last moment, which is the only place that is correct.
19
+
20
+ import { HttpError, codes } from '../errors.js';
21
+
22
+ /**
23
+ * Default request header order, from curl 8.21.0 on the wire. `'*'` is where headers not named
24
+ * here go, in the order they were given — which is where curl puts the caller's own.
25
+ */
26
+ export const CURL_HEADER_ORDER = Object.freeze([
27
+ 'host',
28
+ 'user-agent',
29
+ 'accept',
30
+ 'accept-encoding',
31
+ '*',
32
+ // curl sends no Connection on HTTP/1.1 (keep-alive is the default and it stays silent), so
33
+ // there is no reference position for it. Grouped with the other connection-and-framing headers
34
+ // at the end, which is where curl puts the ones it does send.
35
+ 'connection',
36
+ 'content-length',
37
+ 'content-type',
38
+ ]);
39
+
40
+ /**
41
+ * An ordered, case-preserving header list.
42
+ *
43
+ * Deliberately not a `Headers` subclass and deliberately not backed by one: the whole point is to
44
+ * be the thing `Headers` is not. Lookup and mutation are case-insensitive, as HTTP requires;
45
+ * iteration returns names exactly as they were written, in the order they arrived.
46
+ */
47
+ export class OrderedHeaders {
48
+ /** @param {Headers | Iterable<[string, string]> | Record<string, string> | null} [init] */
49
+ constructor(init = null) {
50
+ /** @type {Array<[string, string]>} name as written, value */
51
+ this._list = [];
52
+ if (init == null) return;
53
+ const pairs =
54
+ typeof (/** @type {any} */ (init)[Symbol.iterator]) === 'function'
55
+ ? /** @type {Iterable<[string, string]>} */ (init)
56
+ : Object.entries(init);
57
+ for (const [name, value] of pairs) this.append(name, value);
58
+ }
59
+
60
+ _indexOf(name) {
61
+ const lower = String(name).toLowerCase();
62
+ return this._list.findIndex(([n]) => n.toLowerCase() === lower);
63
+ }
64
+
65
+ has(name) {
66
+ return this._indexOf(name) !== -1;
67
+ }
68
+
69
+ /** Comma-joined when a field appears more than once, matching `Headers.get`. */
70
+ get(name) {
71
+ const lower = String(name).toLowerCase();
72
+ const hits = this._list.filter(([n]) => n.toLowerCase() === lower).map(([, v]) => v);
73
+ return hits.length ? hits.join(', ') : null;
74
+ }
75
+
76
+ /**
77
+ * Replace IN PLACE when the field is already present, so setting a value does not move a header
78
+ * to the end and silently reorder the request. A caller who wrote `User-Agent` first and then
79
+ * had it overwritten should still see it first.
80
+ */
81
+ set(name, value) {
82
+ const at = this._indexOf(name);
83
+ if (at === -1) {
84
+ this.append(name, value);
85
+ return;
86
+ }
87
+ this._list[at] = [this._list[at][0], String(value)];
88
+ // A repeated field collapses to the first position, as `Headers.set` collapses to one value.
89
+ const lower = String(name).toLowerCase();
90
+ this._list = this._list.filter(([n], i) => i === at || n.toLowerCase() !== lower);
91
+ }
92
+
93
+ append(name, value) {
94
+ const n = String(name);
95
+ if (n === '') throw new HttpError(codes.HTTP_HEADER, 'header name must not be empty', { name });
96
+ this._list.push([n, String(value)]);
97
+ }
98
+
99
+ delete(name) {
100
+ const lower = String(name).toLowerCase();
101
+ this._list = this._list.filter(([n]) => n.toLowerCase() !== lower);
102
+ }
103
+
104
+ /**
105
+ * Put the fields into `order`, which names lowercased header names and may contain `'*'` to mark
106
+ * where everything unnamed goes. Fields keep their relative order within each group, so a
107
+ * caller's own sequence survives.
108
+ *
109
+ * @param {readonly string[]} order
110
+ */
111
+ reorder(order) {
112
+ const star = order.indexOf('*');
113
+ const rank = new Map();
114
+ order.forEach((name, i) => {
115
+ if (name !== '*') rank.set(name, i);
116
+ });
117
+ // With no '*', unnamed fields go last — the same rule the TLS extension order uses.
118
+ const fallback = star === -1 ? order.length : star;
119
+ this._list = this._list
120
+ .map((entry, i) => ({ entry, i, r: rank.get(entry[0].toLowerCase()) ?? fallback }))
121
+ .sort((a, b) => a.r - b.r || a.i - b.i)
122
+ .map((x) => x.entry);
123
+ }
124
+
125
+ /** @returns {Array<[string, string]>} names as written, in order */
126
+ entries() {
127
+ return this._list.map(([n, v]) => [n, v]);
128
+ }
129
+
130
+ /** Lowercased names, for HTTP/2 where RFC 9113 s8.2.1 requires them. */
131
+ lowercased() {
132
+ return this._list.map(([n, v]) => [n.toLowerCase(), v]);
133
+ }
134
+
135
+ [Symbol.iterator]() {
136
+ return this.entries()[Symbol.iterator]();
137
+ }
138
+ }
139
+
140
+ /**
141
+ * Read the caller's header names in the order and case they wrote them, BEFORE anything hands them
142
+ * to `Request`, which is where both are lost.
143
+ *
144
+ * Recovers what is recoverable and no more: an array of pairs or a plain object still carries the
145
+ * caller's order, a `Headers` or a `Request` does not — those were normalised before this package
146
+ * ever saw them, and there is nothing here to reconstruct.
147
+ *
148
+ * @param {RequestInfo | URL} input
149
+ * @param {RequestInit} [init]
150
+ * @returns {Array<[string, string]> | null} null when the caller's order was already gone
151
+ */
152
+ export function callerHeaderOrder(input, init) {
153
+ const raw = init?.headers ?? (input instanceof Request ? null : undefined);
154
+ if (raw == null) return null;
155
+ if (typeof Headers !== 'undefined' && raw instanceof Headers) return null;
156
+ if (Array.isArray(raw)) return raw.map(([n, v]) => [String(n), String(v)]);
157
+ if (typeof raw === 'object') return Object.entries(raw).map(([n, v]) => [n, String(v)]);
158
+ return null;
159
+ }
package/src/client.js CHANGED
@@ -15,6 +15,8 @@ import { ByteReader, ByteWriter, UnexpectedEofError, concat, utf8 } from './util
15
15
  import { serializeRequestHead } from './http1/request.js';
16
16
  import { bodyFraming, readResponseBody, readResponseHead } from './http1/response.js';
17
17
  import { acceptEncodingFor, decodeBody } from './client/decode.js';
18
+ import { OrderedHeaders, CURL_HEADER_ORDER, callerHeaderOrder } from './client/header-order.js';
19
+ import { applyProfile } from './profiles.js';
18
20
  import { CookieJar } from './client/cookies.js';
19
21
  import { DEFAULT_MAX_REDIRECTS, nextRequest, shouldRedirect } from './client/redirect.js';
20
22
  import { ConnectionPool, poolKey } from './pool.js';
@@ -81,7 +83,30 @@ const NULL_BODY_STATUS = new Set([101, 204, 205, 304]);
81
83
  * yours and visible. Measured on the edge: WASM brotli decodes at about 2x native gzip, and
82
84
  * the wire bytes it saves do not pay that back — see the README. The reason to turn it on is
83
85
  * matching a browser's Accept-Encoding, not saving CPU.
86
+ * @property {{ chacha20?: import('./tls/aead.js').AeadOptions['impl'] }} [ciphers] injected AEAD
87
+ * implementations, by capability name. `chacha20` (seal/open, RFC 8439) is what lets
88
+ * TLS_CHACHA20_POLY1305_SHA256 be offered and performed — WebCrypto has no ChaCha20 on this
89
+ * runtime, and a suite offered but not performable is a dead connection if a server selects it,
90
+ * so without this the suite stays out of the ClientHello. Required by `profiles.chrome`.
91
+ * @property {{ x25519mlkem768?: import('./tls/hybrid.js').MlKem768 }} [groups] injected key-exchange
92
+ * implementations, by capability name. `x25519mlkem768` (ML-KEM-768 keygen/encapsulate/
93
+ * decapsulate) is what lets the post-quantum hybrid group be offered and performed; without it
94
+ * the group stays out of the ClientHello. Required by `profiles.chrome`.
84
95
  * @property {boolean} [keepAlive] default true.
96
+ * @property {import('./profiles.js').FingerprintProfile} [profile] one coherent network identity
97
+ * instead of a dozen knobs that can disagree — TLS, HTTP/2, header order and default headers
98
+ * together. Explicit options win over it. A profile that declares capabilities this package
99
+ * cannot perform is REFUSED rather than silently reduced: see `profiles.chrome`.
100
+ * @property {readonly string[]} [headerOrder] request header names, lowercased, in the order to
101
+ * emit them; `'*'` marks where headers not named go, in the order the caller gave them. Defaults
102
+ * to curl's (`CURL_HEADER_ORDER`). The platform `Headers` sorts alphabetically and lowercases, so
103
+ * without this a request goes out with `user-agent` last, which no real client does.
104
+ * @property {string[]} [http2PseudoHeaderOrder] request pseudo-headers in the order to emit them.
105
+ * Defaults to curl's. Any of the four omitted is appended rather than dropped: RFC 9113 s8.3.1
106
+ * makes all four mandatory, so a request missing one is malformed rather than merely unusual.
107
+ * @property {Record<string, 'incremental'|'without'|'never'>} [http2HpackIndexing] per-field HPACK
108
+ * indexing. Which fields enter the dynamic table is read by an Akamai-style h2 fingerprint.
109
+ * Defaults to curl's: everything incremental except `:path`.
85
110
  * @property {Array<[number, number]>} [http2Settings] the HTTP/2 SETTINGS flight, as [id, value]
86
111
  * pairs. Order is significant — an Akamai-style h2 fingerprint reads the ids in the order they
87
112
  * are sent — so this replaces the flight rather than merging into it. Defaults to curl's. The
@@ -106,7 +131,7 @@ export class Client {
106
131
  * @param {ClientOptions} [options]
107
132
  */
108
133
  constructor(options = {}) {
109
- this.options = snapshotOptions(options);
134
+ this.options = snapshotOptions(applyProfile(options));
110
135
  this.pool = new ConnectionPool(options.pool);
111
136
  // HTTP/2 connections are NOT pooled the way h1 is: one connection multiplexes many concurrent
112
137
  // streams, so it is not checked out per request. It lives here, keyed exactly like the h1 pool,
@@ -237,6 +262,9 @@ export function install(options = {}) {
237
262
 
238
263
  async function performFetch(client, input, init) {
239
264
  const o = client.options;
265
+ // Read the caller's header names and case before `Request` normalises them away: the platform
266
+ // `Headers` sorts lexicographically and lowercases, so by the line below both are already gone.
267
+ const callerOrder = callerHeaderOrder(input, init);
240
268
  const request = new Request(input, init);
241
269
  const redirectMode = init?.redirect ?? request.redirect ?? 'follow';
242
270
  const maxRedirects = o.maxRedirects ?? DEFAULT_MAX_REDIRECTS;
@@ -251,7 +279,9 @@ async function performFetch(client, input, init) {
251
279
  let current = {
252
280
  url: new URL(request.url),
253
281
  method: request.method,
254
- headers: new Headers(request.headers),
282
+ // The caller's own list when it survived, so its order and case reach the wire; otherwise the
283
+ // normalised Headers, which is all there is left to work from.
284
+ headers: new OrderedHeaders(callerOrder ?? request.headers),
255
285
  body,
256
286
  };
257
287
  const history = [];
@@ -382,7 +412,7 @@ async function openFreshAndSend(client, current, { hop, key, proxy, trust, tls }
382
412
  tls,
383
413
  alpn,
384
414
  resumption: resumptionFor(client, key),
385
- deps: o.deps,
415
+ deps: tlsDeps(o),
386
416
  deadlines,
387
417
  limits: o.limits ?? {},
388
418
  now: o.now,
@@ -446,6 +476,12 @@ function registerHttp2(client, key, conn) {
446
476
  // `tls`, and one being reachable while the other was not made "the fingerprint is
447
477
  // configurable" only half true.
448
478
  ...(client.options.http2Settings ? { settings: client.options.http2Settings } : {}),
479
+ ...(client.options.http2PseudoHeaderOrder
480
+ ? { pseudoHeaderOrder: client.options.http2PseudoHeaderOrder }
481
+ : {}),
482
+ ...(client.options.http2HpackIndexing
483
+ ? { hpackIndexing: client.options.http2HpackIndexing }
484
+ : {}),
449
485
  onClose: () => {
450
486
  client._h2conns.delete(h2);
451
487
  // Only drop the keyed entry if it is still this connection; a newer one may have replaced it.
@@ -570,6 +606,22 @@ function serverNeverSawIt(err) {
570
606
  return err instanceof TunnelFetchError && err.code === codes.TLS_TRUNCATED && err.detail?.got === 0;
571
607
  }
572
608
 
609
+ /**
610
+ * The TLS deps for a connection: the caller's injectable randomness/keygen, plus the injected
611
+ * crypto implementations (`ciphers` -> `aead`, `groups` -> `kem`) the TLS layer needs to offer and
612
+ * perform the capability-gated ChaCha20 suite and X25519MLKEM768 group. Folded in here rather than
613
+ * carried in `tls`, so they neither enter the pool key nor disable native-fetch delegation — they
614
+ * are injected primitives, not fingerprint configuration.
615
+ * @param {Readonly<import('./client.js').ClientOptions>} o
616
+ * @returns {import('./tls/connect.js').TlsDeps}
617
+ */
618
+ function tlsDeps(o) {
619
+ const deps = { ...(o.deps ?? {}) };
620
+ if (o.ciphers) deps.aead = o.ciphers;
621
+ if (o.groups) deps.kem = o.groups;
622
+ return deps;
623
+ }
624
+
573
625
  async function sendAndReceive(client, conn, current, { key, deadlines, reused }) {
574
626
  const o = client.options;
575
627
  const target = targetFromUrl(current.url);
@@ -611,7 +663,7 @@ async function sendAndReceive(client, conn, current, { key, deadlines, reused })
611
663
  trust: o.trust ?? { mode: 'system' },
612
664
  tls: o.tls ?? {},
613
665
  resumption: resumptionFor(client, key),
614
- deps: o.deps,
666
+ deps: tlsDeps(o),
615
667
  deadlines,
616
668
  limits: o.limits ?? {},
617
669
  now: o.now,
@@ -723,11 +775,15 @@ async function sendAndReceiveH2(client, h2, current, { deadlines }) {
723
775
  */
724
776
  function buildH2Request(client, current, target) {
725
777
  const o = client.options;
726
- const headers = new Headers(current.headers);
778
+ const headers = new OrderedHeaders(current.headers);
727
779
  const defaultPort = current.url.protocol === 'https:' ? 443 : 80;
728
780
  const authority =
729
781
  target.port === defaultPort ? current.url.hostname : `${current.url.hostname}:${target.port}`;
730
782
 
783
+ // Profile headers are defaults: a request that sets its own User-Agent keeps it.
784
+ for (const [name, value] of client.options.profileHeaders ?? []) {
785
+ if (!headers.has(name)) headers.set(name, value);
786
+ }
731
787
  if (!headers.has('accept')) headers.set('accept', '*/*');
732
788
  if (!headers.has('accept-encoding') && o.decompress !== false) {
733
789
  headers.set('accept-encoding', acceptEncodingFor(o.decoders));
@@ -747,12 +803,12 @@ function buildH2Request(client, current, target) {
747
803
  else if (['POST', 'PUT', 'PATCH'].includes(current.method)) headers.set('content-length', '0');
748
804
  else headers.delete('content-length');
749
805
 
750
- // Headers iterates lowercased (RFC 9113 s8.2.1 requires lowercase names on the wire) — which is
751
- // also why h1 loses caller order; h2 is no different here. Pseudo-header ORDER, the part a
752
- // fingerprinter reads, is fixed in http2/connection.js buildRequestFields, not here.
753
- const out = [];
754
- for (const [k, v] of headers) out.push([k, v]);
755
- return { authority, headers: out };
806
+ // Lowercased at the last moment, and only here: RFC 9113 s8.2.1 REQUIRES lowercase field names
807
+ // on an h2 wire and a server must treat an uppercase one as malformed. h1 is the opposite — real
808
+ // clients send `Host:`, not `host:` — which is why the case is carried this far and dropped only
809
+ // on this path. Pseudo-header order is fixed in http2/connection.js buildRequestFields.
810
+ headers.reorder(o.headerOrder ?? CURL_HEADER_ORDER);
811
+ return { authority, headers: headers.lowercased() };
756
812
  }
757
813
 
758
814
  function wantsKeepAlive(headInfo) {
@@ -803,13 +859,19 @@ function requestTarget(url) {
803
859
  }
804
860
 
805
861
  function buildHeaders(client, current, target) {
806
- const headers = new Headers(current.headers);
862
+ const headers = new OrderedHeaders(current.headers);
807
863
  // Host is derived from the URL and never carried across a redirect; the default port is omitted
808
864
  // because a server matching virtual hosts on the literal Host value expects it that way.
809
865
  const defaultPort = current.url.protocol === 'https:' ? 443 : 80;
810
866
  const hostValue =
811
867
  target.port === defaultPort ? current.url.hostname : `${current.url.hostname}:${target.port}`;
812
868
 
869
+ // Profile headers are defaults: a request that sets its own User-Agent keeps it. Applied on both
870
+ // the h1 and h2 paths, which build their headers separately — the h2 side had this and the h1
871
+ // side did not, so a profile's User-Agent reached an h2 request and vanished from an h1 one.
872
+ for (const [name, value] of client.options.profileHeaders ?? []) {
873
+ if (!headers.has(name)) headers.set(name, value);
874
+ }
813
875
  if (!headers.has('accept')) headers.set('accept', '*/*');
814
876
  if (!headers.has('accept-encoding') && client.options.decompress !== false) {
815
877
  // Never advertise br or zstd: the runtime has no DecompressionStream for either, so the
@@ -838,12 +900,11 @@ function buildHeaders(client, current, target) {
838
900
  else if (['POST', 'PUT', 'PATCH'].includes(current.method)) headers.set('content-length', '0');
839
901
  else headers.delete('content-length');
840
902
 
841
- const ordered = [['host', hostValue]];
842
- for (const [k, v] of headers) {
843
- if (k === 'host') continue;
844
- ordered.push([k, v]);
845
- }
846
- return ordered;
903
+ // Host is set rather than prepended so the order below decides where it goes — which for curl
904
+ // is first, but for a caller matching something else may not be.
905
+ headers.set('Host', hostValue);
906
+ headers.reorder(client.options.headerOrder ?? CURL_HEADER_ORDER);
907
+ return headers.entries();
847
908
  }
848
909
 
849
910
  export { CookieJar, ConnectionPool, utf8 };