@rester159/blacktip 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. package/CHANGELOG.md +190 -0
  2. package/README.md +21 -0
  3. package/dist/behavioral/parsers.d.ts +89 -0
  4. package/dist/behavioral/parsers.d.ts.map +1 -0
  5. package/dist/behavioral/parsers.js +223 -0
  6. package/dist/behavioral/parsers.js.map +1 -0
  7. package/dist/blacktip.d.ts +34 -1
  8. package/dist/blacktip.d.ts.map +1 -1
  9. package/dist/blacktip.js +105 -1
  10. package/dist/blacktip.js.map +1 -1
  11. package/dist/diagnostics.d.ts +31 -0
  12. package/dist/diagnostics.d.ts.map +1 -1
  13. package/dist/diagnostics.js +146 -0
  14. package/dist/diagnostics.js.map +1 -1
  15. package/dist/identity-pool.d.ts +160 -0
  16. package/dist/identity-pool.d.ts.map +1 -0
  17. package/dist/identity-pool.js +288 -0
  18. package/dist/identity-pool.js.map +1 -0
  19. package/dist/index.d.ts +7 -2
  20. package/dist/index.d.ts.map +1 -1
  21. package/dist/index.js +7 -1
  22. package/dist/index.js.map +1 -1
  23. package/dist/tls-side-channel.d.ts +82 -0
  24. package/dist/tls-side-channel.d.ts.map +1 -0
  25. package/dist/tls-side-channel.js +241 -0
  26. package/dist/tls-side-channel.js.map +1 -0
  27. package/dist/types.d.ts +15 -0
  28. package/dist/types.d.ts.map +1 -1
  29. package/dist/types.js.map +1 -1
  30. package/docs/akamai-bypass.md +257 -0
  31. package/docs/anti-bot-validation.md +84 -0
  32. package/docs/calibration-validation.md +93 -0
  33. package/docs/identity-pool.md +176 -0
  34. package/docs/tls-side-channel.md +83 -0
  35. package/native/tls-client/go.mod +21 -0
  36. package/native/tls-client/go.sum +36 -0
  37. package/native/tls-client/main.go +216 -0
  38. package/package.json +8 -2
  39. package/scripts/fit-cmu-keystroke.mjs +186 -0
@@ -0,0 +1,83 @@
1
+ # TLS side-channel (v0.3.0)
2
+
3
+ The v0.3.0 answer to "an edge gates the very first request before BlackTip's browser even has a session." BlackTip ships a Go-based daemon built on `bogdanfinn/tls-client` that performs HTTP requests with a real Chrome TLS ClientHello, real H2 frame settings, and real H2 frame order. You use it to make gating requests the browser can't make through itself, then inject the resulting cookies into the browser session before navigating.
4
+
5
+ ## When you need this
6
+
7
+ Most BlackTip flows do not need the TLS side-channel. The browser's own TLS via `channel: 'chrome'` is real Chrome and passes every detector we've validated against. The side-channel is for the cases where it isn't enough:
8
+
9
+ 1. **First-request edge gating.** Some Akamai-protected sites refuse to serve a session cookie to a navigation that doesn't already have one. You hit them via the side-channel first (which goes through `bm_s` → sensor data POST → `bm_sv` issuance), then inject the resulting cookies into the browser and navigate normally.
10
+ 2. **Cross-platform User-Agent spoofing.** You're running on Linux but want the target to see Windows. The browser's TLS comes from the host OS Chrome, so you can't fake the platform without a TLS rewriter. The side-channel can. v0.2.0's L016 fix removed the broken UA-override path; this is the supported alternative.
11
+ 3. **API-level operations.** You want to call a JSON API the site exposes, but the API edge enforces the same TLS profile as the browser. Use the side-channel to call the API without paying browser-render overhead per request.
12
+ 4. **Pre-warming for proxy rotation.** You're rotating residential proxies and need to "warm" each new IP with a Chrome-TLS handshake before the browser session uses it.
13
+
14
+ ## Build the daemon
15
+
16
+ The daemon is a small Go program in `native/tls-client/`. Go is the only build dependency. Install Go from https://go.dev/dl/ then:
17
+
18
+ ```bash
19
+ cd native/tls-client
20
+ go build -o blacktip-tls . # Linux / macOS
21
+ go build -o blacktip-tls.exe . # Windows
22
+ ```
23
+
24
+ The build produces a single ~14 MB statically-linked binary. BlackTip resolves it via `native/tls-client/blacktip-tls[.exe]` automatically; override with `BLACKTIP_TLS_BIN=/abs/path/to/binary` if you want to ship it elsewhere.
25
+
26
+ ## Usage
27
+
28
+ ```typescript
29
+ import { BlackTip } from '@rester159/blacktip';
30
+
31
+ const bt = new BlackTip({ logLevel: 'info' });
32
+ await bt.launch();
33
+
34
+ // 1. Make the gating request through the side-channel.
35
+ const resp = await bt.fetchWithTls({
36
+ url: 'https://www.opentable.com/',
37
+ headers: {
38
+ 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36',
39
+ 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
40
+ 'Accept-Language': 'en-US,en;q=0.9',
41
+ },
42
+ });
43
+ console.log('TLS fetch status:', resp.status);
44
+ console.log('Earned cookies:', resp.cookies.map(c => c.name));
45
+
46
+ // 2. Inject the cookies into the browser session.
47
+ const injected = await bt.injectTlsCookies(resp, 'https://www.opentable.com/');
48
+ console.log('Injected', injected, 'cookies');
49
+
50
+ // 3. Navigate normally — the browser carries the side-channel-earned tokens.
51
+ await bt.navigate('https://www.opentable.com/');
52
+ await bt.waitForStable();
53
+ // ... rest of flow
54
+
55
+ await bt.close(); // Closes the TLS daemon too
56
+ ```
57
+
58
+ ## Architecture
59
+
60
+ - **Wire protocol**: newline-delimited JSON over the daemon's stdin/stdout. Each request has a string `id`; responses match by id, so multiple in-flight `fetch()` calls don't interleave.
61
+ - **Daemon lifecycle**: spawned lazily on first `fetchWithTls()` call, kept alive until `bt.close()`. No subprocess startup cost per request.
62
+ - **Concurrency**: the daemon handles each request in its own Go goroutine. The Node side dispatches them in parallel and reassembles by id.
63
+ - **Profile selection**: defaults to `chrome_133`. Override per-request via `bt.fetchWithTls({ url, profile: 'chrome_124' })`. Available profiles are whatever `bogdanfinn/tls-client/profiles` ships — at the time of writing: `chrome_120`, `chrome_124`, `chrome_131`, `chrome_133`, `firefox_120`, `safari_ios_16_0`.
64
+ - **Body encoding**: bodies are base64 on the wire to avoid newline / Unicode issues with the line-delimited protocol. The wrapper handles encoding/decoding transparently.
65
+
66
+ ## Validated TLS fingerprint
67
+
68
+ Run via the integration test (`tests/tls-side-channel.integration.test.ts`):
69
+
70
+ ```
71
+ JA4: t13d1516h2_8daaf6152771_d8a2da3f94cd
72
+ First cipher: TLS_GREASE (0x5A5A)
73
+ HTTP/2 Akamai fingerprint: 1:65536;2:0;4:6291456;6:262144|15663105|0|m,a,s,p
74
+ ```
75
+
76
+ This is byte-for-byte identical to real Chrome 133. The leading `t13d` JA4 prefix confirms TLS 1.3 with Chrome's count signature; the GREASE first cipher confirms proper GREASE rotation; the H2 fingerprint matches Chrome's frame settings and frame order (`m,a,s,p` = method/authority/scheme/path).
77
+
78
+ ## Caveats
79
+
80
+ - **Not a Chrome browser.** The side-channel is HTTP-only — no JavaScript execution, no DOM, no Cookie JAR persistence beyond what you inject manually. You use it to acquire tokens, not to drive a flow.
81
+ - **Akamai sensor data is not bypassed.** A first request to an Akamai-protected URL through the side-channel returns 403 with `bm_s`/`bm_ss`/`bm_so` set — the same place a browser would be at after one request. Akamai expects a sensor data POST to follow before serving 200s. The side-channel collects the session cookies; the sensor POST is not yet automated. You can either (a) inject the bm_s cookies into the browser and let the browser do the sensor POST (this is what the OpenTable validation flow does), or (b) implement the sensor POST yourself.
82
+ - **Always set the User-Agent header.** The Go HTTP client defaults to `Go-http-client/2.0` if you don't override it. That's a textbook automation tell. The wrapper auto-sets `Accept-Language` to `en-US,en;q=0.9` if missing, but UA is your job.
83
+ - **Linux/macOS support is built but not validated in CI.** The `go build` command produces native binaries for whatever platform you run it on; the Node-side `resolveBinaryPath()` picks the right name based on `process.platform`. If you ship to a server, build the daemon in the same OS the server runs.
@@ -0,0 +1,21 @@
1
+ module github.com/rester159/blacktip/native/tls-client
2
+
3
+ go 1.26.2
4
+
5
+ require (
6
+ github.com/andybalholm/brotli v1.2.0 // indirect
7
+ github.com/bdandy/go-errors v1.2.2 // indirect
8
+ github.com/bdandy/go-socks4 v1.2.3 // indirect
9
+ github.com/bogdanfinn/fhttp v0.6.8 // indirect
10
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
11
+ github.com/bogdanfinn/tls-client v1.14.0 // indirect
12
+ github.com/bogdanfinn/utls v1.7.7-barnius // indirect
13
+ github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
14
+ github.com/klauspost/compress v1.18.2 // indirect
15
+ github.com/quic-go/qpack v0.6.0 // indirect
16
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
17
+ golang.org/x/crypto v0.46.0 // indirect
18
+ golang.org/x/net v0.48.0 // indirect
19
+ golang.org/x/sys v0.39.0 // indirect
20
+ golang.org/x/text v0.32.0 // indirect
21
+ )
@@ -0,0 +1,36 @@
1
+ github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
2
+ github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
3
+ github.com/bdandy/go-errors v1.2.2 h1:WdFv/oukjTJCLa79UfkGmwX7ZxONAihKu4V0mLIs11Q=
4
+ github.com/bdandy/go-errors v1.2.2/go.mod h1:NkYHl4Fey9oRRdbB1CoC6e84tuqQHiqrOcZpqFEkBxM=
5
+ github.com/bdandy/go-socks4 v1.2.3 h1:Q6Y2heY1GRjCtHbmlKfnwrKVU/k81LS8mRGLRlmDlic=
6
+ github.com/bdandy/go-socks4 v1.2.3/go.mod h1:98kiVFgpdogR8aIGLWLvjDVZ8XcKPsSI/ypGrO+bqHI=
7
+ github.com/bogdanfinn/fhttp v0.6.8 h1:LiQyHOY3i0QoxxNB7nq27/nGNNbtPj0fuBPozhR7Ws4=
8
+ github.com/bogdanfinn/fhttp v0.6.8/go.mod h1:A+EKDzMx2hb4IUbMx4TlkoHnaJEiLl8r/1Ss1Y+5e5M=
9
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls h1:tV6eDEiRbRCcepALSzxR94JUVD3N3ACIiRLgyc2Ep8s=
10
+ github.com/bogdanfinn/quic-go-utls v1.0.9-utls/go.mod h1:aHph9B9H9yPOt5xnhWKSOum27DJAqpiHzwX+gjvaXcg=
11
+ github.com/bogdanfinn/tls-client v1.14.0 h1:vyk7Cn4BIvLAGVuMfb0tP22OqogfO1lYamquQNEZU1A=
12
+ github.com/bogdanfinn/tls-client v1.14.0/go.mod h1:LsU6mXVn8MOFDwTkyRfI7V1BZM1p0wf2ZfZsICW/1fM=
13
+ github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5ZlrCR7fOLU=
14
+ github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
15
+ github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
16
+ github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
17
+ github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
18
+ github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
19
+ github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
20
+ github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
21
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
22
+ github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
23
+ golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
24
+ golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
25
+ golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
26
+ golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
27
+ golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
28
+ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
29
+ golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
30
+ golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
31
+ golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
32
+ golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
33
+ golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
34
+ golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
35
+ golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
36
+ golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -0,0 +1,216 @@
1
+ // BlackTip TLS side-channel daemon.
2
+ //
3
+ // Reads newline-delimited JSON requests from stdin, performs HTTP
4
+ // requests with a real Chrome TLS fingerprint via bogdanfinn/tls-client,
5
+ // and writes newline-delimited JSON responses to stdout.
6
+ //
7
+ // This is the v0.3.0 answer to the question: "what do I do when an edge
8
+ // gates the very first request before BlackTip's browser has a session?"
9
+ // You use this daemon to make the gating request — it presents a real
10
+ // Chrome TLS ClientHello, real H2 frames, and real headers — capture
11
+ // the cookies and tokens it gets back, and inject them into the
12
+ // browser session before the user-driven flow continues.
13
+ //
14
+ // Wire format:
15
+ //
16
+ // Request: {"id":"<string>","url":"<string>","method":"<GET|POST|...>","headers":{"...":"..."},"body":"<string base64>","timeoutMs":15000,"profile":"chrome_133"}
17
+ // Response: {"id":"<string>","ok":true,"status":200,"headers":{"...":["...","..."]},"body":"<string base64>","finalUrl":"<string>","durationMs":123}
18
+ // OR
19
+ // {"id":"<string>","ok":false,"error":"<message>","durationMs":123}
20
+ //
21
+ // One JSON object per line in both directions. The Node parent reads
22
+ // stdout line-by-line and matches responses by id. The daemon stays
23
+ // alive across many requests so we don't pay subprocess startup cost
24
+ // per call.
25
+ package main
26
+
27
+ import (
28
+ "bufio"
29
+ "encoding/base64"
30
+ "encoding/json"
31
+ "fmt"
32
+ "io"
33
+ "os"
34
+ "strings"
35
+ "sync"
36
+ "time"
37
+
38
+ http "github.com/bogdanfinn/fhttp"
39
+ tls_client "github.com/bogdanfinn/tls-client"
40
+ "github.com/bogdanfinn/tls-client/profiles"
41
+ )
42
+
43
+ type request struct {
44
+ ID string `json:"id"`
45
+ URL string `json:"url"`
46
+ Method string `json:"method"`
47
+ Headers map[string]string `json:"headers"`
48
+ Body string `json:"body"`
49
+ TimeoutMs int `json:"timeoutMs"`
50
+ Profile string `json:"profile"`
51
+ }
52
+
53
+ type response struct {
54
+ ID string `json:"id"`
55
+ OK bool `json:"ok"`
56
+ Status int `json:"status,omitempty"`
57
+ Headers map[string][]string `json:"headers,omitempty"`
58
+ Body string `json:"body,omitempty"`
59
+ FinalURL string `json:"finalUrl,omitempty"`
60
+ DurationMs int64 `json:"durationMs"`
61
+ Error string `json:"error,omitempty"`
62
+ }
63
+
64
+ // resolveProfile maps a profile name to a tls-client ClientProfile.
65
+ // Defaults to the latest Chrome at the time of writing.
66
+ func resolveProfile(name string) profiles.ClientProfile {
67
+ switch strings.ToLower(name) {
68
+ case "chrome_120":
69
+ return profiles.Chrome_120
70
+ case "chrome_124":
71
+ return profiles.Chrome_124
72
+ case "chrome_131":
73
+ return profiles.Chrome_131
74
+ case "chrome_133":
75
+ return profiles.Chrome_133
76
+ case "firefox_120":
77
+ return profiles.Firefox_120
78
+ case "safari_ios_16_0":
79
+ return profiles.Safari_IOS_16_0
80
+ default:
81
+ return profiles.Chrome_133
82
+ }
83
+ }
84
+
85
+ // buildClient constructs a tls-client with the requested profile and timeout.
86
+ // We rebuild on every request because timeout is per-request and the cost
87
+ // is negligible compared to the network round-trip.
88
+ func buildClient(profile profiles.ClientProfile, timeoutMs int) (tls_client.HttpClient, error) {
89
+ if timeoutMs <= 0 {
90
+ timeoutMs = 15000
91
+ }
92
+ options := []tls_client.HttpClientOption{
93
+ tls_client.WithTimeoutSeconds(timeoutMs / 1000),
94
+ tls_client.WithClientProfile(profile),
95
+ tls_client.WithNotFollowRedirects(),
96
+ }
97
+ return tls_client.NewHttpClient(tls_client.NewNoopLogger(), options...)
98
+ }
99
+
100
+ func handle(req request) response {
101
+ start := time.Now()
102
+ durationFor := func() int64 { return time.Since(start).Milliseconds() }
103
+
104
+ if req.URL == "" {
105
+ return response{ID: req.ID, OK: false, Error: "url is required", DurationMs: durationFor()}
106
+ }
107
+ method := req.Method
108
+ if method == "" {
109
+ method = "GET"
110
+ }
111
+
112
+ client, err := buildClient(resolveProfile(req.Profile), req.TimeoutMs)
113
+ if err != nil {
114
+ return response{ID: req.ID, OK: false, Error: "buildClient: " + err.Error(), DurationMs: durationFor()}
115
+ }
116
+
117
+ var bodyReader io.Reader
118
+ if req.Body != "" {
119
+ decoded, decErr := base64.StdEncoding.DecodeString(req.Body)
120
+ if decErr != nil {
121
+ return response{ID: req.ID, OK: false, Error: "body base64 decode: " + decErr.Error(), DurationMs: durationFor()}
122
+ }
123
+ bodyReader = strings.NewReader(string(decoded))
124
+ }
125
+
126
+ httpReq, err := http.NewRequest(method, req.URL, bodyReader)
127
+ if err != nil {
128
+ return response{ID: req.ID, OK: false, Error: "NewRequest: " + err.Error(), DurationMs: durationFor()}
129
+ }
130
+
131
+ for k, v := range req.Headers {
132
+ httpReq.Header.Set(k, v)
133
+ }
134
+ // If the caller didn't set Accept-Language, fall back to a Chrome default.
135
+ if httpReq.Header.Get("Accept-Language") == "" {
136
+ httpReq.Header.Set("Accept-Language", "en-US,en;q=0.9")
137
+ }
138
+
139
+ resp, err := client.Do(httpReq)
140
+ if err != nil {
141
+ return response{ID: req.ID, OK: false, Error: "Do: " + err.Error(), DurationMs: durationFor()}
142
+ }
143
+ defer resp.Body.Close()
144
+
145
+ bodyBytes, err := io.ReadAll(resp.Body)
146
+ if err != nil {
147
+ return response{ID: req.ID, OK: false, Error: "read body: " + err.Error(), DurationMs: durationFor()}
148
+ }
149
+
150
+ headers := make(map[string][]string, len(resp.Header))
151
+ for k, v := range resp.Header {
152
+ headers[k] = v
153
+ }
154
+
155
+ finalURL := req.URL
156
+ if resp.Request != nil && resp.Request.URL != nil {
157
+ finalURL = resp.Request.URL.String()
158
+ }
159
+
160
+ return response{
161
+ ID: req.ID,
162
+ OK: true,
163
+ Status: resp.StatusCode,
164
+ Headers: headers,
165
+ Body: base64.StdEncoding.EncodeToString(bodyBytes),
166
+ FinalURL: finalURL,
167
+ DurationMs: durationFor(),
168
+ }
169
+ }
170
+
171
+ func main() {
172
+ scanner := bufio.NewScanner(os.Stdin)
173
+ // Allow large request bodies (e.g. POSTed forms with file fields).
174
+ scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
175
+
176
+ // Stdout writes are mutex-protected so concurrent handlers don't
177
+ // interleave their JSON lines.
178
+ var stdoutMu sync.Mutex
179
+ emit := func(r response) {
180
+ out, err := json.Marshal(r)
181
+ if err != nil {
182
+ out = []byte(fmt.Sprintf(`{"id":%q,"ok":false,"error":"marshal failed: %s"}`, r.ID, err.Error()))
183
+ }
184
+ stdoutMu.Lock()
185
+ defer stdoutMu.Unlock()
186
+ os.Stdout.Write(out)
187
+ os.Stdout.Write([]byte("\n"))
188
+ }
189
+
190
+ var wg sync.WaitGroup
191
+ for scanner.Scan() {
192
+ line := append([]byte(nil), scanner.Bytes()...)
193
+ if len(line) == 0 {
194
+ continue
195
+ }
196
+ var req request
197
+ if err := json.Unmarshal(line, &req); err != nil {
198
+ emit(response{ID: "", OK: false, Error: "unmarshal: " + err.Error()})
199
+ continue
200
+ }
201
+ // Handle requests concurrently — the Go TLS client is goroutine-safe.
202
+ wg.Add(1)
203
+ go func(r request) {
204
+ defer wg.Done()
205
+ emit(handle(r))
206
+ }(req)
207
+ }
208
+ if err := scanner.Err(); err != nil {
209
+ fmt.Fprintln(os.Stderr, "scan error:", err)
210
+ os.Exit(1)
211
+ }
212
+ // Wait for any in-flight requests to drain before exiting. Without
213
+ // this, closing stdin races the goroutines and the parent never
214
+ // sees the response.
215
+ wg.Wait()
216
+ }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@rester159/blacktip",
3
- "version": "0.2.0",
3
+ "version": "0.4.0",
4
4
  "description": "Stealth browser instrument for AI agents. Real Chrome + patchright CDP stealth + human-calibrated behavioral simulation. Every action is indistinguishable from a human.",
5
5
  "main": "dist/index.js",
6
6
  "types": "dist/index.d.ts",
@@ -21,8 +21,14 @@
21
21
  "dist",
22
22
  "README.md",
23
23
  "AGENTS.md",
24
+ "CHANGELOG.md",
24
25
  "LICENSE",
25
- "examples"
26
+ "examples",
27
+ "docs",
28
+ "native/tls-client/main.go",
29
+ "native/tls-client/go.mod",
30
+ "native/tls-client/go.sum",
31
+ "scripts"
26
32
  ],
27
33
  "scripts": {
28
34
  "build": "tsc",
@@ -0,0 +1,186 @@
1
+ #!/usr/bin/env node
2
+ /**
3
+ * Fit a behavioral profile against the real CMU Keystroke Dynamics
4
+ * dataset and report a held-out validation result.
5
+ *
6
+ * This is the v0.3.0 calibration validation script. It:
7
+ *
8
+ * 1. Loads `data/cmu-keystroke/DSL-StrongPasswordData.csv` (51 subjects
9
+ * × 8 sessions × 50 reps = 20,400 phrases of `.tie5Roanl`).
10
+ * 2. Splits subjects 80/20 into training and held-out sets.
11
+ * 3. Fits a `CalibratedProfile` against the training subjects.
12
+ * 4. Reports the fitted distribution parameters.
13
+ * 5. Computes a Kolmogorov–Smirnov-style distribution-similarity score
14
+ * (max CDF distance) of the fit against (a) the training set,
15
+ * (b) the held-out set, and (c) BlackTip's canonical HUMAN_PROFILE.
16
+ * A good fit beats the canonical profile on the held-out set.
17
+ * 6. Writes the resulting profile to
18
+ * `data/cmu-keystroke/calibrated-profile.json` so users can load
19
+ * it in their own code without re-running the fit every time.
20
+ *
21
+ * Run with: node scripts/fit-cmu-keystroke.mjs
22
+ */
23
+
24
+ import { readFileSync, writeFileSync } from 'node:fs';
25
+ import { join, dirname } from 'node:path';
26
+ import { fileURLToPath } from 'node:url';
27
+ import { parseCmuKeystrokeCsv } from '../dist/behavioral/parsers.js';
28
+ import { fitTypingDynamics, fitMouseDynamics, deriveProfileConfig } from '../dist/behavioral/calibration.js';
29
+ import { HUMAN_PROFILE } from '../dist/behavioral-engine.js';
30
+
31
+ const root = join(dirname(fileURLToPath(import.meta.url)), '..');
32
+ const csvPath = join(root, 'data', 'cmu-keystroke', 'DSL-StrongPasswordData.csv');
33
+ const outPath = join(root, 'data', 'cmu-keystroke', 'calibrated-profile.json');
34
+
35
+ console.log('Loading CMU CSV from', csvPath);
36
+ const csv = readFileSync(csvPath, 'utf-8');
37
+ const allSessions = parseCmuKeystrokeCsv(csv);
38
+ console.log(`Parsed ${allSessions.length} typing sessions (each is one rep of .tie5Roanl)`);
39
+
40
+ if (allSessions.length === 0) {
41
+ console.error('No sessions parsed — check CSV format');
42
+ process.exit(1);
43
+ }
44
+
45
+ // The phrase has 11 keys; sanity check the first session.
46
+ const first = allSessions[0];
47
+ console.log(`First session: ${first.keystrokes.length} keystrokes, phrase=${first.phrase}`);
48
+ if (first.keystrokes.length !== 11) {
49
+ console.error(`Expected 11 keystrokes per session, got ${first.keystrokes.length}`);
50
+ process.exit(1);
51
+ }
52
+
53
+ // CMU subjects are encoded in the source CSV but not exposed by the
54
+ // parser (which throws away subject IDs). For an honest 80/20 train/test
55
+ // split we re-read the CSV's first column ourselves.
56
+ const lines = csv.trim().split(/\r?\n/);
57
+ const subjectsInOrder = lines.slice(1).map((l) => l.split(',')[0]);
58
+ const uniqueSubjects = [...new Set(subjectsInOrder)];
59
+ console.log(`Found ${uniqueSubjects.length} unique subjects`);
60
+
61
+ // Deterministic 80/20 split — sort then slice, so re-runs are stable.
62
+ const trainSubjects = new Set(uniqueSubjects.slice(0, Math.floor(uniqueSubjects.length * 0.8)));
63
+ const testSubjects = new Set(uniqueSubjects.slice(Math.floor(uniqueSubjects.length * 0.8)));
64
+ console.log(`Train: ${trainSubjects.size} subjects, Test: ${testSubjects.size} subjects`);
65
+
66
+ const trainSessions = [];
67
+ const testSessions = [];
68
+ for (let i = 0; i < allSessions.length; i++) {
69
+ const subj = subjectsInOrder[i];
70
+ if (trainSubjects.has(subj)) trainSessions.push(allSessions[i]);
71
+ else if (testSubjects.has(subj)) testSessions.push(allSessions[i]);
72
+ }
73
+ console.log(`Train sessions: ${trainSessions.length}, Test sessions: ${testSessions.length}`);
74
+
75
+ // Fit
76
+ console.log('\nFitting typing dynamics on training set...');
77
+ const trainTyping = fitTypingDynamics(trainSessions);
78
+ console.log(` Hold time: mean=${trainTyping.holdTime.mean.toFixed(1)}ms p5=${trainTyping.holdTime.p5.toFixed(1)} p50=${trainTyping.holdTime.p50.toFixed(1)} p95=${trainTyping.holdTime.p95.toFixed(1)}`);
79
+ console.log(` Flight time: mean=${trainTyping.flightTime.mean.toFixed(1)}ms p5=${trainTyping.flightTime.p5.toFixed(1)} p50=${trainTyping.flightTime.p50.toFixed(1)} p95=${trainTyping.flightTime.p95.toFixed(1)}`);
80
+ console.log(` Digraphs fit: ${Object.keys(trainTyping.digraphFlightTime).length}`);
81
+ console.log(` Mistake rate: ${(trainTyping.mistakeRate * 100).toFixed(2)}%`);
82
+
83
+ // Mouse fit isn't applicable to CMU keystroke data — leave it at the
84
+ // canonical defaults so the derived ProfileConfig is well-formed.
85
+ const mouseFit = fitMouseDynamics([]);
86
+
87
+ // Held-out evaluation: extract raw flight and hold samples from each set,
88
+ // then compare empirical CDFs.
89
+ const flightsFromSessions = (sessions) => {
90
+ const out = [];
91
+ for (const s of sessions) for (let i = 1; i < s.keystrokes.length; i++) out.push(s.keystrokes[i].flightTimeMs);
92
+ return out;
93
+ };
94
+ const holdsFromSessions = (sessions) => {
95
+ const out = [];
96
+ for (const s of sessions) for (const k of s.keystrokes) out.push(k.holdTimeMs);
97
+ return out;
98
+ };
99
+
100
+ const ksDistance = (a, b) => {
101
+ // Empirical KS distance: max |F_a(x) - F_b(x)| over the merged sample set.
102
+ const sortedA = [...a].sort((x, y) => x - y);
103
+ const sortedB = [...b].sort((x, y) => x - y);
104
+ const all = [...new Set([...sortedA, ...sortedB])].sort((x, y) => x - y);
105
+ const cdf = (sorted, x) => {
106
+ let lo = 0, hi = sorted.length;
107
+ while (lo < hi) { const mid = (lo + hi) >> 1; if (sorted[mid] <= x) lo = mid + 1; else hi = mid; }
108
+ return lo / sorted.length;
109
+ };
110
+ let maxDiff = 0;
111
+ for (const x of all) {
112
+ const d = Math.abs(cdf(sortedA, x) - cdf(sortedB, x));
113
+ if (d > maxDiff) maxDiff = d;
114
+ }
115
+ return maxDiff;
116
+ };
117
+
118
+ const trainFlights = flightsFromSessions(trainSessions);
119
+ const testFlights = flightsFromSessions(testSessions);
120
+ const trainHolds = holdsFromSessions(trainSessions);
121
+ const testHolds = holdsFromSessions(testSessions);
122
+
123
+ // Synthesize samples from BlackTip's canonical profile to compare against.
124
+ // HUMAN_PROFILE.typingSpeedMs is a [min, max] uniform-ish range — sample
125
+ // 5000 values uniformly to build a synthetic distribution.
126
+ const sampleUniform = (lo, hi, n) => {
127
+ const out = [];
128
+ for (let i = 0; i < n; i++) out.push(lo + Math.random() * (hi - lo));
129
+ return out;
130
+ };
131
+ const canonicalFlights = sampleUniform(HUMAN_PROFILE.typingSpeedMs[0], HUMAN_PROFILE.typingSpeedMs[1], 5000);
132
+ const canonicalHolds = sampleUniform(HUMAN_PROFILE.clickDwellMs?.[0] ?? 40, HUMAN_PROFILE.clickDwellMs?.[1] ?? 100, 5000);
133
+
134
+ // Synthesize "calibrated" samples from the fitted distribution by
135
+ // sampling within [p5, p95]. This mirrors what BehavioralEngine will
136
+ // actually emit when configured with the fitted ProfileConfig.
137
+ const calibratedFlights = sampleUniform(trainTyping.flightTime.p5, trainTyping.flightTime.p95, 5000);
138
+ const calibratedHolds = sampleUniform(trainTyping.holdTime.p5, trainTyping.holdTime.p95, 5000);
139
+
140
+ const ksCanonicalFlight = ksDistance(testFlights, canonicalFlights);
141
+ const ksCalibratedFlight = ksDistance(testFlights, calibratedFlights);
142
+ const ksCanonicalHold = ksDistance(testHolds, canonicalHolds);
143
+ const ksCalibratedHold = ksDistance(testHolds, calibratedHolds);
144
+
145
+ console.log('\n=== Held-out KS distance (lower = closer to real human distribution) ===');
146
+ console.log(`Flight time:`);
147
+ console.log(` Canonical HUMAN_PROFILE [${HUMAN_PROFILE.typingSpeedMs[0]}, ${HUMAN_PROFILE.typingSpeedMs[1]}]ms: KS=${ksCanonicalFlight.toFixed(4)}`);
148
+ console.log(` Calibrated [p5=${trainTyping.flightTime.p5.toFixed(0)}, p95=${trainTyping.flightTime.p95.toFixed(0)}]ms: KS=${ksCalibratedFlight.toFixed(4)}`);
149
+ console.log(` Improvement: ${(ksCanonicalFlight - ksCalibratedFlight).toFixed(4)} (${((1 - ksCalibratedFlight / ksCanonicalFlight) * 100).toFixed(1)}% closer)`);
150
+ console.log(`Hold time:`);
151
+ console.log(` Canonical clickDwellMs [${HUMAN_PROFILE.clickDwellMs?.[0]}, ${HUMAN_PROFILE.clickDwellMs?.[1]}]ms: KS=${ksCanonicalHold.toFixed(4)}`);
152
+ console.log(` Calibrated [p5=${trainTyping.holdTime.p5.toFixed(0)}, p95=${trainTyping.holdTime.p95.toFixed(0)}]ms: KS=${ksCalibratedHold.toFixed(4)}`);
153
+ console.log(` Improvement: ${(ksCanonicalHold - ksCalibratedHold).toFixed(4)} (${((1 - ksCalibratedHold / ksCanonicalHold) * 100).toFixed(1)}% closer)`);
154
+
155
+ const profileConfig = deriveProfileConfig(mouseFit, trainTyping);
156
+ const calibrated = {
157
+ name: 'cmu-keystroke-2009',
158
+ source: 'CMU Keystroke Dynamics dataset (Killourhy & Maxion 2009), 80% subject train split',
159
+ fittedAt: new Date().toISOString(),
160
+ trainSubjects: trainSubjects.size,
161
+ testSubjects: testSubjects.size,
162
+ trainSessions: trainSessions.length,
163
+ testSessions: testSessions.length,
164
+ validation: {
165
+ flightTime: {
166
+ canonicalKsDistance: ksCanonicalFlight,
167
+ calibratedKsDistance: ksCalibratedFlight,
168
+ improvementRatio: 1 - ksCalibratedFlight / ksCanonicalFlight,
169
+ },
170
+ holdTime: {
171
+ canonicalKsDistance: ksCanonicalHold,
172
+ calibratedKsDistance: ksCalibratedHold,
173
+ improvementRatio: 1 - ksCalibratedHold / ksCanonicalHold,
174
+ },
175
+ },
176
+ fits: {
177
+ typing: trainTyping,
178
+ },
179
+ profileConfig,
180
+ };
181
+
182
+ writeFileSync(outPath, JSON.stringify(calibrated, null, 2));
183
+ console.log(`\nWrote calibrated profile to ${outPath}`);
184
+ console.log(`\nUse it via:`);
185
+ console.log(` import calibrated from './data/cmu-keystroke/calibrated-profile.json' with { type: 'json' };`);
186
+ console.log(` const bt = new BlackTip({ behaviorProfile: calibrated.profileConfig });`);