@rester159/blacktip 0.1.0 → 0.4.0

package/CHANGELOG.md

@@ -0,0 +1,190 @@
+# Changelog
+
+All notable changes to **BlackTip** will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.4.0] — 2026-04-10
+
+The "close the remaining gaps" release. v0.4.0 ships everything that had been accumulated since v0.2.0 plus three new pieces that close out the gaps named in the v0.3.0 wrap-up: a Kasada-validated pass on a real armed endpoint (Twitch), an `IdentityPool` for long-running session and identity rotation, and a launch-time IP reputation gate. There is no separate v0.3.0 release on npm — the v0.3.0 work was developed in the same release cycle and rolls into 0.4.0 as one shipment.
+
+### Added — IdentityPool for long-running session rotation
+
+- **`IdentityPool`** in `src/identity-pool.ts` — long-running identity management. An identity is the union of cookies, localStorage, proxy, device profile, behavior profile, locale, and timezone. The pool persists to a JSON file so identities survive restarts. Each identity has a per-domain burn list so an identity blocked on opentable.com is still eligible for amazon.com. Composes `SnapshotManager` (for cookies + storage) and `ProxyPool` (for proxy selection + ban feedback).
+- **Rotation policy** — `maxUses` and `maxAgeMs` auto-burn thresholds, plus `preferLeastRecentlyUsed` for fair distribution across identities.
+- **Proxy feedback loop** — `pool.markBurned(id, reason, 'opentable.com')` reports the ban to the bound `ProxyPool` so the next identity drawn from the pool won't reuse the same proxy on the same target until the ban window decays. This is the loop that was missing in v0.2.0 — proxies could go stale silently because nothing was telling the pool which ones had been blocked.
+- **`pool.applyToConfig(identity, baseConfig?)`** produces a `BlackTipConfig` ready to pass to `new BlackTip(config)`. **`pool.restoreSnapshot(bt, identity)`** rehydrates a saved session into the running browser. **`pool.captureSnapshot(bt, identity)`** saves the post-flow state back into the identity for next time.
+- **16 new unit tests** in `tests/identity-pool.test.ts` covering CRUD, persistence round-trip, acquisition with per-domain burning, rotation policy auto-burn, proxy feedback to ProxyPool, and `applyToConfig`/`clearBurn` semantics.
+- **`docs/identity-pool.md`** — usage guide, API reference, persistence format, and a worked example of an end-to-end flow with identity rotation and proxy feedback.
+
+### Added — IP reputation gate
+
+- **`BlackTipConfig.requireResidentialIp`** — `'throw'`, `'warn'`, or `false` (default). When set, BlackTip runs `bt.checkIpReputation()` immediately after `launch()` and either warns or throws if the egress IP is on a known datacenter ASN. Use `'throw'` in production / CI where a flagged IP would burn a real account; use `'warn'` for local dev.
+- This is the launch-time defensive companion to `IdentityPool`'s outbound proxy selection. Together they make it impossible for a misconfigured BlackTip to silently launch from a datacenter IP and burn a target.
+
+### Added — Kasada validated pass
+
+- **Twitch (Kasada)** validated as a real-world pass. `bt.testAgainstAntiBot('https://www.twitch.tv/')` returns `passed: true` with `vendorSignals: [{ vendor: 'kasada', signal: 'script' }]` — the Kasada client script is detected on the page (proving the target is actually armed) and BlackTip slides past without triggering the challenge interstitial.
+- This closes the only remaining "validated against" gap from the v0.3.0 wrap-up. BlackTip is now validated against eight commercial detector vendors on real targets: Akamai (OpenTable, BestBuy, Walmart), DataDome (Vinted), Cloudflare (Crunchbase, ChatGPT — with `cf_clearance` proof of managed-challenge auto-pass), PerimeterX/HUMAN (Walmart), and Kasada (Twitch).
+
+### Added — multi-vendor diagnostics
+
+- **`bt.testAgainstAntiBot(url)`** — generic multi-vendor anti-bot probe. Recognises Akamai, DataDome, Cloudflare, PerimeterX/HUMAN, Imperva, Kasada, and Arkose. Reports both detected challenges/blocks AND vendor signals (cookies, scripts) on passing pages — so a `passed: true` result on a target with `vendorSignals: ['datadome']` is proof that BlackTip is actually sliding past DataDome, not a false negative on an unprotected URL.
+- **`docs/anti-bot-validation.md`** — multi-vendor scoreboard for the v0.2.0 line with reproduction recipe, the cookie/script signal table, and caveats. Walmart (Akamai + PerimeterX simultaneously), BestBuy (Akamai, corrected from earlier PerimeterX classification), Vinted (DataDome with real catalog rendering), Crunchbase (Cloudflare), Ticketmaster, and OpenTable (Akamai Bot Manager regression) all pass — eight commercial-detector targets, eight passes from a residential connection.
+- **2 new diagnostics integration tests** for `testAgainstAntiBot` — the shape check on a non-protected URL plus a real-DataDome regression that asserts the `datadome` cookie signal is captured against vinted.com.
+
+### Added — Tier 2 calibration parsers (v0.3.0 prep)
+
+- **`src/behavioral/parsers.ts`** — dataset parsers that turn the calibration scaffold from a skeleton into an end-to-end pipeline. Ships:
+  - **`parseCmuKeystrokeCsv()`** — parses the CMU Keystroke Dynamics CSV (Killourhy & Maxion 2009) into `TypingSession[]`. Maps the fixed phrase `.tie5Roanl` plus Return through the CSV's `H.<key>`, `DD.<k1>.<k2>`, `UD.<k1>.<k2>` columns; converts seconds to milliseconds; uses up-down latency as flight time.
+  - **`parseBalabitMouseCsv()`** — parses Balabit Mouse Dynamics Challenge per-session CSVs into `MouseMovement[]`. Segments contiguous Move/Drag rows ending in a Pressed event into one movement; normalises timestamps to ms-since-movement-start; records click coordinates as the target.
+  - **`parseGenericTelemetryJson()`** — bring-your-own-data path for users who export telemetry in the normalized `MouseMovement` / `TypingSession` shapes directly.
+- **15 new unit tests** in `tests/behavioral-parsers.test.ts` covering both parsers' shape, time-unit conversion, header detection, CRLF tolerance, and end-to-end fitting through `fitFromSamples()`. Synthetic fixtures only — neither dataset is bundled (both have no-redistribute terms).
+- The parsers close the gap that had kept "Tier 2 behavioral calibration" on the deferred list since v0.2.0: users can now drop a CMU or Balabit file in and get a `CalibratedProfile` with no ETL of their own.
+
+### Added — TLS side-channel via bogdanfinn/tls-client
+
+- **`bt.fetchWithTls(req)` and `bt.injectTlsCookies(resp, targetUrl)`** — perform an HTTP request through a Go-based daemon built on `bogdanfinn/tls-client` that presents a real Chrome TLS ClientHello, real H2 frame settings, and real H2 frame order. Then inject the resulting cookies into the BlackTip browser session before navigating. Solves the "edge gates the very first request before BlackTip's browser has a session" problem and is the v0.3.0 path for cross-platform UA spoofing now that L016 closed the broken context-level override.
+- **`native/tls-client/main.go`** — newline-delimited JSON daemon. Stays alive across many requests so we don't pay subprocess startup cost per call. Per-request `id` field lets multiple `fetch()` calls run concurrently without interleaving.
+- **`src/tls-side-channel.ts`** — Node-side wrapper. `TlsSideChannel.spawn()` lazily starts the daemon, manages the JSON-line protocol, parses Set-Cookie headers into structured cookies, cleans up on `close()`. The wrapper is also exported standalone for callers that want to use it without a `BlackTip` instance.
+- **`docs/tls-side-channel.md`** — usage guide, build instructions (Go is the only build dep), validated TLS fingerprint (`JA4: t13d1516h2_8daaf6152771_d8a2da3f94cd`, GREASE first cipher, Chrome H2 frame order), and the four scenarios where the side-channel adds value over the browser alone.
+- **4 new integration tests** in `tests/tls-side-channel.integration.test.ts` covering JA4/GREASE/H2 fingerprint, cookie parsing, concurrent requests via per-request IDs, and post-close rejection. Tests skip cleanly if the Go daemon binary isn't built.
+- **End-to-end validated against OpenTable**: side-channel fetch returns 403 with three Akamai bot manager session cookies (`bm_ss`, `bm_s`, `bm_so`) — exactly the place a real browser would be after one request — and the browser session carries those cookies into the subsequent navigation.
+
+### Added — calibration validated against real CMU dataset
+
+- **Real CMU Keystroke Dynamics validation.** The Tier 2 calibration parsers shipped earlier in the v0.3.0 cycle have now been validated end-to-end against the real CMU dataset (Killourhy & Maxion 2009 — 51 subjects × 8 sessions × 50 reps = 20,400 phrases of `.tie5Roanl`). Deterministic 80/20 subject split: train on 40 subjects → fit `TypingFit` → KS-distance compare against the 11 held-out subjects.
+- **Result: calibrated profile beats canonical `HUMAN_PROFILE` by 53% on hold time** (KS distance 0.4297 → 0.2018) and **13.7% on flight time** (KS 0.4811 → 0.4152) on the held-out set. This is the first time BlackTip's behavioral pipeline has been validated end-to-end against a real public dataset; up through v0.2.0 the parameters were sane defaults, v0.3.0 makes them empirically grounded.
+- **`scripts/fit-cmu-keystroke.mjs`** — reproducible validation script. Run with `node scripts/fit-cmu-keystroke.mjs` after downloading the CMU CSV to `data/cmu-keystroke/`. Writes the fitted `CalibratedProfile` to `data/cmu-keystroke/calibrated-profile.json` for users to load via `new BlackTip({ behaviorProfile: calibrated.profileConfig })`.
+- **`docs/calibration-validation.md`** — methodology, fitted parameters, KS-distance table, what the result proves and what it does NOT prove, future calibration sources (Balabit, GREYC-NISLAB, Buffalo Free-Text).
+- **CMU parser fixes**: the original parser used bare characters as CSV column labels but the CMU CSV uses `period` for `.`, `five` for `5`, and `Shift.r` for the capital `R` (which requires a Shift modifier in `.tie5Roanl`). Fixed and the synthetic-fixture tests updated to match.
+
+### Added — diagnostics fixes
+
+- **Cookie detection now reads via the BlackTip cookies API instead of `document.cookie`.** The earlier implementation missed httpOnly cookies — and `cf_clearance`, `__cf_bm`, `_abck`, `datadome`, `bm_sz` are all httpOnly. This caused false negatives where a target was actually protected and BlackTip was passing it but the diagnostic reported `vendorSignals: []`. Fixed: `testAgainstAntiBot` now lists Cloudflare cookie signals on chatgpt.com, vinted.com, crunchbase.com (cf_clearance present, proving each had served and BlackTip had passed a managed challenge silently).
+- **More commercial detector targets validated**: ChatGPT (Cloudflare with cf_clearance present), Canada Goose, Hyatt, Footlocker (no live Kasada cookies on homepages — they may arm only on cart/checkout). Documented in the updated `docs/anti-bot-validation.md`.
+
+### Test suite
+
+- **78 unit tests passing (was 63), zero regressions.** Plus 4 new TLS integration tests (skip if daemon not built), 2 new diagnostics integration tests, 15 new behavioral parser tests.
+
+## [0.2.0] — 2026-04-10
+
+### Defeating Akamai Bot Manager — the L016 fix
+
+The headline change in 0.2.0 is fixing the User-Agent / Sec-Ch-Ua consistency bug that had been silently undermining BlackTip against top-tier commercial detectors since 0.1.0. **OpenTable's Akamai Bot Manager went from blocking us at the edge to letting us into the booking flow on the very next request after the fix landed.** Same machine, same network, same IP.
+
+See `docs/akamai-bypass.md` for the full plan, methodology, and status against each Akamai detection layer.
+
+### Fixed
+
+- **L016 — User-Agent / Sec-Ch-Ua consistency.** `browser-core.ts` was setting `userAgent` at the Playwright context level via `newContext({userAgent: ...})`. Playwright's `userAgent` option overrides the `User-Agent` HTTP header but does NOT update the `Sec-Ch-Ua` / `Sec-Ch-Ua-Mobile` / `Sec-Ch-Ua-Platform` client hint headers — those come from the actual Chromium binary version. The result was BlackTip broadcasting `User-Agent: Chrome/125` and `Sec-Ch-Ua: "Google Chrome";v="146"` simultaneously, an inconsistency real Chrome NEVER produces and that Akamai Bot Manager catches as a textbook spoofing tell. Fix: removed the `userAgent` context override entirely; real Chrome's natural User-Agent comes through and matches Sec-Ch-Ua.
+- This bug had been silently invalidating BlackTip's stealth against high-tier detectors. CreepJS / bot.sannysoft / browserleaks / fingerprint.com don't cross-check UA against client hints, so they didn't catch it. Akamai / DataDome / PerimeterX do, and they did.
+- **Side effect:** cross-platform UA spoofing (declaring a `desktop-macos` profile while running on Linux) is no longer supported by default. v0.3.0 will reintroduce it via `setExtraHTTPHeaders` for callers who need it.
+
+### Added
+
+- **`bt.captureFingerprint()`** — captures TLS, HTTP/2, and HTTP header fingerprint via `tls.peet.ws/api/all` and `httpbin.org/headers`. Returns a structured `FingerprintSnapshot` with the critical `headers.uaConsistent` flag for verifying L016 is fixed in any given install.
+- **`bt.checkIpReputation()`** — queries the active session's egress IP via `ipinfo.io`, scores it against known datacenter / residential ASN patterns (Amazon, Google Cloud, Azure, OVH, DigitalOcean, Hetzner, Linode, Vultr — and major residential ISPs), returns an `IpReputationResult` with `isDatacenter` / `isResidential` flags and free-form notes.
+- **`bt.testAgainstAkamai(url)`** — visits an Akamai-protected URL and reports the result with diagnosis. Recognizes the Akamai Access Denied page format, extracts the reference number, and returns a suggested next step. Use as a regression check in CI or interactively when troubleshooting.
+- **`bt.warmSession({sites?, dwellMsRange?})`** — visits a sequence of "normal" sites with realistic dwell times and small scrolls before the target navigation. Accumulates cookies, populates History, and triggers natural behavioral signals so the target sees a session that already looks human.
+- **`BlackTipConfig.userDataDir`** — persistent Chrome user data directory. When set, BlackTip uses `chromium.launchPersistentContext()` instead of the default fresh context, so cookies / localStorage / history / visited sites accumulate across runs. Critical for sites with "first request from unknown session" challenges.
+- **`docs/akamai-bypass.md`** — comprehensive 7-phase plan for defeating Akamai Bot Manager, with current passing/failing matrix per detection layer, validated targets, recipes, and a troubleshooting checklist for users who hit blocks.
+- **10 new diagnostics integration tests** in `tests/v04-diagnostics.integration.test.ts`. The most important is the L016 consistency check, which serves as a regression guard so we never reintroduce the UA / Sec-Ch-Ua mismatch.
+
+### Validated against
+
+- **OpenTable** (Akamai Bot Manager): blocked at every URL in 0.1.0 with Access Denied references; passing as of 0.2.0. Confirmed against the deep-link booking page for Gjelina (Venice).
+- All 0.1.0 detector targets continue to pass (bot.sannysoft, CreepJS, tls.peet.ws, browserleaks×4, fingerprint.com, pixelscan, browserscan, nowsecure.nl).
+
+### Test suite
+
+- **162 → 172 tests passing**, zero regressions.
+
+### What's deferred to 0.3.0+
+
+- TLS-rewriting proxy (`bogdanfinn/tls-client` integration) for cross-platform UA spoofing and Chrome version impersonation
+- Akamai sensor data analysis and targeted JS-level patches for sites that probe deeper than headers
+- Tier 2 behavioral calibration against real public datasets (Balabit, CMU Keystroke, GREYC)
+
+## [0.1.0] — 2026-04-10
+
+### First public release
+
+BlackTip is a stealth browser instrument for AI agents. Real Chrome via `patchright` with CDP-level stealth patches, human-calibrated behavioral simulation, and an agent-friendly TCP serve protocol with bundled JSON responses. Passes every free fingerprint detector tested and known public Cloudflare bot-fight test targets.
+
+### Added — core architecture
+
+- **Real Chrome via `channel: 'chrome'`** — uses the installed Chrome Stable binary for an authentic TLS ClientHello with rotating GREASE values and a native HTTP/2 frame order. No TLS proxy required for most use cases.
+- **`patchright`-based CDP stealth** — drop-in Playwright replacement with patches that neutralize `Runtime.Enable` detection, automation string artifacts, and Error-stack hooks that vanilla Playwright leaks.
+- **Device profiles** — `desktop-windows`, `desktop-macos`, `desktop-linux` with matching user agents, hardware concurrency, plugins, fonts, and WebGL vendor/renderer strings.
+- **Behavioral engine** — Bézier mouse paths with Fitts' Law movement time, digraph-aware typing with realistic typo-and-correction patterns, scroll deceleration curves, normal-distribution sampling via Box-Muller, and reading pause estimation tied to the profile's WPM range.
+- **Retry engine** — six-strategy cascade (`standard`, `wait`, `reload`, `altSelector`, `scroll`, `clearOverlays`) with event emission on each retry.
+- **Seeded noise layers** — canvas and audio fingerprints receive profile-seeded noise via Mulberry32 PRNG so they're stable cross-session but unique per profile, matching how real hardware variance looks.
+
+### Added — agent primitives
+
+- **`bt.waitForStable({networkIdleMs, domIdleMs, maxMs})`** — waits until no network requests and no DOM mutations for a window. Replaces fixed `setTimeout` sleeps with a real page-settled signal.
+- **`bt.waitForText(text, {timeout})`** — polls body `innerText` for a target string. For server-rendered confirmations, OCR completion, and async content.
+- **`bt.inspect(selector)`** — returns `{exists, visible, tagName, text, attributes, boundingBox}` in one call. Replaces multiple hand-written `executeJS` queries for diagnostics.
+- **`bt.listOptions(selectorOrBaseId)`** — enumerates Angular/React-style custom dropdown options via the `{baseId}_option-{n}` pattern.
+- **`bt.networkSince(ms, pattern?)`** and **`bt.didRequestFireSince(pattern, ms)`** — filtered Performance API resource entries and a boolean convenience for "did my submit actually reach the server?" diagnostics. Critical for avoiding burned retries on lockout-protected forms.
+- **`bt.dismissOverlays()`** — proactively hides fixed/sticky overlays matching known patterns (Intercom, Drift, Zendesk, OneTrust, Medallia, cookie banners).
+- **`bt.pauseForInput({prompt, validate?, timeoutMs?})`** — first-class MFA / user-in-the-loop support. Emits a `paused` frame to the client via the serve protocol; resumes when the client sends `RESUME <id>\n<value>`.
+- **`bt.findInShadowDom(cssSelector, {timeout?})`** — recursive walker over open shadow roots. Supports modern component libraries (Lit, Stencil, Material Web Components).
+- **`bt.download(selector, {saveTo})`** — click-to-download with returned `{path, size, suggestedFilename, url}` metadata.
+
+### Added — click robustness
+
+- **Live bounding-box re-read before `mouse.click()`** — re-captures the target's current box immediately before the click. If the element moved more than ~5 pixels during the mouse-movement phase (DOM reflow, async scripts, layout shifts), performs a short correction move so the click lands on the right place. Fixes the L011 coordinate-reflow issue.
+- **Pre-click hit verification** — uses `document.elementFromPoint` to verify the click coordinates actually land on the intended interactive element. If an overlay is covering the target, dismisses it and falls back to `locator.click({force: true})`.
+- **Auto-importance detection on click/clickText/clickRole** — buttons whose visible text matches `Submit`, `Pay`, `Confirm`, `Place Order`, `Delete`, `Remove`, `Checkout`, `Purchase`, etc. automatically receive `importance: 'high'` which triggers longer pre-action hesitation (2–3× base pause plus a hesitation spike). Matches the long-tail distribution behavioral biometrics systems expect on consequential actions. Callers can override with explicit `importance`.
+- **`BlackTipFrame.type()` iframe hardening** — ported the `Control+A + Backspace + keyboard.type + inputValue` verification pattern from the main `type()` method to the iframe variant. Ready for Stripe Elements, Braintree Hosted Fields, and other framework-driven iframes.
+
+### Added — serve mode and CLI
+
+- **Bundled JSON responses** — every `send` command returns `{ok, result, url, title, screenshotPath, screenshotB64, screenshotBytes, durationMs, error?}` in one frame. Cuts round-trips by ~60% for linear flows.
+- **Batched commands** — `BATCH\n<json array>` runs multiple commands sequentially and returns an array of bundles, stopping on first failure.
+- **CLI flags** — `--file <path>`, `--stdin`, `--pretty`, `--port` eliminate shell-escape hell for complex JavaScript commands.
+- **`npx blacktip` subcommands** — `serve`, `send`, `batch`, `resume`, `pending`, `exec` with per-command help.
+
+### Added — infrastructure
+
+- **`ProxyPool`** — round-robin / random / least-used strategies, per-domain ban tracking with decay window, reputation snapshot for observability, URL formatters for BrightData / Oxylabs / Smartproxy.
+- **`SnapshotManager`** — `capture()` and `restore()` for session migration across proxies or machines. Serializes cookies, localStorage, sessionStorage, and active URL into a JSON blob.
+- **Observability** — `attachObservability(bt, exporters)` wires BlackTip's EventEmitter events into a `StructuredEvent` shape compatible with OpenTelemetry attribute data model. Ships `JsonlFileExporter` and `ConsoleExporter` as reference implementations. No OTel SDK dependency.
+- **Behavioral calibration scaffold** — `src/behavioral/calibration.ts` with `fitDistribution`, `fitFittsLaw` (OLS), `fitMouseDynamics`, `fitTypingDynamics`, `fitFromSamples`, `deriveProfileConfig`. Ready to ingest real mouse-dynamics and keystroke-dynamics datasets (Balabit Mouse Dynamics Challenge, CMU Keystroke Dynamics, GREYC-NISLAB) via user-supplied parser wrappers.
+
+### Detector results
+
+Captured against free public detectors as of the release:
+
+- **bot.sannysoft.com** — 31 passed / 0 failed across 57 rows
+- **tls.peet.ws** — JA4 `t13d1516h2_...`, first cipher rotates `TLS_GREASE (0x...)` matching real Chrome 122/124
+- **CreepJS** — Grade A, 0% headless, 0% stealth, 31% like-headless (unavoidable desktop false-positive)
+- **browserleaks.com/canvas** — signature shared with 100 of 298,939 browsers (blending in, not unique)
+- **browserleaks.com/webgl** — real GPU via ANGLE, not SwiftShader
+- **browserleaks.com/webrtc** — no RFC1918 local IP leak
+- **browserleaks.com/javascript** — Chrome 122 on Win32, en-US, America/New_York, 1920×1080 all consistent
+- **fingerprint.com/demo** — passes
+- **pixelscan.net** — passes
+- **browserscan.net** — identified as "Chrome on Windows 10", not flagged
+
+Real-target validation:
+
+- **nowsecure.nl** (Cloudflare bot-fight test, nodriver author's public benchmark) — passes without challenge
+- **antoinevastel.com/bots** (ex-DataDome VP of Research) — loads without block
+- **Anthem.com** (Okta MFA, Angular SPA, real medical claim submission) — full end-to-end flow successful
+
+### Notes
+
+- Always runs headful. `headless: true` in config is silently ignored — there is no real-headless path that passes serious detectors.
+- Real Chrome must be installed on the host for the preferred `channel: 'chrome'` path. patchright's bundled Chromium is the fallback.
+- Scoped-name fix: the initial planned unscoped name `blacktip` was blocked by npm's anti-typosquatting policy (too similar to the pre-existing `black-tip` package). Released as `@rester159/blacktip` instead.
+
+[Unreleased]: https://github.com/rester159/blacktip/compare/v0.4.0...HEAD
+[0.4.0]: https://github.com/rester159/blacktip/compare/v0.2.0...v0.4.0
+[0.2.0]: https://github.com/rester159/blacktip/compare/v0.1.0...v0.2.0
+[0.1.0]: https://github.com/rester159/blacktip/releases/tag/v0.1.0

package/README.md

@@ -1,5 +1,17 @@
+<p align="center">
+  <img src="assets/logo.svg" alt="BlackTip" width="360">
+</p>
+
 # BlackTip
 
+[![npm version](https://img.shields.io/npm/v/%40rester159%2Fblacktip.svg?color=cb3837&label=npm)](https://www.npmjs.com/package/@rester159/blacktip)
+[![npm downloads](https://img.shields.io/npm/dm/%40rester159%2Fblacktip.svg?color=cb3837)](https://www.npmjs.com/package/@rester159/blacktip)
+[![CI](https://github.com/rester159/blacktip/actions/workflows/ci.yml/badge.svg)](https://github.com/rester159/blacktip/actions/workflows/ci.yml)
+[![license](https://img.shields.io/npm/l/%40rester159%2Fblacktip.svg?color=blue)](LICENSE)
+[![GitHub stars](https://img.shields.io/github/stars/rester159/blacktip?style=social)](https://github.com/rester159/blacktip/stargazers)
+[![TypeScript](https://img.shields.io/badge/TypeScript-5.0+-3178C6?logo=typescript&logoColor=white)](https://www.typescriptlang.org/)
+[![Node](https://img.shields.io/node/v/%40rester159%2Fblacktip.svg?color=5fa04e&logo=node.js&logoColor=white)](https://nodejs.org/)
+
 **Stealth browser instrument for AI agents.** Real Chrome + patchright CDP stealth + human-calibrated behavioral simulation. BlackTip is the hands, you (or your agent) are the brain.
 
 BlackTip is not an agent. It does not parse natural language, does not plan, and does not decide what to click. It exposes primitives — `navigate`, `click`, `type`, `scroll`, `screenshot`, `waitForStable` — and wraps every action in human behavior that defeats bot detection.
@@ -37,6 +49,22 @@ BlackTip's architecture:
 - nowsecure.nl (Cloudflare bot-fight test, nodriver author's public benchmark) — passes without challenge
 - antoinevastel.com/bots (ex-DataDome VP of Research) — loads without block
 - Anthem.com (Okta MFA, Angular SPA, real insurance claim submission) — end-to-end flow successful
+- Walmart.com (Akamai + PerimeterX simultaneously) — passes both layers
+- BestBuy.com (Akamai) — passes
+- Vinted.com (DataDome) — real catalog renders, datadome cookie present
+- Crunchbase.com (Cloudflare) — passes, cf_clearance issued silently
+- ChatGPT.com (Cloudflare) — passes, cf_clearance issued silently
+- Twitch.tv (Kasada) — passes, kasada client script detected on the page
+- Ticketmaster.com — passes
+- OpenTable.com Gjelina deep link (Akamai Bot Manager) — passes; full booking endpoint with real time slots
+
+**v0.4.0 also ships:**
+- **`IdentityPool`** — long-running session and identity rotation. An identity is the union of cookies, localStorage, proxy, device profile, behavior profile, locale, and timezone. Persists to a JSON file so identities survive restarts. Per-domain burn list — an identity blocked on one site is still eligible for others. Composes `SnapshotManager` and `ProxyPool` with a feedback loop that auto-bans burned identities' proxies in the pool. See **[docs/identity-pool.md](docs/identity-pool.md)**.
+- **`BlackTipConfig.requireResidentialIp`** — `'throw'` / `'warn'` / `false`. Runs `bt.checkIpReputation()` on launch and refuses (or warns) if the egress IP is on a known datacenter ASN. The launch-time defensive companion to `IdentityPool`'s outbound proxy selection.
+- **Go-based TLS side-channel daemon (`bt.fetchWithTls`)** that performs HTTP requests with a real Chrome TLS ClientHello, H2 frame settings, and frame order via `bogdanfinn/tls-client`. Use to make gating requests the browser can't make through itself, then inject the resulting cookies into the browser session before navigating. JA4 `t13d1516h2_8daaf6152771_d8a2da3f94cd`, GREASE first cipher, exact Chrome H2 fingerprint. See **[docs/tls-side-channel.md](docs/tls-side-channel.md)**.
+- **Behavioral profile calibrated against the real CMU Keystroke Dynamics dataset.** **53% closer to real human hold-time distribution** than the canonical defaults on a held-out subject set (51 subjects, 80/20 split). See **[docs/calibration-validation.md](docs/calibration-validation.md)** for methodology and reproduction.
+
+See **[docs/anti-bot-validation.md](docs/anti-bot-validation.md)** for the live multi-vendor scoreboard with vendor-signal verification (proves each target is actually protected, not a false negative on an unprotected URL).
 
 ---
 
@@ -147,10 +175,32 @@ See `AGENTS.md` for the full agent-facing reference, including the decision tree
 | `bt.cookies()` / `bt.setCookies()` / `bt.clearCookies()` | Cookie jar |
 | `bt.executeJS(script)` | Raw JS evaluation |
 | `bt.pauseForInput({prompt, validate?, timeoutMs?})` | User-in-the-loop (MFA) |
+| `bt.captureFingerprint()` | TLS / HTTP2 / header snapshot via tls.peet.ws + httpbin (v0.2.0) |
+| `bt.checkIpReputation()` | Egress IP, ASN, datacenter/residential heuristic (v0.2.0) |
+| `bt.testAgainstAkamai(url)` | Akamai-protected target probe with diagnosis (v0.2.0) |
+| `bt.testAgainstAntiBot(url)` | Multi-vendor probe — detects Akamai, DataDome, Cloudflare, PerimeterX, Imperva, Kasada, Arkose, plus vendor signals on a passing page (v0.2.0) |
+| `bt.fetchWithTls(req)` | Perform an HTTP request via a Go-based `bogdanfinn/tls-client` daemon with a real Chrome TLS ClientHello, H2 frame settings, and frame order. Use for first-request edge gating and cross-platform UA spoofing. Requires the daemon binary at `native/tls-client/` (build with `go build .`) (v0.3.0) |
+| `bt.injectTlsCookies(resp, targetUrl?)` | Inject cookies returned by `fetchWithTls()` into the browser session, filtered by target eTLD+1 (v0.3.0) |
+
+Plus `IdentityPool` (v0.4.0) for long-running session and identity rotation across many flows. See **[docs/identity-pool.md](docs/identity-pool.md)**.
+| `bt.warmSession({sites?, dwellMsRange?})` | Pre-target warm-up — visit normal sites first (v0.2.0) |
 | `bt.serve(port?)` | Start TCP command server |
 
 Plus `SnapshotManager`, `ProxyPool`, `attachObservability`, and the calibration module — see the TypeScript types for details.
 
+### Akamai Bot Manager
+
+As of v0.2.0, BlackTip passes Akamai Bot Manager on validated targets including OpenTable. The headline fix (L016) was a User-Agent / Sec-Ch-Ua client hint consistency bug that had been silently undermining stealth against top-tier commercial detectors since v0.1.0. See **[docs/akamai-bypass.md](docs/akamai-bypass.md)** for the full plan, methodology, current status by detection layer, and a troubleshooting checklist for users who hit blocks.
+
+Quick verification that you're on a fixed build:
+
+```typescript
+const fp = await bt.captureFingerprint();
+if (!fp.headers.uaConsistent) {
+  throw new Error('UA / Sec-Ch-Ua mismatch — upgrade BlackTip to v0.2.0+');
+}
+```
+
 ---
 
 ## Configuration
@@ -168,6 +218,10 @@ new BlackTip({
   screenResolution: { width: 1920, height: 1080 },
   proxy: 'http://user:pass@host:port',
   chromiumPath: '/custom/chrome',
+  // v0.2.0 — persistent Chrome profile for cookies/history continuity
+  // across BlackTip runs. Critical for sites with "first request from
+  // unknown session" challenges (Akamai, DataDome, PerimeterX).
+  userDataDir: './.bt-profile',
 });
 ```
 
@@ -196,6 +250,47 @@ Changes to BlackTip's `.ts` files recompile to `dist/` on save, and the consumin
 
 ---
 
+## Server deployment
+
+BlackTip runs real Chrome in headful mode — on servers you use **Xvfb** (virtual X framebuffer) so Chrome renders to a virtual display instead of a physical monitor. This is the standard approach used by every serious stealth tool; running Chrome with `--headless` is NOT supported because headless mode is detectable at many fingerprint levels.
+
+The repo ships production-ready deployment artifacts:
+
+- **`Dockerfile`** — production image with Chrome Stable, Xvfb, Node 20, and all runtime dependencies. Build with `docker build -t blacktip:latest .`.
+- **`docker-compose.yml`** — one-command local / dev environment. `docker compose up --build`.
+- **`deploy/systemd/xvfb.service`** and **`deploy/systemd/blacktip.service`** — systemd unit files for bare-metal VPS deployments.
+- **`deploy/README.md`** — full deployment guide covering Docker, systemd, AWS / GCP / DigitalOcean / Hetzner / Fly.io, sizing, cost, monitoring, and troubleshooting.
+
+**Quick Docker start:**
+
+```bash
+docker build -t blacktip:latest .
+docker run --rm -it \
+  --shm-size=2gb \
+  --cap-add=SYS_ADMIN \
+  -p 9779:9779 \
+  blacktip:latest
+```
+
+**Quick bare-metal start (Ubuntu 22.04):**
+
+```bash
+# System deps + Chrome + Node
+sudo apt-get update && sudo apt-get install -y xvfb libnss3 libatk-bridge2.0-0 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxrandr2 libgbm1 libxss1 libasound2 fonts-liberation libvulkan1
+curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
+sudo apt-get install -y nodejs
+wget -qO /tmp/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb && sudo apt install -y /tmp/chrome.deb
+
+# Run under Xvfb
+export DISPLAY=:99
+Xvfb :99 -screen 0 1920x1080x24 &
+node your-app.js
+```
+
+See **[deploy/README.md](deploy/README.md)** for the full guide including systemd setup, cloud provider specifics, resource sizing, and troubleshooting.
+
+---
+
 ## What BlackTip is NOT
 
 - **Not an agent.** It doesn't plan or decide. A human or an LLM drives it.

package/dist/behavioral/parsers.d.ts

@@ -0,0 +1,89 @@
+/**
+ * Dataset parsers for behavioral calibration.
+ *
+ * The calibration scaffold (`./calibration.ts`) defines normalized
+ * `MouseMovement` and `TypingSession` shapes plus distribution fitters.
+ * What was missing through v0.2.0 was the bridge from real public datasets
+ * into those shapes — without parsers, the scaffold required every user
+ * to write their own ETL, which kept "Tier 2 behavioral calibration" on
+ * the deferred list.
+ *
+ * This module ships parsers for the two datasets we point users at
+ * most often:
+ *
+ *   1. **CMU Keystroke Dynamics** (Killourhy & Maxion, 2009).
+ *      Free for research. CSV format with columns `subject`, `sessionIndex`,
+ *      `rep`, then for the fixed phrase `.tie5Roanl` a tuple of
+ *      `H.<key>` (hold), `DD.<k1>.<k2>` (down-down latency), and
+ *      `UD.<k1>.<k2>` (up-down = flight time). 51 subjects × 8 sessions
+ *      × 50 repetitions = 20,400 phrases.
+ *
+ *   2. **Balabit Mouse Dynamics Challenge** (Antal & Egyed-Zsigmond, 2014).
+ *      Free for research. Per-session CSV with columns
+ *      `record_timestamp`, `client_timestamp`, `button`, `state`, `x`, `y`.
+ *      Each row is a single mouse event; consecutive `Mouse` rows
+ *      followed by a `Pressed` row form one movement.
+ *
+ * Plus a generic JSON loader for users with their own telemetry exported
+ * in the `MouseMovement` / `TypingSession` shapes directly.
+ *
+ * The parsers do NOT download the datasets — both have ToS that say "do
+ * not redistribute". Users acquire the data themselves and feed the file
+ * contents in as a string. We just turn raw text into normalized samples.
+ */
+import type { MouseMovement, TypingSession } from './calibration.js';
+/** The fixed phrase typed by every CMU subject. Used to map column index → key. */
+export declare const CMU_PHRASE = ".tie5Roanl";
+/**
+ * Parse the CMU Keystroke Dynamics CSV (DSL-StrongPasswordData.csv).
+ *
+ * Each row is one repetition of the phrase `.tie5Roanl` plus Return,
+ * yielding 11 keys, 11 hold-times, 10 down-down and 10 up-down latencies.
+ * We normalize each row into one `TypingSession` of 11 keystrokes.
+ *
+ * The CSV looks like:
+ *   subject,sessionIndex,rep,H.period,DD.period.t,UD.period.t,H.t,DD.t.i,UD.t.i,H.i,...
+ *   s002,1,1,0.1491,0.3979,0.2488,0.1069,0.1674,0.0605,...
+ *
+ * All time values are in **seconds** in the source file; we convert to
+ * milliseconds.
+ *
+ * Returns one `TypingSession` per CSV row. Pass these straight into
+ * `fitTypingDynamics()`.
+ */
+export declare function parseCmuKeystrokeCsv(csvText: string): TypingSession[];
+/**
+ * Parse a single Balabit Mouse Dynamics session CSV.
+ *
+ * Each row is one mouse event:
+ *   record_timestamp,client_timestamp,button,state,x,y
+ *   1424866316.93,0.000,NoButton,Move,1192,529
+ *   1424866316.99,0.064,NoButton,Move,1183,532
+ *   ...
+ *   1424866317.84,0.911,Left,Pressed,820,440
+ *
+ * We segment into movements: each contiguous run of `Move`/`Drag` rows
+ * followed by a `Pressed` row becomes one `MouseMovement`. The press row
+ * is the click target. Movements that don't end in a press are also
+ * recorded but with `endedWithClick: false` — these are navigation moves.
+ *
+ * Time is normalized to milliseconds since the first sample of the
+ * movement.
+ */
+export declare function parseBalabitMouseCsv(csvText: string): MouseMovement[];
+/**
+ * Generic loader for users who already export their telemetry in the
+ * normalized shapes. Accepts JSON of the form:
+ *
+ *   { "movements": MouseMovement[], "sessions": TypingSession[] }
+ *
+ * Either field may be absent. Returns the parsed object with empty
+ * defaults filled in. This is the "bring your own data" path — most
+ * production users will write a tiny exporter on their own telemetry
+ * pipeline that emits this shape, then feed it through `fitFromSamples()`.
+ */
+export declare function parseGenericTelemetryJson(jsonText: string): {
+    movements: MouseMovement[];
+    sessions: TypingSession[];
+};
+//# sourceMappingURL=parsers.d.ts.map

package/dist/behavioral/parsers.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"parsers.d.ts","sourceRoot":"","sources":["../../src/behavioral/parsers.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAgCG;AAEH,OAAO,KAAK,EAAmB,aAAa,EAAe,aAAa,EAAE,MAAM,kBAAkB,CAAC;AAInG,mFAAmF;AACnF,eAAO,MAAM,UAAU,eAAe,CAAC;AAEvC;;;;;;;;;;;;;;;;GAgBG;AACH,wBAAgB,oBAAoB,CAAC,OAAO,EAAE,MAAM,GAAG,aAAa,EAAE,CAoDrE;AAoBD;;;;;;;;;;;;;;;;;GAiBG;AACH,wBAAgB,oBAAoB,CAAC,OAAO,EAAE,MAAM,GAAG,aAAa,EAAE,CA0DrE;AAID;;;;;;;;;;GAUG;AACH,wBAAgB,yBAAyB,CAAC,QAAQ,EAAE,MAAM,GAAG;IAC3D,SAAS,EAAE,aAAa,EAAE,CAAC;IAC3B,QAAQ,EAAE,aAAa,EAAE,CAAC;CAC3B,CASA"}