tls-forge 0.1.0rc5__tar.gz

tls_forge-0.1.0rc5/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Ihar Kazlouski
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

tls_forge-0.1.0rc5/NOTICE

@@ -0,0 +1,48 @@
+tls-forge
+Copyright 2026 Ihar Kazlouski
+
+This product includes software developed by Ihar Kazlouski.
+
+Licensed under the Apache License, Version 2.0 (the "License"); see LICENSE.
+Section 4 of that licence obliges anyone who copies, modifies or redistributes
+this work — in source or binary form, and an npm package containing a compiled
+binary is a binary form — to retain the copyright notice above and to carry this
+NOTICE file with it.
+
+
+WHAT IS NOT COVERED BY THE GRANT
+--------------------------------
+
+Two files in this tree are recordings of another program's output rather than
+the author's work, and the Apache-2.0 grant above is not a claim over them:
+
+  testdata/chrome151-clienthello.bin
+      A TLS ClientHello as Google Chrome 151 put it on the wire, captured on the
+      author's own machine and kept as a test fixture. It is a protocol message
+      — a measurement of what a program sent — not a copy of that program.
+
+  profile/data/chrome_151/linux.json
+  profile/data/chrome_151/macos.json
+  profile/data/chrome_151/windows.json
+      The same ClientHello together with the HTTP/2 settings, header order and
+      user-agent string observed from the same browser, one file per platform.
+      The user-agent strings are Google's; the rest is measurement.
+
+None of these files contains any part of Chrome, and nothing here is endorsed by
+or affiliated with Google. Everything else in the tree — the library, the echo
+server, the command line, the Node client, the tests and the prose — is the
+author's work and is covered by LICENSE.
+
+
+THIRD-PARTY SOFTWARE
+--------------------
+
+tls-forge compiles to a single static binary, so its dependencies travel inside
+it — including inside the binaries published in the npm packages. Their licences
+are reproduced in full in THIRD-PARTY-NOTICES.txt, which must travel with any
+distributed copy for the same reason this file must.
+
+One of them is worth naming here. github.com/bogdanfinn/fhttp ships no licence
+file of its own; its sources carry the Go Authors' copyright header and it is a
+fork of the Go standard library's net/http, so the Go licence is the one that
+applies. It is reproduced in THIRD-PARTY-NOTICES.txt rather than left as a gap.

tls_forge-0.1.0rc5/PKG-INFO

@@ -0,0 +1,336 @@
+Metadata-Version: 2.5
+Name: tls-forge
+Version: 0.1.0rc5
+Summary: Scraping HTTP client that gets past TLS fingerprinting — sends a real browser's JA3/JA4, HTTP/2 and header order
+Project-URL: Homepage, https://github.com/Sec-CH-Lemon/tls-forge
+Project-URL: Source, https://github.com/Sec-CH-Lemon/tls-forge
+Project-URL: Changelog, https://github.com/Sec-CH-Lemon/tls-forge/blob/main/CHANGELOG.md
+Author: Ihar Kazlouski
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+License-File: THIRD-PARTY-NOTICES.txt
+Keywords: anti-bot,bot-detection,browser,cloudflare,fingerprint,http2,impersonate,ja3,ja4,scraper,scraping,tls,tls-fingerprint,utls,web-scraping
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: test
+Requires-Dist: coverage>=7; extra == 'test'
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# TLS Forge for Python
+
+`tls-forge` is a typed Python HTTP client whose network fingerprint comes from
+a measured browser profile. It controls the TLS ClientHello, HTTP/2 settings and
+header order that Python's OpenSSL-based clients cannot reproduce by changing a
+`User-Agent`.
+
+The Python package handles application orchestration and communicates with one
+long-lived `tls-forge daemon` process. The project overview, CLI and explanation
+of TLS fingerprinting are in the
+[main README](https://github.com/Sec-CH-Lemon/tls-forge#readme).
+
+It does not execute page JavaScript and does not solve CAPTCHAs or managed
+challenges. It addresses rejection caused specifically by TLS and HTTP
+fingerprint mismatch.
+
+## Requirements and installation
+
+- Python 3.9 or newer;
+- a supported platform wheel or Go 1.26.7+ for a custom transport.
+
+```bash
+pip install tls-forge
+```
+
+Import the package as `tlsforge`, without the hyphen:
+
+```python
+import tlsforge
+```
+
+Platform wheels include the transport binary. Installation does not run a
+compiler, download an executable through a postinstall hook, or install runtime
+Python dependencies.
+
+To use a custom build:
+
+```bash
+export TLSFORGE_BIN=/absolute/path/to/tls-forge
+```
+
+## Basic use
+
+```python
+import tlsforge
+
+with tlsforge.Client(profile="chrome") as client:
+    response = client.get("https://tls.browserleaks.com/json")
+    print(response.status)
+    print(response.json())
+```
+
+The transport is resolved during construction and started lazily on the first
+request. Prefer the context-manager form so the child process is always closed.
+
+## Constructor options
+
+```python
+client = tlsforge.Client(
+    profile="chrome_151",
+    proxy="http://user:pass@proxy.example:8080",
+    timeout=45,
+    binary="/absolute/path/to/tls-forge",
+    insecure=False,
+    cookie_file="cookies.json",
+    cookie_set="warm-eu",
+    on_stderr=lambda line: print(line),
+)
+```
+
+| Option | Type and default | Meaning |
+|---|---|---|
+| `profile` | `str | None`, CLI default | Profile name or JSON profile path passed to `tls-forge daemon --profile`. |
+| `proxy` | `str | None`, direct | HTTP, HTTPS or SOCKS proxy URL, optionally with credentials. |
+| `timeout` | `float`, `45.0` | Per-request deadline in seconds. It must be positive and finite. The Go transport receives an additional 15-second grace period so both timeout layers do not race. |
+| `binary` | `str | os.PathLike | None`, auto | Explicit path to the transport executable. A missing explicit path raises `BinaryNotFound` and does not fall through. |
+| `insecure` | `bool`, `False` | Skip upstream certificate verification. Use only for controlled endpoints. |
+| `cookie_file` | path or `None` | Load a warmed session from project JSON, browser-export JSON or Netscape cookies.txt. |
+| `cookie_set` | `str | None` | Select a named set from a multi-session file; when omitted, the daemon chooses one at random. Requires `cookie_file`; without it, `Client(...)` raises `ValueError`. |
+| `on_stderr` | callable or `None` | Receive transport diagnostics one line at a time. Callback exceptions are ignored so diagnostics cannot break the reader thread. |
+
+`tlsforge.DEFAULT_TIMEOUT` contains the default Python-side timeout.
+
+`Client()` without `profile` uses the daemon's default profile: a locally
+installed `local` profile when available and otherwise the latest bundled
+`chrome` profile.
+
+## Request methods
+
+### `get(url, *, headers=None, order=None, cookies=None)`
+
+```python
+response = client.get(
+    "https://example.com/page",
+    headers={
+        "referer": "https://example.com/",
+        "accept-language": "en-GB,en;q=0.9",
+    },
+    order=["referer", "accept-language"],
+    cookies=["session=abc"],
+)
+```
+
+### `post(url, body="", *, headers=None, order=None, cookies=None)`
+
+```python
+response = client.post(
+    "https://example.com/api",
+    '{"hello":"world"}',
+    headers={"content-type": "application/json"},
+    order=["content-type"],
+)
+```
+
+### `request(url, *, method="GET", headers=None, order=None, body=None, cookies=None)`
+
+```python
+response = client.request(
+    "https://example.com/resource",
+    method="DELETE",
+    headers={"authorization": "Bearer token"},
+    order=["authorization"],
+    cookies=["session=abc"],
+)
+```
+
+| Request argument | Type and default | Meaning |
+|---|---|---|
+| `url` | `str`, required | Absolute HTTP or HTTPS URL. |
+| `method` | `str`, `"GET"` | HTTP method. `get()` and `post()` set it automatically. |
+| `headers` | `Mapping[str, str] | None` | Headers layered over the profile. Existing profile names retain their browser position. |
+| `order` | `Sequence[str] | None` | Order for caller-only headers. Profile headers keep their measured browser positions; unnamed caller headers follow in sorted order. |
+| `body` | `str | bytes | None` | Request body. `bytes` is sent as base64 so it arrives intact; `None` omits the field. |
+| `cookies` | `Iterable[str] | None` | `name=value` pairs added to the daemon's jar before the request. |
+
+Do not manually set the `cookie` header unless replacing the jar's complete
+output is intentional. `cookies` preserves values received earlier in the
+session.
+
+## Response
+
+Requests return a frozen `tlsforge.Response` record. Its attributes cannot be
+reassigned; treat the `headers` mapping as response data rather than mutating
+it:
+
+```python
+Response(
+    status=200,
+    url="https://example.com/final",
+    body="<html>...</html>",
+    headers={
+        "content-type": ("text/html; charset=utf-8",),
+        "set-cookie": ("a=1", "b=2"),
+    },
+    cookies=("a=1", "b=2"),
+)
+```
+
+| Field or method | Meaning |
+|---|---|
+| `status` | HTTP status code. Non-2xx responses are still returned normally. |
+| `url` | Final URL after redirects. |
+| `body` | Decompressed response body as `str`. Bytes that are not valid UTF-8 appear as U+FFFD. |
+| `content` | The same body as `bytes`, exactly as it arrived. Use this for images, archives, or anything that is not text. |
+| `headers` | Header names mapped to tuples of field values. Repeated `set-cookie` fields remain separate. |
+| `cookies` | Tuple of `name=value` pairs held for the final URL. |
+| `ok` | `True` for status codes from 200 through 299. |
+| `json(**kwargs)` | Parse `body` with `json.loads` and return its result. |
+
+The SDK accepts the old daemon protocol where a header value was a scalar
+string, but always normalises the public result to `tuple[str, ...]`.
+
+`Response.json()` raises the standard `json.JSONDecodeError` when the body is
+not JSON.
+
+## Exceptions
+
+All package-specific errors inherit from `tlsforge.TLSForgeError`.
+
+```python
+try:
+    response = client.get(url)
+except tlsforge.BinaryNotFound:
+    # Fix the installation or binary path.
+    ...
+except tlsforge.Timeout:
+    # Retry or increase the deadline.
+    ...
+except tlsforge.RequestFailed:
+    # DNS, connection, proxy or TLS request failure.
+    ...
+except tlsforge.TransportError:
+    # Process startup, exit, pipe or protocol failure.
+    ...
+```
+
+| Exception | Meaning |
+|---|---|
+| `TLSForgeError` | Base class for package errors. |
+| `BinaryNotFound` | No usable transport binary was found, or an explicit path was missing. |
+| `RequestFailed` | The daemon ran, but the HTTP request failed. |
+| `Timeout` | The Python deadline passed. Also inherits from built-in `TimeoutError`. |
+| `TransportError` | The daemon could not start, exited, could not be written to, or returned invalid protocol data. |
+
+Invalid caller values such as an empty URL, non-positive timeout or non-callable
+`on_stderr` raise standard `ValueError` or `TypeError`.
+
+The client remains usable after a request timeout or transport failure. The next
+request starts a fresh daemon while monotonically increasing request ids prevent
+a late response from being attached to the wrong call.
+
+## Identity, threads and close
+
+One `Client` is one browser identity:
+
+- one transport process;
+- one TLS and HTTP profile;
+- one cookie jar;
+- one proxy and exit IP.
+
+The client is thread-safe but serialises requests with a lock. Calls from
+multiple threads queue instead of overlapping. Use a pool of clients for
+parallel scraping, normally one per proxy:
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+from contextlib import ExitStack
+from itertools import cycle
+
+import tlsforge
+
+with ExitStack() as stack:
+    clients = [
+        stack.enter_context(tlsforge.Client(proxy=proxy))
+        for proxy in proxies
+    ]
+    turns = cycle(clients)
+    with ThreadPoolExecutor(max_workers=len(clients)) as pool:
+        pages = list(pool.map(lambda url: next(turns).get(url), urls))
+```
+
+`close()` is idempotent and final. It waits for the current request, stops and
+reaps the daemon, and prevents later requests from silently creating a new
+identity. A finalizer also cleans up a forgotten client, but deterministic
+`with` or `close()` usage is preferred.
+
+## Warmed cookie sessions
+
+`cookie_file` accepts the same formats as the CLI:
+
+- a project JSON file containing multiple named sets;
+- a bare JSON array of sets;
+- a flat browser-extension cookie export;
+- Netscape cookies.txt.
+
+```python
+with tlsforge.Client(
+    cookie_file="cookies.json",
+    cookie_set="warm-eu",
+) as client:
+    response = client.get("https://example.com/")
+```
+
+Expired cookies are removed by the daemon before a set is seeded. Cookie
+domain, path, secure, HttpOnly and expiry attributes are retained by the Go
+cookie jar.
+
+The Python SDK currently loads but does not write session files. Use the CLI
+`fetch --save-cookies` or `batch --save-cookies` when the final jar must be
+persisted.
+
+## Binary resolution
+
+The exported `tlsforge.resolve_binary(explicit=None)` function and `Client` use
+this order:
+
+1. constructor `binary` option;
+2. `TLSFORGE_BIN` environment variable;
+3. binary bundled in the platform wheel;
+4. `bin/tls-forge` from a surrounding source checkout;
+5. `tls-forge` on `PATH`.
+
+```python
+import tlsforge
+
+print(tlsforge.resolve_binary())
+print(tlsforge.bundled_binary())
+print(tlsforge.exe_name())
+```
+
+An explicit path that does not exist raises `BinaryNotFound` rather than
+silently selecting a different version.
+
+## Development and typing
+
+The package includes `py.typed`, so its annotations are available to static
+type checkers.
+
+From the repository root:
+
+```bash
+make python-test
+```
+
+The suite uses a stand-in daemon and requires 100% statement and branch
+coverage. It does not need a Go build or network access.
+
+The project is licensed under Apache-2.0; see
+[`LICENSE`](https://github.com/Sec-CH-Lemon/tls-forge/blob/main/LICENSE),
+[`NOTICE`](https://github.com/Sec-CH-Lemon/tls-forge/blob/main/NOTICE) and
+[`THIRD-PARTY-NOTICES.txt`](https://github.com/Sec-CH-Lemon/tls-forge/blob/main/THIRD-PARTY-NOTICES.txt).