forgemesh-pay-per-crawl 0.1.2__tar.gz

forgemesh_pay_per_crawl-0.1.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ForgeMesh Labs
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

forgemesh_pay_per_crawl-0.1.2/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.4
+Name: forgemesh-pay-per-crawl
+Version: 0.1.2
+Summary: Charge AI crawlers and bots per page with x402 USDC micropayments on Base. ASGI, WSGI and Django middleware; you keep 100%.
+Author-email: ForgeMesh Labs <hello@forgemesh.io>
+License-Expression: MIT
+Project-URL: Homepage, https://forgemesh.io
+Project-URL: Repository, https://github.com/forgemeshlabs/pay-per-crawl-py
+Project-URL: Issues, https://github.com/forgemeshlabs/pay-per-crawl-py/issues
+Project-URL: PHP-WordPress-edition, https://github.com/forgemeshlabs/forgemesh-pay-per-crawl
+Keywords: x402,mpp,pay-per-crawl,402,ai-crawlers,usdc,base,fastapi,flask,django,asgi,wsgi
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware
+Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pycryptodome>=3.20
+Provides-Extra: httpx
+Requires-Dist: httpx>=0.27; extra == "httpx"
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == "test"
+Requires-Dist: httpx>=0.27; extra == "test"
+Requires-Dist: fastapi>=0.110; extra == "test"
+Requires-Dist: uvicorn>=0.29; extra == "test"
+Requires-Dist: flask>=3; extra == "test"
+Requires-Dist: django>=4.2; extra == "test"
+Dynamic: license-file
+
+# ForgeMesh Pay-per-Crawl for Python
+
+Charge AI crawlers and bots per page with [x402](https://x402.org) USDC micropayments on Base, paid straight to your wallet. Humans and verified search engines read free. Middleware for **FastAPI / Starlette (ASGI)**, **Flask and any WSGI app**, and **Django**, on one framework-free core.
+
+This is the Python edition of the [WordPress plugin](https://github.com/forgemeshlabs/forgemesh-pay-per-crawl); behaviour is identical.
+
+- **Humans see nothing different.** Browsers get the page as always.
+- **Verified search engines stay free.** Googlebot, Bingbot, DuckDuckBot, Applebot and others are checked by reverse DNS, so SEO is untouched. A crawler that only *claims* to be Googlebot is treated as a script.
+- **AI crawlers, SEO crawlers and scripts get `402 Payment Required`** with an x402 challenge (v2 `PAYMENT-REQUIRED` header + v1 JSON body) and an MPP `WWW-Authenticate: Payment` challenge. They pay USDC on Base and retry; the page is served only after the payment settles on-chain.
+- **You keep 100%.** The payer's USDC goes directly to your wallet. No custody, no account, no API key, no revenue share.
+- **Discovery built in.** Serves `/.well-known/x402.json`, `/llms.txt` and `/agents.txt`.
+
+## Install
+
+```
+pip install forgemesh-pay-per-crawl
+```
+
+Python 3.10+. One dependency (`pycryptodome`, for keccak256). `httpx` is used for facilitator calls if installed, otherwise the standard library. Start with `mode="monitor"` (records what bots would have paid, never blocks), then switch to `mode="gate"`.
+
+## FastAPI / Starlette
+
+```python
+from forgemesh_pay_per_crawl import Config
+from forgemesh_pay_per_crawl.asgi import PayPerCrawlMiddleware
+
+app.add_middleware(PayPerCrawlMiddleware,
+    config=Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
+```
+
+## Flask (and any WSGI app)
+
+```python
+from forgemesh_pay_per_crawl import Config
+from forgemesh_pay_per_crawl.wsgi import PayPerCrawlWSGI
+
+app.wsgi_app = PayPerCrawlWSGI(app.wsgi_app,
+    Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
+```
+
+## Django
+
+```python
+# settings.py
+MIDDLEWARE = ["forgemesh_pay_per_crawl.django.PayPerCrawlMiddleware", *MIDDLEWARE]  # near the top
+PAY_PER_CRAWL = {"pay_to": "0xYourWalletOnBase", "mode": "gate", "price_usd": 0.01, "base_url": "https://example.com"}
+```
+
+## Settings
+
+All are `Config(...)` keyword arguments (or keys of `PAY_PER_CRAWL`).
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `pay_to` | required in gate mode | Your `0x` wallet on Base. |
+| `mode` | `"monitor"` | `"monitor"` records only; `"gate"` answers 402. |
+| `price_usd` | `0.01` | Price per page fetch, paid in USDC. |
+| `facilitator_url` | PayAI (`https://facilitator.payai.network`, no key) | Any x402 facilitator that supports Base. `facilitator_auth` sets an `Authorization` header if yours needs one. |
+| `class_policy` | search/social/monitor free; seo/ai/automation/bot paid | Per class: `"free"`, `"paid"` or `"block"`. |
+| `bot_overrides` | `{}` | Per bot name, e.g. `{"GPTBot": "free", "Bytespider": "block"}`. |
+| `allow_paths` | `[]` | Path prefixes that are never gated. `/robots.txt`, `/sitemap*`, `/llms.txt`, `/agents.txt`, `/.well-known/`, `/favicon.ico` always are. |
+| `base_url` | from the request | Your public origin. Set it behind a proxy or load balancer. |
+| `trust_proxy_header` | `""` | Header with the real client IP (`"CF-Connecting-IP"`, `"X-Forwarded-For"`) so search-engine verification sees the right address. |
+| `mpp_secret` | random per process | HMAC key for MPP challenge ids. Set a stable value if you run more than one worker. |
+| `gated_methods` | `("GET", "HEAD")` | HEAD is gated like GET (402 with no body). |
+| `site_name`, `site_description`, `pages` | | Used in the discovery files; `pages` is a list (or callable returning one) of `(title, url)` for `llms.txt`. |
+| `on_event` | `None` | Callback receiving a dict for every classified request (status, bot, class, rail, tx hash, payer). |
+
+Bot classes: `human`, `search`, `social`, `monitor`, `seo`, `ai`, `automation`, `bot`. The catalog covers 85 crawlers and tools; see `core/catalog.py`.
+
+Multiple worker processes: pass `lock=FileNonceLock("/var/tmp/fmppc")` to `Gate(...)` (and use `PayPerCrawlMiddleware(app, gate=gate)`) so a payment authorization cannot be replayed on another worker, and set a stable `mpp_secret`. The default lock is in-process memory.
+
+### Behind Cloudflare
+
+If your site is proxied by Cloudflare, set **AI Crawl Control** (and any Bot Fight Mode / AI Labyrinth / "Block AI bots" rule) to **Allow** for the crawlers you want to charge. Cloudflare answers those requests at the edge before they ever reach your app, so the gate would never see them. Set `trust_proxy_header="CF-Connecting-IP"` and `base_url` to your public origin.
+
+## How a payment works
+
+1. A bot requests a page and gets `402` with the price, asset (USDC on Base) and your wallet, in the `PAYMENT-REQUIRED` header (x402 v2), the JSON body (x402 v1, with `invalidReason` when a payment was rejected) and `WWW-Authenticate: Payment` (MPP).
+2. The bot signs a USDC transfer authorization (EIP-3009) and retries with it in `PAYMENT-SIGNATURE` (v2), `X-PAYMENT` (v1) or `Authorization: Payment` (MPP).
+3. The middleware checks recipient, amount and expiry locally, locks the authorization nonce against replay, then posts it to the **facilitator** (`/verify`, then `/settle`). The facilitator verifies and submits it on-chain; it never holds funds. v1 payments are forwarded in v1 shape.
+4. The page is served only when a transaction hash comes back, with the receipt in `PAYMENT-RESPONSE`, `X-PAYMENT-RESPONSE` and (MPP) `Payment-Receipt`.
+
+The package never contacts ForgeMesh. Its only outbound requests are to the facilitator you chose, and only when a payment is presented. (The optional Pro reporter below is the one exception, and is off without a key.)
+
+## Money flow
+
+The payer's USDC moves wallet to wallet on Base, from the payer to the `pay_to` address you configured. There is no ForgeMesh account, no custody, no settlement service and no revenue share: the site owner keeps 100%. The facilitator (PayAI by default) charges the seller no fee at the time of writing; check your facilitator's terms.
+
+## Using the pieces directly
+
+```python
+from forgemesh_pay_per_crawl import Classifier, Config, Gate, Request, X402, build_challenge
+
+Classifier().classify("Mozilla/5.0 (compatible; GPTBot/1.2)").cls        # "ai"
+gate = Gate(Config(pay_to="0x...", mode="gate"))
+decision = gate.handle(Request(method="GET", path="/x", headers={"user-agent": "curl/8"}, host="example.com"))
+decision.response   # a Response (status, headers, body) to send, or None to serve the page
+decision.headers    # headers to add to the page you serve (payment receipt, Link)
+```
+
+`Gate.handle` is blocking (it can call the facilitator); in async code run it in a thread, as the ASGI middleware does. The facilitator transport is injectable (`Gate(config, transport=fn)` with `fn(url, body, headers) -> dict`).
+
+## Pro (optional)
+
+With a ForgeMesh Pay-per-Crawl Pro license (`pro_key="fmppc_..."`) the package reports daily bot rollups and settlements to the hosted dashboard at proxy.forgemesh.io and pulls the crawler-intel feed into the classifier. Nothing is sent without a key. The rollups are held in memory per process, so with several workers each reports its own counts.
+
+## Verified
+
+- Unit and integration tests: `pytest` (keccak vectors incl. block-boundary cases, classifier with rDNS, 402 shapes, MPP challenge/credential round trip, facilitator flow, replay, HEAD, file lock, Pro sync, all three adapters, and an integration matrix against live uvicorn/FastAPI and Flask servers with a mock facilitator).
+- Real settlements on Base mainnet through PayAI, October 2026, from the FastAPI example gated at $0.01: x402 v1 ([tx](https://basescan.org/tx/0x49f0f05b3509545e5134aa6cf091ff7eab4e4f1e1b9aef9bf28280c231e2a945)) and x402 v2 ([tx](https://basescan.org/tx/0x15d95ad90c9530e4816016f65681ddd4a79730373634c991c57afa2b88a8b44b)).
+
+## Development
+
+```
+python -m venv .venv && .venv/bin/pip install -e ".[test]"
+.venv/bin/pytest
+python -m build
+```
+
+MIT. Made by [ForgeMesh Labs](https://forgemesh.io) (hello@forgemesh.io).

forgemesh_pay_per_crawl-0.1.2/README.md

@@ -0,0 +1,121 @@
+# ForgeMesh Pay-per-Crawl for Python
+
+Charge AI crawlers and bots per page with [x402](https://x402.org) USDC micropayments on Base, paid straight to your wallet. Humans and verified search engines read free. Middleware for **FastAPI / Starlette (ASGI)**, **Flask and any WSGI app**, and **Django**, on one framework-free core.
+
+This is the Python edition of the [WordPress plugin](https://github.com/forgemeshlabs/forgemesh-pay-per-crawl); behaviour is identical.
+
+- **Humans see nothing different.** Browsers get the page as always.
+- **Verified search engines stay free.** Googlebot, Bingbot, DuckDuckBot, Applebot and others are checked by reverse DNS, so SEO is untouched. A crawler that only *claims* to be Googlebot is treated as a script.
+- **AI crawlers, SEO crawlers and scripts get `402 Payment Required`** with an x402 challenge (v2 `PAYMENT-REQUIRED` header + v1 JSON body) and an MPP `WWW-Authenticate: Payment` challenge. They pay USDC on Base and retry; the page is served only after the payment settles on-chain.
+- **You keep 100%.** The payer's USDC goes directly to your wallet. No custody, no account, no API key, no revenue share.
+- **Discovery built in.** Serves `/.well-known/x402.json`, `/llms.txt` and `/agents.txt`.
+
+## Install
+
+```
+pip install forgemesh-pay-per-crawl
+```
+
+Python 3.10+. One dependency (`pycryptodome`, for keccak256). `httpx` is used for facilitator calls if installed, otherwise the standard library. Start with `mode="monitor"` (records what bots would have paid, never blocks), then switch to `mode="gate"`.
+
+## FastAPI / Starlette
+
+```python
+from forgemesh_pay_per_crawl import Config
+from forgemesh_pay_per_crawl.asgi import PayPerCrawlMiddleware
+
+app.add_middleware(PayPerCrawlMiddleware,
+    config=Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
+```
+
+## Flask (and any WSGI app)
+
+```python
+from forgemesh_pay_per_crawl import Config
+from forgemesh_pay_per_crawl.wsgi import PayPerCrawlWSGI
+
+app.wsgi_app = PayPerCrawlWSGI(app.wsgi_app,
+    Config(pay_to="0xYourWalletOnBase", mode="gate", price_usd=0.01, base_url="https://example.com"))
+```
+
+## Django
+
+```python
+# settings.py
+MIDDLEWARE = ["forgemesh_pay_per_crawl.django.PayPerCrawlMiddleware", *MIDDLEWARE]  # near the top
+PAY_PER_CRAWL = {"pay_to": "0xYourWalletOnBase", "mode": "gate", "price_usd": 0.01, "base_url": "https://example.com"}
+```
+
+## Settings
+
+All are `Config(...)` keyword arguments (or keys of `PAY_PER_CRAWL`).
+
+| Setting | Default | Meaning |
+|---|---|---|
+| `pay_to` | required in gate mode | Your `0x` wallet on Base. |
+| `mode` | `"monitor"` | `"monitor"` records only; `"gate"` answers 402. |
+| `price_usd` | `0.01` | Price per page fetch, paid in USDC. |
+| `facilitator_url` | PayAI (`https://facilitator.payai.network`, no key) | Any x402 facilitator that supports Base. `facilitator_auth` sets an `Authorization` header if yours needs one. |
+| `class_policy` | search/social/monitor free; seo/ai/automation/bot paid | Per class: `"free"`, `"paid"` or `"block"`. |
+| `bot_overrides` | `{}` | Per bot name, e.g. `{"GPTBot": "free", "Bytespider": "block"}`. |
+| `allow_paths` | `[]` | Path prefixes that are never gated. `/robots.txt`, `/sitemap*`, `/llms.txt`, `/agents.txt`, `/.well-known/`, `/favicon.ico` always are. |
+| `base_url` | from the request | Your public origin. Set it behind a proxy or load balancer. |
+| `trust_proxy_header` | `""` | Header with the real client IP (`"CF-Connecting-IP"`, `"X-Forwarded-For"`) so search-engine verification sees the right address. |
+| `mpp_secret` | random per process | HMAC key for MPP challenge ids. Set a stable value if you run more than one worker. |
+| `gated_methods` | `("GET", "HEAD")` | HEAD is gated like GET (402 with no body). |
+| `site_name`, `site_description`, `pages` | | Used in the discovery files; `pages` is a list (or callable returning one) of `(title, url)` for `llms.txt`. |
+| `on_event` | `None` | Callback receiving a dict for every classified request (status, bot, class, rail, tx hash, payer). |
+
+Bot classes: `human`, `search`, `social`, `monitor`, `seo`, `ai`, `automation`, `bot`. The catalog covers 85 crawlers and tools; see `core/catalog.py`.
+
+Multiple worker processes: pass `lock=FileNonceLock("/var/tmp/fmppc")` to `Gate(...)` (and use `PayPerCrawlMiddleware(app, gate=gate)`) so a payment authorization cannot be replayed on another worker, and set a stable `mpp_secret`. The default lock is in-process memory.
+
+### Behind Cloudflare
+
+If your site is proxied by Cloudflare, set **AI Crawl Control** (and any Bot Fight Mode / AI Labyrinth / "Block AI bots" rule) to **Allow** for the crawlers you want to charge. Cloudflare answers those requests at the edge before they ever reach your app, so the gate would never see them. Set `trust_proxy_header="CF-Connecting-IP"` and `base_url` to your public origin.
+
+## How a payment works
+
+1. A bot requests a page and gets `402` with the price, asset (USDC on Base) and your wallet, in the `PAYMENT-REQUIRED` header (x402 v2), the JSON body (x402 v1, with `invalidReason` when a payment was rejected) and `WWW-Authenticate: Payment` (MPP).
+2. The bot signs a USDC transfer authorization (EIP-3009) and retries with it in `PAYMENT-SIGNATURE` (v2), `X-PAYMENT` (v1) or `Authorization: Payment` (MPP).
+3. The middleware checks recipient, amount and expiry locally, locks the authorization nonce against replay, then posts it to the **facilitator** (`/verify`, then `/settle`). The facilitator verifies and submits it on-chain; it never holds funds. v1 payments are forwarded in v1 shape.
+4. The page is served only when a transaction hash comes back, with the receipt in `PAYMENT-RESPONSE`, `X-PAYMENT-RESPONSE` and (MPP) `Payment-Receipt`.
+
+The package never contacts ForgeMesh. Its only outbound requests are to the facilitator you chose, and only when a payment is presented. (The optional Pro reporter below is the one exception, and is off without a key.)
+
+## Money flow
+
+The payer's USDC moves wallet to wallet on Base, from the payer to the `pay_to` address you configured. There is no ForgeMesh account, no custody, no settlement service and no revenue share: the site owner keeps 100%. The facilitator (PayAI by default) charges the seller no fee at the time of writing; check your facilitator's terms.
+
+## Using the pieces directly
+
+```python
+from forgemesh_pay_per_crawl import Classifier, Config, Gate, Request, X402, build_challenge
+
+Classifier().classify("Mozilla/5.0 (compatible; GPTBot/1.2)").cls        # "ai"
+gate = Gate(Config(pay_to="0x...", mode="gate"))
+decision = gate.handle(Request(method="GET", path="/x", headers={"user-agent": "curl/8"}, host="example.com"))
+decision.response   # a Response (status, headers, body) to send, or None to serve the page
+decision.headers    # headers to add to the page you serve (payment receipt, Link)
+```
+
+`Gate.handle` is blocking (it can call the facilitator); in async code run it in a thread, as the ASGI middleware does. The facilitator transport is injectable (`Gate(config, transport=fn)` with `fn(url, body, headers) -> dict`).
+
+## Pro (optional)
+
+With a ForgeMesh Pay-per-Crawl Pro license (`pro_key="fmppc_..."`) the package reports daily bot rollups and settlements to the hosted dashboard at proxy.forgemesh.io and pulls the crawler-intel feed into the classifier. Nothing is sent without a key. The rollups are held in memory per process, so with several workers each reports its own counts.
+
+## Verified
+
+- Unit and integration tests: `pytest` (keccak vectors incl. block-boundary cases, classifier with rDNS, 402 shapes, MPP challenge/credential round trip, facilitator flow, replay, HEAD, file lock, Pro sync, all three adapters, and an integration matrix against live uvicorn/FastAPI and Flask servers with a mock facilitator).
+- Real settlements on Base mainnet through PayAI, October 2026, from the FastAPI example gated at $0.01: x402 v1 ([tx](https://basescan.org/tx/0x49f0f05b3509545e5134aa6cf091ff7eab4e4f1e1b9aef9bf28280c231e2a945)) and x402 v2 ([tx](https://basescan.org/tx/0x15d95ad90c9530e4816016f65681ddd4a79730373634c991c57afa2b88a8b44b)).
+
+## Development
+
+```
+python -m venv .venv && .venv/bin/pip install -e ".[test]"
+.venv/bin/pytest
+python -m build
+```
+
+MIT. Made by [ForgeMesh Labs](https://forgemesh.io) (hello@forgemesh.io).

forgemesh_pay_per_crawl-0.1.2/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "forgemesh-pay-per-crawl"
+version = "0.1.2"
+description = "Charge AI crawlers and bots per page with x402 USDC micropayments on Base. ASGI, WSGI and Django middleware; you keep 100%."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "ForgeMesh Labs", email = "hello@forgemesh.io" }]
+keywords = ["x402", "mpp", "pay-per-crawl", "402", "ai-crawlers", "usdc", "base", "fastapi", "flask", "django", "asgi", "wsgi"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Internet :: WWW/HTTP :: WSGI :: Middleware",
+  "Topic :: Internet :: WWW/HTTP :: Dynamic Content",
+  "Typing :: Typed",
+]
+dependencies = ["pycryptodome>=3.20"]
+
+[project.optional-dependencies]
+httpx = ["httpx>=0.27"]
+test = ["pytest>=8", "httpx>=0.27", "fastapi>=0.110", "uvicorn>=0.29", "flask>=3", "django>=4.2"]
+
+[project.urls]
+Homepage = "https://forgemesh.io"
+Repository = "https://github.com/forgemeshlabs/pay-per-crawl-py"
+Issues = "https://github.com/forgemeshlabs/pay-per-crawl-py/issues"
+PHP-WordPress-edition = "https://github.com/forgemeshlabs/forgemesh-pay-per-crawl"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+forgemesh_pay_per_crawl = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src", "."]

forgemesh_pay_per_crawl-0.1.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

forgemesh_pay_per_crawl-0.1.2/src/forgemesh_pay_per_crawl/__init__.py

@@ -0,0 +1,3 @@
+"""ForgeMesh Pay-per-Crawl for Python: charge AI crawlers per page with x402 USDC on Base."""
+from .core import *  # noqa: F401,F403
+from .core.x402 import VERSION as __version__  # noqa: F401

forgemesh_pay_per_crawl-0.1.2/src/forgemesh_pay_per_crawl/asgi.py

@@ -0,0 +1,64 @@
+"""ASGI middleware for FastAPI, Starlette and any ASGI app.
+
+    app.add_middleware(PayPerCrawlMiddleware, config=Config(pay_to="0x...", mode="gate"))
+"""
+from __future__ import annotations
+
+import asyncio
+from typing import Optional
+
+from .core import Config, Gate, Request, Response
+
+
+def _request(scope) -> Request:
+    headers: dict[str, str] = {}
+    for k, v in scope.get("headers", []):
+        name = k.decode("latin-1").lower()
+        val = v.decode("latin-1")
+        headers[name] = f"{headers[name]}, {val}" if name in headers else val
+    client = scope.get("client") or ("", 0)
+    host = headers.get("host") or ""
+    if not host:
+        server = scope.get("server") or ("", None)
+        host = server[0] + (f":{server[1]}" if server[1] else "")
+    return Request(
+        method=scope.get("method", "GET"),
+        path=(scope.get("raw_path") or scope.get("path", "/").encode()).decode("latin-1").split("?")[0] or "/",
+        query=scope.get("query_string", b"").decode("latin-1"),
+        scheme=scope.get("scheme", "http"),
+        host=host,
+        headers=headers,
+        ip=client[0] or "",
+    )
+
+
+class PayPerCrawlMiddleware:
+    def __init__(self, app, config: Optional[Config] = None, gate: Optional[Gate] = None, **settings):
+        self.app = app
+        self.gate = gate or Gate(config or Config(**settings))
+
+    async def __call__(self, scope, receive, send):
+        if scope["type"] != "http":
+            await self.app(scope, receive, send)
+            return
+        decision = await asyncio.to_thread(self.gate.handle, _request(scope))
+        if decision.response is not None:
+            r: Response = decision.response
+            await send({"type": "http.response.start", "status": r.status,
+                        "headers": [(k.lower().encode("latin-1"), v.encode("latin-1")) for k, v in r.headers]})
+            await send({"type": "http.response.body", "body": r.body})
+            return
+        if not decision.headers:
+            await self.app(scope, receive, send)
+            return
+        extra = [(k.lower().encode("latin-1"), v.encode("latin-1")) for k, v in decision.headers]
+        names = {k for k, _ in extra}
+
+        async def send_with_headers(message):
+            if message["type"] == "http.response.start":
+                # Ours win over same-named headers from the app (receipts, no-store).
+                kept = [(k, v) for k, v in message.get("headers", []) if k.lower() not in names or k.lower() == b"link"]
+                message = {**message, "headers": kept + extra}
+            await send(message)
+
+        await self.app(scope, receive, send_with_headers)

forgemesh_pay_per_crawl-0.1.2/src/forgemesh_pay_per_crawl/core/__init__.py

@@ -0,0 +1,42 @@
+"""Framework-free core of ForgeMesh Pay-per-Crawl."""
+from .classifier import CLASSES, Classification, Classifier, SystemResolver
+from .config import DEFAULT_CLASS_POLICY, Config
+from .discovery import PATHS as DISCOVERY_PATHS
+from .discovery import Discovery
+from .gate import Gate
+from .http import Decision, Request, Response
+from .keccak import keccak256, keccak256_hex
+from .locks import FileNonceLock, MemoryNonceLock
+from .pro import ProReporter, Stats
+from .x402 import TransportError, X402, default_transport, usd_to_units
+
+__all__ = [
+    "CLASSES", "Classification", "Classifier", "SystemResolver", "Config", "DEFAULT_CLASS_POLICY", "Discovery",
+    "DISCOVERY_PATHS", "Gate", "Decision", "Request", "Response", "keccak256", "keccak256_hex", "FileNonceLock",
+    "MemoryNonceLock", "ProReporter", "Stats", "TransportError", "X402", "default_transport", "usd_to_units",
+    "classify", "build_challenge", "extract_payment", "precheck", "verify_and_settle",
+]
+
+
+# Function-style entry points over the classes, for people who want the pieces.
+def classify(ua, headers=None, ip="", classifier=None):
+    return (classifier or Classifier()).classify(ua, headers, ip)
+
+
+def build_challenge(x402: X402, resource_url, description=""):
+    """(PAYMENT-REQUIRED value, v1 body dict, WWW-Authenticate value) for one resource."""
+    from .x402 import b64_encode_json
+    return (b64_encode_json(x402.envelope_v2(resource_url, description)),
+            x402.body_v1(resource_url, description), x402.mpp_challenge(description))
+
+
+def extract_payment(x402, headers, resource_url):
+    return x402.extract_payment(headers, resource_url)
+
+
+def precheck(x402, payment, resource_url=""):
+    return x402.precheck(payment, resource_url)
+
+
+def verify_and_settle(x402, payment, resource_url):
+    return x402.verify_and_settle(payment, resource_url)

forgemesh_pay_per_crawl-0.1.2/src/forgemesh_pay_per_crawl/core/catalog.py

@@ -0,0 +1,97 @@
+# Generated from the PHP reference catalog; edit by hand from here on.
+from __future__ import annotations
+
+# name, class, match needles, rDNS suffixes that prove the operator (search engines).
+CATALOG: list[dict] = [
+    # Search engines (free by default; verified by rDNS).
+    {"name": 'Googlebot', "class": 'search', "match": ['googlebot'], "verify": ['.googlebot.com', '.google.com']},
+    {"name": 'Google-InspectionTool', "class": 'search', "match": ['google-inspectiontool'], "verify": ['.googlebot.com', '.google.com']},
+    {"name": 'GoogleOther', "class": 'ai', "match": ['googleother'], "verify": ['.googlebot.com', '.google.com']},
+    {"name": 'Bingbot', "class": 'search', "match": ['bingbot'], "verify": ['.search.msn.com']},
+    {"name": 'DuckDuckBot', "class": 'search', "match": ['duckduckbot'], "verify": []},
+    {"name": 'Applebot', "class": 'search', "match": ['applebot'], "verify": ['.applebot.apple.com']},
+    {"name": 'YandexBot', "class": 'search', "match": ['yandexbot', 'yandex.com/bots'], "verify": ['.yandex.ru', '.yandex.net', '.yandex.com']},
+    {"name": 'Baiduspider', "class": 'search', "match": ['baiduspider'], "verify": ['.baidu.com', '.baidu.jp']},
+    {"name": 'Yahoo Slurp', "class": 'search', "match": ['yahoo! slurp'], "verify": ['.crawl.yahoo.net']},
+    {"name": 'SeznamBot', "class": 'search', "match": ['seznambot'], "verify": ['.seznam.cz']},
+    {"name": 'Qwantbot', "class": 'search', "match": ['qwantbot', 'qwantify'], "verify": []},
+    # AI crawlers and assistants (paid by default).
+    {"name": 'GPTBot', "class": 'ai', "match": ['gptbot']},
+    {"name": 'ChatGPT-User', "class": 'ai', "match": ['chatgpt-user']},
+    {"name": 'OAI-SearchBot', "class": 'ai', "match": ['oai-searchbot']},
+    {"name": 'ClaudeBot', "class": 'ai', "match": ['claudebot', 'anthropic-ai']},
+    {"name": 'Claude-User', "class": 'ai', "match": ['claude-user']},
+    {"name": 'Claude-SearchBot', "class": 'ai', "match": ['claude-searchbot']},
+    {"name": 'PerplexityBot', "class": 'ai', "match": ['perplexitybot']},
+    {"name": 'Perplexity-User', "class": 'ai', "match": ['perplexity-user']},
+    {"name": 'Google-Extended', "class": 'ai', "match": ['google-extended']},
+    {"name": 'Gemini-Deep-Research', "class": 'ai', "match": ['gemini-deep-research']},
+    {"name": 'Amazonbot', "class": 'ai', "match": ['amazonbot']},
+    {"name": 'Applebot-Extended', "class": 'ai', "match": ['applebot-extended']},
+    {"name": 'Meta-ExternalAgent', "class": 'ai', "match": ['meta-externalagent', 'meta-externalfetcher', 'facebookbot']},
+    {"name": 'Bytespider', "class": 'ai', "match": ['bytespider', 'tiktokspider']},
+    {"name": 'CCBot', "class": 'ai', "match": ['ccbot']},
+    {"name": 'cohere-ai', "class": 'ai', "match": ['cohere-ai', 'cohere-training-data-crawler']},
+    {"name": 'MistralAI-User', "class": 'ai', "match": ['mistralai-user']},
+    {"name": 'DuckAssistBot', "class": 'ai', "match": ['duckassistbot']},
+    {"name": 'YouBot', "class": 'ai', "match": ['youbot']},
+    {"name": 'Diffbot', "class": 'ai', "match": ['diffbot']},
+    {"name": 'ImagesiftBot', "class": 'ai', "match": ['imagesiftbot']},
+    {"name": 'Omgili', "class": 'ai', "match": ['omgili', 'webzio']},
+    {"name": 'AI2Bot', "class": 'ai', "match": ['ai2bot']},
+    {"name": 'PetalBot', "class": 'ai', "match": ['petalbot']},
+    {"name": 'Timpibot', "class": 'ai', "match": ['timpibot']},
+    {"name": 'iAskBot', "class": 'ai', "match": ['iaskspider', 'iaskbot']},
+    {"name": 'Kagibot', "class": 'ai', "match": ['kagibot']},
+    {"name": 'DeepSeekBot', "class": 'ai', "match": ['deepseekbot']},
+    {"name": 'Grok', "class": 'ai', "match": ['xai-grok', 'grokbot']},
+    {"name": 'Brave Search', "class": 'ai', "match": ['bravebot']},
+    {"name": 'Firecrawl', "class": 'ai', "match": ['firecrawl']},
+    {"name": 'Jina Reader', "class": 'ai', "match": ['jina', 'readerbot']},
+    {"name": 'Exa', "class": 'ai', "match": ['exabot', 'exa.ai']},
+    {"name": 'Tavily', "class": 'ai', "match": ['tavily']},
+    {"name": 'Browserbase', "class": 'ai', "match": ['browserbase']},
+    # SEO crawlers (paid by default: commercial, high volume).
+    {"name": 'AhrefsBot', "class": 'seo', "match": ['ahrefsbot']},
+    {"name": 'SemrushBot', "class": 'seo', "match": ['semrushbot']},
+    {"name": 'MJ12bot', "class": 'seo', "match": ['mj12bot']},
+    {"name": 'DotBot', "class": 'seo', "match": ['dotbot']},
+    {"name": 'DataForSeoBot', "class": 'seo', "match": ['dataforseobot']},
+    {"name": 'BLEXBot', "class": 'seo', "match": ['blexbot']},
+    {"name": 'Screaming Frog', "class": 'seo', "match": ['screaming frog']},
+    {"name": 'SiteAuditBot', "class": 'seo', "match": ['siteauditbot']},
+    # Social link previews (free by default: humans see these).
+    {"name": 'Facebook preview', "class": 'social', "match": ['facebookexternalhit']},
+    {"name": 'Twitterbot', "class": 'social', "match": ['twitterbot']},
+    {"name": 'LinkedInBot', "class": 'social', "match": ['linkedinbot']},
+    {"name": 'Slackbot', "class": 'social', "match": ['slackbot', 'slack-imgproxy']},
+    {"name": 'Discordbot', "class": 'social', "match": ['discordbot']},
+    {"name": 'TelegramBot', "class": 'social', "match": ['telegrambot']},
+    {"name": 'WhatsApp', "class": 'social', "match": ['whatsapp']},
+    {"name": 'Pinterestbot', "class": 'social', "match": ['pinterestbot', 'pinterest/']},
+    {"name": 'Redditbot', "class": 'social', "match": ['redditbot']},
+    {"name": 'Mastodon', "class": 'social', "match": ['mastodon']},
+    {"name": 'Bluesky', "class": 'social', "match": ['bluesky cardyb']},
+    {"name": 'iMessage preview', "class": 'social', "match": ['facebot twitterbot']},
+    # Monitors and feed readers (free by default).
+    {"name": 'UptimeRobot', "class": 'monitor', "match": ['uptimerobot']},
+    {"name": 'Pingdom', "class": 'monitor', "match": ['pingdom']},
+    {"name": 'StatusCake', "class": 'monitor', "match": ['statuscake']},
+    {"name": 'Better Uptime', "class": 'monitor', "match": ['betteruptime', 'better uptime']},
+    {"name": 'Feed reader', "class": 'monitor', "match": ['feedly', 'feedbin', 'inoreader', 'newsblur', 'feedfetcher']},
+    {"name": 'Internet Archive', "class": 'monitor', "match": ['archive.org_bot', 'ia_archiver']},
+    {"name": 'WordPress', "class": 'monitor', "match": ['wordpress/']},
+    # Generic automation (paid by default).
+    {"name": 'curl', "class": 'automation', "match": ['curl/']},
+    {"name": 'wget', "class": 'automation', "match": ['wget/']},
+    {"name": 'python-requests', "class": 'automation', "match": ['python-requests', 'python-urllib', 'aiohttp', 'httpx/']},
+    {"name": 'Scrapy', "class": 'automation', "match": ['scrapy']},
+    {"name": 'Go-http-client', "class": 'automation', "match": ['go-http-client']},
+    {"name": 'Node fetch', "class": 'automation', "match": ['node-fetch', 'undici', 'axios/', 'got (']},
+    {"name": 'Java HTTP', "class": 'automation', "match": ['java/', 'okhttp', 'apache-httpclient']},
+    {"name": 'Headless browser', "class": 'automation', "match": ['headlesschrome', 'phantomjs', 'puppeteer', 'playwright', 'selenium']},
+    {"name": 'HTTPie', "class": 'automation', "match": ['httpie']},
+    {"name": 'PostmanRuntime', "class": 'automation', "match": ['postmanruntime']},
+    {"name": 'libwww', "class": 'automation', "match": ['libwww', 'lwp-trivial']},
+    {"name": 'x402 client', "class": 'automation', "match": ['x402', 'mpp/']},
+]