x402-cleanweb-agent 1.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- x402_cleanweb_agent-1.2.0/PKG-INFO +268 -0
- x402_cleanweb_agent-1.2.0/README.md +238 -0
- x402_cleanweb_agent-1.2.0/agent_tools.py +253 -0
- x402_cleanweb_agent-1.2.0/autonomous_agent_client.py +228 -0
- x402_cleanweb_agent-1.2.0/mcp_server.py +331 -0
- x402_cleanweb_agent-1.2.0/polygon_agent.py +278 -0
- x402_cleanweb_agent-1.2.0/pyproject.toml +62 -0
- x402_cleanweb_agent-1.2.0/setup.cfg +4 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/PKG-INFO +268 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/SOURCES.txt +12 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/dependency_links.txt +1 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/entry_points.txt +3 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/requires.txt +7 -0
- x402_cleanweb_agent-1.2.0/x402_cleanweb_agent.egg-info/top_level.txt +4 -0
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: x402-cleanweb-agent
|
|
3
|
+
Version: 1.2.0
|
|
4
|
+
Summary: Web3 x402 Micropayment MCP & AI Agent Tool Suite for LLM Data Extraction on Polygon Mainnet
|
|
5
|
+
Author-email: x402 AI Agent Team <developer@x402.agent>
|
|
6
|
+
License: MIT
|
|
7
|
+
Project-URL: Homepage, https://x402-cleanweb-agent.onrender.com
|
|
8
|
+
Project-URL: Documentation, https://x402-cleanweb-agent.onrender.com/docs
|
|
9
|
+
Project-URL: Repository, https://github.com/nohosa001-pixel/x402-cleanweb-agent
|
|
10
|
+
Project-URL: Issues, https://github.com/nohosa001-pixel/x402-cleanweb-agent/issues
|
|
11
|
+
Keywords: ai-agent,x402,mcp,polygon,web3,langchain,crewai,smolagents,micropayments,web-scraper,llm-ready
|
|
12
|
+
Classifier: Development Status :: 5 - Production/Stable
|
|
13
|
+
Classifier: Intended Audience :: Developers
|
|
14
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
15
|
+
Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
|
|
16
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Description-Content-Type: text/markdown
|
|
23
|
+
Requires-Dist: mcp>=1.0.0
|
|
24
|
+
Requires-Dist: web3>=6.0.0
|
|
25
|
+
Requires-Dist: beautifulsoup4>=4.12.0
|
|
26
|
+
Requires-Dist: requests>=2.31.0
|
|
27
|
+
Requires-Dist: python-dotenv>=1.0.0
|
|
28
|
+
Requires-Dist: youtube-transcript-api>=0.6.2
|
|
29
|
+
Requires-Dist: pypdf>=4.0.0
|
|
30
|
+
|
|
31
|
+
# ⚡ x402-cleanweb-agent
|
|
32
|
+
|
|
33
|
+
> **Turn any messy webpage, YouTube video, or PDF paper into pure, LLM-ready clean Markdown on Polygon.**
|
|
34
|
+
> *Zero Sign-up. Zero Subscriptions. True Machine-to-Machine HTTP 402 Micropayments for Autonomous AI Agents.*
|
|
35
|
+
|
|
36
|
+
[](https://x402-cleanweb-agent.onrender.com)
|
|
37
|
+
[](https://x402-cleanweb-agent.onrender.com/docs)
|
|
38
|
+
[](https://www.python.org/)
|
|
39
|
+
[](https://polygonscan.com/token/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359)
|
|
40
|
+
[](https://opensource.org/licenses/MIT)
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## 💡 Why x402-cleanweb-agent?
|
|
45
|
+
|
|
46
|
+
Traditional web scraping and data extraction APIs force expensive **$49/month subscriptions** and complex API key management.
|
|
47
|
+
|
|
48
|
+
`x402-cleanweb-agent` solves this for autonomous AI agents, scrapers, and developers:
|
|
49
|
+
|
|
50
|
+
- ❌ **No Monthly Subscriptions**: Pay only for what you query ($0.005 ~ $0.05 per call in USDC).
|
|
51
|
+
- ❌ **No Sign-ups or API Keys**: Native **HTTP 402 Payment Required** machine-to-machine protocol.
|
|
52
|
+
- 🤖 **Zero-Human AI Agent Ready**: AI agents with a crypto wallet can autonomously buy data 24/7.
|
|
53
|
+
- ⚡ **Sub-Second Speed**: Strips ads, tracking scripts, and clutter, returning pure, structured Markdown.
|
|
54
|
+
- 🪙 **Ultra-Low Gas Fees**: Powered by **Polygon PoS** (< $0.005 network gas).
|
|
55
|
+
- 📊 **Token Savings Engine**: Calculates raw vs. cleaned token reduction (avg. 60~85% savings) and estimated LLM prompt cost savings ($).
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## 🚀 Live Demo & Service Endpoints
|
|
60
|
+
|
|
61
|
+
- 🌐 **Web3 DApp UI**: [https://x402-cleanweb-agent.onrender.com](https://x402-cleanweb-agent.onrender.com)
|
|
62
|
+
- 📚 **Swagger API Docs**: [https://x402-cleanweb-agent.onrender.com/docs](https://x402-cleanweb-agent.onrender.com/docs)
|
|
63
|
+
- 📂 **GitHub Repo**: [https://github.com/nohosa001-pixel/x402-cleanweb-agent](https://github.com/nohosa001-pixel/x402-cleanweb-agent)
|
|
64
|
+
|
|
65
|
+
| Service | Endpoint | Pricing | Output & Description |
|
|
66
|
+
| :--- | :--- | :--- | :--- |
|
|
67
|
+
| **🌐 Clean Web** | `GET /api/v1/clean-web` | **0.01 USDC** | Ad/Noise removal + AI-ready Markdown + **Token Savings Analytics** |
|
|
68
|
+
| **🎬 YouTube Transcript** | `GET /api/v1/clean-youtube` | **0.02 USDC** | Full video **transcripts with timestamps** formatted in Markdown |
|
|
69
|
+
| **📑 PDF Paper & Report** | `GET /api/v1/clean-pdf` | **0.05 USDC** | arXiv papers & earnings reports converted into **structured Markdown** |
|
|
70
|
+
| **📝 Pure Plain Text** | `GET /api/v1/clean-text` | **0.005 USDC** | Ultra-lightweight raw text extraction for fast vector indexing |
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## 🤖 Zero-Human Autonomous AI Agent Integration & Tools
|
|
75
|
+
|
|
76
|
+
AI agents with a Polygon wallet (Private Key) can autonomously handle payment and data extraction with **zero human intervention** and **built-in Budget Guard protection**:
|
|
77
|
+
|
|
78
|
+
### 1. Ready-to-Use Agent Toolkit (CrewAI / smolagents / LangChain)
|
|
79
|
+
|
|
80
|
+
```python
|
|
81
|
+
from agent_tools import get_x402_agent_tools, X402AgentToolkit
|
|
82
|
+
|
|
83
|
+
# 1. Initialize toolkit with spending limits
|
|
84
|
+
toolkit = X402AgentToolkit(
|
|
85
|
+
private_key="0xYOUR_AGENT_PRIVATE_KEY",
|
|
86
|
+
max_daily_budget_usdc=1.0 # Budget Guard protects against infinite loops
|
|
87
|
+
)
|
|
88
|
+
|
|
89
|
+
# 2. Get ready-to-use callable tools
|
|
90
|
+
tools = toolkit.get_tools_list()
|
|
91
|
+
|
|
92
|
+
# 3. Direct execution
|
|
93
|
+
web_data = toolkit.clean_web("https://example.com/article")
|
|
94
|
+
yt_data = toolkit.clean_youtube("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
|
|
95
|
+
pdf_data = toolkit.clean_pdf("https://arxiv.org/pdf/2301.00001.pdf")
|
|
96
|
+
|
|
97
|
+
# 4. Inspect spending report
|
|
98
|
+
print(toolkit.get_spending_report())
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### 2. Machine-Readable Agent Discovery Protocol
|
|
102
|
+
|
|
103
|
+
Autonomous agents and crawlers can discover services, pricing, and on-chain payment contracts dynamically:
|
|
104
|
+
|
|
105
|
+
- **Agent Manifest**: `GET https://x402-cleanweb-agent.onrender.com/.well-known/agent.json`
|
|
106
|
+
- **Real-time Pricing Catalog**: `GET https://x402-cleanweb-agent.onrender.com/api/v1/agent/pricing-catalog`
|
|
107
|
+
|
|
108
|
+
|
|
109
|
+
---
|
|
110
|
+
|
|
111
|
+
## 🛠️ How It Works (M2M Architecture)
|
|
112
|
+
|
|
113
|
+
```mermaid
|
|
114
|
+
sequenceDiagram
|
|
115
|
+
autonumber
|
|
116
|
+
actor Agent as Autonomous AI Agent
|
|
117
|
+
participant Server as x402 Gateway (FastAPI)
|
|
118
|
+
participant Polygon as Polygon Mainnet (Bor RPC)
|
|
119
|
+
participant Scraper as AI Data Cleaning Engine
|
|
120
|
+
|
|
121
|
+
Agent->>Server: GET /api/v1/clean-web?url=https://example.com
|
|
122
|
+
Note over Server: Check X-Payment-Tx header
|
|
123
|
+
Server-->>Agent: 402 Payment Required (Chain ID: 137, Recipient, Amount)
|
|
124
|
+
|
|
125
|
+
Agent->>Polygon: Send USDC Transfer (e.g. 0.01 USDC)
|
|
126
|
+
Polygon-->>Agent: Return Tx Hash (0xabc...123)
|
|
127
|
+
|
|
128
|
+
Agent->>Server: GET /api/v1/clean-web?url=... with Header [X-Payment-Tx: 0xabc...123]
|
|
129
|
+
Server->>Polygon: Verify Receipt, Event Logs, Recipient & Nonce
|
|
130
|
+
Polygon-->>Server: Tx Confirmed (Status: 1)
|
|
131
|
+
|
|
132
|
+
Server->>Scraper: Sanitize and Structure to Clean Markdown
|
|
133
|
+
Scraper-->>Server: Return Clean Markdown + Token Analytics
|
|
134
|
+
Server-->>Agent: 200 OK (Clean Markdown & Analytics JSON)
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
---
|
|
138
|
+
|
|
139
|
+
## ⚡ Direct cURL Quickstart
|
|
140
|
+
|
|
141
|
+
```bash
|
|
142
|
+
# Step 1: Query without payment to inspect 402 payment requirements
|
|
143
|
+
curl -i -X GET "https://x402-cleanweb-agent.onrender.com/api/v1/clean-web?url=https://example.com"
|
|
144
|
+
|
|
145
|
+
# Step 2: After sending USDC on Polygon, query with transaction hash
|
|
146
|
+
curl -X GET "https://x402-cleanweb-agent.onrender.com/api/v1/clean-web?url=https://example.com" \
|
|
147
|
+
-H "X-Payment-Tx: 0x<YOUR_POLYGON_TX_HASH>"
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
### Sample Response
|
|
151
|
+
|
|
152
|
+
```json
|
|
153
|
+
{
|
|
154
|
+
"status": "success",
|
|
155
|
+
"service": "clean-web",
|
|
156
|
+
"source_url": "https://example.com",
|
|
157
|
+
"title": "Example Domain",
|
|
158
|
+
"markdown_content": "# Example Domain\n\nThis domain is for use in illustrative examples...",
|
|
159
|
+
"token_analytics": {
|
|
160
|
+
"raw_html_estimated_tokens": 1250,
|
|
161
|
+
"clean_markdown_estimated_tokens": 280,
|
|
162
|
+
"tokens_saved": 970,
|
|
163
|
+
"token_savings_percentage": "77.6%",
|
|
164
|
+
"estimated_llm_cost_saved_usd": "$0.0029"
|
|
165
|
+
},
|
|
166
|
+
"processing_time_seconds": 0.38
|
|
167
|
+
}
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
## 🔌 Model Context Protocol (MCP) Setup
|
|
173
|
+
|
|
174
|
+
### Option 1: 1-Click Auto Installer (Recommended)
|
|
175
|
+
Automatically configures Claude Desktop & Cursor without editing JSON files:
|
|
176
|
+
|
|
177
|
+
```bash
|
|
178
|
+
# Windows
|
|
179
|
+
install_mcp.bat
|
|
180
|
+
|
|
181
|
+
# macOS / Linux
|
|
182
|
+
python install_mcp.py
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
---
|
|
186
|
+
|
|
187
|
+
### Option 2: Run via `uvx` (No installation needed)
|
|
188
|
+
Add directly to your `claude_desktop_config.json` or Cursor:
|
|
189
|
+
|
|
190
|
+
```json
|
|
191
|
+
{
|
|
192
|
+
"mcpServers": {
|
|
193
|
+
"polygon-x402-cleanweb": {
|
|
194
|
+
"command": "uvx",
|
|
195
|
+
"args": ["--from", "git+https://github.com/nohosa001-pixel/x402-cleanweb-agent", "x402-agent"]
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
203
|
+
### Option 3: Manual Local Configuration
|
|
204
|
+
|
|
205
|
+
```json
|
|
206
|
+
{
|
|
207
|
+
"mcpServers": {
|
|
208
|
+
"polygon-x402-cleanweb": {
|
|
209
|
+
"command": "python",
|
|
210
|
+
"args": ["-u", "/absolute/path/to/x402-micro-agent/mcp_server.py"],
|
|
211
|
+
"env": {
|
|
212
|
+
"PYTHONUNBUFFERED": "1",
|
|
213
|
+
"POLYGON_RPC_URL": "https://polygon-bor-rpc.publicnode.com",
|
|
214
|
+
"SERVER_WALLET_ADDRESS": "0x255F9991233f86B29dB847c8d5b8CB9915e80dCf",
|
|
215
|
+
"USDC_CONTRACT_ADDRESS": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359"
|
|
216
|
+
}
|
|
217
|
+
}
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Exposed MCP Tools
|
|
223
|
+
|
|
224
|
+
- `get_payment_info()`: Retrieve pricing tiers and recipient address.
|
|
225
|
+
- `fetch_clean_markdown(url, payment_tx_hash)`: Clean Web scraper (0.01 USDC).
|
|
226
|
+
- `fetch_youtube_transcript(url, language, payment_tx_hash)`: YouTube transcript extractor (0.02 USDC).
|
|
227
|
+
- `fetch_pdf_markdown(url, payment_tx_hash)`: PDF research paper converter (0.05 USDC).
|
|
228
|
+
- `fetch_plain_text(url, payment_tx_hash)`: Lightweight text scraper (0.005 USDC).
|
|
229
|
+
|
|
230
|
+
|
|
231
|
+
---
|
|
232
|
+
|
|
233
|
+
## 🛠️ Local Development & Running
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
# 1. Clone repository
|
|
237
|
+
git clone https://github.com/nohosa001-pixel/x402-cleanweb-agent.git
|
|
238
|
+
cd x402-cleanweb-agent
|
|
239
|
+
|
|
240
|
+
# 2. Setup virtual environment
|
|
241
|
+
python -m venv .venv
|
|
242
|
+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
|
243
|
+
|
|
244
|
+
# 3. Install dependencies
|
|
245
|
+
pip install -r requirements.txt
|
|
246
|
+
|
|
247
|
+
# 4. Start local server
|
|
248
|
+
python main.py
|
|
249
|
+
# Server running at: http://localhost:8000
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
---
|
|
253
|
+
|
|
254
|
+
## 📜 On-Chain Contract & Network Details
|
|
255
|
+
|
|
256
|
+
- **Network**: Polygon Mainnet (Chain ID: `137`)
|
|
257
|
+
- **Token Contract (USDC)**: [`0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359`](https://polygonscan.com/token/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359)
|
|
258
|
+
- **Recipient Treasury**: `0x255F9991233f86B29dB847c8d5b8CB9915e80dCf`
|
|
259
|
+
- **Standard**: [HTTP 402 Payment Required](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402)
|
|
260
|
+
|
|
261
|
+
---
|
|
262
|
+
|
|
263
|
+
## 🤝 Contributing & License
|
|
264
|
+
|
|
265
|
+
Contributions and suggestions are welcome!
|
|
266
|
+
Feel free to open an issue or pull request on GitHub: [https://github.com/nohosa001-pixel/x402-cleanweb-agent/issues](https://github.com/nohosa001-pixel/x402-cleanweb-agent/issues)
|
|
267
|
+
|
|
268
|
+
Distributed under the **MIT License**.
|
|
@@ -0,0 +1,238 @@
|
|
|
1
|
+
# ⚡ x402-cleanweb-agent
|
|
2
|
+
|
|
3
|
+
> **Turn any messy webpage, YouTube video, or PDF paper into pure, LLM-ready clean Markdown on Polygon.**
|
|
4
|
+
> *Zero Sign-up. Zero Subscriptions. True Machine-to-Machine HTTP 402 Micropayments for Autonomous AI Agents.*
|
|
5
|
+
|
|
6
|
+
[](https://x402-cleanweb-agent.onrender.com)
|
|
7
|
+
[](https://x402-cleanweb-agent.onrender.com/docs)
|
|
8
|
+
[](https://www.python.org/)
|
|
9
|
+
[](https://polygonscan.com/token/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359)
|
|
10
|
+
[](https://opensource.org/licenses/MIT)
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 💡 Why x402-cleanweb-agent?
|
|
15
|
+
|
|
16
|
+
Traditional web scraping and data extraction APIs force expensive **$49/month subscriptions** and complex API key management.
|
|
17
|
+
|
|
18
|
+
`x402-cleanweb-agent` solves this for autonomous AI agents, scrapers, and developers:
|
|
19
|
+
|
|
20
|
+
- ❌ **No Monthly Subscriptions**: Pay only for what you query ($0.005 ~ $0.05 per call in USDC).
|
|
21
|
+
- ❌ **No Sign-ups or API Keys**: Native **HTTP 402 Payment Required** machine-to-machine protocol.
|
|
22
|
+
- 🤖 **Zero-Human AI Agent Ready**: AI agents with a crypto wallet can autonomously buy data 24/7.
|
|
23
|
+
- ⚡ **Sub-Second Speed**: Strips ads, tracking scripts, and clutter, returning pure, structured Markdown.
|
|
24
|
+
- 🪙 **Ultra-Low Gas Fees**: Powered by **Polygon PoS** (< $0.005 network gas).
|
|
25
|
+
- 📊 **Token Savings Engine**: Calculates raw vs. cleaned token reduction (avg. 60~85% savings) and estimated LLM prompt cost savings ($).
|
|
26
|
+
|
|
27
|
+
---
|
|
28
|
+
|
|
29
|
+
## 🚀 Live Demo & Service Endpoints
|
|
30
|
+
|
|
31
|
+
- 🌐 **Web3 DApp UI**: [https://x402-cleanweb-agent.onrender.com](https://x402-cleanweb-agent.onrender.com)
|
|
32
|
+
- 📚 **Swagger API Docs**: [https://x402-cleanweb-agent.onrender.com/docs](https://x402-cleanweb-agent.onrender.com/docs)
|
|
33
|
+
- 📂 **GitHub Repo**: [https://github.com/nohosa001-pixel/x402-cleanweb-agent](https://github.com/nohosa001-pixel/x402-cleanweb-agent)
|
|
34
|
+
|
|
35
|
+
| Service | Endpoint | Pricing | Output & Description |
|
|
36
|
+
| :--- | :--- | :--- | :--- |
|
|
37
|
+
| **🌐 Clean Web** | `GET /api/v1/clean-web` | **0.01 USDC** | Ad/Noise removal + AI-ready Markdown + **Token Savings Analytics** |
|
|
38
|
+
| **🎬 YouTube Transcript** | `GET /api/v1/clean-youtube` | **0.02 USDC** | Full video **transcripts with timestamps** formatted in Markdown |
|
|
39
|
+
| **📑 PDF Paper & Report** | `GET /api/v1/clean-pdf` | **0.05 USDC** | arXiv papers & earnings reports converted into **structured Markdown** |
|
|
40
|
+
| **📝 Pure Plain Text** | `GET /api/v1/clean-text` | **0.005 USDC** | Ultra-lightweight raw text extraction for fast vector indexing |
|
|
41
|
+
|
|
42
|
+
---
|
|
43
|
+
|
|
44
|
+
## 🤖 Zero-Human Autonomous AI Agent Integration & Tools
|
|
45
|
+
|
|
46
|
+
AI agents with a Polygon wallet (Private Key) can autonomously handle payment and data extraction with **zero human intervention** and **built-in Budget Guard protection**:
|
|
47
|
+
|
|
48
|
+
### 1. Ready-to-Use Agent Toolkit (CrewAI / smolagents / LangChain)
|
|
49
|
+
|
|
50
|
+
```python
|
|
51
|
+
from agent_tools import get_x402_agent_tools, X402AgentToolkit
|
|
52
|
+
|
|
53
|
+
# 1. Initialize toolkit with spending limits
|
|
54
|
+
toolkit = X402AgentToolkit(
|
|
55
|
+
private_key="0xYOUR_AGENT_PRIVATE_KEY",
|
|
56
|
+
max_daily_budget_usdc=1.0 # Budget Guard protects against infinite loops
|
|
57
|
+
)
|
|
58
|
+
|
|
59
|
+
# 2. Get ready-to-use callable tools
|
|
60
|
+
tools = toolkit.get_tools_list()
|
|
61
|
+
|
|
62
|
+
# 3. Direct execution
|
|
63
|
+
web_data = toolkit.clean_web("https://example.com/article")
|
|
64
|
+
yt_data = toolkit.clean_youtube("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
|
|
65
|
+
pdf_data = toolkit.clean_pdf("https://arxiv.org/pdf/2301.00001.pdf")
|
|
66
|
+
|
|
67
|
+
# 4. Inspect spending report
|
|
68
|
+
print(toolkit.get_spending_report())
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
### 2. Machine-Readable Agent Discovery Protocol
|
|
72
|
+
|
|
73
|
+
Autonomous agents and crawlers can discover services, pricing, and on-chain payment contracts dynamically:
|
|
74
|
+
|
|
75
|
+
- **Agent Manifest**: `GET https://x402-cleanweb-agent.onrender.com/.well-known/agent.json`
|
|
76
|
+
- **Real-time Pricing Catalog**: `GET https://x402-cleanweb-agent.onrender.com/api/v1/agent/pricing-catalog`
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## 🛠️ How It Works (M2M Architecture)
|
|
82
|
+
|
|
83
|
+
```mermaid
|
|
84
|
+
sequenceDiagram
|
|
85
|
+
autonumber
|
|
86
|
+
actor Agent as Autonomous AI Agent
|
|
87
|
+
participant Server as x402 Gateway (FastAPI)
|
|
88
|
+
participant Polygon as Polygon Mainnet (Bor RPC)
|
|
89
|
+
participant Scraper as AI Data Cleaning Engine
|
|
90
|
+
|
|
91
|
+
Agent->>Server: GET /api/v1/clean-web?url=https://example.com
|
|
92
|
+
Note over Server: Check X-Payment-Tx header
|
|
93
|
+
Server-->>Agent: 402 Payment Required (Chain ID: 137, Recipient, Amount)
|
|
94
|
+
|
|
95
|
+
Agent->>Polygon: Send USDC Transfer (e.g. 0.01 USDC)
|
|
96
|
+
Polygon-->>Agent: Return Tx Hash (0xabc...123)
|
|
97
|
+
|
|
98
|
+
Agent->>Server: GET /api/v1/clean-web?url=... with Header [X-Payment-Tx: 0xabc...123]
|
|
99
|
+
Server->>Polygon: Verify Receipt, Event Logs, Recipient & Nonce
|
|
100
|
+
Polygon-->>Server: Tx Confirmed (Status: 1)
|
|
101
|
+
|
|
102
|
+
Server->>Scraper: Sanitize and Structure to Clean Markdown
|
|
103
|
+
Scraper-->>Server: Return Clean Markdown + Token Analytics
|
|
104
|
+
Server-->>Agent: 200 OK (Clean Markdown & Analytics JSON)
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## ⚡ Direct cURL Quickstart
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
# Step 1: Query without payment to inspect 402 payment requirements
|
|
113
|
+
curl -i -X GET "https://x402-cleanweb-agent.onrender.com/api/v1/clean-web?url=https://example.com"
|
|
114
|
+
|
|
115
|
+
# Step 2: After sending USDC on Polygon, query with transaction hash
|
|
116
|
+
curl -X GET "https://x402-cleanweb-agent.onrender.com/api/v1/clean-web?url=https://example.com" \
|
|
117
|
+
-H "X-Payment-Tx: 0x<YOUR_POLYGON_TX_HASH>"
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
### Sample Response
|
|
121
|
+
|
|
122
|
+
```json
|
|
123
|
+
{
|
|
124
|
+
"status": "success",
|
|
125
|
+
"service": "clean-web",
|
|
126
|
+
"source_url": "https://example.com",
|
|
127
|
+
"title": "Example Domain",
|
|
128
|
+
"markdown_content": "# Example Domain\n\nThis domain is for use in illustrative examples...",
|
|
129
|
+
"token_analytics": {
|
|
130
|
+
"raw_html_estimated_tokens": 1250,
|
|
131
|
+
"clean_markdown_estimated_tokens": 280,
|
|
132
|
+
"tokens_saved": 970,
|
|
133
|
+
"token_savings_percentage": "77.6%",
|
|
134
|
+
"estimated_llm_cost_saved_usd": "$0.0029"
|
|
135
|
+
},
|
|
136
|
+
"processing_time_seconds": 0.38
|
|
137
|
+
}
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## 🔌 Model Context Protocol (MCP) Setup
|
|
143
|
+
|
|
144
|
+
### Option 1: 1-Click Auto Installer (Recommended)
|
|
145
|
+
Automatically configures Claude Desktop & Cursor without editing JSON files:
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
# Windows
|
|
149
|
+
install_mcp.bat
|
|
150
|
+
|
|
151
|
+
# macOS / Linux
|
|
152
|
+
python install_mcp.py
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
### Option 2: Run via `uvx` (No installation needed)
|
|
158
|
+
Add directly to your `claude_desktop_config.json` or Cursor:
|
|
159
|
+
|
|
160
|
+
```json
|
|
161
|
+
{
|
|
162
|
+
"mcpServers": {
|
|
163
|
+
"polygon-x402-cleanweb": {
|
|
164
|
+
"command": "uvx",
|
|
165
|
+
"args": ["--from", "git+https://github.com/nohosa001-pixel/x402-cleanweb-agent", "x402-agent"]
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
---
|
|
172
|
+
|
|
173
|
+
### Option 3: Manual Local Configuration
|
|
174
|
+
|
|
175
|
+
```json
|
|
176
|
+
{
|
|
177
|
+
"mcpServers": {
|
|
178
|
+
"polygon-x402-cleanweb": {
|
|
179
|
+
"command": "python",
|
|
180
|
+
"args": ["-u", "/absolute/path/to/x402-micro-agent/mcp_server.py"],
|
|
181
|
+
"env": {
|
|
182
|
+
"PYTHONUNBUFFERED": "1",
|
|
183
|
+
"POLYGON_RPC_URL": "https://polygon-bor-rpc.publicnode.com",
|
|
184
|
+
"SERVER_WALLET_ADDRESS": "0x255F9991233f86B29dB847c8d5b8CB9915e80dCf",
|
|
185
|
+
"USDC_CONTRACT_ADDRESS": "0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359"
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
### Exposed MCP Tools
|
|
193
|
+
|
|
194
|
+
- `get_payment_info()`: Retrieve pricing tiers and recipient address.
|
|
195
|
+
- `fetch_clean_markdown(url, payment_tx_hash)`: Clean Web scraper (0.01 USDC).
|
|
196
|
+
- `fetch_youtube_transcript(url, language, payment_tx_hash)`: YouTube transcript extractor (0.02 USDC).
|
|
197
|
+
- `fetch_pdf_markdown(url, payment_tx_hash)`: PDF research paper converter (0.05 USDC).
|
|
198
|
+
- `fetch_plain_text(url, payment_tx_hash)`: Lightweight text scraper (0.005 USDC).
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
203
|
+
## 🛠️ Local Development & Running
|
|
204
|
+
|
|
205
|
+
```bash
|
|
206
|
+
# 1. Clone repository
|
|
207
|
+
git clone https://github.com/nohosa001-pixel/x402-cleanweb-agent.git
|
|
208
|
+
cd x402-cleanweb-agent
|
|
209
|
+
|
|
210
|
+
# 2. Setup virtual environment
|
|
211
|
+
python -m venv .venv
|
|
212
|
+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
|
213
|
+
|
|
214
|
+
# 3. Install dependencies
|
|
215
|
+
pip install -r requirements.txt
|
|
216
|
+
|
|
217
|
+
# 4. Start local server
|
|
218
|
+
python main.py
|
|
219
|
+
# Server running at: http://localhost:8000
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
## 📜 On-Chain Contract & Network Details
|
|
225
|
+
|
|
226
|
+
- **Network**: Polygon Mainnet (Chain ID: `137`)
|
|
227
|
+
- **Token Contract (USDC)**: [`0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359`](https://polygonscan.com/token/0x3c499c542cEF5E3811e1192ce70d8cC03d5c3359)
|
|
228
|
+
- **Recipient Treasury**: `0x255F9991233f86B29dB847c8d5b8CB9915e80dCf`
|
|
229
|
+
- **Standard**: [HTTP 402 Payment Required](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/402)
|
|
230
|
+
|
|
231
|
+
---
|
|
232
|
+
|
|
233
|
+
## 🤝 Contributing & License
|
|
234
|
+
|
|
235
|
+
Contributions and suggestions are welcome!
|
|
236
|
+
Feel free to open an issue or pull request on GitHub: [https://github.com/nohosa001-pixel/x402-cleanweb-agent/issues](https://github.com/nohosa001-pixel/x402-cleanweb-agent/issues)
|
|
237
|
+
|
|
238
|
+
Distributed under the **MIT License**.
|