spectrawl 0.2.0 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +26 -5
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -19,7 +19,14 @@ npm install spectrawl
|
|
|
19
19
|
|
|
20
20
|
**Auth** — Persistent cookie storage (SQLite), multi-account management, automatic cookie refresh, expiry alerts.
|
|
21
21
|
|
|
22
|
-
**Act** —
|
|
22
|
+
**Act** — 24 platform adapters covering 30+ sites:
|
|
23
|
+
- **Content platforms:** X, Reddit, LinkedIn, Dev.to, Hashnode, IndieHackers, Medium, Hacker News, Quora
|
|
24
|
+
- **Developer:** GitHub (repos, issues, releases), HuggingFace (models, datasets), Discord (bot + webhooks)
|
|
25
|
+
- **Launch/SEO:** Product Hunt, BetaList, AlternativeTo, SaaSHub, DevHunt, AppSumo
|
|
26
|
+
- **Directories:** Generic adapter for MicroLaunch, Uneed, Peerlist, Fazier, BetaPage, LaunchingNext, StartupStash, SideProjectors, TAIFT, Futurepedia, Crunchbase, G2, StackShare, YouTube
|
|
27
|
+
- Rate limiting, content dedup, dead letter queue for retries.
|
|
28
|
+
|
|
29
|
+
**Proxy** — Rotating proxy server. One endpoint (`localhost:8080`) for all your tools. Round-robin, random, or least-used strategies. Health checking with auto-failover.
|
|
23
30
|
|
|
24
31
|
## Quick Start
|
|
25
32
|
|
|
@@ -123,6 +130,19 @@ Configure the cascade in `spectrawl.json`:
|
|
|
123
130
|
| Hashnode | GraphQL API | post |
|
|
124
131
|
| LinkedIn | Cookie API (Voyager) | post |
|
|
125
132
|
| IndieHackers | Browser automation | post, comment, upvote |
|
|
133
|
+
| Medium | REST API | post (markdown) |
|
|
134
|
+
| GitHub | REST v3 | repo, file, issue, release |
|
|
135
|
+
| Discord | Bot API + webhooks | send, thread |
|
|
136
|
+
| Product Hunt | GraphQL v2 | launch, comment, upvote |
|
|
137
|
+
| Hacker News | Cookie/form POST | submit, comment, upvote |
|
|
138
|
+
| YouTube | Data API v3 | comment, playlist, update |
|
|
139
|
+
| Quora | Browser automation | answer, question |
|
|
140
|
+
| HuggingFace | Hub API | repo, model card, upload |
|
|
141
|
+
| BetaList | REST API | submit |
|
|
142
|
+
| AlternativeTo | Browser automation | submit |
|
|
143
|
+
| SaaSHub | Browser automation | submit |
|
|
144
|
+
| DevHunt | Browser automation | submit |
|
|
145
|
+
| **30+ Directories** | Generic adapter | submit (MicroLaunch, Uneed, TAIFT, Futurepedia, Crunchbase, G2, etc.) |
|
|
126
146
|
|
|
127
147
|
## Configuration
|
|
128
148
|
|
|
@@ -141,10 +161,11 @@ Configure the cascade in `spectrawl.json`:
|
|
|
141
161
|
"scrapeTtl": 24
|
|
142
162
|
},
|
|
143
163
|
"proxy": {
|
|
144
|
-
"
|
|
145
|
-
"
|
|
146
|
-
"
|
|
147
|
-
|
|
164
|
+
"localPort": 8080,
|
|
165
|
+
"strategy": "round-robin",
|
|
166
|
+
"upstreams": [
|
|
167
|
+
{ "url": "http://user:pass@proxy1.example.com:8080" }
|
|
168
|
+
]
|
|
148
169
|
},
|
|
149
170
|
"camoufox": {
|
|
150
171
|
"url": "http://localhost:9869"
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "spectrawl",
|
|
3
|
-
"version": "0.2.
|
|
4
|
-
"description": "The unified web layer for AI agents. Search, browse,
|
|
3
|
+
"version": "0.2.1",
|
|
4
|
+
"description": "The unified web layer for AI agents. Search (6 engines), stealth browse (Camoufox + Playwright), auth (cookies, multi-account), act (24 adapters, 30+ platforms), proxy rotation. Self-hosted, free.",
|
|
5
5
|
"main": "src/index.js",
|
|
6
6
|
"types": "index.d.ts",
|
|
7
7
|
"bin": {
|