spectrawl 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 FayAndXan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,179 @@
+# Spectrawl
+
+The unified web layer for AI agents. Search, browse, authenticate, and act on platforms — one tool, self-hosted, free.
+
+## What It Does
+
+AI agents need to interact with the web. That means searching, browsing pages, logging into platforms, and posting content. Today you duct-tape together Playwright + Tavily + cookie managers + platform-specific scripts. Spectrawl replaces all of that.
+
+```
+npm install spectrawl
+```
+
+**Search** — 6 engines in a cascade: SearXNG → DuckDuckGo → Brave → Serper → Google CSE → Jina. Tries free/unlimited first, falls through to quota-based. Dual scraping (Jina Reader + readability). Optional LLM summarization.
+
+**Browse** — Stealth browsing with anti-detection out of the box. Three tiers:
+1. `playwright-extra` + stealth plugin (default, works immediately)
+2. Camoufox binary — engine-level anti-fingerprint (`npx spectrawl install-stealth`)
+3. Remote Camoufox service (for existing deployments)
+
+**Auth** — Persistent cookie storage (SQLite), multi-account management, automatic cookie refresh, expiry alerts.
+
+**Act** — Post to X, Reddit, Dev.to, Hashnode, LinkedIn, IndieHackers. Rate limiting, content dedup, dead letter queue for retries.
+
+## Quick Start
+
+```bash
+npm install spectrawl
+npx spectrawl init          # create spectrawl.json config
+npx spectrawl search "your query"
+```
+
+### As a Library
+
+```js
+const { Spectrawl } = require('spectrawl')
+const web = new Spectrawl()
+
+// Search
+const results = await web.search('best practices for node.js APIs')
+console.log(results.sources)      // [{ url, title, snippet, content }]
+console.log(results.answer)       // LLM summary (if configured)
+
+// Browse with stealth
+const page = await web.browse('https://example.com')
+console.log(page.content)         // extracted text
+console.log(page.engine)          // 'stealth-playwright' or 'camoufox'
+
+// Act on platforms
+await web.act('x', 'post', {
+  text: 'Hello from Spectrawl',
+  account: '@myhandle'
+})
+
+// Check auth health
+const accounts = await web.status()
+// [{ platform: 'x', account: '@myhandle', status: 'valid', expiresAt: '...' }]
+```
+
+### HTTP Server
+
+```bash
+npx spectrawl serve --port 3900
+```
+
+```
+POST /search   { "query": "...", "summarize": true }
+POST /browse   { "url": "...", "screenshot": true }
+POST /act      { "platform": "x", "action": "post", "params": { "text": "..." } }
+GET  /status
+GET  /health
+```
+
+### MCP Server
+
+Works with any MCP-compatible agent framework:
+
+```bash
+npx spectrawl mcp
+```
+
+Exposes 5 tools: `web_search`, `web_browse`, `web_act`, `web_auth`, `web_status`.
+
+## Stealth Browsing
+
+Default: `playwright-extra` with stealth plugin patches webdriver detection, navigator properties, canvas/WebGL fingerprinting, and plugin enumeration. Works for ~90% of sites.
+
+For deeper anti-detection:
+
+```bash
+npx spectrawl install-stealth
+```
+
+Downloads the [Camoufox](https://github.com/daijro/camoufox) browser — a patched Firefox with engine-level anti-fingerprint. Spectrawl auto-detects and uses it.
+
+## Search Engines
+
+| Engine | Free Tier | Default |
+|--------|-----------|---------|
+| SearXNG | Unlimited (self-hosted) | ✅ |
+| DuckDuckGo | Unlimited | ✅ |
+| Brave | 2,000/month | ✅ |
+| Serper | 2,500/month | Fallback |
+| Google CSE | 100/day | Fallback |
+| Jina Reader | Unlimited | Fallback |
+
+Configure the cascade in `spectrawl.json`:
+
+```json
+{
+  "search": {
+    "cascade": ["searxng", "ddg", "brave", "serper", "google-cse", "jina"]
+  }
+}
+```
+
+## Platform Adapters
+
+| Platform | Auth Method | Actions |
+|----------|-------------|---------|
+| X/Twitter | GraphQL Cookie + OAuth 1.0a | post |
+| Reddit | Cookie API (oauth.reddit.com) | post, comment |
+| Dev.to | REST API (API key) | post |
+| Hashnode | GraphQL API | post |
+| LinkedIn | Cookie API (Voyager) | post |
+| IndieHackers | Browser automation | post, comment, upvote |
+
+## Configuration
+
+`spectrawl.json`:
+
+```json
+{
+  "port": 3900,
+  "search": {
+    "cascade": ["ddg", "brave"],
+    "scrapeTop": 3
+  },
+  "cache": {
+    "path": "./data/cache.db",
+    "searchTtl": 1,
+    "scrapeTtl": 24
+  },
+  "proxy": {
+    "host": "proxy.example.com",
+    "port": "8080",
+    "username": "user",
+    "password": "pass"
+  },
+  "camoufox": {
+    "url": "http://localhost:9869"
+  },
+  "rateLimit": {
+    "x": { "postsPerHour": 3, "minDelayMs": 60000 },
+    "reddit": { "postsPerHour": 5 }
+  }
+}
+```
+
+## Environment Variables
+
+```
+BRAVE_API_KEY       Brave Search API key
+SERPER_API_KEY      Serper.dev API key
+GOOGLE_CSE_KEY      Google Custom Search API key
+GOOGLE_CSE_CX       Google Custom Search engine ID
+JINA_API_KEY        Jina Reader API key (optional)
+SEARXNG_URL         SearXNG instance URL (default: http://localhost:8888)
+CAMOUFOX_URL        Remote Camoufox service URL
+OPENAI_API_KEY      For LLM summarization
+ANTHROPIC_API_KEY   For LLM summarization
+```
+
+## License
+
+MIT
+
+## Part of xanOS
+
+Spectrawl is the web layer for [xanOS](https://github.com/FayAndXan/xanOS) — the autonomous content engine. Use it standalone or as part of the full stack.

package/index.d.ts

@@ -0,0 +1,90 @@
+declare module 'spectrawl' {
+  export interface SearchResult {
+    sources: Array<{
+      url: string
+      title: string
+      snippet?: string
+      content?: string
+    }>
+    answer?: string
+    engine: string
+    cached: boolean
+  }
+
+  export interface BrowseResult {
+    content?: string
+    html?: string
+    screenshot?: Buffer
+    url: string
+    title: string
+    engine: 'stealth-playwright' | 'camoufox' | 'remote-camoufox' | 'playwright'
+    cached: boolean
+    cookies?: any[]
+  }
+
+  export interface ActResult {
+    success: boolean
+    error?: string
+    detail?: string
+    suggestion?: string
+    retryAfter?: number
+    url?: string
+    [key: string]: any
+  }
+
+  export interface AccountStatus {
+    platform: string
+    account: string
+    status: 'valid' | 'expiring' | 'expired' | 'unknown'
+    expiresAt?: string
+    cookieCount?: number
+  }
+
+  export interface SearchOptions {
+    summarize?: boolean
+    minResults?: number
+    noCache?: boolean
+  }
+
+  export interface BrowseOptions {
+    screenshot?: boolean
+    fullPage?: boolean
+    html?: boolean
+    extract?: boolean
+    stealth?: boolean
+    camoufox?: boolean
+    noCache?: boolean
+    saveCookies?: boolean
+    _cookies?: any[]
+  }
+
+  export interface ActParams {
+    text?: string
+    title?: string
+    body?: string
+    account?: string
+    group?: string
+    postUrl?: string
+    tweetId?: string
+    mediaIds?: string[]
+    _cookies?: any[]
+    [key: string]: any
+  }
+
+  export class Spectrawl {
+    constructor(config?: any)
+    search(query: string, opts?: SearchOptions): Promise<SearchResult>
+    browse(url: string, opts?: BrowseOptions): Promise<BrowseResult>
+    act(platform: string, action: string, params?: ActParams): Promise<ActResult>
+    status(): Promise<AccountStatus[]>
+    close(): Promise<void>
+  }
+
+  export function installStealth(): Promise<{
+    path: string
+    binary?: string
+    version: string
+  }>
+
+  export function isStealthInstalled(): boolean
+}

package/package.json

@@ -0,0 +1,53 @@
+{
+  "name": "spectrawl",
+  "version": "0.1.0",
+  "description": "The unified web layer for AI agents. Search, browse, authenticate, act — one tool, self-hosted, free.",
+  "main": "src/index.js",
+  "types": "index.d.ts",
+  "bin": {
+    "spectrawl": "./src/cli.js"
+  },
+  "files": [
+    "src/",
+    "index.d.ts",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "dev": "node src/server.js",
+    "start": "node src/server.js",
+    "test": "node --test test/*.test.js"
+  },
+  "keywords": [
+    "ai-agent",
+    "web-scraping",
+    "browser-automation",
+    "search-api",
+    "mcp",
+    "stealth-browser",
+    "cookie-management",
+    "tavily-alternative",
+    "anti-detect",
+    "camoufox",
+    "playwright"
+  ],
+  "author": "FayAndXan",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/FayAndXan/spectrawl"
+  },
+  "homepage": "https://github.com/FayAndXan/spectrawl#readme",
+  "bugs": {
+    "url": "https://github.com/FayAndXan/spectrawl/issues"
+  },
+  "engines": {
+    "node": ">=20.0.0"
+  },
+  "dependencies": {
+    "better-sqlite3": "^11.0.0",
+    "playwright": "^1.50.0",
+    "playwright-extra": "^4.3.6",
+    "puppeteer-extra-plugin-stealth": "^2.11.2"
+  }
+}

package/src/act/adapters/devto.js

@@ -0,0 +1,103 @@
+const https = require('https')
+
+/**
+ * Dev.to platform adapter.
+ * Uses official REST API — simple, no cookies needed.
+ */
+class DevtoAdapter {
+  async execute(action, params, ctx) {
+    switch (action) {
+      case 'post':
+        return this._post(params, ctx)
+      case 'update':
+        return this._update(params, ctx)
+      default:
+        throw new Error(`Unsupported Dev.to action: ${action}`)
+    }
+  }
+
+  async _post(params, ctx) {
+    const { title, body, tags, published, account } = params
+    
+    const apiKey = await this._getApiKey(account, ctx)
+    
+    const article = {
+      article: {
+        title,
+        body_markdown: body,
+        published: published !== false,
+        tags: tags || []
+      }
+    }
+
+    const data = await postJson('https://dev.to/api/articles', JSON.stringify(article), {
+      'api-key': apiKey,
+      'Content-Type': 'application/json'
+    })
+
+    return { articleId: data.id, url: data.url, slug: data.slug }
+  }
+
+  async _update(params, ctx) {
+    const { articleId, title, body, tags, published, account } = params
+    
+    const apiKey = await this._getApiKey(account, ctx)
+    
+    const article = { article: {} }
+    if (title) article.article.title = title
+    if (body) article.article.body_markdown = body
+    if (tags) article.article.tags = tags
+    if (published !== undefined) article.article.published = published
+
+    const data = await putJson(`https://dev.to/api/articles/${articleId}`, JSON.stringify(article), {
+      'api-key': apiKey,
+      'Content-Type': 'application/json'
+    })
+
+    return { articleId: data.id, url: data.url }
+  }
+
+  async _getApiKey(account, ctx) {
+    // Try to get API key from auth store
+    const creds = await ctx.auth.getCookies('devto', account)
+    if (creds?.apiKey) return creds.apiKey
+    
+    // Try env
+    if (process.env.DEVTO_API_KEY) return process.env.DEVTO_API_KEY
+    
+    throw new Error('Dev.to API key not configured. Run: spectrawl login devto --api-key YOUR_KEY')
+  }
+}
+
+function postJson(url, body, headers) {
+  return jsonRequest('POST', url, body, headers)
+}
+
+function putJson(url, body, headers) {
+  return jsonRequest('PUT', url, body, headers)
+}
+
+function jsonRequest(method, url, body, headers) {
+  return new Promise((resolve, reject) => {
+    const urlObj = new URL(url)
+    const opts = {
+      hostname: urlObj.hostname,
+      path: urlObj.pathname,
+      method,
+      headers: { ...headers, 'Content-Length': Buffer.byteLength(body) }
+    }
+    const req = https.request(opts, res => {
+      let data = ''
+      res.on('data', c => data += c)
+      res.on('end', () => {
+        try { resolve(JSON.parse(data)) }
+        catch (e) { reject(new Error(`Invalid Dev.to response: ${data.slice(0, 200)}`)) }
+      })
+    })
+    req.on('error', reject)
+    req.write(body)
+    req.end()
+  })
+}
+
+module.exports = { DevtoAdapter }

package/src/act/adapters/hashnode.js

@@ -0,0 +1,89 @@
+const https = require('https')
+
+/**
+ * Hashnode platform adapter.
+ * Uses official GraphQL API.
+ */
+class HashnodeAdapter {
+  async execute(action, params, ctx) {
+    switch (action) {
+      case 'post':
+        return this._post(params, ctx)
+      default:
+        throw new Error(`Unsupported Hashnode action: ${action}`)
+    }
+  }
+
+  async _post(params, ctx) {
+    const { title, body, tags, publicationId, account } = params
+    const apiKey = await this._getApiKey(account, ctx)
+
+    const query = `
+      mutation PublishPost($input: PublishPostInput!) {
+        publishPost(input: $input) {
+          post { id, slug, url, title }
+        }
+      }
+    `
+
+    const variables = {
+      input: {
+        title,
+        contentMarkdown: body,
+        publicationId: publicationId || await this._getPublicationId(apiKey),
+        tags: (tags || []).map(t => ({ slug: t.toLowerCase().replace(/\s+/g, '-'), name: t }))
+      }
+    }
+
+    const data = await graphql(apiKey, query, variables)
+
+    if (data.errors) {
+      throw new Error(`Hashnode error: ${data.errors[0]?.message}`)
+    }
+
+    const post = data.data?.publishPost?.post
+    return { postId: post?.id, url: post?.url, slug: post?.slug }
+  }
+
+  async _getPublicationId(apiKey) {
+    const query = `query { me { publications(first: 1) { edges { node { id } } } } }`
+    const data = await graphql(apiKey, query)
+    return data.data?.me?.publications?.edges?.[0]?.node?.id
+  }
+
+  async _getApiKey(account, ctx) {
+    const creds = await ctx.auth.getCookies('hashnode', account)
+    if (creds?.apiKey) return creds.apiKey
+    if (process.env.HASHNODE_API_KEY) return process.env.HASHNODE_API_KEY
+    throw new Error('Hashnode API key not configured')
+  }
+}
+
+function graphql(apiKey, query, variables = {}) {
+  return new Promise((resolve, reject) => {
+    const body = JSON.stringify({ query, variables })
+    const opts = {
+      hostname: 'gql.hashnode.com',
+      path: '/',
+      method: 'POST',
+      headers: {
+        'Authorization': apiKey,
+        'Content-Type': 'application/json',
+        'Content-Length': Buffer.byteLength(body)
+      }
+    }
+    const req = https.request(opts, res => {
+      let data = ''
+      res.on('data', c => data += c)
+      res.on('end', () => {
+        try { resolve(JSON.parse(data)) }
+        catch (e) { reject(new Error('Invalid Hashnode response')) }
+      })
+    })
+    req.on('error', reject)
+    req.write(body)
+    req.end()
+  })
+}
+
+module.exports = { HashnodeAdapter }