@crowdlisten/harness 1.0.0

package/AGENTS.md

@@ -0,0 +1,167 @@
+# CrowdListen — Agent Reference
+
+Unified MCP server for AI agents. Planning, social listening, skill packs, and knowledge management with progressive disclosure.
+
+## Quick Start
+
+```bash
+npx @crowdlisten/harness login
+```
+
+Auto-configures MCP for Claude Code, Cursor, Gemini CLI, Codex, OpenClaw, Amp.
+
+### Manual MCP config (stdio)
+```json
+{ "crowdlisten": { "command": "npx", "args": ["-y", "@crowdlisten/harness"] } }
+```
+
+### Remote MCP config (Streamable HTTP)
+```json
+{
+  "crowdlisten": {
+    "url": "https://mcp.crowdlisten.com/mcp",
+    "headers": {
+      "Authorization": "Bearer YOUR_API_KEY"
+    }
+  }
+}
+```
+
+## Interfaces
+
+| Interface | Access | Best for |
+|-----------|--------|----------|
+| MCP stdio | `npx @crowdlisten/harness` — ~41 tools | Local agents |
+| MCP HTTP | `POST https://mcp.crowdlisten.com/mcp` | Remote agents, cloud |
+| REST | `POST https://mcp.crowdlisten.com/tools/{name}` | Non-MCP integrations |
+| OpenAPI | `GET https://mcp.crowdlisten.com/openapi.json` | Docs, code gen |
+| CLI | `npx @crowdlisten/harness login/setup/serve/openapi` | Auth, config, hosting |
+| Web UI | `npx @crowdlisten/harness context` → localhost:3847 | Visual context extraction |
+
+## Progressive Disclosure
+
+You start with **4 tools**. Activate skill packs to unlock more:
+
+```
+list_skill_packs()                              → see available packs
+activate_skill_pack({ pack_id: "planning" })    → unlocks 11 task tools
+activate_skill_pack({ pack_id: "social-listening" }) → unlocks 7 social tools
+```
+
+After activation, new tools appear automatically via `tools/list_changed`.
+
+## Always-On Tools (4)
+
+- **list_skill_packs**(include_virtual?) — List all packs with status (active/available), tool counts
+- **activate_skill_pack**(pack_id) — Activate a pack to unlock its tools. For SKILL.md packs, returns workflow instructions.
+- **remember**(type, title, content) — Save context across sessions. Types: preference, decision, pattern, insight, style
+- **recall**(type?, search?, limit?) — Retrieve saved context blocks
+
+## Skill Packs
+
+| Pack | Tools | Description |
+|------|-------|-------------|
+| **core** (always on) | 4 | Discovery + memory |
+| **planning** | 11 | Tasks, plans, progress tracking |
+| **knowledge** | 3 | Project knowledge base |
+| **social-listening** | 7 | Search social platforms (free) |
+| **audience-analysis** | 6 | AI analysis (CROWDLISTEN_API_KEY) |
+| **sessions** | 3 | Multi-agent coordination |
+| **setup** | 5 | Board management |
+| **legacy** | 6 | Previous-gen context extraction |
+
+Plus: 8 native SKILL.md workflow packs (competitive-analysis, content-creator, etc.)
+
+## Planning Pack (11 tools)
+
+- **list_tasks**(board_id?, status?, limit?) — List board tasks
+- **get_task**(task_id) — Full task details
+- **create_task**(title, description?, priority?) — Create task
+- **update_task**(task_id, ...) — Update task fields
+- **claim_task**(task_id, executor?, branch?) — Start work, get context
+- **complete_task**(task_id, summary?) — Mark done
+- **delete_task**(task_id) — Remove task
+- **log_progress**(task_id, message) — Track execution
+- **create_plan**(task_id, approach, ...) — Draft execution plan
+- **get_plan**(task_id) — View plan with history
+- **update_plan**(plan_id, ...) — Iterate, submit for review
+
+## Knowledge Pack (3 tools)
+
+- **query_context**(search?, type?, tags?) — Search decisions, patterns, learnings
+- **add_context**(type, title, body, ...) — Write to knowledge base
+- **record_learning**(task_id, title, body, promote?) — Capture learning
+
+## Social Listening Pack (7 tools, free)
+
+- **search_content**(platform, query, limit?) — Search posts across platforms
+- **get_content_comments**(platform, contentId, limit?) — Get comments/replies
+- **get_trending_content**(platform, limit?) — Trending posts
+- **get_user_content**(platform, userId, limit?) — User's recent posts
+- **get_platform_status**() — Available platforms + capabilities
+- **health_check**() — Platform connectivity status
+- **extract_url**(url, mode?, limit?) — Vision extraction from any URL
+
+Platforms: reddit, twitter, tiktok, instagram, youtube, moltbook
+
+## Audience Analysis Pack (6 tools, CROWDLISTEN_API_KEY)
+
+- **analyze_content**(platform, contentId, analysisDepth?) — Sentiment, themes, tensions
+- **cluster_opinions**(platform, contentId, clusterCount?) — Opinion clustering
+- **enrich_content**(platform, contentId, question?) — Intent/stance analysis
+- **deep_analyze**(platform, contentId, analysisDepth?) — Full audience intelligence
+- **extract_insights**(platform, contentId, categories?) — Pain points, feature requests
+- **research_synthesis**(query, platforms?, depth?) — Cross-platform research
+
+## Analysis (5 tools) — requires CROWDLISTEN_API_KEY
+
+- **run_analysis**(project_id, question, platforms?, max_results?) — Run audience analysis across Reddit, YouTube, TikTok, Twitter, Instagram, Xiaohongshu. Streams results.
+- **continue_analysis**(analysis_id, question) — Follow-up question on existing analysis.
+- **get_analysis**(analysis_id) — Get full analysis results with themes, sentiment, quotes.
+- **list_analyses**(project_id, limit?) — List analyses for a project.
+- **generate_specs**(project_id, analysis_id?, spec_type?) — Generate feature requests, user stories, acceptance criteria from analysis.
+
+## Content & Vectors (4 tools) — requires CROWDLISTEN_API_KEY
+
+- **ingest_content**(project_id, content, source_url?, title?, metadata?) — Ingest content into vector store.
+- **search_vectors**(project_id, query, limit?, threshold?) — Semantic search across ingested content.
+- **get_content_stats**(project_id) — Document count, chunks, storage usage.
+- **delete_content**(content_id) — Delete content document and embeddings.
+
+## Document Generation (2 tools) — requires CROWDLISTEN_API_KEY
+
+- **generate_prd**(project_id, analysis_ids?, template?, sections?) — Generate PRD from analysis. Templates: standard, lean, technical, marketing.
+- **update_prd_section**(document_id, section, instructions?, content?) — Update a specific PRD section.
+
+## Sessions (3 tools) — Parallel Agents
+
+- **start_session**(task_id, executor?, focus) — Start additional parallel session.
+- **list_sessions**(task_id, status?) — List sessions showing status and focus.
+- **update_session**(session_id, status?, focus?) — Update session: idle, running, completed, failed, stopped.
+
+## LLM Proxy (2 tools) — free, no API key
+
+- **llm_complete**(prompt, model?, max_tokens?, temperature?, system?) — LLM completion through CrowdListen. Default: gpt-4o-mini.
+- **list_llm_models**() — List available models and capabilities.
+
+## Agent Network (3 tools) — mixed auth
+
+- **register_agent**(name, capabilities?, executor?) — Register in agent network. Free.
+- **get_capabilities**() — List network capabilities. Free.
+- **submit_analysis**(agent_id, analysis_id, summary) — Share analysis results. Requires API key.
+
+## Core Workflow
+
+```
+list_skill_packs → activate_skill_pack("planning")
+→ list_tasks → claim_task → query_context → create_plan
+→ [human review] → execute → record_learning → complete_task
+```
+
+## Privacy
+
+- PII redacted locally before LLM calls
+- Context stored locally (~/.crowdlisten/)
+- User's own API keys for extraction
+- No data syncs without explicit user action
+- Agent-proxied tools go through `agent.crowdlisten.com` with your API key

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 CrowdListen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,153 @@
+# CrowdListen
+
+> Give your AI agent crowd context — analyzed intelligence from what real users say, what markets think, and what communities want.
+
+![CrowdListen — Give your agent evidence, not guesses](docs/images/hero.png)
+
+[English](README.md) | [中文文档](README-CN.md) | [한국어](README-KO.md) | [Español](README-ES.md)
+
+## The Problem
+
+AI agents don't know what your users think. Every session starts from scratch — no awareness of what people say on Reddit, no signal from TikTok comments, no synthesis of forum discussions. You end up copy-pasting feedback manually and watching your agent make decisions without the one input that matters most: what real people think.
+
+CrowdListen fixes this with a four-step loop:
+
+1. **Listen** — search Reddit, YouTube, TikTok, Twitter/X, Instagram, Xiaohongshu, and forums
+2. **Analyze** — cluster opinions by theme, extract pain points, synthesize cross-platform reports
+3. **Remember** — save analyzed insights with semantic embeddings, not raw posts
+4. **Recall** — any agent retrieves crowd context via natural language, across sessions and devices
+
+Any agent — Claude Code, Cursor, Gemini CLI, Codex — can recall this later. The intelligence compounds across sessions and across agents. That's crowd context.
+
+## Get Started
+
+One command. Your browser opens, you sign in, and your agents are configured automatically:
+
+```bash
+npx @crowdlisten/harness login
+```
+
+Auto-configures MCP for **Claude Code, Cursor, Gemini CLI, Codex, Amp, and OpenClaw**. No env vars, no JSON editing, no API keys to manage. Restart your agent after login.
+
+### Manual Setup
+
+Add to your agent's MCP config:
+
+```json
+{
+  "mcpServers": {
+    "crowdlisten": {
+      "command": "npx",
+      "args": ["-y", "@crowdlisten/harness"]
+    }
+  }
+}
+```
+
+For remote access, use the HTTP transport:
+
+```json
+{
+  "mcpServers": {
+    "crowdlisten": {
+      "url": "https://mcp.crowdlisten.com/mcp",
+      "headers": {
+        "Authorization": "Bearer YOUR_API_KEY"
+      }
+    }
+  }
+}
+```
+
+## What You Can Do
+
+| Capability | What it does | How it works |
+|---|---|---|
+| **Search social platforms** | Search Reddit, YouTube, TikTok, Twitter/X, Instagram, Xiaohongshu from one tool | Returns structured posts with engagement metrics, timestamps, and author info — same format regardless of platform |
+| **Analyze audience signal** | Cluster opinions, extract pain points, generate cross-platform reports | AI groups comments by theme, scores sentiment, identifies competitive signals |
+| **Save and recall across sessions** | Semantic memory that follows you across agents and devices | Your agent saves with `save`, retrieves with `recall` using meaning-based search — not keyword matching |
+| **Plan and track work** | Tasks, execution plans, progress tracking | Your agent claims tasks, drafts plans with assumptions and risks, logs progress |
+| **Get specs from crowd feedback** | Turn crowd intelligence into implementation-ready specs | Specs include evidence citations, acceptance criteria, and confidence scores |
+| **Extract from any website** | Screenshot any URL and get structured data back | Vision mode sends screenshots to an LLM — works on forums, paywalled sites, anything with a URL |
+
+## How It Works
+
+![CrowdListen Pipeline — Raw Crowd Signals to Agent Delivery](docs/images/pipeline.jpg)
+
+Your agent starts with **5 core tools** and activates skill packs on demand. No restart required — new tools appear instantly.
+
+### Skill Packs
+
+| Pack | Tools | What it does | Free? |
+|------|:-----:|---|:---:|
+| **core** (always on) | 5 | Semantic memory, discovery, preferences | Yes |
+| **social-listening** | 7 | Search Reddit, TikTok, YouTube, Twitter, Instagram, Xiaohongshu | Yes |
+| **audience-analysis** | 6 | Opinion clustering, deep analysis, insight extraction, research synthesis | API key |
+| **planning** | 11 | Tasks, execution plans, progress tracking | Yes |
+| **spec-delivery** | 3 | Browse and claim actionable specs from crowd feedback | Yes |
+| **sessions** | 3 | Multi-agent coordination | Yes |
+| **analysis** | 5 | Run full analyses, generate specs from results | API key |
+| **content** | 4 | Ingest content, vector search | API key |
+| **generation** | 2 | PRD generation | API key |
+| **llm** | 2 | Free LLM completion proxy | Yes |
+| **agent-network** | 3 | Register agents, discover capabilities | Mixed |
+
+Plus 8 **workflow packs** (competitive-analysis, content-strategy, market-research, ux-research, and more) that deliver expert methodology when activated.
+
+Full tool reference: **[docs/TOOLS.md](docs/TOOLS.md)**
+
+### Platforms
+
+| Platform | Setup | Notes |
+|---|---|---|
+| Reddit | None | Works immediately |
+| TikTok, Instagram, Xiaohongshu | `npx playwright install chromium` | Browser-based extraction |
+| Twitter/X | `TWITTER_USERNAME` + `TWITTER_PASSWORD` in `.env` | Credential-based |
+| YouTube | `YOUTUBE_API_KEY` in `.env` | API key required |
+| Vision mode (any URL) | Any one of: `ANTHROPIC_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY` | Screenshots + LLM extraction |
+
+### Supported Agents
+
+**Auto-configured on login:** Claude Code, Cursor, Gemini CLI, Codex, Amp, OpenClaw
+
+**Also works with (manual config):** Copilot, Droid, Qwen Code, OpenCode
+
+## CLI
+
+```bash
+npx @crowdlisten/harness login          # Sign in + auto-configure agents
+npx @crowdlisten/harness setup          # Re-run auto-configure
+npx @crowdlisten/harness serve          # Start HTTP server on :3848
+
+npx crowdlisten search reddit "AI agents" --limit 20
+npx crowdlisten vision https://news.ycombinator.com --limit 10
+npx crowdlisten trending reddit --limit 10
+```
+
+## Privacy
+
+- PII redacted locally before LLM calls
+- Memories stored with row-level security — users can only access their own data
+- Local fallback when cloud is unavailable
+- Your own API keys for LLM extraction
+- No data syncs without explicit action
+- MIT open-source and inspectable
+
+## Development
+
+```bash
+git clone https://github.com/Crowdlisten/crowdlisten_harness.git
+cd crowdlisten_harness
+npm install && npm run build
+npm test    # 210+ tests via Vitest
+```
+
+For agent-readable capability descriptions and example workflows, see [AGENTS.md](AGENTS.md).
+
+## Contributing
+
+Highest-value contributions: new platform adapters (Threads, Bluesky, Hacker News, Product Hunt, Mastodon) and extraction fixes.
+
+## License
+
+MIT — [crowdlisten.com](https://crowdlisten.com)

package/dist/agent-proxy.d.ts

@@ -0,0 +1,24 @@
+/**
+ * Agent Proxy — Shared HTTP client for agent.crowdlisten.com
+ *
+ * All agent-proxied tools route through these helpers.
+ * Supports POST, GET, and SSE streaming endpoints.
+ */
+/**
+ * Returns the CROWDLISTEN_API_KEY or throws.
+ * Tools that require auth call this; free tools skip it.
+ */
+export declare function requireApiKey(): string;
+export declare function agentPost(path: string, body: Record<string, unknown>, apiKey?: string): Promise<unknown>;
+export declare function agentGet(path: string, apiKey?: string): Promise<unknown>;
+export declare function agentDelete(path: string, apiKey?: string): Promise<unknown>;
+/**
+ * Connects to an SSE endpoint, collects all events, and returns
+ * the aggregated result. Used for long-running operations like
+ * analysis/run and PRD generation.
+ *
+ * Collects `data:` lines until the stream closes or times out.
+ * If the last event contains a JSON object with a "status" field,
+ * that's treated as the final result. Otherwise returns all events.
+ */
+export declare function agentStream(path: string, body: Record<string, unknown>, apiKey?: string, timeoutMs?: number): Promise<unknown>;

package/dist/agent-proxy.js

@@ -0,0 +1,140 @@
+/**
+ * Agent Proxy — Shared HTTP client for agent.crowdlisten.com
+ *
+ * All agent-proxied tools route through these helpers.
+ * Supports POST, GET, and SSE streaming endpoints.
+ */
+const AGENT_BASE = process.env.CROWDLISTEN_AGENT_URL || "https://agent.crowdlisten.com";
+// ─── Auth ──────────────────────────────────────────────────────────────────
+/**
+ * Returns the CROWDLISTEN_API_KEY or throws.
+ * Tools that require auth call this; free tools skip it.
+ */
+export function requireApiKey() {
+    const key = process.env.CROWDLISTEN_API_KEY;
+    if (!key) {
+        throw new Error("CROWDLISTEN_API_KEY is required for this tool. " +
+            "Get one at https://crowdlisten.com/settings/api-keys");
+    }
+    return key;
+}
+function headers(apiKey) {
+    const h = {
+        "Content-Type": "application/json",
+        Accept: "application/json",
+    };
+    if (apiKey)
+        h["Authorization"] = `Bearer ${apiKey}`;
+    return h;
+}
+// ─── POST ──────────────────────────────────────────────────────────────────
+export async function agentPost(path, body, apiKey) {
+    const url = `${AGENT_BASE}${path}`;
+    const res = await fetch(url, {
+        method: "POST",
+        headers: headers(apiKey),
+        body: JSON.stringify(body),
+    });
+    if (!res.ok) {
+        const text = await res.text().catch(() => "");
+        throw new Error(`Agent API error ${res.status} on POST ${path}: ${text || res.statusText}`);
+    }
+    return res.json();
+}
+// ─── GET ───────────────────────────────────────────────────────────────────
+export async function agentGet(path, apiKey) {
+    const url = `${AGENT_BASE}${path}`;
+    const res = await fetch(url, {
+        method: "GET",
+        headers: headers(apiKey),
+    });
+    if (!res.ok) {
+        const text = await res.text().catch(() => "");
+        throw new Error(`Agent API error ${res.status} on GET ${path}: ${text || res.statusText}`);
+    }
+    return res.json();
+}
+// ─── DELETE ────────────────────────────────────────────────────────────────
+export async function agentDelete(path, apiKey) {
+    const url = `${AGENT_BASE}${path}`;
+    const res = await fetch(url, {
+        method: "DELETE",
+        headers: headers(apiKey),
+    });
+    if (!res.ok) {
+        const text = await res.text().catch(() => "");
+        throw new Error(`Agent API error ${res.status} on DELETE ${path}: ${text || res.statusText}`);
+    }
+    return res.json();
+}
+// ─── SSE Stream → Aggregated Result ────────────────────────────────────────
+/**
+ * Connects to an SSE endpoint, collects all events, and returns
+ * the aggregated result. Used for long-running operations like
+ * analysis/run and PRD generation.
+ *
+ * Collects `data:` lines until the stream closes or times out.
+ * If the last event contains a JSON object with a "status" field,
+ * that's treated as the final result. Otherwise returns all events.
+ */
+export async function agentStream(path, body, apiKey, timeoutMs = 120_000) {
+    const url = `${AGENT_BASE}${path}`;
+    const controller = new AbortController();
+    const timer = setTimeout(() => controller.abort(), timeoutMs);
+    try {
+        const res = await fetch(url, {
+            method: "POST",
+            headers: {
+                ...headers(apiKey),
+                Accept: "text/event-stream",
+            },
+            body: JSON.stringify(body),
+            signal: controller.signal,
+        });
+        if (!res.ok) {
+            const text = await res.text().catch(() => "");
+            throw new Error(`Agent API error ${res.status} on SSE ${path}: ${text || res.statusText}`);
+        }
+        // Read SSE stream
+        const reader = res.body?.getReader();
+        if (!reader)
+            throw new Error("No response body from SSE endpoint");
+        const decoder = new TextDecoder();
+        const events = [];
+        let buffer = "";
+        while (true) {
+            const { done, value } = await reader.read();
+            if (done)
+                break;
+            buffer += decoder.decode(value, { stream: true });
+            const lines = buffer.split("\n");
+            buffer = lines.pop() || "";
+            for (const line of lines) {
+                if (line.startsWith("data: ")) {
+                    const data = line.slice(6).trim();
+                    if (data === "[DONE]")
+                        continue;
+                    try {
+                        events.push(JSON.parse(data));
+                    }
+                    catch {
+                        events.push(data);
+                    }
+                }
+            }
+        }
+        // Return the last event if it looks like a final result, otherwise all events
+        if (events.length === 0)
+            return { status: "completed", events: [] };
+        const last = events[events.length - 1];
+        if (typeof last === "object" &&
+            last !== null &&
+            "status" in last) {
+            return last;
+        }
+        return { status: "completed", events };
+    }
+    finally {
+        clearTimeout(timer);
+    }
+}