sc-research 1.0.12 → 1.0.14

package/README.md

@@ -1,109 +1,178 @@
-# Social Media Research Skill
+# sc-research
 
-> **"The Skill provides the catch; the Agent cooks the meal."**
+AI-ready social research toolkit for Reddit + X with a template installer for agent workflows.
 
-A **Headless Data Provider** for AI Agents. It researches topics across **Reddit** and **X (Twitter)**, returning raw, normalized, high-engagement discussion data for your Agent to analyze.
+Use it to:
 
-Now includes a **CLI installer** so you can drop the full research workflow into a Claude Code project with a single command.
+- fetch normalized social discussion data,
+- run deep or quick research from CLI,
+- scaffold command/skill templates into agent folders,
+- visualize classified outputs in a local dashboard.
 
-## Features
+## Why This Project
 
-- **Dynamic Source Detection**: Automatically switches between Reddit and X based on available API keys (`OPENAI_API_KEY`, `XAI_API_KEY`).
-- **Unified Data Model**: Normalizes data from different platforms into a single `ResearchItem` schema.
-- **AI-Native Output**: optimized for LLM consumption (JSON).
-- **Engagement-Based Sorting**: Prioritizes content with high community interaction.
-- **Multi-Platform Bootstrap**: Install AI assistant templates into any repo via CLI (`claude`, `cursor`, `windsurf`, `antigravity`).
+`sc-research` is designed for **AI-agent pipelines** where research and analysis are separated:
 
-## Installation (NPM + CLI)
+- **Fetch layer**: collect raw social signals (`reddit_data.json`, `x_data.json`)
+- **Analysis layer**: agent skills produce strict `classified_*.json` files
+- **Presentation layer**: dashboard auto-loads available classified files
+
+This keeps the data pipeline composable and tool-agnostic.
+
+## Install
 
 ```bash
 npm install -g sc-research
 ```
 
-Then, in any project you want to bootstrap:
+Check version:
 
 ```bash
-cd /path/to/your/project
+sc-research --version
+```
 
-# Preview what init would do
-sc-research init --ai claude,cursor --dry-run
+## Quick Start
 
-# Apply one platform
-sc-research init --ai claude
+### 1) Configure API keys
 
-# Apply multiple platforms in one run
-sc-research init --ai claude,cursor
+Create `.sc-research` in your project root:
 
-# Apply all supported platforms
-sc-research init --ai all
+```env
+OPENAI_API_KEY=...
+XAI_API_KEY=...
 ```
 
-This now installs SC-Research templates for selected platform(s) only.
-It does not read, write, or update `package.json`.
+Notes:
+
+- If a key is missing, that source is skipped.
+- You can override env path with `SC_RESEARCH_ENV_FILE`.
+
+### 2) Run research
+
+```bash
+# quick
+sc-research research "best budget iem"
+
+# deep
+sc-research research:deep "best budget iem"
 
-Template architecture:
-- `templates/base/` is the canonical source for command/skill content
-- `templates/platforms/*.json` defines per-platform output rules used during `init`
+# deep discovery mode
+sc-research research:deep "DISCOVERY_WEEKLY" --mode=discovery
+```
 
-## Usage with Claude Code
+### 3) Visualize
 
-1. **Run init once per project**
+```bash
+sc-research visualize
+```
 
-   ```bash
-   sc-research init --ai claude
-   ```
+Dashboard reads available `classified_*.json` files and enables tabs accordingly.
 
-   Then run commands directly from your terminal:
-   - `sc-research research "<topic>"`
-   - `sc-research research:deep "<topic>"`
-   - `sc-research visualize`
+## CLI Commands
 
-2. **Open the project in Claude Code**
+```bash
+sc-research init --ai <targets> [--dry-run] [--force] [--project-root <path>]
+sc-research research "<topic>" [--source=reddit|x|both] [--from=YYYY-MM-DD --to=YYYY-MM-DD] [--mode=discovery]
+sc-research research:deep "<topic>" [same flags as research]
+sc-research visualize [path/to/classified_rank.json]
+```
 
-3. **Use the commands:**
+Supported init targets:
 
-   - `/research <topic>` – Quick research + answer
-   - `/deep-research <topic>` – Deep fetch + best template routing
-   - `/rank` – Rank existing data into `classified_rank.json`
-   - `/sentiment` – Build sentiment report
-   - `/trend` – Build timeline report
-   - `/controversy` – Build controversy map
-   - `/discovery` – Discover viral topics
-   - `/visualize` – Launch dashboard
+- `claude`
+- `cursor`
+- `windsurf`
+- `antigravity`
+- `all`
 
-## Direct Bun CLI (Optional, existing behavior)
+## Agent Template Bootstrap
 
-### CLI
+Install templates into a target project:
 
 ```bash
-# Research a topic
-bun run research "Best budget IEM 2024"
+cd /path/to/your/project
+
+# preview
+sc-research init --ai claude,cursor --dry-run
 
-# Specify output file
-bun run research "Best budget IEM 2024" --out=./my_results.json
+# apply
+sc-research init --ai claude,cursor
 ```
 
-### Programmatic
+Canonical template source:
+
+- `templates/base/`
+
+Platform mapping config:
+
+- `templates/platforms/*.json`
+
+## Data Outputs
+
+### Raw fetch outputs
+
+- `reddit_data.json`
+- `x_data.json`
+
+### Classified analysis outputs
+
+- `classified_rank.json`
+- `classified_sentiment.json`
+- `classified_trend.json`
+- `classified_controversy.json`
+- `classified_discovery.json`
+
+## Discovery Mode (Special Flow)
 
-```typescript
-import { SocialMediaResearch } from 'social-media-research-skill';
+Discovery has dedicated behavior in runtime:
 
-const researcher = new SocialMediaResearch();
-const results = await researcher.search("Best budget IEM 2024");
+- `DISCOVERY_WEEKLY` + discovery mode fetches Reddit weekly trending (`r/popular/top`).
+- Other discovery topics map to candidate subreddits, then fetch subreddit top/search.
+- X still runs through X search flow when X is enabled.
 
-console.log(results);
+Recommended commands:
+
+```bash
+# broad weekly discovery
+sc-research research:deep "DISCOVERY_WEEKLY" --mode=discovery
+
+# topic-focused discovery
+sc-research research:deep "wireless earbuds" --mode=discovery
 ```
 
-## Configuration
+## Repository Structure
+
+```text
+src/
+   core/            # fetch clients + research/data services
+   cli/             # init system, adapters, template rendering
+   entries/         # CLI entrypoints (research, visualize, cli)
+templates/
+   base/            # source-of-truth command + skill templates
+   platforms/       # platform folder mapping config
+web/               # dashboard frontend
+docs/              # usage notes and orchestrator test cases
+```
 
-Set the following environment variables in your `.sc-research` file:
+## Development
 
-```env
-OPENAI_API_KEY=sk-... # Required for Reddit (URL Discovery)
-XAI_API_KEY=...       # Required for X (Twitter)
+```bash
+# run source entrypoints
+bun run research "topic"
+bun run research:deep "topic"
+bun run visualize
+
+# tests
+bun test
+
+# build all distributables
+bun run build
 ```
 
-Optional override: set `SC_RESEARCH_ENV_FILE` to use a different env file path.
+## Notes
+
+- Requires Node.js `>=20` for packaged CLI runtime.
+- Project uses Bun scripts for local development/build.
 
 ## License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sc-research",
-  "version": "1.0.12",
+  "version": "1.0.14",
   "description": "Headless Social Media Research Data Provider for AI Agents",
   "type": "module",
   "main": "dist/index.js",

package/templates/base/commands/controversy.md

@@ -1,28 +1,16 @@
 ---
-description: Find controversial debates in existing research data.
+description: Generate a controversy map from social media data.
 ---
 
-1. Check for data existence
+This command produces a controversy analysis. It follows the orchestrated pipeline with controversy as the target.
 
-    > Ensure `reddit_data.json` or `x_data.json` exists in the current directory.
+1. **Ensure raw data exists**
 
-2. Check freshness for the current request
+   > Check if `reddit_data.json` or `x_data.json` exists. If missing or stale, delegate to the `social_media_fetch` skill to fetch fresh data with deep depth.
 
-    > Confirm raw data matches the requested topic and date window (if provided). If it does not match, re-fetch with **deep research**: `sc-research research:deep "TOPIC"`.
+2. **Run analysis**
 
-3. Run the controversy skill
+   > Read the `social_media_controversy` skill instructions and follow them to generate `classified_controversy.json`.
 
-    > Use the `social_media_controversy` skill to identify polarizing topics, extract opposing quotes, and generate `classified_controversy.json`.
-
-4. Validate output schema
-
-    > Ensure `classified_controversy.json` includes:
-    >
-    > - `topic`
-    > - `overall_divisiveness`
-    > - `controversies` array
-    > - each controversy contains `topic`, `heat_score`, `divisiveness`, `side_a`, and `side_b`
-    > - each side contains `position`, `supporter_count`, and `sample_quote` with `text`, `author`, `link`
-
-5. Display the results
-    > Read `classified_controversy.json` and display controversies with side-by-side opposing views and heat scores.
+3. **Present results**
+   > Display controversies with side-by-side opposing views and heat scores from `classified_controversy.json`.

package/templates/base/commands/deep-research.md

@@ -1,25 +1,14 @@
 ---
-description: Deeply research a topic and route to the best analysis template. This is the default fetch depth for all templates.
+description: Run deep research on a topic and produce a classified analysis report. This command triggers the full orchestrated pipeline.
 ---
 
-1. Run the deep research fetcher
+This command runs the full research pipeline. Follow the `using_social_media_research` orchestrator skill to execute all steps.
 
-    > `sc-research research:deep "ARGUMENTS"` (you may also add `--from=YYYY-MM-DD --to=YYYY-MM-DD` for a custom window)
+**Pipeline overview** (orchestrator handles the details):
 
-2. Route to the right analysis mode
+1. **Resolve intent** — determine which analysis type fits the user's question (rank, sentiment, trend, controversy, discovery, or all).
+2. **Fetch data** — the orchestrator delegates to `social_media_fetch` to run `sc-research research:deep "ARGUMENTS"` (add `--from=YYYY-MM-DD --to=YYYY-MM-DD` if the user specifies a date range).
+3. **Analyze** — the orchestrator delegates to the matched worker skill(s) to produce `classified_*.json` output.
+4. **Present results** — display the analysis summary to the user.
 
-    > Follow **Intent → Template Routing** rules in `using_social_media_research`.
-    >
-    > - If the user explicitly asks for "full analysis", "everything", or "all views": run all 4 templates in order (`rank` -> `sentiment` -> `trend` -> `controversy`).
-    > - Otherwise: pick the SINGLE most suitable template for the user's question.
-
-3. Validate outputs before presenting
-
-    > Confirm the generated `classified_*.json` file(s):
-    >
-    > - Match the requested topic (or close variant of the same topic).
-    > - Match requested date window when `--from/--to` was provided.
-    > - Include required fields from `../skills/social_media_schema/SKILL.md`.
-
-4. Display the results
-    > Present whichever template output(s) were selected after validation.
+Read the `using_social_media_research` skill now and follow its Step 1 through Step 6.

package/templates/base/commands/discovery.md

@@ -1,27 +1,19 @@
 ---
-description: Discover viral topics and emerging themes from existing research data.
+description: Discover viral topics and emerging themes from social media data.
 ---
 
-1. Check for data existence
+This command produces a discovery analysis. It follows the orchestrated pipeline with discovery as the target.
 
-    > Ensure `reddit_data.json` or `x_data.json` exists in the current directory.
+1. **Ensure raw data exists**
 
-2. Check freshness for the current request
+   > Check if `reddit_data.json` or `x_data.json` exists. If missing or stale, delegate to the `social_media_fetch` skill to fetch data with `--mode=discovery`:
+   >
+   > - Broad weekly feed: topic = `DISCOVERY_WEEKLY`
+   > - Topic-focused: topic = user's query
 
-    > Confirm raw data matches the requested topic and date window (if provided). If it does not match, re-fetch with **deep research**: `sc-research research:deep "TOPIC" --mode=discovery`.
+2. **Run analysis**
 
-3. Run the discovery skill
+   > Read the `social_media_discovery` skill instructions and follow them to generate `classified_discovery.json`.
 
-    > Use the `social_media_discovery` skill to cluster posts by topic and generate `classified_discovery.json`.
-
-4. Validate output schema
-
-    > Ensure `classified_discovery.json` includes:
-    >
-    > - `period`
-    > - `total_posts_analyzed`
-    > - `trending_topics` array
-    > - per topic: `id`, `topic_name`, `description`, `category`, `engagement_score`, `sentiment`, `key_posts`
-
-5. Display the results
-    > Read `classified_discovery.json` and display the discovered topics with their engagement scores, sentiment, and top posts.
+3. **Present results**
+   > Display discovered topics with engagement scores, sentiment, and top posts from `classified_discovery.json`.

package/templates/base/commands/quick.md

@@ -2,15 +2,14 @@
 description: Quick research a topic and get a fast text answer (Reddit only, skips X).
 ---
 
-1. Run the quick research fetcher (Reddit only)
+1. **Fetch data (Reddit only)**
 
-   > `sc-research research "ARGUMENTS" --source=reddit`
-   >
-   > This fetches Reddit data only and skips X for speed. Use `/deep-research` if you need deeper multi-source analysis.
+   > Delegate to the `social_media_fetch` skill with: topic = `"ARGUMENTS"`, depth = `quick`, source = `reddit`.
+   > This fetches Reddit data only and skips X for speed. Use `/deep-research` for deeper multi-source analysis.
 
-2. Read the raw data
+2. **Read the raw data**
 
    > Read `reddit_data.json` only. Ignore `x_data.json` even if it exists from a prior run.
 
-3. Provide a concise answer
-   > Synthesize a 3-5 sentence answer directly from the Reddit data. Mention the community favorite, 1-2 alternatives, and a real user quote. Do NOT generate any classified files.
+3. **Provide a concise answer**
+   > Synthesize a 3–5 sentence answer directly from the Reddit data. Mention the community favorite, 1–2 alternatives, and a real user quote. Do NOT generate any classified files.

package/templates/base/commands/rank.md

@@ -1,27 +1,16 @@
 ---
-description: Rank and classify existing research data.
+description: Generate a ranking report from social media data.
 ---
 
-1. Check for data existence
+This command produces a ranked analysis. It follows the orchestrated pipeline with rank as the target.
 
-    > Ensure `reddit_data.json` or `x_data.json` exists in the current directory.
+1. **Ensure raw data exists**
 
-2. Check freshness for the current request
+   > Check if `reddit_data.json` or `x_data.json` exists. If missing or stale, delegate to the `social_media_fetch` skill to fetch fresh data with deep depth.
 
-    > Confirm raw data matches the requested topic and date window (if provided). If it does not match, re-fetch with **deep research**: `sc-research research:deep "TOPIC"`.
+2. **Run analysis**
 
-3. Run the ranking skill
+   > Read the `social_media_rank` skill instructions and follow them to generate `classified_rank.json`.
 
-    > Use the `social_media_rank` skill to generate `classified_rank.json` from the existing data files.
-
-4. Validate output schema
-
-    > Ensure `classified_rank.json` includes:
-    >
-    > - `topic`
-    > - `key_insights` (array)
-    > - `products` (array)
-    > - per product: `rank`, `name`, `sentiment`, `mentions`, `estimated_engagement_score`, `consensus`, `pros`, `cons`
-
-5. Display the results
-    > Read `classified_rank.json` and display key insights plus a summary of the ranked products.
+3. **Present results**
+   > Display key insights and ranked products from `classified_rank.json`.

package/templates/base/commands/research.md

@@ -1,10 +1,16 @@
-1. Run the research fetcher
+---
+description: Research a topic and get a concise text answer.
+---
 
-    > `sc-research research "ARGUMENTS"` (optionally add `--from=YYYY-MM-DD --to=YYYY-MM-DD` to focus on a specific date range)
+1. **Fetch data**
 
-2. Analyze the results
+   > Delegate to the `social_media_fetch` skill with **quick** depth:
+   > Topic = `"ARGUMENTS"`, depth = `quick`.
+   > Optionally add `--from=YYYY-MM-DD --to=YYYY-MM-DD` if user specifies a date range.
 
-    > Read the generated `reddit_data.json` and `x_data.json`.
+2. **Analyze the results**
 
-3. Provide an answer
-    > Based on the data, provide a concise answer including the Community Favorite, decent alternatives, and a representative quote.
+   > Read the generated `reddit_data.json` and `x_data.json`.
+
+3. **Provide an answer**
+   > Based on the data, provide a concise answer including the Community Favorite, decent alternatives, and a representative quote. Do not produce any `classified_*.json` file.

package/templates/base/commands/sentiment.md

@@ -1,28 +1,16 @@
 ---
-description: Analyze sentiment from existing research data.
+description: Generate a sentiment analysis report from social media data.
 ---
 
-1. Check for data existence
+This command produces a sentiment breakdown. It follows the orchestrated pipeline with sentiment as the target.
 
-    > Ensure `reddit_data.json` or `x_data.json` exists in the current directory.
+1. **Ensure raw data exists**
 
-2. Check freshness for the current request
+   > Check if `reddit_data.json` or `x_data.json` exists. If missing or stale, delegate to the `social_media_fetch` skill to fetch fresh data with deep depth.
 
-    > Confirm raw data matches the requested topic and date window (if provided). If it does not match, re-fetch with **deep research**: `sc-research research:deep "TOPIC"`.
+2. **Run analysis**
 
-3. Run the sentiment skill
+   > Read the `social_media_sentiment` skill instructions and follow them to generate `classified_sentiment.json`.
 
-    > Use the `social_media_sentiment` skill to analyze the raw data and generate `classified_sentiment.json`.
-
-4. Validate output schema
-
-    > Ensure `classified_sentiment.json` includes:
-    >
-    > - `topic`
-    > - `overall_mood`
-    > - `distribution` (`very_positive`, `positive`, `mixed`, `negative`)
-    > - `by_source` with both `reddit` and `x`
-    > - `product_sentiments`
-
-5. Display the results
-    > Read `classified_sentiment.json` and display the overall mood, sentiment distribution, and per-product sentiment breakdown.
+3. **Present results**
+   > Display overall mood, sentiment distribution, and per-product breakdown from `classified_sentiment.json`.

package/templates/base/commands/trend.md

@@ -1,27 +1,16 @@
 ---
-description: Analyze discussion trends over time from existing research data.
+description: Generate a trend timeline report from social media data.
 ---
 
-1. Check for data existence
+This command produces a trend analysis. It follows the orchestrated pipeline with trend as the target.
 
-    > Ensure `reddit_data.json` or `x_data.json` exists in the current directory.
+1. **Ensure raw data exists**
 
-2. Check freshness for the current request
+   > Check if `reddit_data.json` or `x_data.json` exists. If missing or stale, delegate to the `social_media_fetch` skill to fetch fresh data with deep depth.
 
-    > Confirm raw data matches the requested topic and date window (if provided). If it does not match, re-fetch with **deep research**: `sc-research research:deep "TOPIC"`.
+2. **Run analysis**
 
-3. Run the trend skill
+   > Read the `social_media_trend` skill instructions and follow them to generate `classified_trend.json`.
 
-    > Use the `social_media_trend` skill to parse dates, choose an adaptive granularity (day/week/month), and generate `classified_trend.json`.
-
-4. Validate output schema
-
-    > Ensure `classified_trend.json` includes:
-    >
-    > - `topic`
-    > - `date_range` with `from` and `to`
-    > - `timeline` entries with `period`, `post_count`, `total_engagement`, `reddit_posts`, `x_posts`
-    > - `key_moments` entries with `date`, `event`, `significance`
-
-5. Display the results
-    > Read `classified_trend.json` and display timeline activity, engagement trends, and key moments.
+3. **Present results**
+   > Display timeline activity, engagement trends, and key moments from `classified_trend.json`.

package/templates/base/commands/visualize.md

@@ -1,14 +1,14 @@
 ---
-description: Launch the visualization dashboard.
+description: Launch the visualization dashboard for classified research data.
 ---
 
-1. Check for classified data
+1. **Check for classified data**
 
-    > Ensure at least one `classified_*.json` file exists (e.g., `classified_rank.json`, `classified_sentiment.json`, `classified_trend.json`, `classified_controversy.json`). If none exist, suggest running `/rank`, `/sentiment`, `/trend`, or `/controversy` first.
+   > Verify at least one `classified_*.json` file exists (`classified_rank.json`, `classified_sentiment.json`, `classified_trend.json`, `classified_controversy.json`, `classified_discovery.json`). If none exist, suggest running `/deep-research` first.
 
-2. Launch the visualizer
+2. **Launch the dashboard**
 
-    > `sc-research visualize`
+   > `sc-research visualize`
 
-3. Inform the user
-    > Tell the user to open the URL provided by the command (usually http://localhost:5173). The dashboard will show tabs for each available classified file.
+3. **Inform the user**
+   > Tell the user to open the URL provided by the command (usually `http://localhost:5173`). The dashboard shows tabs for each available classified file.

package/templates/base/skills/social_media_controversy.md

@@ -1,36 +1,19 @@
 ---
 name: social_media_controversy
-description: Analyze existing Reddit/X raw data to identify divisive topics and generate `classified_controversy.json` with strict `ControversyData` output. Use for debate, disagreement, or polarizing-topic requests.
+description: Analysis-only worker. Reads existing raw Reddit/X data to identify divisive topics and generates `classified_controversy.json` with strict `ControversyData` output. Use for debate, disagreement, or polarizing-topic requests.
 ---
 
 # Social Media Controversy Skill
 
-This worker maps where social media opinions conflict and presents both sides with evidence.
+This worker maps where social media opinions conflict and presents both sides with evidence. It performs **analysis only** — fetching is handled by the orchestrator via `social_media_fetch`.
 
-## Required Inputs
+## Prerequisites
 
-Use existing raw files only:
+The following files must already exist (produced by `social_media_fetch`):
 
-- `reddit_data.json`
-- `x_data.json`
+- `reddit_data.json` and/or `x_data.json`
 
-At least one valid source file must exist.
-
-## Command Execution Flow
-
-Use this sequence for controversy analysis:
-
-1. Fetch or refresh raw data (outside this worker):
-
-- `sc-research research:deep "TOPIC"`
-- Optional filters: `--source=reddit|x|both --from=YYYY-MM-DD --to=YYYY-MM-DD`
-
-2. Run this `social_media_controversy` worker to generate `classified_controversy.json`.
-3. Optional visualization:
-
-- `sc-research visualize`
-
-If raw files are missing, stale, or mismatched for the requested topic/date range, run step 1 first.
+At least one valid source file must be present. If both are missing, **stop and report failure** — do not attempt to fetch data.
 
 ## Step 1: Preflight Validation
 
@@ -94,6 +77,53 @@ Save strict JSON to:
 
 - `classified_controversy.json`
 
+## Output Type Contract
+
+Your output MUST match this exact shape. The dashboard detects controversy data by checking for `overall_divisiveness` (string) + `controversies` (array). Missing either field = broken tab.
+
+```json
+{
+  "topic": "AI replacing developers",
+  "overall_divisiveness": "High",
+  "controversies": [
+    {
+      "topic": "Will AI make junior devs obsolete?",
+      "heat_score": 82,
+      "divisiveness": "High",
+      "side_a": {
+        "position": "AI will eliminate most entry-level coding jobs within 5 years",
+        "supporter_count": 45,
+        "sample_quotes": [
+          {
+            "text": "Why would a company hire juniors when Copilot can do 80% of their work?",
+            "author": "u/startup_cto",
+            "link": "https://reddit.com/r/programming/comments/ghi789"
+          }
+        ]
+      },
+      "side_b": {
+        "position": "AI is a tool that augments developers, not replaces them",
+        "supporter_count": 62,
+        "sample_quotes": [
+          {
+            "text": "Every generation says the same thing. We still need people who understand the problem domain.",
+            "author": "u/senior_eng_20yr",
+            "link": "https://reddit.com/r/ExperiencedDevs/comments/jkl012"
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+
+### Enum Rules for Controversy
+
+- `overall_divisiveness`: **Title Case** — `"Low"`, `"Medium"`, `"High"`. NEVER use `"low"`, `"medium"`, `"high"`.
+- `divisiveness` on each controversy: **Title Case** — same as above.
+- `heat_score`: integer `0-100`. NEVER use a value outside this range.
+- `supporter_count`: integer `>= 0`. Use `0` if unknown, NEVER use null.
+
 ## Final Validation Checklist
 
 - JSON parse succeeds.