sc-research 1.0.3 → 1.0.4

package/CLAUDE.md

@@ -1,4 +1,4 @@
-# Communities Research Skill
+# Social Media Research Skill
 
 > **"The Skill provides the catch; the Agent cooks the meal."**
 
@@ -15,7 +15,7 @@ This project is a headless social media research tool designed for **Claude Code
 This project uses `bun`.
 
 ### Environment Variables
-Required in `.env`:
+Required in `.sc-research`:
 - `OPENAI_API_KEY`: For Reddit URL discovery and general LLM ops
 - `XAI_API_KEY`: (Optional) For X/Twitter data
 
@@ -38,16 +38,16 @@ Required in `.env`:
 
 ## 🏗️ Architecture
 
-- **Orchestrator**: `using_communities_research` (The routing brain)
+- **Orchestrator**: `using_social_media_research` (The routing brain)
 - **Workers**: 
-  - `communities_fetch` (Data provider)
-  - `communities_rank` (Ranking analysis)
-  - `communities_sentiment` (Sentiment analysis)
-  - `communities_trend` (Timeline analysis)
-  - `communities_controversy` (Debate analysis)
-  - `communities_discovery` (Viral topic clustering)
-  - `communities_visualize` (Dashboard launcher)
-  - `communities_research_test` (Legacy/fixed-link test helper)
+  - `social_media_fetch` (Data provider)
+  - `social_media_rank` (Ranking analysis)
+  - `social_media_sentiment` (Sentiment analysis)
+  - `social_media_trend` (Timeline analysis)
+  - `social_media_controversy` (Debate analysis)
+  - `social_media_discovery` (Viral topic clustering)
+  - `social_media_visualize` (Dashboard launcher)
+  - `social_media_research_test` (Legacy/fixed-link test helper)
 
 ## 🧪 Testing Rules
 

package/README.md

@@ -99,13 +99,15 @@ console.log(results);
 
 ## Configuration
 
-Set the following environment variables in your `.env` file:
+Set the following environment variables in your `.sc-research` file:
 
 ```env
 OPENAI_API_KEY=sk-... # Required for Reddit (URL Discovery)
 XAI_API_KEY=...       # Required for X (Twitter)
 ```
 
+Optional override: set `SC_RESEARCH_ENV_FILE` to use a different env file path.
+
 ## License
 
 MIT

package/dist/cli.js

@@ -401,7 +401,7 @@ async function runInit(opts) {
   logger.info(`  Scripts overwritten: ${packageSummary.scriptsOverwritten}`);
   logger.info("");
   logger.info("Next steps:");
-  logger.info("  1) Ensure your project has required env vars set:");
+  logger.info("  1) Ensure your project has required env vars set (for example in .sc-research):");
   logger.info("     - OPENAI_API_KEY");
   logger.info("     - XAI_API_KEY (optional, for X)");
   logger.info("  2) You can now run from this project:");
@@ -492,6 +492,8 @@ function sanitizePackageName(name) {
 }
 
 // src/entries/cli.ts
+var DEFAULT_ENV_FILE = ".sc-research";
+var ENV_FILE_OVERRIDE_VAR = "SC_RESEARCH_ENV_FILE";
 async function main() {
   const args = process3.argv.slice(2);
   if (args.length === 1 && (args[0] === "--version" || args[0] === "-v") || args[0] === "version") {
@@ -627,7 +629,7 @@ async function runPackagedEntry(entryFile, args) {
   const logger = new Logger;
   const entryPath = resolvePackagedEntry(entryFile);
   const resolvedEntryPath = entryPath ?? logger.exitWithError(`Internal error: could not find packaged entry "${entryFile}". Rebuild/reinstall sc-research and retry.`);
-  const dotEnvVars = loadDotEnv(process3.cwd());
+  const dotEnvVars = loadProjectEnv(process3.cwd());
   const result = spawnSync("node", [resolvedEntryPath, ...args], {
     stdio: "inherit",
     env: { ...dotEnvVars, ...process3.env }
@@ -651,8 +653,9 @@ function resolvePackagedEntry(entryFile) {
   }
   return null;
 }
-function loadDotEnv(cwd) {
-  const envPath = path5.join(cwd, ".env");
+function loadProjectEnv(cwd) {
+  const configuredPath = process3.env[ENV_FILE_OVERRIDE_VAR]?.trim();
+  const envPath = configuredPath ? path5.isAbsolute(configuredPath) ? configuredPath : path5.join(cwd, configuredPath) : path5.join(cwd, DEFAULT_ENV_FILE);
   if (!fs5.existsSync(envPath))
     return {};
   const out = {};

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sc-research",
-  "version": "1.0.3",
+  "version": "1.0.4",
   "description": "Headless Social Media Research Data Provider for AI Agents",
   "type": "module",
   "main": "dist/index.js",
@@ -15,11 +15,13 @@
     "sc-research": "dist/cli.js"
   },
   "scripts": {
-    "research": "bun src/entries/research.ts",
-    "research:deep": "bun src/entries/research.ts --depth=deep",
-    "visualize": "bun src/entries/visualize.ts",
-    "test:fixed": "bun src/test/manual-fixed-links.ts",
-    "test:x": "bun src/test/manual-xai.ts",
+    "research": "bun --env-file=.sc-research src/entries/research.ts",
+    "research:deep": "bun --env-file=.sc-research src/entries/research.ts --depth=deep",
+    "init": "node dist/cli.js init",
+    "init:all": "node dist/cli.js init --ai all",
+    "visualize": "bun --env-file=.sc-research src/entries/visualize.ts",
+    "test:fixed": "bun --env-file=.sc-research src/test/manual-fixed-links.ts",
+    "test:x": "bun --env-file=.sc-research src/test/manual-xai.ts",
     "build:index": "bun build src/entries/research.ts --outfile dist/index.js --target node",
     "build:visualize": "bun build src/entries/visualize.ts --outdir dist --target node",
     "build:test:fixed": "bun build src/test/manual-fixed-links.ts --outdir dist/test --target node",

package/templates/base/commands/controversy.md

@@ -12,7 +12,7 @@ description: Find controversial debates in existing research data.
 
 3. Run the controversy skill
 
-    > Use the `communities_controversy` skill to identify polarizing topics, extract opposing quotes, and generate `classified_controversy.json`.
+    > Use the `social_media_controversy` skill to identify polarizing topics, extract opposing quotes, and generate `classified_controversy.json`.
 
 4. Validate output schema
 

package/templates/base/commands/deep-research.md

@@ -8,7 +8,7 @@ description: Deeply research a topic and route to the best analysis template. Th
 
 2. Route to the right analysis mode
 
-    > Follow **Intent → Template Routing** rules in `using_communities_research`.
+    > Follow **Intent → Template Routing** rules in `using_social_media_research`.
     >
     > - If the user explicitly asks for "full analysis", "everything", or "all views": run all 4 templates in order (`rank` -> `sentiment` -> `trend` -> `controversy`).
     > - Otherwise: pick the SINGLE most suitable template for the user's question.

package/templates/base/commands/discovery.md

@@ -12,7 +12,7 @@ description: Discover viral topics and emerging themes from existing research da
 
 3. Run the discovery skill
 
-    > Use the `communities_discovery` skill to cluster posts by topic and generate `classified_discovery.json`.
+    > Use the `social_media_discovery` skill to cluster posts by topic and generate `classified_discovery.json`.
 
 4. Validate output schema
 

package/templates/base/commands/rank.md

@@ -12,7 +12,7 @@ description: Rank and classify existing research data.
 
 3. Run the ranking skill
 
-    > Use the `communities_rank` skill to generate `classified_rank.json` from the existing data files.
+    > Use the `social_media_rank` skill to generate `classified_rank.json` from the existing data files.
 
 4. Validate output schema
 

package/templates/base/commands/sentiment.md

@@ -12,7 +12,7 @@ description: Analyze sentiment from existing research data.
 
 3. Run the sentiment skill
 
-    > Use the `communities_sentiment` skill to analyze the raw data and generate `classified_sentiment.json`.
+    > Use the `social_media_sentiment` skill to analyze the raw data and generate `classified_sentiment.json`.
 
 4. Validate output schema
 

package/templates/base/commands/trend.md

@@ -12,7 +12,7 @@ description: Analyze discussion trends over time from existing research data.
 
 3. Run the trend skill
 
-    > Use the `communities_trend` skill to parse dates, choose an adaptive granularity (day/week/month), and generate `classified_trend.json`.
+    > Use the `social_media_trend` skill to parse dates, choose an adaptive granularity (day/week/month), and generate `classified_trend.json`.
 
 4. Validate output schema
 

package/templates/base/manifest.json

@@ -1,65 +1,65 @@
 {
-    "version": 1,
-    "templates": [
-        {
-            "id": "communities_fetch",
-            "kind": "skill",
-            "description": "Worker skill that fetches raw discussion data from Reddit and X (Twitter) for a given topic. Returns raw JSON files.",
-            "bodyFile": "skills/communities_fetch.md"
-        },
-    {
-      "id": "communities_rank",
+  "version": 1,
+  "templates": [
+    {
+      "id": "social_media_fetch",
+      "kind": "skill",
+      "description": "Worker skill that fetches raw discussion data from Reddit and X (Twitter) for a given topic. Returns raw JSON files.",
+      "bodyFile": "skills/social_media_fetch.md"
+    },
+    {
+      "id": "social_media_rank",
       "kind": "skill",
       "description": "Analyze raw social media data (Reddit/X) to produce a ranked, classified report with strict JSON output.",
-      "bodyFile": "skills/communities_rank.md"
+      "bodyFile": "skills/social_media_rank.md"
     },
     {
-      "id": "communities_sentiment",
+      "id": "social_media_sentiment",
       "kind": "skill",
       "description": "Worker skill that analyzes raw social media data to produce a sentiment breakdown report with strict JSON output.",
-      "bodyFile": "skills/communities_sentiment.md"
+      "bodyFile": "skills/social_media_sentiment.md"
     },
     {
-      "id": "communities_trend",
+      "id": "social_media_trend",
       "kind": "skill",
       "description": "Worker skill that analyzes raw social media data to produce a trend timeline report with strict JSON output.",
-      "bodyFile": "skills/communities_trend.md"
+      "bodyFile": "skills/social_media_trend.md"
     },
     {
-      "id": "communities_controversy",
+      "id": "social_media_controversy",
       "kind": "skill",
       "description": "Worker skill that analyzes raw social media data to identify polarizing topics and produce a controversy map with strict JSON output.",
-      "bodyFile": "skills/communities_controversy.md"
+      "bodyFile": "skills/social_media_controversy.md"
     },
     {
-      "id": "communities_discovery",
+      "id": "social_media_discovery",
       "kind": "skill",
       "description": "Worker skill that analyzes raw social media data to discover and cluster high-signal emerging topics.",
-      "bodyFile": "skills/communities_discovery.md"
+      "bodyFile": "skills/social_media_discovery.md"
     },
     {
-      "id": "communities_visualize",
+      "id": "social_media_visualize",
       "kind": "skill",
       "description": "Worker skill that launches a local web dashboard to visualize all available classified research data.",
-      "bodyFile": "skills/communities_visualize.md"
+      "bodyFile": "skills/social_media_visualize.md"
     },
     {
-      "id": "communities_research_test",
+      "id": "social_media_research_test",
       "kind": "skill",
-      "description": "Test the communities research skill with fixed Reddit links (no API key needed). Fetches data and returns JSON for AI classification.",
-      "bodyFile": "skills/communities_research_test.md"
+      "description": "Test the social media research skill with fixed Reddit links (no API key needed). Fetches data and returns JSON for AI classification.",
+      "bodyFile": "skills/social_media_research_test.md"
     },
     {
-      "id": "using_communities_research",
+      "id": "using_social_media_research",
       "kind": "skill",
-      "description": "Orchestrator skill that understands user intent and routes to the most suitable analysis template.",
-      "bodyFile": "skills/using_communities_research.md"
+      "description": "Entrypoint router for social-media research questions. Use when user asks what Reddit/X users think (best/top, compare, sentiment, trend, controversy, discovery, quick summary, full analysis) and route to the correct worker + fetch strategy.",
+      "bodyFile": "skills/using_social_media_research.md"
     },
-        {
-            "id": "research",
-            "kind": "command",
-            "description": "Research a topic using the Quick Answer flow.",
-            "bodyFile": "commands/research.md"
+    {
+      "id": "research",
+      "kind": "command",
+      "description": "Research a topic using the Quick Answer flow.",
+      "bodyFile": "commands/research.md"
     },
     {
       "id": "quick",
@@ -114,6 +114,6 @@
       "kind": "command",
       "description": "Run fixed-link test research pipeline.",
       "bodyFile": "commands/test-research.md"
-        }
-    ]
+    }
+  ]
 }

package/templates/base/skills/social_media_controversy.md

@@ -0,0 +1,94 @@
+---
+name: social_media_controversy
+description: Analyze existing Reddit/X raw data to identify divisive topics and generate `classified_controversy.json` with strict `ControversyData` output. Use for debate, disagreement, or polarizing-topic requests.
+---
+
+# Social Media Controversy Skill
+
+This worker maps where social media opinions conflict and presents both sides with evidence.
+
+## Required Inputs
+
+Use existing raw files only:
+
+- `reddit_data.json`
+- `x_data.json`
+
+At least one valid source file must exist.
+
+## Step 1: Preflight Validation
+
+1. Parse each available source file.
+2. Confirm top-level `items` arrays.
+3. Skip malformed records and track skipped count.
+4. Stop if no usable discussion text remains.
+
+## Step 2: Lock Output Schema
+
+Read `web/src/types.ts` and treat `ControversyData` as source of truth.
+
+Required top-level fields:
+
+- `topic`
+- `overall_divisiveness`
+- `controversies`
+
+## Step 3: Identify Genuine Controversies
+
+Find 2-5 topics with clear opposing viewpoints.
+
+Good signals:
+
+- explicit disagreement language
+- conflicting claims about the same product/theme
+- platform splits (Reddit vs X)
+- high-engagement threads arguing opposite positions
+
+Do not manufacture controversy where consensus is strong.
+
+## Step 4: Structure Each Controversy
+
+For each controversy:
+
+- `topic`: concise debate label
+- `heat_score`: `0-100` intensity score based on volume + engagement + disagreement strength
+- `divisiveness`: `"Low" | "Medium" | "High"`
+- `side_a` and `side_b` each with:
+  - `position`
+  - `supporter_count` (best estimate; `0` if unknown)
+  - `sample_quotes` (up to 3 real quotes)
+
+Quote rules:
+
+- Use real text, real author, real link.
+- Prefer diverse arguments per side (not duplicates).
+- If limited evidence exists, include fewer quotes instead of fabricating.
+
+## Step 5: Set Overall Divisiveness
+
+Set `overall_divisiveness` from the full set:
+
+- `"High"` if multiple high-heat, strongly split controversies exist
+- `"Medium"` for mixed or moderate splits
+- `"Low"` if disagreements are minor or sparse
+
+## Step 6: Write Output
+
+Save strict JSON to:
+
+- `classified_controversy.json`
+
+## Final Validation Checklist
+
+- JSON parse succeeds.
+- Object matches `ControversyData` shape.
+- Enum values use allowed casing exactly.
+- Array fields are arrays (never null/undefined).
+- Every quote references real source evidence.
+
+## Critical Rules
+
+1. **No external fetch**: analyze existing data only.
+2. **No fabricated arguments**: only report controversies present in source text.
+3. **No fabricated citations**: quote text, author, and link must be real.
+4. **Schema strictness**: if instructions conflict with `types.ts`, `types.ts` wins.

package/templates/base/skills/social_media_discovery.md

@@ -0,0 +1,91 @@
+---
+name: social_media_discovery
+description: Analyze existing Reddit/X raw data to find emerging or viral themes and generate `classified_discovery.json` with strict `DiscoveryData` output.
+---
+
+# Social Media Discovery Skill
+
+This worker clusters noisy social discussions into trend themes that can be visualized.
+
+## Required Inputs
+
+Use existing raw files only:
+
+- `reddit_data.json`
+- `x_data.json`
+
+At least one valid source file must exist.
+
+## Step 1: Preflight Validation
+
+1. Parse each available source file.
+2. Confirm top-level `items` arrays.
+3. Skip malformed records and track skipped count.
+4. Stop if no usable posts remain.
+
+## Step 2: Lock Schema
+
+Read `web/src/types.ts` and follow `DiscoveryData` and `DiscoveryTopic` exactly.
+
+Important enum constraints:
+
+- `sentiment`: `"positive" | "negative" | "neutral" | "mixed"`
+- `platform`: `"reddit" | "x"`
+
+## Step 3: Cluster Topics
+
+Create 3-8 meaningful topic clusters from actual post content.
+
+- Merge near-duplicate themes.
+- Ignore clear spam/noise.
+- Prefer clusters with both relevance and engagement.
+
+Each cluster should have:
+
+- `id` (stable slug-like id)
+- `topic_name`
+- `description`
+- `category`
+
+## Step 4: Compute Topic Metrics
+
+For each cluster:
+
+- `engagement_score` = sum of per-item engagement where item engagement =
+  `max(score, likes, upvotes, 0) + comments + shares`
+- `sentiment` using allowed discovery sentiment enum
+
+Also compute top-level:
+
+- `total_posts_analyzed`
+- `period` (always string, derived from available date range)
+
+## Step 5: Attach Evidence Content
+
+For each topic include:
+
+- `key_posts` (high-signal posts, preferably 1-3 entries)
+- `highlight_comments` (up to 3 real excerpts with author/link/platform)
+
+Use real source text only. If evidence is limited, include fewer entries instead of fabricating.
+
+## Step 6: Write Output
+
+Save strict JSON to:
+
+- `classified_discovery.json`
+
+## Final Validation Checklist
+
+- JSON parse succeeds.
+- Output matches `DiscoveryData` shape.
+- All enum values are valid and correctly cased.
+- Array fields are arrays (never null/undefined).
+- Every quote/comment/link is traceable to raw input data.
+
+## Critical Rules
+
+1. **No external fetch**: do not run data collection here.
+2. **No fabricated clusters or quotes**: everything must map to real evidence.
+3. **Schema strictness**: `types.ts` is authoritative.
+4. **Graceful fallback**: use empty arrays for missing optional evidence; never invent content.

package/templates/base/skills/social_media_fetch.md

@@ -0,0 +1,82 @@
+---
+name: social_media_fetch
+description: Worker skill that fetches raw discussion data from Reddit and X into `reddit_data.json` and `x_data.json`. Use before running rank, sentiment, trend, controversy, or discovery analysis.
+---
+
+# Social Media Fetch Skill
+
+This worker is the data-ingestion step for the pipeline. It fetches raw social data only and does not classify or analyze it.
+
+## Inputs and Outputs
+
+- **Input**: User topic and optional filters (`source`, `from/to`, `mode`)
+- **Output files** (project root):
+  - `reddit_data.json`
+  - `x_data.json`
+
+At least one output file must be produced for a successful fetch.
+
+## Step 1: Choose Fetch Mode
+
+- **Quick mode** (faster, lighter coverage):
+  - `sc-research research "TOPIC"`
+- **Deep mode** (default for analysis workflows):
+  - `sc-research research:deep "TOPIC"`
+- **Discovery mode** (theme clustering data):
+  - `sc-research research:deep "TOPIC" --mode=discovery`
+
+## Step 2: Build Command with Optional Flags
+
+Use flags only when requested:
+
+- `--source=reddit|x|both`
+- `--from=YYYY-MM-DD --to=YYYY-MM-DD`
+- `--mode=discovery`
+
+Examples:
+
+```bash
+sc-research research:deep "wireless earbuds"
+sc-research research:deep "wireless earbuds" --source=reddit --from=2025-01-01 --to=2025-12-31
+sc-research research:deep "wireless earbuds" --mode=discovery --source=both
+```
+
+## Step 3: Validate Fetch Results
+
+After running the command, verify each produced source file:
+
+1. File exists.
+2. JSON is parseable.
+3. Top-level `items` exists and is an array.
+4. `query` and `dateRange` are present.
+5. Items include usable fields (`text`, `author`, `url`, `date`, `engagement`).
+
+If a source was explicitly requested but its file is missing or malformed, report the failure clearly.
+
+## Step 4: Return a Fetch Summary
+
+Return:
+
+- topic
+- selected mode
+- selected sources
+- date range used
+- item count per source
+- any missing-source or partial-result warnings
+
+## Critical Rules
+
+1. **No analysis here**: do not rank/classify in this skill.
+2. **No fabricated data**: do not create synthetic posts to fill gaps.
+3. **Prefer deep mode for analysis pipelines**: quick mode is for explicit quick-answer requests.
+4. **Fail loudly on malformed output**: do not continue as if fetch succeeded when validation fails.
+
+## Error Handling
+
+| Scenario | Symptom | Action |
+|---|---|---|
+| Missing `OPENAI_API_KEY` | Auth failure on Reddit fetch | Set valid `OPENAI_API_KEY` in `.sc-research` |
+| Missing `XAI_API_KEY` | X file missing/empty while Reddit succeeds | Set `XAI_API_KEY` in `.sc-research` or continue Reddit-only |
+| No relevant results | `items` is empty | Broaden topic keywords and retry |
+| Rate limit / transient API failure | Timeout or provider error | Wait, then retry once with same parameters |
+| Malformed output | JSON parse failure or missing `items` | Re-run fetch; if repeated, report failure explicitly |

package/templates/base/skills/social_media_rank.md

@@ -0,0 +1,94 @@
+---
+name: social_media_rank
+description: Analyze existing Reddit/X raw data and generate `classified_rank.json` using the strict `ClassifiedData` schema. Use for ranking, best-of, compare, or recommendation requests.
+---
+
+# Social Media Ranking Skill
+
+This worker converts raw discussion data into a ranked report suitable for the dashboard.
+
+## Required Inputs
+
+Use existing files only:
+
+- `reddit_data.json`
+- `x_data.json`
+
+At least one valid source file must exist.
+
+## Step 1: Preflight Validation
+
+Before analysis:
+
+1. Confirm at least one input file exists.
+2. Parse each existing file as JSON.
+3. Confirm top-level `items` is an array.
+4. Ignore malformed items, but keep count of dropped items for transparency.
+
+If both sources are missing or invalid, stop and report the failure.
+
+## Step 2: Lock Output Schema
+
+Read `web/src/types.ts` and treat it as source of truth.
+
+- Output type must match `ClassifiedData`.
+- Product entries must match `Product`.
+- If this file and `types.ts` conflict, `types.ts` wins.
+
+## Step 3: Build Product Candidates
+
+From raw text, identify products/topics that are actually discussed.
+
+- Prefer candidates with repeated mentions across multiple posts.
+- Merge obvious aliases into a single canonical product name.
+- Keep the final ranked set to 1-5 products.
+
+## Step 4: Rank with Consistent Signals
+
+For each candidate product:
+
+- **mentions**: count relevant references across items.
+- **estimated_engagement_score**: sum per-item engagement where item engagement = `max(score, likes, upvotes, 0) + comments + shares`.
+- **sentiment**: one of `"Very Positive" | "Positive" | "Mixed" | "Negative"`.
+- **consensus**: short evidence-based summary sentence.
+- **pros/cons**: distilled from real discussion content.
+
+Sort by strongest combined community support (mentions + engagement + sentiment quality), then assign contiguous ranks starting at 1.
+
+## Step 5: Extract Highlight Quotes
+
+For each ranked product, include up to 3 real quotes:
+
+- Quote must come from raw data text.
+- Include real `author` and real `link`.
+- Set `context` to `"pro"`, `"con"`, or `"general"`.
+- Prefer a balanced set of contexts when evidence exists.
+
+Do not fabricate quotes. If fewer than 3 valid quotes exist, include fewer.
+
+## Step 6: Write Output
+
+Save final JSON to:
+
+- `classified_rank.json`
+
+Required top-level fields:
+
+- `topic`
+- `products`
+- `key_insights`
+
+## Final Validation Checklist
+
+- JSON is parseable.
+- `products` is an array with rank values `1..N` and no duplicates.
+- Every product includes all required fields from `Product`.
+- `key_insights` is non-empty and based on observed evidence.
+- No null for array fields (`pros`, `cons`, `highlight_quotes`).
+
+## Critical Rules
+
+1. **No external fetch**: do not run new research in this skill.
+2. **Schema strictness**: dashboard expects `ClassifiedData` shape exactly.
+3. **Evidence-first ranking**: never rank based on assumptions alone.
+4. **No fabricated citations**: quotes, authors, and links must be real.