reddit-harvest 0.1.0

package/LICENSE

@@ -0,0 +1,23 @@
+MIT License
+
+Copyright (c) 2025 anonrose
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+

package/README.md

@@ -0,0 +1,272 @@
+# reddit-harvest
+
+[![npm version](https://img.shields.io/npm/v/reddit-harvest.svg)](https://www.npmjs.com/package/reddit-harvest)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Node.js Version](https://img.shields.io/node/v/reddit-harvest.svg)](https://nodejs.org)
+
+Harvest subreddit posts and comments into structured corpus files for product research, with advanced filtering, deduplication, OpenAI-powered analysis, and an interactive terminal explorer.
+
+## Features
+
+- 📥 **Harvest** posts from multiple subreddits (hot, new, top, or search)
+- 🔍 **Filter** by score, comments, date range
+- 🔄 **Deduplicate** across runs to avoid re-harvesting
+- 📄 **Export** as plain text or structured JSONL
+- 🤖 **Analyze** with OpenAI to extract pain points, personas, and product opportunities
+- 🧭 **Explore** results interactively in your terminal
+
+---
+
+## Installation
+
+```bash
+npm install -g reddit-harvest
+```
+
+Or with pnpm:
+
+```bash
+pnpm add -g reddit-harvest
+```
+
+---
+
+## Quick Start
+
+### 1. Set up credentials
+
+Create a `.env` file (or copy from `env.example`):
+
+```bash
+# Reddit API (required)
+REDDIT_CLIENT_ID=your_client_id
+REDDIT_CLIENT_SECRET=your_client_secret
+REDDIT_REFRESH_TOKEN=your_refresh_token
+REDDIT_USER_AGENT=reddit-harvest/1.0
+
+# OpenAI (optional, for analysis)
+OPENAI_API_KEY=your_openai_key
+OPENAI_MODEL=gpt-4o-mini
+```
+
+### 2. Harvest posts
+
+```bash
+reddit-harvest harvest --subreddits "startups,Entrepreneur" --limit 50
+```
+
+### 3. Analyze with OpenAI
+
+```bash
+reddit-harvest harvest --subreddits "startups" --limit 50 --analyze
+```
+
+### 4. Explore results
+
+```bash
+reddit-harvest explore --latest
+```
+
+---
+
+## Commands
+
+### `harvest` - Download subreddit content
+
+```bash
+reddit-harvest harvest --subreddits "startups,SaaS" --listing top --time week --limit 100
+```
+
+**Options:**
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--subreddits` | required | Comma-separated list of subreddits |
+| `--listing` | `hot` | `hot`, `new`, or `top` |
+| `--time` | `week` | Time range for top: `hour`, `day`, `week`, `month`, `year`, `all` |
+| `--limit` | `25` | Max posts per subreddit |
+| `--search` | - | Search query (uses Reddit search instead of listing) |
+| `--minScore` | - | Skip posts below this score |
+| `--minComments` | - | Skip posts with fewer comments |
+| `--after` | - | Only posts after this date (ISO format) |
+| `--before` | - | Only posts before this date (ISO format) |
+| `--includeComments` | `false` | Include top-level comments |
+| `--commentLimit` | `50` | Max comments per post |
+| `--format` | `txt` | Output format: `txt` or `jsonl` |
+| `--dedupe` | `false` | Skip previously harvested posts |
+| `--analyze` | `false` | Run OpenAI analysis after harvest |
+| `--quoteFidelity` | `false` | Require supporting quotes for all claims |
+
+### `analyze` - Analyze existing corpus
+
+```bash
+reddit-harvest analyze --input outputs/corpus.jsonl
+```
+
+**Options:**
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--input` | required | Path to corpus file (`.txt` or `.jsonl`) |
+| `--outDir` | `outputs` | Output directory |
+| `--quoteFidelity` | `false` | Require supporting quotes |
+
+### `explore` - Interactive browser
+
+```bash
+reddit-harvest explore --latest
+```
+
+**Options:**
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--dir` | `outputs` | Directory containing analysis files |
+| `--latest` | `false` | Auto-select most recent analysis |
+
+---
+
+## Output Files
+
+After running with `--analyze`, you get:
+
+| File | Description |
+|------|-------------|
+| `<timestamp>-r_<subreddit>.txt` | Raw corpus (or `.jsonl`) |
+| `<timestamp>-analysis.md` | Full research synthesis |
+| `<timestamp>-opportunities.json` | Structured product opportunities |
+
+### Opportunities JSON structure
+
+```json
+[{
+  "id": "opp-1",
+  "title": "Automated customer discovery tool",
+  "targetUser": "Solo founders",
+  "problem": "Spending too much time on manual outreach",
+  "currentWorkaround": "Cold emails and LinkedIn DMs",
+  "proposedSolution": "AI-powered lead qualification",
+  "confidence": "medium",
+  "supportingQuotes": [{ "text": "I spend 4 hours a day...", "permalink": "..." }],
+  "risks": ["Crowded market"],
+  "mvpExperiment": "Landing page with email capture"
+}]
+```
+
+---
+
+## Examples
+
+### Full product research workflow
+
+```bash
+# Harvest with filters and analysis
+reddit-harvest harvest \
+  --subreddits "startups,Entrepreneur,SaaS" \
+  --listing top \
+  --time month \
+  --limit 100 \
+  --minScore 5 \
+  --includeComments \
+  --format jsonl \
+  --dedupe \
+  --analyze \
+  --quoteFidelity
+
+# Explore the results
+reddit-harvest explore --latest
+```
+
+### Daily harvesting with deduplication
+
+```bash
+# First run
+reddit-harvest harvest --subreddits "startups" --limit 100 --dedupe --format jsonl
+
+# Later runs skip already-harvested posts
+reddit-harvest harvest --subreddits "startups" --limit 100 --dedupe --format jsonl
+```
+
+### Search for specific topics
+
+```bash
+reddit-harvest harvest \
+  --subreddits "startups" \
+  --search "finding first customers" \
+  --limit 50 \
+  --analyze
+```
+
+---
+
+## Programmatic Usage
+
+```javascript
+import {
+  createRedditClient,
+  harvestSubredditsToFiles,
+  analyzeCorpus
+} from 'reddit-harvest';
+
+// Harvest
+const reddit = createRedditClient();
+const result = await harvestSubredditsToFiles({
+  reddit,
+  subreddits: ['startups'],
+  outDir: './outputs',
+  limit: 50,
+  format: 'jsonl'
+});
+
+// Analyze
+const analysis = await analyzeCorpus({
+  posts: result.allPosts,
+  subreddits: ['startups'],
+  outDir: './outputs'
+});
+
+console.log(analysis.opportunities);
+```
+
+---
+
+## Reddit API Setup
+
+1. Go to [Reddit Apps](https://www.reddit.com/prefs/apps)
+2. Create a "script" type application
+3. Note your `client_id` and `client_secret`
+4. Generate a refresh token using the OAuth flow
+
+---
+
+## Environment Variables
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `REDDIT_CLIENT_ID` | Yes | Reddit app client ID |
+| `REDDIT_CLIENT_SECRET` | Yes | Reddit app client secret |
+| `REDDIT_REFRESH_TOKEN` | Yes | OAuth refresh token |
+| `REDDIT_USER_AGENT` | Yes | User agent string |
+| `OPENAI_API_KEY` | For analysis | OpenAI API key |
+| `OPENAI_MODEL` | No | Model to use (default: `gpt-4o-mini`) |
+
+---
+
+## Notes
+
+- **Rate limits**: Reddit rate limits API requests. The default delay is 1100ms between requests.
+- **API costs**: OpenAI analysis costs money. Use `--limit` to control corpus size.
+- **PII**: Be careful what you store/share from Reddit content.
+- **Reddit ToS**: Don't use for spam, harassment, or violating Reddit's terms.
+
+---
+
+## Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request.
+
+---
+
+## License
+
+MIT © [anonrose](https://github.com/anonrose)

package/env.example

@@ -0,0 +1,15 @@
+# Reddit credentials (recommended: "installed app" + refresh token)
+REDDIT_USER_AGENT="reddit-analysis/0.1 by your_username"
+REDDIT_CLIENT_ID="your_client_id"
+REDDIT_CLIENT_SECRET="your_client_secret"
+REDDIT_REFRESH_TOKEN="your_refresh_token"
+
+# Optional (only needed for certain auth flows; refresh-token flow preferred)
+# REDDIT_USERNAME="your_reddit_username"
+# REDDIT_PASSWORD="your_reddit_password"
+
+# OpenAI (optional; only needed if using --analyze or `npm run analyze`)
+OPENAI_API_KEY="sk-..."
+OPENAI_MODEL="gpt-4o-mini"
+
+

package/package.json

@@ -0,0 +1,68 @@
+{
+  "name": "reddit-harvest",
+  "version": "0.1.0",
+  "type": "module",
+  "description": "Harvest subreddit posts into structured corpus files for product research, with filtering, deduplication, OpenAI analysis, and interactive exploration.",
+  "license": "MIT",
+  "author": "anonrose",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/anonrose/reddit-harvest.git"
+  },
+  "bugs": {
+    "url": "https://github.com/anonrose/reddit-harvest/issues"
+  },
+  "homepage": "https://github.com/anonrose/reddit-harvest#readme",
+  "keywords": [
+    "reddit",
+    "snoowrap",
+    "openai",
+    "product-research",
+    "research",
+    "cli",
+    "harvest",
+    "scraper",
+    "analysis"
+  ],
+  "bin": {
+    "reddit-harvest": "src/cli.js"
+  },
+  "files": [
+    "src/",
+    "README.md",
+    "LICENSE",
+    "env.example"
+  ],
+  "publishConfig": {
+    "access": "public"
+  },
+  "engines": {
+    "node": ">=18.18.0",
+    "pnpm": ">=9"
+  },
+  "exports": {
+    ".": "./src/index.js"
+  },
+  "dependencies": {
+    "@inquirer/prompts": "^7.2.1",
+    "chalk": "^5.4.1",
+    "dotenv": "^16.4.5",
+    "openai": "^4.73.1",
+    "ora": "^8.2.0",
+    "snoowrap": "^1.23.0",
+    "yargs": "^17.7.2",
+    "zod": "^3.24.1"
+  },
+  "devDependencies": {
+    "vitest": "^2.1.8"
+  },
+  "scripts": {
+    "harvest": "node src/cli.js harvest",
+    "analyze": "node src/cli.js analyze",
+    "explore": "node src/cli.js explore",
+    "harvest:analyze": "node src/cli.js harvest --analyze",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "test:coverage": "vitest run --coverage"
+  }
+}

package/src/cli.js

@@ -0,0 +1,277 @@
+#!/usr/bin/env node
+import path from "node:path";
+import yargs from "yargs/yargs";
+import { hideBin } from "yargs/helpers";
+import fs from "node:fs/promises";
+
+import { loadEnv } from "./env.js";
+import { createLogger } from "./logger.js";
+import { createRedditClient } from "./redditClient.js";
+import { harvestSubredditsToFiles, formatPostsToText } from "./redditHarvest.js";
+import { normalizeSubredditsArg, ensureDir } from "./utils.js";
+import { analyzeCorpus, analyzeFileToMarkdown } from "./openaiAnalyze.js";
+import { createDedupeTracker, resetDedupeIndex } from "./dedupe.js";
+import { runExplorer } from "./explorer.js";
+
+loadEnv({ argv: hideBin(process.argv) });
+
+function exitWithError(err) {
+  // eslint-disable-next-line no-console
+  console.error(err?.stack || err?.message || String(err));
+  process.exit(1);
+}
+
+async function runHarvest(argv) {
+  const logger = createLogger({ verbose: Boolean(argv.verbose) });
+  const subreddits = normalizeSubredditsArg(argv.subreddits);
+  if (subreddits.length === 0) throw new Error(`--subreddits is required (e.g. "startups,Entrepreneur")`);
+
+  const outDir = argv.outDir ? path.resolve(argv.outDir) : path.resolve("outputs");
+  await ensureDir(outDir);
+
+  // Handle --resetDedupe
+  if (argv.resetDedupe) {
+    await resetDedupeIndex(outDir);
+    logger.success("Dedupe index reset");
+  }
+
+  const reddit = createRedditClient({ requestDelayMs: Number(argv.requestDelayMs) || 1100 });
+
+  const listing = argv.listing;
+  const time = argv.time;
+  const limit = Number(argv.limit);
+  const includeComments = Boolean(argv.includeComments);
+  const commentLimit = Number(argv.commentLimit);
+  const commentDepth = Number(argv.commentDepth);
+  const format = argv.format || "txt";
+
+  // Filters
+  const minScore = argv.minScore != null ? Number(argv.minScore) : null;
+  const minComments = argv.minComments != null ? Number(argv.minComments) : null;
+  const after = argv.after || null;
+  const before = argv.before || null;
+  const search = argv.search || null;
+
+  // Dedupe
+  let dedupeIndex = null;
+  if (argv.dedupe) {
+    dedupeIndex = await createDedupeTracker(outDir);
+    if (dedupeIndex.existingCount > 0) {
+      logger.info(`Dedupe index loaded: ${dedupeIndex.existingCount} existing post(s)`);
+    }
+  }
+
+  let activeSpinner = null;
+  let activeSubreddit = null;
+  const spinForSubreddit = (sr, text) => {
+    if (!activeSpinner || activeSubreddit !== sr) {
+      if (activeSpinner) activeSpinner.stop();
+      activeSubreddit = sr;
+      activeSpinner = logger.spinner(text).start();
+    } else {
+      activeSpinner.text = text;
+    }
+  };
+
+  const result = await harvestSubredditsToFiles({
+    reddit,
+    subreddits,
+    outDir,
+    listing,
+    time,
+    limit,
+    search,
+    minScore,
+    minComments,
+    after,
+    before,
+    includeComments,
+    commentLimit,
+    commentDepth,
+    dedupeIndex,
+    format,
+    onProgress: (e) => {
+      if (e.type === "subreddit_start") {
+        const mode = e.search ? `search: "${e.search}"` : `${e.listing}${e.listing === "top" ? `/${e.time}` : ""}`;
+        spinForSubreddit(e.subreddit, `Fetching r/${e.subreddit} (${mode})…`);
+        return;
+      }
+      if (e.type === "posts_fetched") {
+        spinForSubreddit(e.subreddit, `r/${e.subreddit}: fetched ${e.totalPosts} post(s)…`);
+        return;
+      }
+      if (e.type === "posts_filtered") {
+        spinForSubreddit(e.subreddit, `r/${e.subreddit}: ${e.totalPosts} post(s) after filtering…`);
+        return;
+      }
+      if (e.type === "dedupe_skipped") {
+        logger.debug(`Skipped ${e.skipped} duplicate post(s) in r/${e.subreddit}`);
+        return;
+      }
+      if (e.type === "post_progress") {
+        spinForSubreddit(e.subreddit, `r/${e.subreddit}: post ${e.index}/${e.total}${includeComments ? " (+comments)" : ""}`);
+        logger.debug(`post ${e.index}/${e.total}: ${String(e.title || "").slice(0, 120)}`);
+        return;
+      }
+      if (e.type === "comments_expand_start") {
+        spinForSubreddit(e.subreddit, `r/${e.subreddit}: post ${e.index}/${e.total}: loading comments…`);
+        return;
+      }
+      if (e.type === "comments_expand_error") {
+        logger.debug(`comments error on post ${e.index}/${e.total}: ${e.error}`);
+        return;
+      }
+      if (e.type === "file_written") {
+        if (activeSpinner) {
+          activeSpinner.succeed(`r/${e.subreddit}: wrote ${e.filePath} (${e.postCount} posts)`);
+          activeSpinner = null;
+          activeSubreddit = null;
+        } else {
+          logger.success(`r/${e.subreddit}: wrote ${e.filePath} (${e.postCount} posts)`);
+        }
+      }
+    }
+  });
+
+  // Save dedupe index
+  if (dedupeIndex) {
+    await dedupeIndex.save();
+    if (dedupeIndex.newCount > 0) {
+      logger.info(`Dedupe index updated: +${dedupeIndex.newCount} new post(s)`);
+    }
+  }
+
+  const totalPosts = result.outputs.reduce((sum, o) => sum + o.postCount, 0);
+  logger.success(`Wrote ${result.outputs.length} file(s) to ${outDir} (${totalPosts} total posts)`);
+  for (const o of result.outputs) {
+    logger.info(`  r/${o.subreddit}: ${o.filePath} (${o.postCount} posts, ${o.textLength} chars)`);
+  }
+
+  if (argv.analyze) {
+    const analyzeSpinner = logger.spinner("Preparing corpus for analysis…").start();
+
+    // Use structured posts for better analysis
+    const analysisOpts = {
+      posts: result.allPosts,
+      subreddits,
+      quoteFidelity: Boolean(argv.quoteFidelity),
+      outDir,
+      timestamp: result.timestamp,
+      onProgress: (e) => {
+        if (e.type === "subreddit_analysis_start") {
+          analyzeSpinner.text = `OpenAI: analyzing r/${e.subreddit}…`;
+        } else if (e.type === "analyze_chunk_start") {
+          analyzeSpinner.text = `OpenAI: chunk ${e.index}/${e.total}…`;
+        } else if (e.type === "analyze_synthesis_start") {
+          analyzeSpinner.text = "OpenAI: synthesizing…";
+        } else if (e.type === "tagging_start") {
+          analyzeSpinner.text = "OpenAI: extracting tags…";
+        } else if (e.type === "opportunities_start") {
+          analyzeSpinner.text = "OpenAI: generating opportunities…";
+        }
+      }
+    };
+
+    const analysisResult = await analyzeCorpus(analysisOpts);
+
+    analyzeSpinner.succeed(`Analysis complete!`);
+    logger.info(`  Analysis: ${analysisResult.analysisPath}`);
+    if (analysisResult.opportunitiesPath) {
+      logger.info(`  Opportunities: ${analysisResult.opportunitiesPath}`);
+    }
+  }
+}
+
+async function runAnalyze(argv) {
+  const logger = createLogger({ verbose: Boolean(argv.verbose) });
+  if (!argv.input) throw new Error(`--input is required (path to a .txt or .jsonl corpus file)`);
+  const inputPath = path.resolve(argv.input);
+  const outDir = argv.outDir ? path.resolve(argv.outDir) : path.resolve("outputs");
+  const quoteFidelity = Boolean(argv.quoteFidelity);
+
+  const sp = logger.spinner("Analyzing corpus with OpenAI…").start();
+  const result = await analyzeFileToMarkdown({
+    inputPath,
+    outDir,
+    quoteFidelity,
+    onProgress: (e) => {
+      if (e.type === "analyze_chunk_start") {
+        sp.text = `OpenAI: chunk ${e.index}/${e.total}…`;
+      } else if (e.type === "analyze_synthesis_start") {
+        sp.text = "OpenAI: synthesizing…";
+      } else if (e.type === "tagging_start") {
+        sp.text = "OpenAI: extracting tags…";
+      } else if (e.type === "opportunities_start") {
+        sp.text = "OpenAI: generating opportunities…";
+      }
+    }
+  });
+
+  sp.succeed(`Analysis complete!`);
+  logger.info(`  Analysis: ${result.analysisPath}`);
+  if (result.opportunitiesPath) {
+    logger.info(`  Opportunities: ${result.opportunitiesPath}`);
+  }
+}
+
+async function runExplore(argv) {
+  const dir = argv.dir ? path.resolve(argv.dir) : path.resolve("outputs");
+  const latest = Boolean(argv.latest);
+
+  await runExplorer({ dir, latest });
+}
+
+yargs(hideBin(process.argv))
+  .scriptName("reddit-harvest")
+  .command(
+    "harvest",
+    "Download subreddit content and write corpus files",
+    (y) =>
+      y
+        .option("subreddits", { type: "string", demandOption: true, describe: "Comma-separated list, e.g. startups,Entrepreneur" })
+        .option("listing", { choices: ["hot", "new", "top"], default: "hot", describe: "Which listing to pull" })
+        .option("time", { choices: ["hour", "day", "week", "month", "year", "all"], default: "week", describe: "Time range (top only)" })
+        .option("limit", { type: "number", default: 25, describe: "Posts per subreddit" })
+        .option("search", { type: "string", describe: "Search query (uses Reddit search instead of listing)" })
+        .option("minScore", { type: "number", describe: "Skip posts below this score" })
+        .option("minComments", { type: "number", describe: "Skip posts with fewer comments" })
+        .option("after", { type: "string", describe: "Only posts after this date (ISO format)" })
+        .option("before", { type: "string", describe: "Only posts before this date (ISO format)" })
+        .option("includeComments", { type: "boolean", default: false, describe: "Include top-level comments" })
+        .option("commentLimit", { type: "number", default: 50, describe: "Max comments per post (best-effort)" })
+        .option("commentDepth", { type: "number", default: 1, describe: "Reply depth when expanding comments" })
+        .option("outDir", { type: "string", default: "outputs", describe: "Output directory" })
+        .option("format", { choices: ["txt", "jsonl"], default: "txt", describe: "Output format" })
+        .option("dedupe", { type: "boolean", default: false, describe: "Skip previously harvested posts" })
+        .option("resetDedupe", { type: "boolean", default: false, describe: "Clear the dedupe index before harvesting" })
+        .option("requestDelayMs", { type: "number", default: 1100, describe: "Delay between Reddit API requests (ms)" })
+        .option("analyze", { type: "boolean", default: false, describe: "Run OpenAI synthesis after harvesting" })
+        .option("quoteFidelity", { type: "boolean", default: false, describe: "Require supporting quotes for all claims" })
+        .option("verbose", { type: "boolean", default: false, describe: "Verbose debug logging" }),
+    (argv) => runHarvest(argv).catch(exitWithError)
+  )
+  .command(
+    "analyze",
+    "Run OpenAI analysis on an existing corpus file",
+    (y) =>
+      y
+        .option("input", { type: "string", demandOption: true, describe: "Path to a .txt or .jsonl corpus file" })
+        .option("outDir", { type: "string", default: "outputs", describe: "Output directory" })
+        .option("quoteFidelity", { type: "boolean", default: false, describe: "Require supporting quotes for all claims" })
+        .option("verbose", { type: "boolean", default: false, describe: "Verbose debug logging" }),
+    (argv) => runAnalyze(argv).catch(exitWithError)
+  )
+  .command(
+    "explore",
+    "Interactively browse analysis results",
+    (y) =>
+      y
+        .option("dir", { type: "string", default: "outputs", describe: "Directory containing analysis files" })
+        .option("latest", { type: "boolean", default: false, describe: "Auto-select the most recent analysis" }),
+    (argv) => runExplore(argv).catch(exitWithError)
+  )
+  .demandCommand(1)
+  .help()
+  .strict()
+  .parse();
+