signalforge-cli 0.4.1__tar.gz

signalforge_cli-0.4.1/.env.example

@@ -0,0 +1,18 @@
+# Backboard API key — get yours at https://app.backboard.io
+BACKBOARD_API_KEY=
+
+# Persisted automatically after first run — do not delete
+DEVPOST_ASSISTANT_ID=
+
+# Devpost session cookie (_devpost) for authenticated endpoints (e.g. /participants)
+# Copy the _devpost cookie value from your browser DevTools after logging in
+DEVPOST_SESSION=
+
+# GitHub personal access token for higher API rate limits (5000/hr vs 60/hr)
+# Generate at https://github.com/settings/tokens (no scopes needed, public data only)
+GITHUB_TOKEN=
+
+# Customer.io Track API credentials (required for --emit-events)
+# Find at https://fly.customer.io/settings/api_credentials
+CUSTOMERIO_SITE_ID=
+CUSTOMERIO_API_KEY=

signalforge_cli-0.4.1/.gitignore

@@ -0,0 +1,13 @@
+.env
+*.csv
+.cursor
+backboard*.md
+uv.lock
+start.sh
+.venv/
+__pycache__/
+*.db
+.backboard/
+dist/
+*.egg-info/
+emails/

signalforge_cli-0.4.1/PKG-INFO

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: signalforge-cli
+Version: 0.4.1
+Summary: CLI for extracting emails and sending customer.io events
+Requires-Python: >=3.11
+Requires-Dist: backboard-sdk>=1.5.9
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pydantic>=2.7.0
+Requires-Dist: python-dotenv>=1.0.1
+Description-Content-Type: text/markdown
+
+# SignalForge
+
+```
+   _____ _                   ________                    
+  / ___/(_)___ _____  ____ _/ / ____/___  _________ ____ 
+  \__ \/ / __ `/ __ \/ __ `/ / /_  / __ \/ ___/ __ `/ _ \
+ ___/ / / /_/ / / / / /_/ / / __/ / /_/ / /  / /_/ /  __/
+/____/_/\__, /_/ /_/\__,_/_/_/    \____/_/   \__, /\___/ 
+       /____/                               /____/       
+```
+
+SignalForge is a CLI toolkit for mining developer signals from public sources,
+enriching them with emails, storing results in SQLite, and emitting Customer.io events.
+
+Commands:
+
+| Command | Purpose |
+|---|---|
+| `signalforge` | Search Devpost projects by keyword, enrich with emails, export CSV |
+| `signalforge-participants` | Scrape a single hackathon's participant list, export CSV |
+| `signalforge-harvest` | Walk the hackathon listing, scrape all participants, store in SQLite, emit delta events |
+| `signalforge-github-forks` | Mine GitHub fork owners and optionally enrich with emails |
+| `signalforge-rb2b` | Import RB2B visitor CSVs, store in SQLite, emit `visited_site` events |
+
+## Requirements
+
+- Python 3.11+
+- [`uv`](https://docs.astral.sh/uv/)
+- A [Backboard](https://app.backboard.io) API key (for `signalforge` only)
+
+## Install
+
+```bash
+uv sync
+```
+
+## Environment
+
+Copy `.env.example` → `.env` and fill in:
+
+| Variable | Required for | Notes |
+|---|---|---|
+| `BACKBOARD_API_KEY` | `signalforge` | Backboard account key |
+| `DEVPOST_ASSISTANT_ID` | auto | Persisted on first run |
+| `DEVPOST_SESSION` | `signalforge-participants`, `signalforge-harvest` | `_devpost` cookie from browser DevTools |
+| `GITHUB_TOKEN` | optional | GitHub PAT for 5000 req/hr (vs 60). No scopes needed |
+| `CUSTOMERIO_SITE_ID` | `--emit-events` | Customer.io Track API |
+| `CUSTOMERIO_API_KEY` | `--emit-events` | Customer.io Track API |
+
+---
+
+## signalforge
+
+Search Devpost projects by keyword, enrich each with detail page + author email, export CSV.
+
+```bash
+uv run signalforge "ai agents" --output results.csv
+uv run signalforge "climate tech" "developer tools" -o results.csv
+
+# Or via start.sh
+./start.sh "ai agents" --output results.csv
+```
+
+---
+
+## signalforge-participants
+
+Scrape a single hackathon's participant list and export to CSV.
+
+```bash
+# First time — pass session cookie
+uv run signalforge-participants "https://authorizedtoact.devpost.com/participants" \
+  --jwt "<_devpost cookie value>" -o participants.csv
+
+# Reuse saved session from .env
+uv run signalforge-participants "https://authorizedtoact.devpost.com/participants" -o out.csv
+
+# Skip email enrichment
+uv run signalforge-participants "https://..." --no-email -o out.csv
+
+# Emit Customer.io events after scrape
+uv run signalforge-participants "https://..." --emit-events -o out.csv
+```
+
+---
+
+## signalforge-harvest
+
+Automated pipeline: walk the hackathon listing → scrape participants → store in SQLite → emit Customer.io events for delta (new) participants.
+
+### Basic usage
+
+```bash
+# Scrape 3 pages of open hackathons (27 hackathons), enrich new participants, emit events
+uv run signalforge-harvest --emit-events
+
+# Fast first run — scrape without email enrichment
+uv run signalforge-harvest --no-email
+```
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--pages N` | `3` | Number of hackathon listing pages to fetch (9 per page) |
+| `--hackathons N` | `0` (all) | Only process the first N hackathons from the listing |
+| `--jwt TOKEN` | `.env` | Devpost `_devpost` session cookie |
+| `--db PATH` | `devpost_harvest.db` | SQLite database path |
+| `--status {open,ended,upcoming}` | `open` | Hackathon status filter (repeatable) |
+| `--max-participants N` | `0` (unlimited) | Cap participants scraped per hackathon |
+| `--no-email` | off | Skip email enrichment entirely (even for new participants) |
+| `--emit-events` | off | Emit Customer.io events for unemitted participants during scrape |
+| `--emit-unsent` | off | Skip scraping — just emit events for all unsent participants in DB |
+| `--rescrape` | off | Re-scrape hackathons already scraped in a previous run |
+
+### How it works
+
+```
+Phase 1: Discover hackathons
+  GET /api/hackathons?status[]=open → paginated JSON listing
+
+Phase 2: Per hackathon
+  2a. Fast scan — scrape all participant pages (no enrichment, ~1 req per 20 participants)
+  2b. Upsert into SQLite → detect delta (new participants not previously in DB)
+  2c. Email-enrich delta only — GitHub API + link walking (skipped with --no-email)
+  2d. Emit Customer.io events for unemitted participants (only with --emit-events)
+```
+
+### Delta logic
+
+On subsequent runs, the fast scan re-fetches participant lists but only new participants
+(not previously in SQLite) get the expensive email enrichment. Already-emitted participants
+are never re-emitted. This makes re-runs fast and safe to repeat.
+
+### Common workflows
+
+```bash
+# Initial bulk scrape (no events yet)
+uv run signalforge-harvest --pages 5
+
+# Emit all unsent events from the DB (no scraping, no JWT needed)
+uv run signalforge-harvest --emit-unsent
+
+# Quick delta check on first hackathon only
+uv run signalforge-harvest --hackathons 1 --rescrape --emit-events
+
+# Re-scan all hackathons for new participants, enrich + emit
+uv run signalforge-harvest --rescrape --emit-events
+
+# Include ended hackathons
+uv run signalforge-harvest --status open --status ended
+
+# Fast delta scan (skip email enrichment for new participants too)
+uv run signalforge-harvest --rescrape --no-email
+```
+
+### SQLite schema
+
+The database (`devpost_harvest.db`) has two tables:
+
+- **`hackathons`** — id, url, title, org, state, dates, registrations, prize, themes.
+  `last_scraped_at` is set after participants are scraped.
+- **`participants`** — (hackathon_url, username) primary key, enrichment fields,
+  `first_seen_at`, `last_seen_at`, `event_emitted_at`.
+
+### Customer.io events
+
+Event name: `devpost_hackathon`. Uses participant email as the Customer.io user ID.
+
+Event data: hackathon_url, hackathon_title, username, name, specialty, profile_url, github_url, linkedin_url.
+
+Email templates in `emails/` use `{{customer.first_name}}` and `{{event.*}}` Liquid variables.
+
+---
+
+## signalforge-github-forks
+
+Mine fork owners and enrich with emails (optional), stored in the same SQLite DB
+under a synthetic `hackathon_url` like `github:forks:owner/repo`.
+
+```bash
+# Presets
+uv run signalforge-github-forks --preset mem0 --emit-events
+uv run signalforge-github-forks --preset supermemory --no-email
+
+# Custom repo
+uv run signalforge-github-forks --repo owner/repo --limit 1000 --mode first_n
+```
+
+---
+
+## signalforge-rb2b
+
+Import RB2B visitor exports into SQLite and emit Customer.io `visited_site` events
+for identified visitors.
+
+```bash
+# Import CSV(s) and emit events for newly added identified visitors
+uv run signalforge-rb2b daily_2026-03-*.csv --emit-events
+
+# Emit any unsent identified visitors from the DB
+uv run signalforge-rb2b --emit-unsent
+```
+
+---
+
+## Development
+
+```bash
+uv run python -m devpost_scraper.cli "ai agents" --output out.csv
+```

signalforge_cli-0.4.1/README.md

@@ -0,0 +1,211 @@
+# SignalForge
+
+```
+   _____ _                   ________                    
+  / ___/(_)___ _____  ____ _/ / ____/___  _________ ____ 
+  \__ \/ / __ `/ __ \/ __ `/ / /_  / __ \/ ___/ __ `/ _ \
+ ___/ / / /_/ / / / / /_/ / / __/ / /_/ / /  / /_/ /  __/
+/____/_/\__, /_/ /_/\__,_/_/_/    \____/_/   \__, /\___/ 
+       /____/                               /____/       
+```
+
+SignalForge is a CLI toolkit for mining developer signals from public sources,
+enriching them with emails, storing results in SQLite, and emitting Customer.io events.
+
+Commands:
+
+| Command | Purpose |
+|---|---|
+| `signalforge` | Search Devpost projects by keyword, enrich with emails, export CSV |
+| `signalforge-participants` | Scrape a single hackathon's participant list, export CSV |
+| `signalforge-harvest` | Walk the hackathon listing, scrape all participants, store in SQLite, emit delta events |
+| `signalforge-github-forks` | Mine GitHub fork owners and optionally enrich with emails |
+| `signalforge-rb2b` | Import RB2B visitor CSVs, store in SQLite, emit `visited_site` events |
+
+## Requirements
+
+- Python 3.11+
+- [`uv`](https://docs.astral.sh/uv/)
+- A [Backboard](https://app.backboard.io) API key (for `signalforge` only)
+
+## Install
+
+```bash
+uv sync
+```
+
+## Environment
+
+Copy `.env.example` → `.env` and fill in:
+
+| Variable | Required for | Notes |
+|---|---|---|
+| `BACKBOARD_API_KEY` | `signalforge` | Backboard account key |
+| `DEVPOST_ASSISTANT_ID` | auto | Persisted on first run |
+| `DEVPOST_SESSION` | `signalforge-participants`, `signalforge-harvest` | `_devpost` cookie from browser DevTools |
+| `GITHUB_TOKEN` | optional | GitHub PAT for 5000 req/hr (vs 60). No scopes needed |
+| `CUSTOMERIO_SITE_ID` | `--emit-events` | Customer.io Track API |
+| `CUSTOMERIO_API_KEY` | `--emit-events` | Customer.io Track API |
+
+---
+
+## signalforge
+
+Search Devpost projects by keyword, enrich each with detail page + author email, export CSV.
+
+```bash
+uv run signalforge "ai agents" --output results.csv
+uv run signalforge "climate tech" "developer tools" -o results.csv
+
+# Or via start.sh
+./start.sh "ai agents" --output results.csv
+```
+
+---
+
+## signalforge-participants
+
+Scrape a single hackathon's participant list and export to CSV.
+
+```bash
+# First time — pass session cookie
+uv run signalforge-participants "https://authorizedtoact.devpost.com/participants" \
+  --jwt "<_devpost cookie value>" -o participants.csv
+
+# Reuse saved session from .env
+uv run signalforge-participants "https://authorizedtoact.devpost.com/participants" -o out.csv
+
+# Skip email enrichment
+uv run signalforge-participants "https://..." --no-email -o out.csv
+
+# Emit Customer.io events after scrape
+uv run signalforge-participants "https://..." --emit-events -o out.csv
+```
+
+---
+
+## signalforge-harvest
+
+Automated pipeline: walk the hackathon listing → scrape participants → store in SQLite → emit Customer.io events for delta (new) participants.
+
+### Basic usage
+
+```bash
+# Scrape 3 pages of open hackathons (27 hackathons), enrich new participants, emit events
+uv run signalforge-harvest --emit-events
+
+# Fast first run — scrape without email enrichment
+uv run signalforge-harvest --no-email
+```
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--pages N` | `3` | Number of hackathon listing pages to fetch (9 per page) |
+| `--hackathons N` | `0` (all) | Only process the first N hackathons from the listing |
+| `--jwt TOKEN` | `.env` | Devpost `_devpost` session cookie |
+| `--db PATH` | `devpost_harvest.db` | SQLite database path |
+| `--status {open,ended,upcoming}` | `open` | Hackathon status filter (repeatable) |
+| `--max-participants N` | `0` (unlimited) | Cap participants scraped per hackathon |
+| `--no-email` | off | Skip email enrichment entirely (even for new participants) |
+| `--emit-events` | off | Emit Customer.io events for unemitted participants during scrape |
+| `--emit-unsent` | off | Skip scraping — just emit events for all unsent participants in DB |
+| `--rescrape` | off | Re-scrape hackathons already scraped in a previous run |
+
+### How it works
+
+```
+Phase 1: Discover hackathons
+  GET /api/hackathons?status[]=open → paginated JSON listing
+
+Phase 2: Per hackathon
+  2a. Fast scan — scrape all participant pages (no enrichment, ~1 req per 20 participants)
+  2b. Upsert into SQLite → detect delta (new participants not previously in DB)
+  2c. Email-enrich delta only — GitHub API + link walking (skipped with --no-email)
+  2d. Emit Customer.io events for unemitted participants (only with --emit-events)
+```
+
+### Delta logic
+
+On subsequent runs, the fast scan re-fetches participant lists but only new participants
+(not previously in SQLite) get the expensive email enrichment. Already-emitted participants
+are never re-emitted. This makes re-runs fast and safe to repeat.
+
+### Common workflows
+
+```bash
+# Initial bulk scrape (no events yet)
+uv run signalforge-harvest --pages 5
+
+# Emit all unsent events from the DB (no scraping, no JWT needed)
+uv run signalforge-harvest --emit-unsent
+
+# Quick delta check on first hackathon only
+uv run signalforge-harvest --hackathons 1 --rescrape --emit-events
+
+# Re-scan all hackathons for new participants, enrich + emit
+uv run signalforge-harvest --rescrape --emit-events
+
+# Include ended hackathons
+uv run signalforge-harvest --status open --status ended
+
+# Fast delta scan (skip email enrichment for new participants too)
+uv run signalforge-harvest --rescrape --no-email
+```
+
+### SQLite schema
+
+The database (`devpost_harvest.db`) has two tables:
+
+- **`hackathons`** — id, url, title, org, state, dates, registrations, prize, themes.
+  `last_scraped_at` is set after participants are scraped.
+- **`participants`** — (hackathon_url, username) primary key, enrichment fields,
+  `first_seen_at`, `last_seen_at`, `event_emitted_at`.
+
+### Customer.io events
+
+Event name: `devpost_hackathon`. Uses participant email as the Customer.io user ID.
+
+Event data: hackathon_url, hackathon_title, username, name, specialty, profile_url, github_url, linkedin_url.
+
+Email templates in `emails/` use `{{customer.first_name}}` and `{{event.*}}` Liquid variables.
+
+---
+
+## signalforge-github-forks
+
+Mine fork owners and enrich with emails (optional), stored in the same SQLite DB
+under a synthetic `hackathon_url` like `github:forks:owner/repo`.
+
+```bash
+# Presets
+uv run signalforge-github-forks --preset mem0 --emit-events
+uv run signalforge-github-forks --preset supermemory --no-email
+
+# Custom repo
+uv run signalforge-github-forks --repo owner/repo --limit 1000 --mode first_n
+```
+
+---
+
+## signalforge-rb2b
+
+Import RB2B visitor exports into SQLite and emit Customer.io `visited_site` events
+for identified visitors.
+
+```bash
+# Import CSV(s) and emit events for newly added identified visitors
+uv run signalforge-rb2b daily_2026-03-*.csv --emit-events
+
+# Emit any unsent identified visitors from the DB
+uv run signalforge-rb2b --emit-unsent
+```
+
+---
+
+## Development
+
+```bash
+uv run python -m devpost_scraper.cli "ai agents" --output out.csv
+```

signalforge_cli-0.4.1/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "signalforge-cli"
+version = "0.4.1"
+description = "CLI for extracting emails and sending customer.io events"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+  "backboard-sdk>=1.5.9",
+  "beautifulsoup4>=4.12.0",
+  "httpx>=0.27.0",
+  "pydantic>=2.7.0",
+  "python-dotenv>=1.0.1",
+]
+
+[project.scripts]
+signalforge = "devpost_scraper.cli:main"
+signalforge-participants = "devpost_scraper.cli:participants_main"
+signalforge-harvest = "devpost_scraper.cli:harvest_main"
+signalforge-github-forks = "devpost_scraper.cli:github_forks_main"
+signalforge-rb2b = "devpost_scraper.cli:rb2b_main"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/devpost_scraper"]
+
+[tool.uv]
+package = true

signalforge_cli-0.4.1/src/devpost_scraper/backboard_client.py

@@ -0,0 +1,135 @@
+from __future__ import annotations
+
+import json
+import os
+from typing import Any, Awaitable, Callable, Mapping
+
+from backboard import BackboardClient
+from backboard.exceptions import BackboardAPIError
+
+ToolHandler = Callable[[dict[str, Any]], Awaitable[dict[str, Any]]]
+
+
+class BackboardClientError(Exception):
+    """Raised when a Backboard operation fails."""
+
+
+def build_client() -> BackboardClient:
+    api_key = os.getenv("BACKBOARD_API_KEY", "").strip()
+    if not api_key:
+        raise BackboardClientError(
+            "Missing required environment variable `BACKBOARD_API_KEY`."
+        )
+    return BackboardClient(api_key=api_key)
+
+
+async def ensure_assistant(
+    client: BackboardClient,
+    *,
+    assistant_id: str | None,
+    name: str,
+    system_prompt: str,
+    tools: list[dict[str, Any]],
+) -> str:
+    if assistant_id:
+        return assistant_id
+    assistant = await client.create_assistant(
+        name=name,
+        system_prompt=system_prompt,
+        tools=tools,
+    )
+    return str(assistant.assistant_id)
+
+
+async def _collect_stream(stream: Any) -> dict[str, Any]:
+    """Drain a streaming add_message response into a unified result dict."""
+    content_parts: list[str] = []
+    tool_calls: list[Any] = []
+    run_id: str | None = None
+    status = "completed"
+
+    async for chunk in stream:
+        t = chunk.get("type")
+        if t == "content_streaming":
+            content_parts.append(chunk.get("content", ""))
+        elif t == "tool_submit_required":
+            status = "REQUIRES_ACTION"
+            run_id = chunk.get("run_id")
+            tool_calls = chunk.get("tool_calls", [])
+        elif t == "run_ended":
+            if chunk.get("status") not in (None, "completed"):
+                raise BackboardClientError(
+                    f"Run ended with status: {chunk.get('status')}"
+                )
+
+    return {
+        "content": "".join(content_parts) or None,
+        "status": status,
+        "tool_calls": tool_calls,
+        "run_id": run_id,
+    }
+
+
+async def run_in_thread(
+    client: BackboardClient,
+    *,
+    assistant_id: str,
+    user_message: str,
+    tool_handlers: Mapping[str, ToolHandler],
+    llm_provider: str = "openai",
+    model_name: str = "gpt-4o-mini",
+    max_tool_rounds: int = 6,
+) -> str:
+    """Create a thread, send a message via streaming, execute the tool loop."""
+    thread = await client.create_thread(assistant_id)
+
+    stream = await client.add_message(
+        thread_id=thread.thread_id,
+        content=user_message,
+        stream=True,
+        llm_provider=llm_provider,
+        model_name=model_name,
+    )
+    result = await _collect_stream(stream)
+
+    rounds = 0
+    while result["status"] == "REQUIRES_ACTION":
+        rounds += 1
+        if rounds > max_tool_rounds:
+            raise BackboardClientError(
+                f"Tool loop exceeded {max_tool_rounds} rounds — aborting."
+            )
+        if not result["run_id"]:
+            raise BackboardClientError("REQUIRES_ACTION without run_id.")
+        if not result["tool_calls"]:
+            raise BackboardClientError("REQUIRES_ACTION without tool_calls.")
+
+        tool_outputs = []
+        for tc in result["tool_calls"]:
+            name = tc["function"]["name"] if isinstance(tc, dict) else tc.function.name
+            args_raw = (
+                tc["function"].get("arguments", "{}")
+                if isinstance(tc, dict)
+                else (tc.function.arguments or "{}")
+            )
+            args = args_raw if isinstance(args_raw, dict) else json.loads(args_raw or "{}")
+            tc_id = tc["id"] if isinstance(tc, dict) else tc.id
+
+            handler = tool_handlers.get(name)
+            if handler is None:
+                raise BackboardClientError(f"No handler registered for tool `{name}`.")
+
+            call_result = await handler(args)
+            tool_outputs.append({"tool_call_id": tc_id, "output": json.dumps(call_result)})
+
+        stream = await client.submit_tool_outputs(
+            thread_id=thread.thread_id,
+            run_id=result["run_id"],
+            tool_outputs=tool_outputs,
+            stream=True,
+        )
+        result = await _collect_stream(stream)
+
+    if not result["content"]:
+        raise BackboardClientError("Run completed without content.")
+    return result["content"]