devpost-scraper 0.1.0__tar.gz → 0.3.0__tar.gz

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/.env.example

@@ -11,3 +11,8 @@ DEVPOST_SESSION=
 # GitHub personal access token for higher API rate limits (5000/hr vs 60/hr)
 # Generate at https://github.com/settings/tokens (no scopes needed, public data only)
 GITHUB_TOKEN=
+
+# Customer.io Track API credentials (required for --emit-events)
+# Find at https://fly.customer.io/settings/api_credentials
+CUSTOMERIO_SITE_ID=
+CUSTOMERIO_API_KEY=

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/.gitignore

@@ -6,6 +6,8 @@ uv.lock
 start.sh
 .venv/
 __pycache__/
+*.db
 .backboard/
 dist/
-*.egg-info/
+*.egg-info/
+emails/

devpost_scraper-0.3.0/PKG-INFO

@@ -0,0 +1,181 @@
+Metadata-Version: 2.4
+Name: devpost-scraper
+Version: 0.3.0
+Summary: CLI for extracting Devpost data with Backboard tool-calling and exporting results to CSV.
+Requires-Python: >=3.11
+Requires-Dist: backboard-sdk>=1.5.9
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: pydantic>=2.7.0
+Requires-Dist: python-dotenv>=1.0.1
+Description-Content-Type: text/markdown
+
+# Devpost Scraper
+
+CLI toolkit for extracting Devpost hackathon data, enriching participants with emails,
+storing results in SQLite, and emitting Customer.io events.
+
+Three commands:
+
+| Command | Purpose |
+|---|---|
+| `devpost-scraper` | Search Devpost projects by keyword, enrich with emails, export CSV |
+| `devpost-participants` | Scrape a single hackathon's participant list, export CSV |
+| `devpost-harvest` | Walk the hackathon listing, scrape all participants, store in SQLite, emit delta events |
+
+## Requirements
+
+- Python 3.11+
+- [`uv`](https://docs.astral.sh/uv/)
+- A [Backboard](https://app.backboard.io) API key (for `devpost-scraper` only)
+
+## Install
+
+```bash
+uv sync
+```
+
+## Environment
+
+Copy `.env.example` → `.env` and fill in:
+
+| Variable | Required for | Notes |
+|---|---|---|
+| `BACKBOARD_API_KEY` | `devpost-scraper` | Backboard account key |
+| `DEVPOST_ASSISTANT_ID` | auto | Persisted on first run |
+| `DEVPOST_SESSION` | `devpost-participants`, `devpost-harvest` | `_devpost` cookie from browser DevTools |
+| `GITHUB_TOKEN` | optional | GitHub PAT for 5000 req/hr (vs 60). No scopes needed |
+| `CUSTOMERIO_SITE_ID` | `--emit-events` | Customer.io Track API |
+| `CUSTOMERIO_API_KEY` | `--emit-events` | Customer.io Track API |
+
+---
+
+## devpost-scraper
+
+Search Devpost projects by keyword, enrich each with detail page + author email, export CSV.
+
+```bash
+uv run devpost-scraper "ai agents" --output results.csv
+uv run devpost-scraper "climate tech" "developer tools" -o results.csv
+
+# Or via start.sh
+./start.sh "ai agents" --output results.csv
+```
+
+---
+
+## devpost-participants
+
+Scrape a single hackathon's participant list and export to CSV.
+
+```bash
+# First time — pass session cookie
+uv run devpost-participants "https://authorizedtoact.devpost.com/participants" \
+  --jwt "<_devpost cookie value>" -o participants.csv
+
+# Reuse saved session from .env
+uv run devpost-participants "https://authorizedtoact.devpost.com/participants" -o out.csv
+
+# Skip email enrichment
+uv run devpost-participants "https://..." --no-email -o out.csv
+
+# Emit Customer.io events after scrape
+uv run devpost-participants "https://..." --emit-events -o out.csv
+```
+
+---
+
+## devpost-harvest
+
+Automated pipeline: walk the hackathon listing → scrape participants → store in SQLite → emit Customer.io events for delta (new) participants.
+
+### Basic usage
+
+```bash
+# Scrape 3 pages of open hackathons (27 hackathons), enrich new participants, emit events
+uv run devpost-harvest --emit-events
+
+# Fast first run — scrape without email enrichment
+uv run devpost-harvest --no-email
+```
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--pages N` | `3` | Number of hackathon listing pages to fetch (9 per page) |
+| `--hackathons N` | `0` (all) | Only process the first N hackathons from the listing |
+| `--jwt TOKEN` | `.env` | Devpost `_devpost` session cookie |
+| `--db PATH` | `devpost_harvest.db` | SQLite database path |
+| `--status {open,ended,upcoming}` | `open` | Hackathon status filter (repeatable) |
+| `--max-participants N` | `0` (unlimited) | Cap participants scraped per hackathon |
+| `--no-email` | off | Skip email enrichment entirely (even for new participants) |
+| `--emit-events` | off | Emit Customer.io events for unemitted participants during scrape |
+| `--emit-unsent` | off | Skip scraping — just emit events for all unsent participants in DB |
+| `--rescrape` | off | Re-scrape hackathons already scraped in a previous run |
+
+### How it works
+
+```
+Phase 1: Discover hackathons
+  GET /api/hackathons?status[]=open → paginated JSON listing
+
+Phase 2: Per hackathon
+  2a. Fast scan — scrape all participant pages (no enrichment, ~1 req per 20 participants)
+  2b. Upsert into SQLite → detect delta (new participants not previously in DB)
+  2c. Email-enrich delta only — GitHub API + link walking (skipped with --no-email)
+  2d. Emit Customer.io events for unemitted participants (only with --emit-events)
+```
+
+### Delta logic
+
+On subsequent runs, the fast scan re-fetches participant lists but only new participants
+(not previously in SQLite) get the expensive email enrichment. Already-emitted participants
+are never re-emitted. This makes re-runs fast and safe to repeat.
+
+### Common workflows
+
+```bash
+# Initial bulk scrape (no events yet)
+uv run devpost-harvest --pages 5
+
+# Emit all unsent events from the DB (no scraping, no JWT needed)
+uv run devpost-harvest --emit-unsent
+
+# Quick delta check on first hackathon only
+uv run devpost-harvest --hackathons 1 --rescrape --emit-events
+
+# Re-scan all hackathons for new participants, enrich + emit
+uv run devpost-harvest --rescrape --emit-events
+
+# Include ended hackathons
+uv run devpost-harvest --status open --status ended
+
+# Fast delta scan (skip email enrichment for new participants too)
+uv run devpost-harvest --rescrape --no-email
+```
+
+### SQLite schema
+
+The database (`devpost_harvest.db`) has two tables:
+
+- **`hackathons`** — id, url, title, org, state, dates, registrations, prize, themes.
+  `last_scraped_at` is set after participants are scraped.
+- **`participants`** — (hackathon_url, username) primary key, enrichment fields,
+  `first_seen_at`, `last_seen_at`, `event_emitted_at`.
+
+### Customer.io events
+
+Event name: `devpost_hackathon`. Uses participant email as the Customer.io user ID.
+
+Event data: hackathon_url, hackathon_title, username, name, specialty, profile_url, github_url, linkedin_url.
+
+Email templates in `emails/` use `{{customer.first_name}}` and `{{event.*}}` Liquid variables.
+
+---
+
+## Development
+
+```bash
+uv run python -m devpost_scraper.cli "ai agents" --output out.csv
+```

devpost_scraper-0.3.0/README.md

@@ -0,0 +1,169 @@
+# Devpost Scraper
+
+CLI toolkit for extracting Devpost hackathon data, enriching participants with emails,
+storing results in SQLite, and emitting Customer.io events.
+
+Three commands:
+
+| Command | Purpose |
+|---|---|
+| `devpost-scraper` | Search Devpost projects by keyword, enrich with emails, export CSV |
+| `devpost-participants` | Scrape a single hackathon's participant list, export CSV |
+| `devpost-harvest` | Walk the hackathon listing, scrape all participants, store in SQLite, emit delta events |
+
+## Requirements
+
+- Python 3.11+
+- [`uv`](https://docs.astral.sh/uv/)
+- A [Backboard](https://app.backboard.io) API key (for `devpost-scraper` only)
+
+## Install
+
+```bash
+uv sync
+```
+
+## Environment
+
+Copy `.env.example` → `.env` and fill in:
+
+| Variable | Required for | Notes |
+|---|---|---|
+| `BACKBOARD_API_KEY` | `devpost-scraper` | Backboard account key |
+| `DEVPOST_ASSISTANT_ID` | auto | Persisted on first run |
+| `DEVPOST_SESSION` | `devpost-participants`, `devpost-harvest` | `_devpost` cookie from browser DevTools |
+| `GITHUB_TOKEN` | optional | GitHub PAT for 5000 req/hr (vs 60). No scopes needed |
+| `CUSTOMERIO_SITE_ID` | `--emit-events` | Customer.io Track API |
+| `CUSTOMERIO_API_KEY` | `--emit-events` | Customer.io Track API |
+
+---
+
+## devpost-scraper
+
+Search Devpost projects by keyword, enrich each with detail page + author email, export CSV.
+
+```bash
+uv run devpost-scraper "ai agents" --output results.csv
+uv run devpost-scraper "climate tech" "developer tools" -o results.csv
+
+# Or via start.sh
+./start.sh "ai agents" --output results.csv
+```
+
+---
+
+## devpost-participants
+
+Scrape a single hackathon's participant list and export to CSV.
+
+```bash
+# First time — pass session cookie
+uv run devpost-participants "https://authorizedtoact.devpost.com/participants" \
+  --jwt "<_devpost cookie value>" -o participants.csv
+
+# Reuse saved session from .env
+uv run devpost-participants "https://authorizedtoact.devpost.com/participants" -o out.csv
+
+# Skip email enrichment
+uv run devpost-participants "https://..." --no-email -o out.csv
+
+# Emit Customer.io events after scrape
+uv run devpost-participants "https://..." --emit-events -o out.csv
+```
+
+---
+
+## devpost-harvest
+
+Automated pipeline: walk the hackathon listing → scrape participants → store in SQLite → emit Customer.io events for delta (new) participants.
+
+### Basic usage
+
+```bash
+# Scrape 3 pages of open hackathons (27 hackathons), enrich new participants, emit events
+uv run devpost-harvest --emit-events
+
+# Fast first run — scrape without email enrichment
+uv run devpost-harvest --no-email
+```
+
+### Flags
+
+| Flag | Default | Description |
+|---|---|---|
+| `--pages N` | `3` | Number of hackathon listing pages to fetch (9 per page) |
+| `--hackathons N` | `0` (all) | Only process the first N hackathons from the listing |
+| `--jwt TOKEN` | `.env` | Devpost `_devpost` session cookie |
+| `--db PATH` | `devpost_harvest.db` | SQLite database path |
+| `--status {open,ended,upcoming}` | `open` | Hackathon status filter (repeatable) |
+| `--max-participants N` | `0` (unlimited) | Cap participants scraped per hackathon |
+| `--no-email` | off | Skip email enrichment entirely (even for new participants) |
+| `--emit-events` | off | Emit Customer.io events for unemitted participants during scrape |
+| `--emit-unsent` | off | Skip scraping — just emit events for all unsent participants in DB |
+| `--rescrape` | off | Re-scrape hackathons already scraped in a previous run |
+
+### How it works
+
+```
+Phase 1: Discover hackathons
+  GET /api/hackathons?status[]=open → paginated JSON listing
+
+Phase 2: Per hackathon
+  2a. Fast scan — scrape all participant pages (no enrichment, ~1 req per 20 participants)
+  2b. Upsert into SQLite → detect delta (new participants not previously in DB)
+  2c. Email-enrich delta only — GitHub API + link walking (skipped with --no-email)
+  2d. Emit Customer.io events for unemitted participants (only with --emit-events)
+```
+
+### Delta logic
+
+On subsequent runs, the fast scan re-fetches participant lists but only new participants
+(not previously in SQLite) get the expensive email enrichment. Already-emitted participants
+are never re-emitted. This makes re-runs fast and safe to repeat.
+
+### Common workflows
+
+```bash
+# Initial bulk scrape (no events yet)
+uv run devpost-harvest --pages 5
+
+# Emit all unsent events from the DB (no scraping, no JWT needed)
+uv run devpost-harvest --emit-unsent
+
+# Quick delta check on first hackathon only
+uv run devpost-harvest --hackathons 1 --rescrape --emit-events
+
+# Re-scan all hackathons for new participants, enrich + emit
+uv run devpost-harvest --rescrape --emit-events
+
+# Include ended hackathons
+uv run devpost-harvest --status open --status ended
+
+# Fast delta scan (skip email enrichment for new participants too)
+uv run devpost-harvest --rescrape --no-email
+```
+
+### SQLite schema
+
+The database (`devpost_harvest.db`) has two tables:
+
+- **`hackathons`** — id, url, title, org, state, dates, registrations, prize, themes.
+  `last_scraped_at` is set after participants are scraped.
+- **`participants`** — (hackathon_url, username) primary key, enrichment fields,
+  `first_seen_at`, `last_seen_at`, `event_emitted_at`.
+
+### Customer.io events
+
+Event name: `devpost_hackathon`. Uses participant email as the Customer.io user ID.
+
+Event data: hackathon_url, hackathon_title, username, name, specialty, profile_url, github_url, linkedin_url.
+
+Email templates in `emails/` use `{{customer.first_name}}` and `{{event.*}}` Liquid variables.
+
+---
+
+## Development
+
+```bash
+uv run python -m devpost_scraper.cli "ai agents" --output out.csv
+```

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "devpost-scraper"
-version = "0.1.0"
+version = "0.3.0"
 description = "CLI for extracting Devpost data with Backboard tool-calling and exporting results to CSV."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -15,6 +15,7 @@ dependencies = [
 [project.scripts]
 devpost-scraper = "devpost_scraper.cli:main"
 devpost-participants = "devpost_scraper.cli:participants_main"
+devpost-harvest = "devpost_scraper.cli:harvest_main"
 
 [build-system]
 requires = ["hatchling"]

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/src/devpost_scraper/cli.py

@@ -17,13 +17,15 @@ from devpost_scraper.backboard_client import (
     ensure_assistant,
     run_in_thread,
 )
+from devpost_scraper.customerio import emit_hackathon_events
 from devpost_scraper.csv_export import write_projects
-from devpost_scraper.models import DevpostProject, HackathonParticipant
+from devpost_scraper.models import DevpostProject, Hackathon, HackathonParticipant
 from devpost_scraper.scraper import (
     find_author_email,
     find_participant_email,
     get_hackathon_participants,
     get_project_details,
+    list_hackathons,
     search_projects,
 )
 
@@ -228,6 +230,7 @@ async def _run_participants(
     jwt_token: str,
     output: str | None,
     no_email: bool,
+    emit_events: bool = False,
 ) -> None:
     all_participants: list[HackathonParticipant] = []
     page = 1
@@ -303,6 +306,9 @@ async def _run_participants(
         writer.writerows(rows)
         print(buf.getvalue())
 
+    if emit_events:
+        await emit_hackathon_events(all_participants)
+
 
 def participants_main() -> None:
     load_dotenv(_ENV_FILE, override=True)
@@ -334,6 +340,12 @@ def participants_main() -> None:
         default=False,
         help="Skip email enrichment (faster)",
     )
+    parser.add_argument(
+        "--emit-events",
+        action="store_true",
+        default=False,
+        help="Emit devpost_hackathon events to Customer.io (requires CUSTOMERIO_SITE_ID and CUSTOMERIO_API_KEY in .env)",
+    )
     args = parser.parse_args()
 
     if not args.output:
@@ -360,5 +372,289 @@ def participants_main() -> None:
             jwt_token=jwt_token,
             output=args.output,
             no_email=args.no_email,
+            emit_events=args.emit_events,
+        )
+    )
+
+
+# ---------------------------------------------------------------------------
+# devpost-harvest: walk hackathon listing → scrape participants → delta emit
+# ---------------------------------------------------------------------------
+
+async def _run_harvest(
+    pages: int,
+    jwt_token: str,
+    db_path: str,
+    no_email: bool,
+    emit_events: bool,
+    rescrape: bool,
+    max_participants: int = 0,
+    max_hackathons: int = 0,
+    statuses: list[str] | None = None,
+) -> None:
+    from devpost_scraper.db import HarvestDB
+
+    db = HarvestDB(db_path)
+
+    # Phase 1: discover hackathons
+    all_hackathons: list[Hackathon] = []
+    for page in range(1, pages + 1):
+        print(f"[harvest] Fetching hackathon listing page {page}…", file=sys.stderr)
+        data = await list_hackathons(page=page, statuses=statuses)
+        batch = data.get("hackathons", [])
+        if not batch:
+            print(f"[harvest] No hackathons on page {page}, stopping.", file=sys.stderr)
+            break
+        for raw in batch:
+            h = Hackathon(**raw)
+            if h.invite_only:
+                print(f"  [skip] invite-only: {h.title}", file=sys.stderr)
+                continue
+            db.upsert_hackathon(h)
+            all_hackathons.append(h)
+            if max_hackathons and len(all_hackathons) >= max_hackathons:
+                break
+        print(f"[harvest] Page {page}: {len(batch)} hackathons ({len(all_hackathons)} total)", file=sys.stderr)
+        if max_hackathons and len(all_hackathons) >= max_hackathons:
+            break
+
+    if not all_hackathons:
+        print("[harvest] No hackathons found.", file=sys.stderr)
+        db.close()
+        return
+
+    # Phase 2: for each hackathon, scrape participants
+    total_new = 0
+    total_emitted = 0
+
+    for h in all_hackathons:
+        if not rescrape and db.hackathon_scraped(h.url):
+            print(f"  [cached] {h.title} — already scraped, skipping (use --rescrape to force)", file=sys.stderr)
+            continue
+
+        print(f"\n[harvest] {h.title} ({h.url})", file=sys.stderr)
+        print(f"  registrations: {h.registrations_count}, state: {h.open_state}", file=sys.stderr)
+
+        # Phase 2a: fast scan — scrape all participant pages (no enrichment)
+        participants: list[HackathonParticipant] = []
+        ppage = 1
+        while True:
+            try:
+                data = await get_hackathon_participants(h.url, jwt_token, page=ppage)
+            except Exception as exc:
+                print(f"  [warn] participants fetch failed page {ppage}: {exc}", file=sys.stderr)
+                break
+
+            batch = data.get("participants", [])
+            has_more = data.get("has_more", False)
+
+            if not batch:
+                if ppage == 1:
+                    print(f"  [info] No participants found (may need auth)", file=sys.stderr)
+                break
+
+            if max_participants and len(participants) + len(batch) > max_participants:
+                batch = batch[:max_participants - len(participants)]
+                has_more = False
+
+            for raw in batch:
+                participants.append(
+                    HackathonParticipant(
+                        hackathon_url=h.url,
+                        hackathon_title=h.title,
+                        username=raw.get("username", ""),
+                        name=raw.get("name", ""),
+                        specialty=raw.get("specialty", ""),
+                        profile_url=raw.get("profile_url", ""),
+                    )
+                )
+
+            if not has_more:
+                break
+            ppage += 1
+
+        if not participants:
+            db.mark_hackathon_scraped(h.url)
+            continue
+
+        print(f"  [scan] {len(participants)} participants across {ppage} pages", file=sys.stderr)
+
+        # Phase 2b: upsert → detect delta
+        new_participants = db.upsert_participants(participants)
+        total_new += len(new_participants)
+        print(f"  [db] {len(new_participants)} new, {len(participants) - len(new_participants)} existing", file=sys.stderr)
+
+        # Phase 2c: email-enrich only the delta
+        if new_participants and not no_email:
+            print(f"  [enrich] enriching {len(new_participants)} new participants…", file=sys.stderr)
+            for p in new_participants:
+                if not p.profile_url:
+                    continue
+                try:
+                    email_data = await find_participant_email(p.profile_url)
+                    p.email = email_data.get("email", "")
+                    p.github_url = email_data.get("github_url", "")
+                    p.linkedin_url = email_data.get("linkedin_url", "")
+                    if p.email:
+                        print(f"    [email] {p.email} ← {p.username}", file=sys.stderr)
+                    db.update_participant_enrichment(p)
+                except Exception as exc:
+                    print(f"    [warn] enrich failed for {p.username}: {exc}", file=sys.stderr)
+
+        # Phase 2d: emit events for unemitted participants
+        if emit_events:
+            unemitted = db.get_unemitted_participants(h.url)
+            if unemitted:
+                print(f"  [cio] Emitting events for {len(unemitted)} unemitted participants…", file=sys.stderr)
+                await emit_hackathon_events(unemitted)
+                for p in unemitted:
+                    db.mark_event_emitted(h.url, p.username)
+                total_emitted += len(unemitted)
+
+        db.mark_hackathon_scraped(h.url)
+
+    # Summary
+    stats = db.stats()
+    print(f"\n{'=' * 60}", file=sys.stderr)
+    print(f"[harvest] Done.", file=sys.stderr)
+    print(f"  hackathons in db: {stats['hackathons']}", file=sys.stderr)
+    print(f"  participants in db: {stats['participants']}", file=sys.stderr)
+    print(f"  with email: {stats['with_email']}", file=sys.stderr)
+    print(f"  new this run: {total_new}", file=sys.stderr)
+    print(f"  events emitted (total): {stats['events_emitted']}", file=sys.stderr)
+    if total_emitted:
+        print(f"  events emitted (this run): {total_emitted}", file=sys.stderr)
+    print(f"  db: {db_path}", file=sys.stderr)
+    db.close()
+
+
+async def _run_emit_unsent(db_path: str) -> None:
+    from devpost_scraper.db import HarvestDB
+
+    db = HarvestDB(db_path)
+    unemitted = db.all_unemitted_participants()
+
+    if not unemitted:
+        print("[emit-unsent] No unsent participants with emails in DB.", file=sys.stderr)
+        db.close()
+        return
+
+    print(f"[emit-unsent] {len(unemitted)} participants to emit", file=sys.stderr)
+    await emit_hackathon_events(unemitted)
+
+    for p in unemitted:
+        db.mark_event_emitted(p.hackathon_url, p.username)
+
+    stats = db.stats()
+    print(f"\n[emit-unsent] Done. {len(unemitted)} events emitted.", file=sys.stderr)
+    print(f"  events emitted (total): {stats['events_emitted']}", file=sys.stderr)
+    db.close()
+
+
+def harvest_main() -> None:
+    load_dotenv(_ENV_FILE, override=True)
+
+    parser = argparse.ArgumentParser(
+        prog="devpost-harvest",
+        description=(
+            "Walk the Devpost hackathon listing, scrape participants per hackathon, "
+            "store in SQLite, and emit Customer.io events for new (delta) participants."
+        ),
+    )
+    parser.add_argument(
+        "--pages",
+        type=int,
+        default=3,
+        help="Number of hackathon listing pages to fetch (9 hackathons/page, default: 3)",
+    )
+    parser.add_argument(
+        "--jwt",
+        metavar="TOKEN",
+        default=None,
+        help="Value of the _devpost session cookie. Falls back to DEVPOST_SESSION in .env",
+    )
+    parser.add_argument(
+        "--db",
+        metavar="PATH",
+        default="devpost_harvest.db",
+        help="SQLite database path (default: devpost_harvest.db)",
+    )
+    parser.add_argument(
+        "--no-email",
+        action="store_true",
+        default=False,
+        help="Skip email enrichment (much faster)",
+    )
+    parser.add_argument(
+        "--emit-events",
+        action="store_true",
+        default=False,
+        help="Emit Customer.io events for delta participants during scrape",
+    )
+    parser.add_argument(
+        "--emit-unsent",
+        action="store_true",
+        default=False,
+        help="Skip scraping — just emit Customer.io events for all unsent participants in the DB",
+    )
+    parser.add_argument(
+        "--rescrape",
+        action="store_true",
+        default=False,
+        help="Re-scrape hackathons that were already scraped in a previous run",
+    )
+    parser.add_argument(
+        "--max-participants",
+        type=int,
+        default=0,
+        metavar="N",
+        help="Cap participants scraped per hackathon (0 = unlimited, default: 0)",
+    )
+    parser.add_argument(
+        "--hackathons",
+        type=int,
+        default=0,
+        metavar="N",
+        help="Only process the first N hackathons from the listing (0 = all, default: 0)",
+    )
+    parser.add_argument(
+        "--status",
+        action="append",
+        choices=["open", "ended", "upcoming"],
+        default=None,
+        dest="statuses",
+        help="Hackathon status filter (repeatable, default: open). e.g. --status open --status ended",
+    )
+    args = parser.parse_args()
+
+    if args.statuses is None:
+        args.statuses = ["open"]
+
+    if args.emit_unsent:
+        asyncio.run(_run_emit_unsent(db_path=args.db))
+        return
+
+    jwt_token = args.jwt or os.getenv(_PARTICIPANTS_JWT_KEY, "").strip()
+    if not jwt_token:
+        raise SystemExit(
+            "[error] No session cookie. Pass --jwt TOKEN or set DEVPOST_SESSION in .env\n"
+            "  Copy the _devpost cookie value from browser DevTools → Application → Cookies"
+        )
+
+    if args.jwt:
+        _ENV_FILE.touch(exist_ok=True)
+        set_key(str(_ENV_FILE), _PARTICIPANTS_JWT_KEY, args.jwt)
+
+    asyncio.run(
+        _run_harvest(
+            pages=args.pages,
+            jwt_token=jwt_token,
+            db_path=args.db,
+            no_email=args.no_email,
+            emit_events=args.emit_events,
+            rescrape=args.rescrape,
+            max_participants=args.max_participants,
+            max_hackathons=args.hackathons,
+            statuses=args.statuses,
         )
     )

devpost_scraper-0.3.0/src/devpost_scraper/customerio.py

@@ -0,0 +1,87 @@
+from __future__ import annotations
+
+import logging
+import os
+import sys
+
+import httpx
+
+from devpost_scraper.models import DevpostHackathonEvent, HackathonParticipant
+
+logger = logging.getLogger(__name__)
+
+_TRACK_API_URL = "https://track.customer.io/api/v1"
+_EVENT_NAME = "devpost_hackathon"
+
+
+class CustomerIOService:
+    """Async Customer.io Track API client (httpx + basic auth)."""
+
+    def __init__(self, site_id: str, api_key: str) -> None:
+        self._auth = (site_id, api_key)
+
+    async def identify_user(self, user_id: str, email: str, **attrs: str) -> bool:
+        payload = {"email": email, **{k: v for k, v in attrs.items() if v}}
+        url = f"{_TRACK_API_URL}/customers/{user_id}"
+        async with httpx.AsyncClient() as client:
+            resp = await client.put(url, json=payload, auth=self._auth, timeout=10.0)
+        if resp.status_code == 200:
+            return True
+        logger.error("[cio] identify %s → %s %s", user_id, resp.status_code, resp.text)
+        return False
+
+    async def track_event(self, user_id: str, event_name: str, data: dict) -> bool:
+        url = f"{_TRACK_API_URL}/customers/{user_id}/events"
+        payload = {"name": event_name, "data": data}
+        async with httpx.AsyncClient() as client:
+            resp = await client.post(url, json=payload, auth=self._auth, timeout=10.0)
+        if resp.status_code == 200:
+            return True
+        logger.error("[cio] track %s/%s → %s %s", user_id, event_name, resp.status_code, resp.text)
+        return False
+
+
+def _build_service() -> CustomerIOService:
+    site_id = os.getenv("CUSTOMERIO_SITE_ID", "").strip()
+    api_key = os.getenv("CUSTOMERIO_API_KEY", "").strip()
+    if not site_id or not api_key:
+        raise SystemExit(
+            "[error] CUSTOMERIO_SITE_ID and CUSTOMERIO_API_KEY must be set in .env"
+        )
+    return CustomerIOService(site_id, api_key)
+
+
+async def emit_hackathon_events(participants: list[HackathonParticipant]) -> None:
+    eligible = [p for p in participants if p.email]
+    if not eligible:
+        print("[cio] No participants with emails — skipping event emission", file=sys.stderr)
+        return
+
+    svc = _build_service()
+    sent = 0
+
+    for p in eligible:
+        event = DevpostHackathonEvent(
+            hackathon_url=p.hackathon_url,
+            hackathon_title=p.hackathon_title,
+            username=p.username,
+            name=p.name,
+            specialty=p.specialty,
+            profile_url=p.profile_url,
+            github_url=p.github_url,
+            linkedin_url=p.linkedin_url,
+        )
+
+        name_parts = p.name.split(maxsplit=1)
+        first = name_parts[0] if name_parts else ""
+        last = name_parts[1] if len(name_parts) > 1 else ""
+
+        await svc.identify_user(p.email, email=p.email, first_name=first, last_name=last)
+        ok = await svc.track_event(p.email, _EVENT_NAME, event.model_dump())
+        if ok:
+            sent += 1
+            print(f"  [cio] {_EVENT_NAME} → {p.email}", file=sys.stderr)
+        else:
+            print(f"  [cio] FAILED {p.email}", file=sys.stderr)
+
+    print(f"[cio] Emitted {sent}/{len(eligible)} events", file=sys.stderr)

devpost_scraper-0.3.0/src/devpost_scraper/db.py

@@ -0,0 +1,224 @@
+from __future__ import annotations
+
+import sqlite3
+from datetime import datetime, timezone
+from pathlib import Path
+
+from devpost_scraper.models import Hackathon, HackathonParticipant
+
+_DEFAULT_DB = "devpost_harvest.db"
+
+_SCHEMA = """\
+CREATE TABLE IF NOT EXISTS hackathons (
+    id            INTEGER PRIMARY KEY,
+    url           TEXT UNIQUE NOT NULL,
+    title         TEXT,
+    organization_name TEXT,
+    open_state    TEXT,
+    submission_period_dates TEXT,
+    registrations_count INTEGER,
+    prize_amount  TEXT,
+    themes        TEXT,
+    invite_only   INTEGER,
+    first_seen_at TEXT NOT NULL,
+    last_scraped_at TEXT
+);
+
+CREATE TABLE IF NOT EXISTS participants (
+    hackathon_url TEXT NOT NULL,
+    hackathon_title TEXT,
+    username      TEXT NOT NULL,
+    name          TEXT,
+    specialty     TEXT,
+    profile_url   TEXT,
+    github_url    TEXT,
+    linkedin_url  TEXT,
+    email         TEXT,
+    first_seen_at TEXT NOT NULL,
+    last_seen_at  TEXT NOT NULL,
+    event_emitted_at TEXT,
+    PRIMARY KEY (hackathon_url, username)
+);
+"""
+
+
+class HarvestDB:
+    def __init__(self, db_path: str = _DEFAULT_DB) -> None:
+        self._path = Path(db_path)
+        self._conn = sqlite3.connect(str(self._path))
+        self._conn.row_factory = sqlite3.Row
+        self._conn.executescript(_SCHEMA)
+        self._migrate()
+
+    def _migrate(self) -> None:
+        cols = {r[1] for r in self._conn.execute("PRAGMA table_info(participants)").fetchall()}
+        if "hackathon_title" not in cols:
+            self._conn.execute("ALTER TABLE participants ADD COLUMN hackathon_title TEXT")
+            self._conn.commit()
+
+    def close(self) -> None:
+        self._conn.close()
+
+    def upsert_hackathon(self, h: Hackathon) -> None:
+        now = _now_iso()
+        self._conn.execute(
+            """INSERT INTO hackathons
+                   (id, url, title, organization_name, open_state,
+                    submission_period_dates, registrations_count, prize_amount,
+                    themes, invite_only, first_seen_at)
+               VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+               ON CONFLICT(id) DO UPDATE SET
+                   title=excluded.title,
+                   organization_name=excluded.organization_name,
+                   open_state=excluded.open_state,
+                   submission_period_dates=excluded.submission_period_dates,
+                   registrations_count=excluded.registrations_count,
+                   prize_amount=excluded.prize_amount,
+                   themes=excluded.themes,
+                   invite_only=excluded.invite_only
+            """,
+            (
+                h.id, h.url, h.title, h.organization_name, h.open_state,
+                h.submission_period_dates, h.registrations_count, h.prize_amount,
+                h.themes, int(h.invite_only), now,
+            ),
+        )
+        self._conn.commit()
+
+    def upsert_participants(
+        self, participants: list[HackathonParticipant],
+    ) -> list[HackathonParticipant]:
+        """Insert or update participants. Returns only the NEW ones (not previously seen)."""
+        now = _now_iso()
+        new: list[HackathonParticipant] = []
+
+        for p in participants:
+            existing = self._conn.execute(
+                "SELECT 1 FROM participants WHERE hackathon_url=? AND username=?",
+                (p.hackathon_url, p.username),
+            ).fetchone()
+
+            if existing is None:
+                new.append(p)
+                self._conn.execute(
+                    """INSERT INTO participants
+                           (hackathon_url, hackathon_title, username, name, specialty,
+                            profile_url, github_url, linkedin_url, email,
+                            first_seen_at, last_seen_at)
+                       VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                    """,
+                    (
+                        p.hackathon_url, p.hackathon_title, p.username, p.name,
+                        p.specialty, p.profile_url, p.github_url, p.linkedin_url,
+                        p.email, now, now,
+                    ),
+                )
+            else:
+                self._conn.execute(
+                    """UPDATE participants
+                       SET hackathon_title=?, name=?, specialty=?, profile_url=?,
+                           github_url=?, linkedin_url=?, email=?, last_seen_at=?
+                       WHERE hackathon_url=? AND username=?
+                    """,
+                    (
+                        p.hackathon_title, p.name, p.specialty, p.profile_url,
+                        p.github_url, p.linkedin_url, p.email, now,
+                        p.hackathon_url, p.username,
+                    ),
+                )
+
+        self._conn.commit()
+        return new
+
+    def update_participant_enrichment(self, p: HackathonParticipant) -> None:
+        """Update email/github/linkedin fields for an existing participant."""
+        self._conn.execute(
+            """UPDATE participants
+               SET email=?, github_url=?, linkedin_url=?, last_seen_at=?
+               WHERE hackathon_url=? AND username=?""",
+            (p.email, p.github_url, p.linkedin_url, _now_iso(), p.hackathon_url, p.username),
+        )
+        self._conn.commit()
+
+    def mark_event_emitted(self, hackathon_url: str, username: str) -> None:
+        self._conn.execute(
+            "UPDATE participants SET event_emitted_at=? WHERE hackathon_url=? AND username=?",
+            (_now_iso(), hackathon_url, username),
+        )
+        self._conn.commit()
+
+    def get_unemitted_participants(self, hackathon_url: str) -> list[HackathonParticipant]:
+        """Return participants that have an email but haven't had events emitted yet."""
+        rows = self._conn.execute(
+            """SELECT * FROM participants
+               WHERE hackathon_url=? AND email != '' AND event_emitted_at IS NULL""",
+            (hackathon_url,),
+        ).fetchall()
+        return [
+            HackathonParticipant(
+                hackathon_url=r["hackathon_url"],
+                hackathon_title=r["hackathon_title"] or "",
+                username=r["username"],
+                name=r["name"] or "",
+                specialty=r["specialty"] or "",
+                profile_url=r["profile_url"] or "",
+                github_url=r["github_url"] or "",
+                linkedin_url=r["linkedin_url"] or "",
+                email=r["email"] or "",
+            )
+            for r in rows
+        ]
+
+    def all_unemitted_participants(self) -> list[HackathonParticipant]:
+        """Return all participants across all hackathons with email but no event emitted."""
+        rows = self._conn.execute(
+            "SELECT * FROM participants WHERE email != '' AND event_emitted_at IS NULL"
+        ).fetchall()
+        return [
+            HackathonParticipant(
+                hackathon_url=r["hackathon_url"],
+                hackathon_title=r["hackathon_title"] or "",
+                username=r["username"],
+                name=r["name"] or "",
+                specialty=r["specialty"] or "",
+                profile_url=r["profile_url"] or "",
+                github_url=r["github_url"] or "",
+                linkedin_url=r["linkedin_url"] or "",
+                email=r["email"] or "",
+            )
+            for r in rows
+        ]
+
+    def hackathon_scraped(self, hackathon_url: str) -> bool:
+        row = self._conn.execute(
+            "SELECT last_scraped_at FROM hackathons WHERE url=?",
+            (hackathon_url,),
+        ).fetchone()
+        return row is not None and row["last_scraped_at"] is not None
+
+    def mark_hackathon_scraped(self, hackathon_url: str) -> None:
+        self._conn.execute(
+            "UPDATE hackathons SET last_scraped_at=? WHERE url=?",
+            (_now_iso(), hackathon_url),
+        )
+        self._conn.commit()
+
+    def stats(self) -> dict[str, int]:
+        hcount = self._conn.execute("SELECT COUNT(*) FROM hackathons").fetchone()[0]
+        pcount = self._conn.execute("SELECT COUNT(*) FROM participants").fetchone()[0]
+        emitted = self._conn.execute(
+            "SELECT COUNT(*) FROM participants WHERE event_emitted_at IS NOT NULL"
+        ).fetchone()[0]
+        with_email = self._conn.execute(
+            "SELECT COUNT(*) FROM participants WHERE email != ''"
+        ).fetchone()[0]
+        return {
+            "hackathons": hcount,
+            "participants": pcount,
+            "with_email": with_email,
+            "events_emitted": emitted,
+        }
+
+
+def _now_iso() -> str:
+    return datetime.now(timezone.utc).isoformat()

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/src/devpost_scraper/models.py

@@ -3,10 +3,26 @@ from __future__ import annotations
 from pydantic import BaseModel, ConfigDict
 
 
+class Hackathon(BaseModel):
+    model_config = ConfigDict(extra="ignore")
+
+    id: int
+    url: str
+    title: str = ""
+    organization_name: str = ""
+    open_state: str = ""
+    submission_period_dates: str = ""
+    registrations_count: int = 0
+    prize_amount: str = ""
+    themes: str = ""
+    invite_only: bool = False
+
+
 class HackathonParticipant(BaseModel):
     model_config = ConfigDict(extra="ignore")
 
     hackathon_url: str = ""
+    hackathon_title: str = ""
     username: str = ""
     name: str = ""
     specialty: str = ""
@@ -17,7 +33,20 @@ class HackathonParticipant(BaseModel):
 
     @classmethod
     def fieldnames(cls) -> list[str]:
-        return ["hackathon_url", "username", "name", "specialty", "profile_url", "github_url", "linkedin_url", "email"]
+        return ["hackathon_url", "hackathon_title", "username", "name", "specialty", "profile_url", "github_url", "linkedin_url", "email"]
+
+
+class DevpostHackathonEvent(BaseModel):
+    """Payload for the Customer.io ``devpost_hackathon`` event."""
+
+    hackathon_url: str
+    hackathon_title: str
+    username: str
+    name: str
+    specialty: str
+    profile_url: str
+    github_url: str
+    linkedin_url: str
 
 
 class DevpostProject(BaseModel):

{devpost_scraper-0.1.0 → devpost_scraper-0.3.0}/src/devpost_scraper/scraper.py

@@ -23,6 +23,7 @@ _WALKABLE_DOMAINS = {
 }
 
 _SEARCH_URL = "https://devpost.com/software/search"
+_HACKATHONS_API_URL = "https://devpost.com/api/hackathons"
 _GITHUB_API_URL = "https://api.github.com/users"
 
 
@@ -83,6 +84,58 @@ async def search_projects(query: str, page: int = 1) -> dict[str, Any]:
     }
 
 
+async def list_hackathons(
+    page: int = 1,
+    statuses: list[str] | None = None,
+) -> dict[str, Any]:
+    """Fetch one page of hackathons from the Devpost API.
+    Returns {"hackathons": [...], "total_count": int, "per_page": int}.
+    """
+    if statuses is None:
+        statuses = ["open"]
+
+    params: list[tuple[str, str]] = [("page", str(page))]
+    for s in statuses:
+        params.append(("status[]", s))
+
+    async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:
+        resp = await client.get(
+            _HACKATHONS_API_URL,
+            params=params,
+            headers={**_JSON_HEADERS, "Accept": "application/json"},
+        )
+        resp.raise_for_status()
+        data = resp.json()
+
+    hackathons = []
+    for item in data.get("hackathons", []):
+        themes = item.get("themes") or []
+        theme_names = ", ".join(t.get("name", "") for t in themes if t.get("name"))
+
+        prize_raw = item.get("prize_amount", "") or ""
+        prize_clean = re.sub(r"<[^>]+>", "", prize_raw).strip()
+
+        hackathons.append({
+            "id": item.get("id", 0),
+            "title": item.get("title", ""),
+            "url": item.get("url", "").rstrip("/"),
+            "organization_name": item.get("organization_name", ""),
+            "open_state": item.get("open_state", ""),
+            "submission_period_dates": item.get("submission_period_dates", ""),
+            "registrations_count": item.get("registrations_count", 0),
+            "prize_amount": prize_clean,
+            "themes": theme_names,
+            "invite_only": bool(item.get("invite_only")),
+        })
+
+    meta = data.get("meta", {})
+    return {
+        "hackathons": hackathons,
+        "total_count": meta.get("total_count", 0),
+        "per_page": meta.get("per_page", 9),
+    }
+
+
 async def get_project_details(url: str) -> dict[str, Any]:
     """Fetch a Devpost project page and extract detail fields."""
     async with httpx.AsyncClient(timeout=30.0, follow_redirects=True) as client:

devpost_scraper-0.1.0/PKG-INFO

@@ -1,101 +0,0 @@
-Metadata-Version: 2.4
-Name: devpost-scraper
-Version: 0.1.0
-Summary: CLI for extracting Devpost data with Backboard tool-calling and exporting results to CSV.
-Requires-Python: >=3.11
-Requires-Dist: backboard-sdk>=1.5.9
-Requires-Dist: beautifulsoup4>=4.12.0
-Requires-Dist: httpx>=0.27.0
-Requires-Dist: pydantic>=2.7.0
-Requires-Dist: python-dotenv>=1.0.1
-Description-Content-Type: text/markdown
-
-# Devpost Scraper
-
-CLI for extracting Devpost project data with a Backboard assistant that can call a Devpost MCP tool server and export structured results to CSV.
-
-## Requirements
-
-- Python 3.11+
-- `uv`
-- Node.js / `npx` available on your machine
-- A Backboard API key
-
-## Environment
-
-Create a `.env` file from `.env.example` and set:
-
-- `BACKBOARD_API_KEY`
-- `BACKBOARD_MODEL` (optional)
-- `DEVPOST_ASSISTANT_NAME` (optional)
-
-## MCP server
-
-This project is designed to use a Devpost MCP server with this configuration:
-
-```json
-{
-  "mcpServers": {
-    "devpost": {
-      "command": "npx",
-      "args": ["devpost-mcp-server"]
-    }
-  }
-}
-```
-
-## Install
-
-```bash
-uv sync
-```
-
-## Run
-
-```bash
-uv run devpost-scraper "ai agents" --output ai_agents.csv
-uv run devpost-scraper "developer tools" "climate tech" --output results.csv
-```
-
-You can also use the startup script:
-
-```bash
-./start.sh "ai agents" --output ai_agents.csv
-```
-
-## What it does
-
-1. Creates or reuses a Backboard assistant configured for Devpost extraction.
-2. Creates a thread for the run.
-3. Sends a prompt that asks the assistant to use the Devpost MCP toolset.
-4. Handles tool-calling loops until the assistant returns completed structured content.
-5. Parses the structured JSON result.
-6. Writes the extracted rows to CSV.
-
-## Expected output shape
-
-Each extracted row should contain fields like:
-
-- `search_term`
-- `project_title`
-- `tagline`
-- `project_url`
-- `hackathon_name`
-- `hackathon_url`
-- `summary`
-- `built_with`
-- `prizes`
-- `submission_date`
-- `team_size`
-
-## Notes
-
-- The CLI is intentionally API-heavy and UI-free.
-- The Backboard assistant must have access to the Devpost MCP tools in the environment where it runs.
-- If your Backboard account or environment requires additional tool registration, wire that into the assistant creation flow in the client module.
-
-## Development
-
-```bash
-uv run python -m devpost_scraper.cli "ai agents" --output out.csv
-```

devpost_scraper-0.1.0/README.md

@@ -1,89 +0,0 @@
-# Devpost Scraper
-
-CLI for extracting Devpost project data with a Backboard assistant that can call a Devpost MCP tool server and export structured results to CSV.
-
-## Requirements
-
-- Python 3.11+
-- `uv`
-- Node.js / `npx` available on your machine
-- A Backboard API key
-
-## Environment
-
-Create a `.env` file from `.env.example` and set:
-
-- `BACKBOARD_API_KEY`
-- `BACKBOARD_MODEL` (optional)
-- `DEVPOST_ASSISTANT_NAME` (optional)
-
-## MCP server
-
-This project is designed to use a Devpost MCP server with this configuration:
-
-```json
-{
-  "mcpServers": {
-    "devpost": {
-      "command": "npx",
-      "args": ["devpost-mcp-server"]
-    }
-  }
-}
-```
-
-## Install
-
-```bash
-uv sync
-```
-
-## Run
-
-```bash
-uv run devpost-scraper "ai agents" --output ai_agents.csv
-uv run devpost-scraper "developer tools" "climate tech" --output results.csv
-```
-
-You can also use the startup script:
-
-```bash
-./start.sh "ai agents" --output ai_agents.csv
-```
-
-## What it does
-
-1. Creates or reuses a Backboard assistant configured for Devpost extraction.
-2. Creates a thread for the run.
-3. Sends a prompt that asks the assistant to use the Devpost MCP toolset.
-4. Handles tool-calling loops until the assistant returns completed structured content.
-5. Parses the structured JSON result.
-6. Writes the extracted rows to CSV.
-
-## Expected output shape
-
-Each extracted row should contain fields like:
-
-- `search_term`
-- `project_title`
-- `tagline`
-- `project_url`
-- `hackathon_name`
-- `hackathon_url`
-- `summary`
-- `built_with`
-- `prizes`
-- `submission_date`
-- `team_size`
-
-## Notes
-
-- The CLI is intentionally API-heavy and UI-free.
-- The Backboard assistant must have access to the Devpost MCP tools in the environment where it runs.
-- If your Backboard account or environment requires additional tool registration, wire that into the assistant creation flow in the client module.
-
-## Development
-
-```bash
-uv run python -m devpost_scraper.cli "ai agents" --output out.csv
-```