vlr-stats-scraper 1.4.1 → 2.0.0

package/README.md

@@ -1,170 +1,326 @@
-# vlr-stats-scraper
-
-Scrapes player stats from [vlr.gg](https://www.vlr.gg) and stores them in a local SQLite
-database, grouped by tournament or stage, so you can pull a backlog of players.
-
-## Setup
-
-Requires Node.js 18+ and a [Hyperbrowser](https://hyperbrowser.ai) API key for scraping.
-
-```bash
-npm install vlr-stats-scraper
-```
-
-Create a `.env` file in the directory you run it from:
-
-```
-HYPERBROWSER_API_KEY=your_key_here
-```
-
-## CLI usage
-
-The package installs a `valstats` command.
-
-| Command | What it does |
-| --- | --- |
-| `valstats <link> <year> <event-name>` | Scrape every player on an event page and save them under that event |
-| `valstats --clear-event <event-name>` | Remove one event and its players |
-| `valstats --clear` | Wipe the whole database (asks for confirmation) |
-
-### Scraping an event
-
-```bash
-npx valstats <link> <year> <event-name>
-```
-
-```bash
-npx valstats https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage 2026 "Stage 2"
-```
-
-- `link` — the vlr.gg event/group-stage page to scrape.
-- `year` — the year the event took place.
-- `event-name` — a label for this tournament/stage (used to group players together later).
-
-What happens:
-
-1. If the event name contains the word **"stage"** (case-insensitive), you're prompted to
-   enter a region (e.g. `AMER`, `EMEA`). That region is saved on the event and on every
-   player scraped. Otherwise region is left empty.
-2. Every player profile linked on the event page is collected. Stats are pulled using
-   vlr.gg's **all-time** timespan.
-3. Each player is scraped, saved to `players.db` in your current directory, linked to the
-   event, and printed to the console.
-
-Running it again with an event name and year that are already saved replaces that event's
-players with the freshly scraped ones. A player already in the database is updated with the
-new data rather than duplicated.
-
-### Clearing data
-
-```bash
-npx valstats --clear-event "Stage 2"   # remove one event and its players
-npx valstats --clear                   # wipe the whole database (type "yes" to confirm)
-```
-
-Players that also belong to another event are kept when a single event is cleared.
-
-## What gets scraped
-
-Each player is stored as an object like this:
-
-```js
-{
-  name: 'tenz',                         // from the profile URL
-  team: 'Sentinels',                    // current team
-  past_teams: ['Cloud9', ...],          // previous teams
-  winnings: '$123,456',                 // total event winnings
-  most_played_agents: ['Jett', ...],    // top 3 agents by usage
-  avg_kda: 1.23,                        // average K:D across those top 3 agents
-  nationality: 'Canada',
-  region: 'AMER',                       // from the CLI prompt / argument, or null
-  stats: [                              // full agent stat rows for the top 3 agents,
-    { Agent: 'Jett', 'K:D': '1.25', ... } //   keyed by vlr.gg's column headers
-  ]
-}
-```
-
-## Library usage
-
-```js
-import {
-  scrapePlayer,
-  scrapeWebsite,
-  savePlayer,
-  saveEvent,
-  addPlayerToEvent,
-  getPlayersByEvent,
-  getRandomPlayerFromEvent,
-} from 'vlr-stats-scraper';
-
-const eventId = saveEvent({ name: 'Stage 2', region: 'AMER', year: 2026 });
-
-const links = await scrapeWebsite('https://www.vlr.gg/event/.../group-stage');
-for (const link of links) {
-  const player = await scrapePlayer(link, 'AMER');
-  if (player) {
-    savePlayer(player);
-    addPlayerToEvent(eventId, player.name);
-  }
-}
-
-const roster = getPlayersByEvent('Stage 2');
-const pick = getRandomPlayerFromEvent('Stage 2');
-```
-
-### Exports
-
-#### Scraping
-
-| Function | Returns | Description |
-| --- | --- | --- |
-| `scrapeWebsite(url)` | `Promise<string[]>` | Scrapes an event page and returns every player profile link on it (with `?timespan=all`). |
-| `scrapePlayer(url, region)` | `Promise<Player>` | Scrapes one player's profile page into a player object (see above). `region` is stored as-is. Returns `undefined` if scraping fails. |
-
-#### Players
-
-| Function | Returns | Description |
-| --- | --- | --- |
-| `savePlayer(player)` | — | Inserts a player, or updates them if one with the same name exists. |
-| `getPlayer(name)` | `Player \| null` | Looks up a player by name. |
-| `getAllPlayers()` | `Player[]` | Every player in the database. |
-| `getRandomPlayer()` | `Player \| null` | A random player from the whole database. |
-
-#### Events (tournaments / stages)
-
-| Function | Returns | Description |
-| --- | --- | --- |
-| `saveEvent({ name, url, region, year, stage })` | `number` | Creates or updates an event by name and returns its id. Only `name` is required. |
-| `addPlayerToEvent(eventId, playerName)` | — | Links a saved player to an event (ignored if already linked). |
-| `getEvents()` | `Event[]` | Every saved event. |
-| `getEventByName(name)` | `Event \| undefined` | Looks up an event by name. |
-| `getPlayersByEvent(eventName)` | `Player[]` | All players linked to an event. |
-| `getRandomPlayerFromEvent(eventName)` | `Player \| null` | A random player from one event. |
-
-#### Clearing
-
-| Function | Returns | Description |
-| --- | --- | --- |
-| `emptyEvent(eventName)` | `boolean` | Removes an event's players but keeps the event so it can be re-scraped. `false` if the event doesn't exist. |
-| `clearEvent(eventName)` | `boolean` | Removes an event and its players. `false` if the event doesn't exist. |
-| `clearDatabase()` | — | Deletes every event and player. |
-
-Players that still belong to another event are never deleted by `emptyEvent` or
-`clearEvent`.
-
-## Data storage
-
-Player and event data is stored in a `players.db` SQLite file created in whichever
-directory you run the tool from. It has three tables:
-
-- `players` — one row per player (`past_teams`, `most_played_agents` and `stats` are stored
-  as JSON and parsed back automatically when read).
-- `events` — one row per tournament/stage (`name`, `url`, `region`, `year`, `stage`).
-- `event_players` — links players to the events they were scraped from.
-
-Databases created by older versions (with `tournaments` / `tournament_players` tables) are
-migrated automatically.
-
-## License
-
-ISC
+# vlr-stats-scraper
+
+Scrapes player stats from [vlr.gg](https://www.vlr.gg) and stores them in a local SQLite
+database, grouped by tournament or stage, so you can pull a backlog of players.
+
+## Setup
+
+Requires Node.js 18+ and a [Hyperbrowser](https://hyperbrowser.ai) API key for scraping.
+
+```bash
+npm install vlr-stats-scraper
+```
+
+Create a `.env` file in the directory you run it from:
+
+```
+HYPERBROWSER_API_KEY=your_key_here
+```
+
+## CLI usage
+
+The package installs a `valstats` command.
+
+| Command | What it does |
+| --- | --- |
+| `valstats <link> <year> <event-name>` | Scrape every player on an event page and save them under that event |
+| `valstats --search <name \| link>` | Print the stats of every saved player whose name starts with `name`, or of one player scraped live from a link |
+| `valstats --player <link> <event-name> <year> [region]` | Scrape one player and add them to an event that's already saved |
+| `valstats --clear-event <event-name>` | Remove one event and its players |
+| `valstats --clear` | Wipe the whole database (asks for confirmation) |
+
+### Scraping an event
+
+```bash
+npx valstats <link> <year> <event-name>
+```
+
+```bash
+npx valstats https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage 2026 "Stage 2"
+```
+
+- `link` — the vlr.gg event/group-stage page to scrape.
+- `year` — the year the event took place.
+- `event-name` — a label for this tournament/stage (used to group players together later).
+
+What happens:
+
+1. The event page is scraped for its title and every player profile linked on it.
+2. The region is read from the event title and saved on every player (not on the event):
+
+   | Title contains | Region |
+   | --- | --- |
+   | `America` (e.g. "VCT 2026: Americas Stage 2") | `AMER` |
+   | `EMEA` | `EMEA` |
+   | `Pacific` | `PAC` |
+   | `China` | `CN` |
+   | none of these (e.g. Masters, Champions) | `null` |
+
+3. Each player is scraped using vlr.gg's **all-time** timespan, saved to `players.db` in your
+   current directory under that event, and printed to the console.
+
+If the event page can't be scraped, nothing is saved or deleted.
+
+Running it again with an event name and year that are already saved replaces that event's
+players with the freshly scraped ones. A player who is already saved under a different event
+gets a new, separate row for this event, so their stats from each event are kept apart (see
+[Data storage](#data-storage)).
+
+### Viewing a player's stats
+
+```bash
+npx valstats --search tenz                                    # every saved player with "tenz" in their name
+npx valstats --search ten                                     # matches names starting with "ten": tenz, tenten, ... (not "kitten")
+npx valstats --search https://www.vlr.gg/player/9/tenz        # scrape a player live without saving
+```
+
+- `--search <name>` searches `players.db` and prints every player whose name starts with
+  those characters (case-insensitive). Names are the ones from the vlr.gg profile URL,
+  e.g. `tenz`. Nothing is scraped. A player saved under several events is printed once per
+  event, labelled with the event name and year, e.g. `tenz (Champs, 2026)`.
+- `--search <link>` scrapes that profile live using the all-time timespan. The result is
+  printed but not saved to the database.
+
+For each player, their team, past teams, nationality, region, winnings, top agents and
+average K:D are printed, followed by a table of their stats on their top 3 agents.
+
+### Adding a player to an event by hand
+
+```bash
+npx valstats --player <link> <event-name> <year> [region]
+```
+
+```bash
+npx valstats --player https://www.vlr.gg/player/9/tenz "Stage 2" 2026        # region copied from the event's other players
+npx valstats --player https://www.vlr.gg/player/9/tenz "Stage 2" 2026 AMER   # region set explicitly
+```
+
+Scrapes one player and saves them under an event that's already in `players.db`, e.g. a player
+the event page missed. The event is checked before scraping, so a mistyped name or year fails
+straight away without using a scrape. If you leave out `region`, the player gets the same
+region as the players already saved under that event.
+
+### Clearing data
+
+```bash
+npx valstats --clear-event "Stage 2"   # remove one event and its players
+npx valstats --clear                   # wipe the whole database (type "yes" to confirm)
+```
+
+Clearing one event only removes that event's player rows. The same player's saves under other
+events are kept.
+
+## What gets scraped
+
+Each player is stored as an object like this:
+
+```js
+{
+  name: 'tenz',                         // from the profile URL
+  team: 'Sentinels',                    // current team
+  past_teams: ['Cloud9', ...],          // previous teams
+  winnings: '$123,456',                 // total event winnings
+  most_played_agents: ['Jett', ...],    // top 3 agents by usage
+  avg_kda: 1.23,                        // average K:D across those top 3 agents
+  nationality: 'Canada',
+  region: 'AMER',                       // from the event title, or null
+  stats: [                              // full agent stat rows for the top 3 agents,
+    { Agent: 'Jett', 'K:D': '1.25', ... } //   keyed by vlr.gg's column headers
+  ]
+}
+```
+
+## Library usage
+
+```js
+import {
+  scrapePlayer,
+  scrapeWebsite,
+  savePlayer,
+  saveEvent,
+  getPlayersByEvent,
+  getRandomPlayerFromEvent,
+} from 'vlr-stats-scraper';
+
+const { playerLinks, region } = await scrapeWebsite('https://www.vlr.gg/event/.../group-stage');
+const eventId = saveEvent({ name: 'Stage 2', year: 2026 });
+
+for (const link of playerLinks) {
+  const player = await scrapePlayer(link, region);
+  if (player) {
+    savePlayer(eventId, player);
+  }
+}
+
+const roster = getPlayersByEvent('Stage 2');
+const pick = getRandomPlayerFromEvent('Stage 2');
+```
+
+### Upgrading from 1.x
+
+2.0.0 changes how players are stored and has breaking API changes:
+
+| 1.x | 2.0.0 |
+| --- | --- |
+| `savePlayer(player)` then `addPlayerToEvent(eventId, player.name)` | `savePlayer(eventId, player)`. `addPlayerToEvent` is removed. |
+| `scrapeWebsite(url)` returns `string[]` | Returns `{ playerLinks, region }` |
+| One row per player, shared across events | One row per player **per event** (see [Data storage](#data-storage)) |
+| `valstats --player <name>` | `valstats --search <name>`. `--player` now adds a player to an event. |
+| Region typed in at a prompt and saved on the event | Region read from the event title and saved on each player only |
+| `saveEvent({ name, url, region, year, stage })` | `saveEvent({ name, url, year, stage })` |
+
+Existing `players.db` files are migrated automatically the first time they're opened.
+
+### Exports
+
+#### Scraping
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `scrapeWebsite(url)` | `Promise<{ playerLinks, region }>` | Scrapes an event page. `playerLinks` is every player profile link on it (with `?timespan=all`); `region` is read from the event title (`'AMER'`, `'EMEA'`, `'PAC'`, `'CN'` or `null`). Returns `undefined` if scraping fails. |
+| `scrapePlayer(url, region)` | `Promise<Player>` | Scrapes one player's profile page into a player object (see above). `region` is stored as-is. Returns `undefined` if scraping fails. |
+
+#### Players
+
+A player is saved separately for each event, so the same player can have different stats in
+different events. Players read from the database also have `id`, `event_id`, `event_name` and
+`event_year`.
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `savePlayer(eventId, player)` | `number` | Saves a player's stats for one event and returns the row's id. Re-saving the same player in the same event updates them. |
+| `getPlayerById(id)` | `Player \| null` | Looks up one saved player row by id. |
+| `getPlayer(name)` | `Player \| null` | The player as they were in their most recent event. |
+| `getPlayerHistory(name)` | `Player[]` | Every saved version of a player, one per event, newest first. |
+| `searchPlayers(query)` | `Player[]` | Every player whose name starts with `query` (case-insensitive), sorted by name, one entry per event. |
+| `getAllPlayers()` | `Player[]` | Every player row in the database. |
+| `getRandomPlayer()` | `Player \| null` | A random player row from the whole database. |
+
+#### Events (tournaments / stages)
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `saveEvent({ name, url, year, stage })` | `number` | Creates or updates an event by name and returns its id. Only `name` is required. |
+| `getEvents()` | `Event[]` | Every saved event. |
+| `getEventByName(name)` | `Event \| undefined` | Looks up an event by name. |
+| `getEventId(name, year)` | `number \| undefined` | The id of the event with that name and year. |
+| `getPlayersByEvent(eventName)` | `Player[]` | Every player saved under an event. |
+| `getRandomPlayerFromEvent(eventName)` | `Player \| null` | A random player from one event. |
+
+#### Clearing
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `emptyEvent(eventName)` | `boolean` | Removes an event's players but keeps the event so it can be re-scraped. `false` if the event doesn't exist. |
+| `clearEvent(eventName)` | `boolean` | Removes an event and its players. `false` if the event doesn't exist. |
+| `clearDatabase()` | — | Deletes every event and player. |
+
+`emptyEvent` and `clearEvent` only delete that event's player rows, so the same player's saves in
+other events are kept.
+
+## Data storage
+
+Player and event data is stored in a `players.db` SQLite file created in whichever
+directory you run the tool from. It has two tables, and **every player row belongs to exactly
+one event**:
+
+```
+events                          players
+──────────────                  ────────────────────────────
+id (PK)  ◄───────────────────── event_id   (which event this row is from)
+name (unique)                   id (PK)    (unique per row)
+url                             name
+year                            team, past_teams, winnings,
+stage                           most_played_agents, avg_kda,
+                                nationality, region, stats
+```
+
+- `events` — one row per tournament/stage. Region isn't stored here; it's on each player.
+- `players` — one row per player **per event**. `past_teams`, `most_played_agents` and `stats`
+  are stored as JSON and parsed back automatically when read.
+
+A player's stats change over time, so the same player in two events is saved as two separate
+rows, each with the stats they had when that event was scraped.
+
+### Example
+
+**events**
+
+| id | name    | year |
+|----|---------|------|
+| 1  | Masters | 2025 |
+| 2  | Champs  | 2026 |
+
+**players**
+
+| id | event_id | name  | avg_kda |
+|----|----------|-------|---------|
+| 1  | 1        | tenz  | 1.10    |
+| 2  | 2        | tenz  | 1.35    |
+| 3  | 2        | aspas | 1.40    |
+
+`tenz` has two rows: one for how they played at Masters and one for Champs. Each has its own
+`id` and stats.
+
+### Rules the database enforces
+
+- **One copy of a player per event.** `(event_id, name)` is unique, so re-scraping Champs
+  updates tenz's Champs row instead of adding another one. Their Masters row isn't touched.
+- **Every player row must point to a real event.** Foreign keys are enforced, so saving a
+  player under an event id that doesn't exist fails.
+- **Deleting an event deletes its players.** Player rows are removed along with their event
+  (`ON DELETE CASCADE`).
+
+### Which function reads what
+
+| You want… | Function | What it does |
+|---|---|---|
+| Save a scraped player | `savePlayer(eventId, player)` | Inserts or updates that player's row for the event and returns its `id` |
+| An event's roster | `getPlayersByEvent('Champs')` | Every row whose `event_id` matches that event |
+| A player's latest version | `getPlayer('tenz')` | Their row from the newest event (highest year) |
+| Every version of a player | `getPlayerHistory('tenz')` | All their rows, newest first |
+| One specific row | `getPlayerById(2)` | Looks a row up by `id` |
+| Search by name | `searchPlayers('ten')` | Partial match, one result per player per event |
+| Re-scrape an event | `emptyEvent('Champs')` | Deletes that event's players but keeps the event |
+| Remove an event | `clearEvent('Champs')` | Deletes the event, and its players go with it |
+
+Every player read from the database also has `event_name` and `event_year`, so you can tell
+which event a row came from without a separate lookup.
+
+### What happens when you scrape an event
+
+```
+valstats <link> <year> <event-name>
+        │
+        ▼
+Already saved with this name + year? ──yes──► emptyEvent(name)
+        │                                     (delete its old player rows)
+        ▼                                            │
+saveEvent(...) ◄─────────────────────────────────────┘
+(create or update the event, get its id)
+        │
+        ▼
+For each player on the event page:
+  scrapePlayer(link) ──► savePlayer(eventId, player)
+```
+
+### Migrating older databases
+
+Older versions stored one shared row per player and linked players to events through an
+`event_players` table (or `tournaments` / `tournament_players` before that). These are
+migrated automatically the first time the database is opened:
+
+```
+Before                                   After
+players: tenz                            players: id 1, tenz, event_id 1
+event_players: (1, tenz), (2, tenz)  ──►          id 2, tenz, event_id 2
+```
+
+Each player is copied into a row for every event they were linked to, and `event_players` is
+removed. Older versions overwrote a player's stats on every scrape, so migrated rows for the
+same player all start with the same (most recent) stats. Players who weren't linked to any
+event are dropped.
+
+## License
+
+ISC

package/index.mjs

@@ -1,17 +1,20 @@
-export { default as scrapePlayer } from './scripts/player-info-scraper.mjs';
-export { scrapeWebsite } from './scripts/player-link-scraper.mjs';
-export {
-  savePlayer,
-  getPlayer,
-  getAllPlayers,
-  getRandomPlayer,
-  saveEvent,
-  addPlayerToEvent,
-  getEvents,
-  getEventByName,
-  getPlayersByEvent,
-  getRandomPlayerFromEvent,
-  emptyEvent,
-  clearEvent,
-  clearDatabase,
-} from './scripts/db.mjs';
+export { default as scrapePlayer } from './scripts/player-info-scraper.mjs';
+export { scrapeWebsite } from './scripts/player-link-scraper.mjs';
+export {
+  savePlayer,
+  getPlayerById,
+  getPlayer,
+  getPlayerHistory,
+  searchPlayers,
+  getAllPlayers,
+  getRandomPlayer,
+  saveEvent,
+  getEvents,
+  getEventByName,
+  getEventId,
+  getPlayersByEvent,
+  getRandomPlayerFromEvent,
+  emptyEvent,
+  clearEvent,
+  clearDatabase,
+} from './scripts/db.mjs';