vlr-stats-scraper 1.5.0 → 3.1.0

package/README.md

@@ -1,7 +1,8 @@
 # vlr-stats-scraper
 
 Scrapes player stats from [vlr.gg](https://www.vlr.gg) and stores them in a local SQLite
-database, grouped by tournament or stage, so you can pull a backlog of players.
+database, grouped by tournament or stage, so you can pull a backlog of players or serve them
+from a website.
 
 ## Setup
 
@@ -15,6 +16,7 @@ Create a `.env` file in the directory you run it from:
 
 ```
 HYPERBROWSER_API_KEY=your_key_here
+VLR_DB_PATH=/path/to/players.db   # optional, defaults to ./players.db
 ```
 
 ## CLI usage
@@ -24,8 +26,9 @@ The package installs a `valstats` command.
 | Command | What it does |
 | --- | --- |
 | `valstats <link> <year> <event-name>` | Scrape every player on an event page and save them under that event |
-| `valstats --player <name \| link>` | Print the stats of every saved player whose name contains `name`, or of one player scraped live from a link |
-| `valstats --clear-event <event-name>` | Remove one event and its players |
+| `valstats --search <name \| link>` | Print the stats of every saved player whose name starts with `name`, or of one player scraped live from a link |
+| `valstats --player <link> <event-name> <year> [region]` | Scrape one player and add them to an event that's already saved |
+| `valstats --clear-event <event-name> [year]` | Remove one event and its players |
 | `valstats --clear` | Wipe the whole database (asks for confirmation) |
 
 ### Scraping an event
@@ -40,149 +43,301 @@ npx valstats https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage
 
 - `link` — the vlr.gg event/group-stage page to scrape.
 - `year` — the year the event took place.
-- `event-name` — a label for this tournament/stage (used to group players together later).
+- `event-name` — a label for this tournament/stage. Events are identified by name **and** year,
+  so "Stage 2" 2025 and "Stage 2" 2026 are kept apart.
 
 What happens:
 
-1. If the event name contains the word **"stage"** (case-insensitive), you're prompted to
-   enter a region (e.g. `AMER`, `EMEA`). That region is saved on the event and on every
-   player scraped. Otherwise region is left empty.
-2. Every player profile linked on the event page is collected. Stats are pulled using
-   vlr.gg's **all-time** timespan.
-3. Each player is scraped, saved to `players.db` in your current directory, linked to the
-   event, and printed to the console.
+1. The event page is scraped for its title and every player profile linked on it.
+2. The region is read from the event title and saved on every player (not on the event):
+
+   | Title contains | Region |
+   | --- | --- |
+   | `America` (e.g. "VCT 2026: Americas Stage 2") | `AMER` |
+   | `EMEA` | `EMEA` |
+   | `Pacific` | `PAC` |
+   | `China` | `CN` |
+   | none of these (e.g. Masters, Champions) | `null` |
+
+3. The event's stats page (`vlr.gg/event/stats/<id>`) is scraped once. It gives every player's
+   stats **in this event only**: rating, ACS, K:D, ADR, KAST, agents played, and so on. The stats
+   page covers the whole event, so a link to one stage of it still gets stats from every stage.
+4. Each player's profile is scraped using vlr.gg's **all-time** timespan for their team, past
+   teams, nationality, winnings and career stats on their top 3 agents. Both sets of stats are
+   saved to the database under that event and printed to the console.
+
+If the event page can't be scraped, nothing is saved or deleted. If only the stats page fails,
+players are saved without event stats.
 
 Running it again with an event name and year that are already saved replaces that event's
-players with the freshly scraped ones. A player already in the database is updated with the
-new data rather than duplicated.
+players with the freshly scraped ones.
 
 ### Viewing a player's stats
 
 ```bash
-npx valstats --player tenz                                    # every saved player with "tenz" in their name
-npx valstats --player ten                                     # partial names work too: tenz, tenten, ...
-npx valstats --player https://www.vlr.gg/player/9/tenz        # scrape a player live
+npx valstats --search tenz                                    # every saved player with a name starting "tenz"
+npx valstats --search ten                                     # tenz, tenten, ... (not "kitten")
+npx valstats --search https://www.vlr.gg/player/9/tenz        # scrape a player live without saving
 ```
 
-- Passing a **name** searches `players.db` and prints every player whose name contains
-  those characters (case-insensitive). Names are the ones from the vlr.gg profile URL,
-  e.g. `tenz`. Nothing is scraped.
-- Passing a **vlr.gg player link** scrapes that profile live using the all-time timespan.
-  The result is printed but not saved to the database.
+- `--search <name>` searches the database and prints every player whose name starts with
+  those characters (case-insensitive). Nothing is scraped. A player saved under several events
+  is printed once per event, labelled with the event name and year, e.g. `tenz (Champs, 2026)`.
+- `--search <link>` scrapes that profile live using the all-time timespan. The result is
+  printed but not saved.
+
+For each player, their team, past teams, nationality, region, winnings, top agents, career K:D
+and their stats in that event are printed, followed by a table of their career stats on their
+top 3 agents.
 
-For each player, their team, past teams, nationality, region, winnings, top agents and
-average K:D are printed, followed by a table of their stats on their top 3 agents.
+### Adding a player to an event by hand
+
+```bash
+npx valstats --player <link> <event-name> <year> [region]
+```
+
+```bash
+npx valstats --player https://www.vlr.gg/player/9/tenz "Stage 2" 2026        # region copied from the event's other players
+npx valstats --player https://www.vlr.gg/player/9/tenz "Stage 2" 2026 AMER   # region set explicitly
+```
+
+Scrapes one player, plus the event's stats page for their event stats, and saves them under an
+event that's already saved, e.g. a player the event page missed. The event is checked before
+scraping, so a mistyped name or year fails straight away without using a scrape.
 
 ### Clearing data
 
 ```bash
-npx valstats --clear-event "Stage 2"   # remove one event and its players
-npx valstats --clear                   # wipe the whole database (type "yes" to confirm)
+npx valstats --clear-event "Stage 2" 2026   # remove one event and its players
+npx valstats --clear-event "Stage 2"        # works when only one year has that name
+npx valstats --clear                        # wipe the whole database (type "yes" to confirm)
 ```
 
-Players that also belong to another event are kept when a single event is cleared.
+If several years share the name, `--clear-event` asks you to give the year instead of guessing.
 
 ## What gets scraped
 
-Each player is stored as an object like this:
+`scrapePlayer` returns an object like this:
 
 ```js
 {
-  name: 'tenz',                         // from the profile URL
-  team: 'Sentinels',                    // current team
-  past_teams: ['Cloud9', ...],          // previous teams
-  winnings: '$123,456',                 // total event winnings
-  most_played_agents: ['Jett', ...],    // top 3 agents by usage
-  avg_kda: 1.23,                        // average K:D across those top 3 agents
+  vlr_id: 9,                            // vlr.gg's player id, from the profile URL
+  name: 'tenz',                         // URL slug
+  url: 'https://www.vlr.gg/player/9/tenz/',
+  team: 'Sentinels',                    // current team, or null
+  past_teams: ['Cloud9', ...],
+  winnings: 183650,                     // total event winnings in dollars
+  most_played_agents: ['jett', 'omen', 'reyna'],
+  avg_kda: 1.31,                        // career K:D on those agents: total kills / total deaths
   nationality: 'Canada',
-  region: 'AMER',                       // from the CLI prompt / argument, or null
-  stats: [                              // full agent stat rows for the top 3 agents,
-    { Agent: 'Jett', 'K:D': '1.25', ... } //   keyed by vlr.gg's column headers
+  region: 'AMER',                       // from the event title, or null
+  scraped_at: '2026-10-31T12:00:00.000Z',
+  stats: [                              // career stats on the top 3 agents
+    { agent: 'jett', maps: 207, use_pct: 45, rounds: 4328, rating: 1.17, acs: 262.4, kd: 1.32,
+      kast: 72, adr: 156.4, kpr: 0.92, apr: 0.2, fk_fd: 1.66, kills: 3989, deaths: 3028, ... },
+    ...
   ]
 }
 ```
 
+Players read from the database also have `id`, `event_id`, `event_name`, `event_year` and
+`event_stats` — their stats in that event, or `null`:
+
+```js
+event_stats: {
+  team: '100T',
+  agents: [{ agent: 'phoenix', use_pct: 95 }, { agent: 'omen', use_pct: 5 }],
+  maps: 21, rounds: 449, rating: 1.19, acs: 243, kd: 1.22, kast: 77, adr: 160.5,
+  kpr: 0.83, apr: 0.4, fk_fd: 1.12, fk_pct: 11, fd_pct: 10, hs_pct: 22, clutch_pct: 14,
+  clutches_won: 5, clutches_played: 35, kills_max: 28,
+  kills: 374, deaths: 307, assists: 180, first_kills: 48, first_deaths: 43,
+}
+```
+
+Every stat is a number. Percentages are stored as 0–100 (`kast: 77` means 77%).
+
 ## Library usage
 
 ```js
 import {
-  scrapePlayer,
   scrapeWebsite,
-  savePlayer,
+  scrapeEventStats,
+  scrapePlayer,
   saveEvent,
-  addPlayerToEvent,
+  savePlayer,
   getPlayersByEvent,
-  getRandomPlayerFromEvent,
+  getEventLeaderboard,
 } from 'vlr-stats-scraper';
 
-const eventId = saveEvent({ name: 'Stage 2', region: 'AMER', year: 2026 });
+const link = 'https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage';
+const { playerLinks, region } = await scrapeWebsite(link);
+const eventStats = await scrapeEventStats(link);
+const eventId = saveEvent({ name: 'Stage 2', year: 2026, url: link });
 
-const links = await scrapeWebsite('https://www.vlr.gg/event/.../group-stage');
-for (const link of links) {
-  const player = await scrapePlayer(link, 'AMER');
+for (const playerLink of playerLinks) {
+  const player = await scrapePlayer(playerLink, region);
   if (player) {
-    savePlayer(player);
-    addPlayerToEvent(eventId, player.name);
+    savePlayer(eventId, { ...player, event_stats: eventStats?.get(player.vlr_id) ?? null });
   }
 }
 
-const roster = getPlayersByEvent('Stage 2');
-const pick = getRandomPlayerFromEvent('Stage 2');
+const roster = getPlayersByEvent('Stage 2', 2026);
+const topFive = getEventLeaderboard('Stage 2', 2026, { stat: 'acs', minRounds: 100, limit: 5 });
 ```
 
+### Using it from a website
+
+Scrape on a schedule (cron, a GitHub Action, ...) with the CLI or library, and have the web server
+open the same file read-only:
+
+```js
+import { openDatabase, getPlayer } from 'vlr-stats-scraper';
+
+openDatabase({ path: '/data/players.db', readonly: true });
+const player = getPlayer('tenz');
+```
+
+- The database uses WAL mode, so the site keeps reading while a scrape is writing.
+- A read-only connection never migrates. If the file is from an older version, it throws and asks
+  you to open it once for writing (e.g. run `valstats`) to upgrade it.
+- better-sqlite3 needs a real, persistent disk: a VPS, or Fly.io / Railway / Render with a
+  volume. Serverless hosts like Vercel and Netlify don't keep files between requests; for those,
+  copy the data into a hosted database such as Turso or Postgres.
+
+### Upgrading from 2.x
+
+3.0.0 changes the database layout and has breaking API changes. Existing `players.db` files are
+migrated automatically the first time they're opened for writing.
+
+| 2.x | 3.0.0 |
+| --- | --- |
+| Event names unique on their own; re-scraping "Stage 2" for a new year overwrote the old one | Events unique by name **and** year |
+| `getEventByName(name)`, `getPlayersByEvent(name)`, `emptyEvent(name)`, `clearEvent(name)` | Take an optional `year`: `getPlayersByEvent('Stage 2', 2026)`. Without it, reads use the newest year and deletes refuse to guess if there are several. `getEvent` is the new name for `getEventByName`. |
+| Per-event rows held all-time profile stats | Rows also get `event_stats`, scraped from the event's stats page |
+| Players matched by URL slug | Players matched by `vlr_id`; `getPlayer(9)` also works |
+| `winnings: '$123,456'` | `winnings: 123456` |
+| `stats: [{ Agent: 'jett', 'K:D': '1.25', ... }]` | `stats: [{ agent: 'jett', kd: 1.25, ... }]`, all numbers |
+| `avg_kda` was a plain average of each agent's K:D | Total kills / total deaths on those agents |
+| `players.db` always in the current directory, opened on import | `VLR_DB_PATH` or `openDatabase({ path, readonly })`, opened on first use |
+| `scrapeWebsite` returned `{ playerLinks, region }` | Also returns `title` and `vlrEventId` |
+
+Rows migrated from 2.x have no `vlr_id`, `url`, `scraped_at` or `event_stats` until their event is
+scraped again; re-scraping matches them up by name.
+
 ### Exports
 
 #### Scraping
 
 | Function | Returns | Description |
 | --- | --- | --- |
-| `scrapeWebsite(url)` | `Promise<string[]>` | Scrapes an event page and returns every player profile link on it (with `?timespan=all`). |
-| `scrapePlayer(url, region)` | `Promise<Player>` | Scrapes one player's profile page into a player object (see above). `region` is stored as-is. Returns `undefined` if scraping fails. |
+| `scrapeWebsite(url)` | `Promise<{ playerLinks, region, title, vlrEventId }>` | Scrapes an event page. `playerLinks` is every player profile link on it (with `?timespan=all`); `region` is read from the event title (`'AMER'`, `'EMEA'`, `'PAC'`, `'CN'` or `null`). Returns `undefined` if scraping fails. |
+| `scrapeEventStats(url)` | `Promise<Map<vlrId, EventStats>>` | Scrapes the stats page of the event `url` points at (any event or stage link works). Returns `undefined` if scraping fails. |
+| `eventStatsUrl(url)` | `string \| null` | The stats page URL for an event link. |
+| `scrapePlayer(url, region)` | `Promise<Player>` | Scrapes one player's profile page into a player object (see above). Returns `undefined` if scraping fails. |
+
+#### Connection
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `openDatabase({ path, readonly })` | `Database` | Opens the database file. Both options are optional. Without a call, the first query opens `$VLR_DB_PATH` or `./players.db` for writing. |
+| `closeDatabase()` | — | Closes the connection. |
 
 #### Players
 
+A player is saved separately for each event. Functions that take a player accept a name
+(`'tenz'`, case-insensitive) or a vlr id (`9`).
+
 | Function | Returns | Description |
 | --- | --- | --- |
-| `savePlayer(player)` | — | Inserts a player, or updates them if one with the same name exists. |
-| `getPlayer(name)` | `Player \| null` | Looks up a player by name. |
-| `searchPlayers(query)` | `Player[]` | Every player whose name contains `query` (case-insensitive), sorted by name. |
-| `getAllPlayers()` | `Player[]` | Every player in the database. |
-| `getRandomPlayer()` | `Player \| null` | A random player from the whole database. |
+| `savePlayer(eventId, player)` | `number` | Saves a player for one event and returns the row's id. Re-saving the same player in the same event updates them. Leaving out `event_stats` keeps what's saved; `null` removes it. |
+| `getPlayerById(id)` | `Player \| null` | One saved row by id. |
+| `getPlayer(nameOrVlrId)` | `Player \| null` | The player as they were in their most recent event. |
+| `getPlayerHistory(nameOrVlrId)` | `Player[]` | Every saved version of a player, one per event, newest first. |
+| `searchPlayers(query)` | `Player[]` | Every player whose name starts with `query` (case-insensitive), one entry per event. |
+| `getPlayersByTeam(team, { includePast })` | `Player[]` | Players currently on `team`, plus former players with `includePast: true`. Newest row per player. |
+| `getAllPlayers()` | `Player[]` | Every player row. |
+| `getRandomPlayer()` | `Player \| null` | A random player row. |
+
+#### Leaderboards
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `getEventLeaderboard(name, year, { stat, minRounds, limit, ascending })` | `Player[]` | Players in one event sorted by an `event_stats` column (default `rating`, 20 players). |
+| `getAgentLeaderboard(agent, { stat, minRounds, limit, ascending })` | `Player[]` | Players sorted by their career stats on one agent, e.g. `getAgentLeaderboard('jett', { stat: 'kd', minRounds: 500 })`. Uses each player's newest row. |
 
 #### Events (tournaments / stages)
 
 | Function | Returns | Description |
 | --- | --- | --- |
-| `saveEvent({ name, url, region, year, stage })` | `number` | Creates or updates an event by name and returns its id. Only `name` is required. |
-| `addPlayerToEvent(eventId, playerName)` | — | Links a saved player to an event (ignored if already linked). |
-| `getEvents()` | `Event[]` | Every saved event. |
-| `getEventByName(name)` | `Event \| undefined` | Looks up an event by name. |
-| `getPlayersByEvent(eventName)` | `Player[]` | All players linked to an event. |
-| `getRandomPlayerFromEvent(eventName)` | `Player \| null` | A random player from one event. |
+| `saveEvent({ name, year, url, stage })` | `number` | Creates or updates the event with this name and year and returns its id. Only `name` is required. |
+| `getEvents()` | `Event[]` | Every saved event, newest first. |
+| `getEvent(name, year?)` | `Event \| undefined` | Looks up an event. Without `year`, the newest event with that name. `getEventByName` is an alias. |
+| `getEventId(name, year?)` | `number \| undefined` | The event's id. |
+| `getPlayersByEvent(name, year?)` | `Player[]` | Every player saved under an event. |
+| `getRandomPlayerFromEvent(name, year?)` | `Player \| null` | A random player from one event. |
 
 #### Clearing
 
 | Function | Returns | Description |
 | --- | --- | --- |
-| `emptyEvent(eventName)` | `boolean` | Removes an event's players but keeps the event so it can be re-scraped. `false` if the event doesn't exist. |
-| `clearEvent(eventName)` | `boolean` | Removes an event and its players. `false` if the event doesn't exist. |
+| `emptyEvent(name, year?)` | `boolean` | Removes an event's players but keeps the event so it can be re-scraped. |
+| `clearEvent(name, year?)` | `boolean` | Removes an event and its players. |
 | `clearDatabase()` | — | Deletes every event and player. |
 
-Players that still belong to another event are never deleted by `emptyEvent` or
-`clearEvent`.
+Both return `false` if the event doesn't exist, and throw if `year` is left out and several years
+share the name. They only delete that event's rows, so the same player's saves in other events
+are kept.
 
 ## Data storage
 
-Player and event data is stored in a `players.db` SQLite file created in whichever
-directory you run the tool from. It has three tables:
+```
+events                    players                         agent_stats
+───────────               ───────────────────             ─────────────────────────
+id (PK) ◄──────────────── event_id                ┌────── player_id
+name ┐ unique together    id (PK) ◄───────────────┤       agent, position
+year ┘                    vlr_id                  │       maps, use_pct, rounds, rating,
+url, vlr_event_id         name, url               │       acs, kd, kast, adr, kills, deaths...
+stage                     team, past_teams (JSON) │
+                          winnings, avg_kda       │       event_stats
+                          most_played_agents      │       ─────────────────────────
+                          nationality, region     └────── player_id (PK)
+                          scraped_at                      team, agents (JSON), maps, rounds,
+                                                          rating, acs, kd, kast, adr, hs_pct...
+```
+
+- `events` — one row per tournament/stage, unique by `(name, year)`.
+- `players` — one row per player **per event**, unique by `(event_id, vlr_id)`.
+- `agent_stats` — the player's career stats on their top 3 agents, as of `scraped_at`.
+- `event_stats` — the player's stats in that event only.
+
+Deleting an event deletes its players, and deleting a player deletes their stats
+(`ON DELETE CASCADE`). The schema version is kept in SQLite's `user_version`; older files are
+migrated automatically when opened for writing.
+
+### Example
+
+**events**
+
+| id | name    | year |
+|----|---------|------|
+| 1  | Stage 2 | 2025 |
+| 2  | Stage 2 | 2026 |
+
+**players**
+
+| id | event_id | vlr_id | name |
+|----|----------|--------|------|
+| 1  | 1        | 9      | tenz |
+| 2  | 2        | 9      | tenz |
+
+**event_stats**
 
-- `players` — one row per player (`past_teams`, `most_played_agents` and `stats` are stored
-  as JSON and parsed back automatically when read).
-- `events` — one row per tournament/stage (`name`, `url`, `region`, `year`, `stage`).
-- `event_players` — links players to the events they were scraped from.
+| player_id | rating | acs |
+|-----------|--------|-----|
+| 1         | 1.05   | 221 |
+| 2         | 1.19   | 243 |
 
-Databases created by older versions (with `tournaments` / `tournament_players` tables) are
-migrated automatically.
+`tenz` has a row per event, each with how they played in that event.
 
 ## License
 

package/index.mjs

@@ -1,15 +1,24 @@
 export { default as scrapePlayer } from './scripts/player-info-scraper.mjs';
 export { scrapeWebsite } from './scripts/player-link-scraper.mjs';
+export { scrapeEventStats, eventStatsUrl } from './scripts/event-stats-scraper.mjs';
 export {
+  openDatabase,
+  closeDatabase,
   savePlayer,
+  getPlayerById,
   getPlayer,
+  getPlayerHistory,
   searchPlayers,
   getAllPlayers,
   getRandomPlayer,
+  getPlayersByTeam,
+  getEventLeaderboard,
+  getAgentLeaderboard,
   saveEvent,
-  addPlayerToEvent,
   getEvents,
+  getEvent,
   getEventByName,
+  getEventId,
   getPlayersByEvent,
   getRandomPlayerFromEvent,
   emptyEvent,