vlr-stats-scraper 1.0.1 → 1.2.0

package/README.md

@@ -1,73 +1,170 @@
-# vlr-stats-scraper
-
-Scrapes player stats from [vlr.gg](https://www.vlr.gg) and stores them in a local SQLite
-database, grouped by tournament or stage, so you can pull a backlog of players.
-
-## Setup
-
-Requires a [Hyperbrowser](https://hyperbrowser.ai) API key for scraping.
-
-```bash
-npm install vlr-stats-scraper
-```
-
-Create a `.env` file in the directory you run it from:
-
-```
-HYPERBROWSER_API_KEY=your_key_here
-```
-
-## CLI usage
-
-```bash
-npx valstats <link> <year> <event-name>
-```
-
-```bash
-npx valstats https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage 2026 "Stage 2"
-```
-
-- `link` — the vlr.gg event/group-stage page to scrape.
-- `year` — the year the event took place.
-- `event-name` — a label for this tournament/stage (used to group players together later).
-
-Each player on that event page gets scraped and saved to `players.db` in your current
-directory, linked to that event name.
-
-## Library usage
-
-```js
-import {
-  scrapePlayer,
-  scrapeWebsite,
-  savePlayer,
-  saveEvent,
-  addPlayerToEvent,
-  getPlayersByEvent,
-  getRandomPlayerFromEvent,
-} from 'vlr-stats-scraper';
-
-const links = await scrapeWebsite('https://www.vlr.gg/event/.../group-stage');
-const player = await scrapePlayer(links[0], 'AMER');
-savePlayer(player);
-```
-
-### Exports
-
-- `scrapePlayer(url, region)` — scrapes one player's profile page.
-- `scrapeWebsite(url)` — scrapes an event page for player profile links.
-- `savePlayer(player)` / `getPlayer(name)` / `getAllPlayers()` / `getRandomPlayer()`
-- `saveEvent({ name, url, region, year, stage })` — creates/updates a tournament or stage,
-  returns its id.
-- `addPlayerToEvent(eventId, playerName)`
-- `getEvents()` / `getEventByName(name)`
-- `getPlayersByEvent(eventName)` / `getRandomPlayerFromEvent(eventName)`
-
-## Data storage
-
-Player and event data is stored in a `players.db` SQLite file created in whichever
-directory you run the tool from.
-
-## License
-
-ISC
+# vlr-stats-scraper
+
+Scrapes player stats from [vlr.gg](https://www.vlr.gg) and stores them in a local SQLite
+database, grouped by tournament or stage, so you can pull a backlog of players.
+
+## Setup
+
+Requires Node.js 18+ and a [Hyperbrowser](https://hyperbrowser.ai) API key for scraping.
+
+```bash
+npm install vlr-stats-scraper
+```
+
+Create a `.env` file in the directory you run it from:
+
+```
+HYPERBROWSER_API_KEY=your_key_here
+```
+
+## CLI usage
+
+The package installs a `valstats` command.
+
+| Command | What it does |
+| --- | --- |
+| `valstats <link> <year> <event-name>` | Scrape every player on an event page and save them under that event |
+| `valstats --clear-event <event-name>` | Remove one event and its players |
+| `valstats --clear` | Wipe the whole database (asks for confirmation) |
+
+### Scraping an event
+
+```bash
+npx valstats <link> <year> <event-name>
+```
+
+```bash
+npx valstats https://www.vlr.gg/event/2977/vct-2026-americas-stage-2/group-stage 2026 "Stage 2"
+```
+
+- `link` — the vlr.gg event/group-stage page to scrape.
+- `year` — the year the event took place.
+- `event-name` — a label for this tournament/stage (used to group players together later).
+
+What happens:
+
+1. If the event name contains the word **"stage"** (case-insensitive), you're prompted to
+   enter a region (e.g. `AMER`, `EMEA`). That region is saved on the event and on every
+   player scraped. Otherwise region is left empty.
+2. Every player profile linked on the event page is collected. Stats are pulled using
+   vlr.gg's **all-time** timespan.
+3. Each player is scraped, saved to `players.db` in your current directory, linked to the
+   event, and printed to the console.
+
+Running it again with an event name and year that are already saved replaces that event's
+players with the freshly scraped ones. A player already in the database is updated with the
+new data rather than duplicated.
+
+### Clearing data
+
+```bash
+npx valstats --clear-event "Stage 2"   # remove one event and its players
+npx valstats --clear                   # wipe the whole database (type "yes" to confirm)
+```
+
+Players that also belong to another event are kept when a single event is cleared.
+
+## What gets scraped
+
+Each player is stored as an object like this:
+
+```js
+{
+  name: 'tenz',                         // from the profile URL
+  team: 'Sentinels',                    // current team
+  past_teams: ['Cloud9', ...],          // previous teams
+  winnings: '$123,456',                 // total event winnings
+  most_played_agents: ['Jett', ...],    // top 3 agents by usage
+  avg_kda: 1.23,                        // average K:D across those top 3 agents
+  nationality: 'Canada',
+  region: 'AMER',                       // from the CLI prompt / argument, or null
+  stats: [                              // full agent stat rows for the top 3 agents,
+    { Agent: 'Jett', 'K:D': '1.25', ... } //   keyed by vlr.gg's column headers
+  ]
+}
+```
+
+## Library usage
+
+```js
+import {
+  scrapePlayer,
+  scrapeWebsite,
+  savePlayer,
+  saveEvent,
+  addPlayerToEvent,
+  getPlayersByEvent,
+  getRandomPlayerFromEvent,
+} from 'vlr-stats-scraper';
+
+const eventId = saveEvent({ name: 'Stage 2', region: 'AMER', year: 2026 });
+
+const links = await scrapeWebsite('https://www.vlr.gg/event/.../group-stage');
+for (const link of links) {
+  const player = await scrapePlayer(link, 'AMER');
+  if (player) {
+    savePlayer(player);
+    addPlayerToEvent(eventId, player.name);
+  }
+}
+
+const roster = getPlayersByEvent('Stage 2');
+const pick = getRandomPlayerFromEvent('Stage 2');
+```
+
+### Exports
+
+#### Scraping
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `scrapeWebsite(url)` | `Promise<string[]>` | Scrapes an event page and returns every player profile link on it (with `?timespan=all`). |
+| `scrapePlayer(url, region)` | `Promise<Player>` | Scrapes one player's profile page into a player object (see above). `region` is stored as-is. Returns `undefined` if scraping fails. |
+
+#### Players
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `savePlayer(player)` | — | Inserts a player, or updates them if one with the same name exists. |
+| `getPlayer(name)` | `Player \| null` | Looks up a player by name. |
+| `getAllPlayers()` | `Player[]` | Every player in the database. |
+| `getRandomPlayer()` | `Player \| null` | A random player from the whole database. |
+
+#### Events (tournaments / stages)
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `saveEvent({ name, url, region, year, stage })` | `number` | Creates or updates an event by name and returns its id. Only `name` is required. |
+| `addPlayerToEvent(eventId, playerName)` | — | Links a saved player to an event (ignored if already linked). |
+| `getEvents()` | `Event[]` | Every saved event. |
+| `getEventByName(name)` | `Event \| undefined` | Looks up an event by name. |
+| `getPlayersByEvent(eventName)` | `Player[]` | All players linked to an event. |
+| `getRandomPlayerFromEvent(eventName)` | `Player \| null` | A random player from one event. |
+
+#### Clearing
+
+| Function | Returns | Description |
+| --- | --- | --- |
+| `emptyEvent(eventName)` | `boolean` | Removes an event's players but keeps the event so it can be re-scraped. `false` if the event doesn't exist. |
+| `clearEvent(eventName)` | `boolean` | Removes an event and its players. `false` if the event doesn't exist. |
+| `clearDatabase()` | — | Deletes every event and player. |
+
+Players that still belong to another event are never deleted by `emptyEvent` or
+`clearEvent`.
+
+## Data storage
+
+Player and event data is stored in a `players.db` SQLite file created in whichever
+directory you run the tool from. It has three tables:
+
+- `players` — one row per player (`past_teams`, `most_played_agents` and `stats` are stored
+  as JSON and parsed back automatically when read).
+- `events` — one row per tournament/stage (`name`, `url`, `region`, `year`, `stage`).
+- `event_players` — links players to the events they were scraped from.
+
+Databases created by older versions (with `tournaments` / `tournament_players` tables) are
+migrated automatically.
+
+## License
+
+ISC

package/index.mjs

@@ -11,4 +11,7 @@ export {
   getEventByName,
   getPlayersByEvent,
   getRandomPlayerFromEvent,
+  emptyEvent,
+  clearEvent,
+  clearDatabase,
 } from './scripts/db.mjs';

package/main.mjs

@@ -2,12 +2,51 @@
 import promptSync from 'prompt-sync';
 import { scrapeWebsite } from './scripts/player-link-scraper.mjs';
 import scrapePlayer from './scripts/player-info-scraper.mjs';
-import { savePlayer, saveEvent, addPlayerToEvent } from './scripts/db.mjs';
+import {
+  savePlayer,
+  saveEvent,
+  addPlayerToEvent,
+  getEventByName,
+  emptyEvent,
+  clearEvent,
+  clearDatabase,
+} from './scripts/db.mjs';
 
-const [link, year, eventName] = process.argv.slice(2);
+const args = process.argv.slice(2);
+
+if (args[0] === '--clear') {
+  const prompt = promptSync();
+  const answer = prompt('This deletes every event and player in players.db. Type "yes" to continue: ');
+  if (answer?.trim().toLowerCase() === 'yes') {
+    clearDatabase();
+    console.log('Cleared players.db');
+  } else {
+    console.log('Cancelled');
+  }
+  process.exit(0);
+}
+
+if (args[0] === '--clear-event') {
+  const name = args[1];
+  if (!name) {
+    console.error('Usage: valstats --clear-event <event-name>');
+    process.exit(1);
+  }
+  if (clearEvent(name)) {
+    console.log(`Removed "${name}" and its players from players.db`);
+  } else {
+    console.error(`No event named "${name}" in players.db`);
+    process.exit(1);
+  }
+  process.exit(0);
+}
+
+const [link, year, eventName] = args;
 
 if (!link || !year || !eventName) {
   console.error('Usage: valstats <link> <year> <event-name>');
+  console.error('       valstats --clear-event <event-name>');
+  console.error('       valstats --clear');
   process.exit(1);
 }
 
@@ -19,6 +58,13 @@ if (isStage) {
   region = prompt('Enter the region: ');
 }
 
+// Re-adding an event + year that's already saved replaces its old players.
+const existing = getEventByName(eventName);
+if (existing && existing.year === parseInt(year, 10)) {
+  emptyEvent(eventName);
+  console.log(`"${eventName}" (${year}) is already in players.db, replacing it`);
+}
+
 const eventId = saveEvent({
   name: eventName,
   url: link,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "vlr-stats-scraper",
-  "version": "1.0.1",
+  "version": "1.2.0",
   "description": "Scrapes player stats from vlr.gg and stores them in a local SQLite database, grouped by tournament or stage.",
   "homepage": "https://github.com/OverYander/VLR-Webscraper#readme",
   "bugs": {

package/scripts/db.mjs

@@ -161,6 +161,33 @@ function getRandomPlayerFromEvent(eventName) {
   `).get(eventName));
 }
 
+// Deletes players that no longer belong to any event.
+function deleteOrphanedPlayers() {
+  db.prepare('DELETE FROM players WHERE name NOT IN (SELECT player_name FROM event_players)').run();
+}
+
+// Removes every player from an event but keeps the event itself, so it can be re-scraped.
+// Players that are still in other events are kept.
+const emptyEvent = db.transaction((eventName) => {
+  const event = getEventByName(eventName);
+  if (!event) return false;
+  db.prepare('DELETE FROM event_players WHERE event_id = ?').run(event.id);
+  deleteOrphanedPlayers();
+  return true;
+});
+
+// Removes an event and its players. Players that are still in other events are kept.
+const clearEvent = db.transaction((eventName) => {
+  if (!emptyEvent(eventName)) return false;
+  db.prepare('DELETE FROM events WHERE name = ?').run(eventName);
+  return true;
+});
+
+// Wipes every event and player from the database.
+const clearDatabase = db.transaction(() => {
+  db.exec('DELETE FROM event_players; DELETE FROM events; DELETE FROM players;');
+});
+
 export {
   savePlayer,
   getPlayer,
@@ -172,4 +199,7 @@ export {
   getEventByName,
   getPlayersByEvent,
   getRandomPlayerFromEvent,
+  emptyEvent,
+  clearEvent,
+  clearDatabase,
 };