wsper-js 0.1.1 → 0.1.2-wc1

package/README.md

@@ -8,7 +8,7 @@
 [![npm](https://img.shields.io/npm/v/wsper-js?label=npm)](https://www.npmjs.com/package/wsper-js)
 ![TypeScript](https://img.shields.io/badge/TypeScript-5.8-blue)
 ![Node](https://img.shields.io/badge/Node-%3E%3D18.0.0-green)
-![Tests](https://img.shields.io/badge/tests-314%20passed-brightgreen)
+![Tests](https://img.shields.io/badge/tests-441%20passed-brightgreen)
 ![License](https://img.shields.io/badge/license-GPL--3.0--or--later-blue)
 </div>
 
@@ -83,7 +83,7 @@ const video = await wsper.youtube.getVideo("dQw4w9WgXcQ");
 console.log(track.ok, video.ok);
 ```
 
-`WsperScraper` currently exposes `spotify`, `twitter`, `threads`, `instagram`, `pinterest`, and `youtube`. Other scrapers are exported as named classes.
+`WsperScraper` currently exposes `spotify`, `twitter`, `threads`, `instagram`, `pinterest`, `youtube`, and `cai`. Other scrapers are exported as named classes.
 
 ## Response Shape
 
@@ -126,6 +126,37 @@ if (response.ok && response.data !== null) {
 
 The examples below use public exports from `src/index.ts` and representative response fields from implementation and tests.
 
+### Public-Link Examples
+
+Some scrapers need live public targets. The bundled examples avoid dummy URLs:
+
+```bash
+npx tsx examples/gdrive.example.ts
+npx tsx examples/foreign-news.example.ts
+npx tsx examples/pubgmobile.example.ts
+npx tsx examples/soundcloud.example.ts
+npx tsx examples/usgs-earthquake.example.ts all_day
+npx tsx examples/open-meteo.example.ts -6.2 106.8
+npx tsx examples/restcountries.example.ts Indonesia
+npx tsx examples/hacker-news.example.ts 10
+npx tsx examples/open-library.example.ts dune
+npx tsx examples/gutendex.example.ts alice
+npx tsx examples/world-bank.example.ts IDN SP.POP.TOTL
+npx tsx examples/ecb.example.ts 100 USD
+npx tsx examples/crossref.example.ts "open science"
+npx tsx examples/openalex.example.ts "open science"
+npx tsx examples/ror.example.ts "University of Indonesia"
+npx tsx examples/clinical-trials.example.ts diabetes
+npx tsx examples/pypi.example.ts requests
+npx tsx examples/packagist.example.ts monolog/monolog
+npx tsx examples/osv.example.ts PyPI requests
+npx tsx examples/binance.example.ts BTCUSDT
+```
+
+`GDriveScraper` defaults to a verified public Google Drive PDF in the example. `ForeignNewsScraper` uses the BBC World RSS feed. `PubgMobileScraper` uses the official server-rendered announcement list because the public `news.shtml` page is a client-rendered shell. Scrapers that commonly require a valid session or cookie, such as TeraBox, Pixiv, and Xiaohongshu, require an explicit URL/ID in their examples instead of using placeholder links.
+
+When a live endpoint returns HTTP 200 but no parseable data, these scrapers return `ok: false` with a parser error code instead of returning `ok: true` with empty data.
+
 ### LyricsScraper
 
 Source: LRCLIB JSON API at `https://lrclib.net/api/search`.
@@ -424,54 +455,225 @@ Representative output:
 }
 ```
 
+### CaiScraper
+
+Source: Character.AI WebSocket & API integrations.
+
+```ts
+import { CaiScraper } from "wsper-js";
+
+const scraper = new CaiScraper({
+  credentials: {
+    bearerToken: "YOUR_CHARACTER_AI_TOKEN",
+  },
+});
+
+// 1. Search for a character
+const search = await scraper.searchCharacters("Mario");
+console.log(search.data?.[0]);
+
+// 2. Chat with a character (optionally using a custom voice ID)
+const chat = await scraper.chat({
+  characterId: "rGKdvZewGUZEJFEQPEBMS5JLQOTOrxi-8ByLFsGmgQM",
+  message: "Hello Mario!",
+  voiceId: "4fdd6bc1-c659-4587-b462-53f569b39078",
+});
+console.log(chat.data?.text);
+console.log(chat.data?.audioUrl); // Generated TTS audio URL
+
+// Chat sessions stay connected per character and are reused by later chat()
+// calls. Disconnect them when your app is done with the character.
+await scraper.disconnectCharacterSession("rGKdvZewGUZEJFEQPEBMS5JLQOTOrxi-8ByLFsGmgQM");
+// Or:
+await scraper.disconnectAllCharacterSessions();
+```
+
+Representative output:
+
+```json
+{
+  "text": "It's-a me, Mario!",
+  "audioUrl": "https://storage.googleapis.com/.../voice.mp3",
+  "raw": { ... }
+}
+```
+
 ## Available Scrapers
 
+This table is audited against the public scraper exports in `src/scrapers/index.ts`.
+
 | Scraper | Purpose | Source/API | Auth/Cookie required? | Example file | Notes |
 | --- | --- | --- | --- | --- | --- |
+| `AioScraper` | Resolve public media download options | `allinonedownloader.com` | No | `examples/aio.example.ts` | `download(url)`; use only authorized public media URLs |
 | `AlkitabScraper` | Bible verse search | `alkitab.me` | No | `examples/alkitab.example.ts` | `search(query)` |
 | `AnimeQuoteScraper` | Random anime quote | `otakotaku.com` | No | `examples/anime-quote.example.ts` | `getRandom()` |
 | `AnimeRandomScraper` | Random anime character image | GitHub raw anime dataset | No | `examples/anime-random.example.ts` | `getImage(character)`, `random()` |
+| `ArenaAiScraper` | AI model leaderboard categories and rankings | `api.wulong.dev/arena-ai-leaderboards/v1` | No | `examples/arena-ai.example.ts` | `getCategories`, `getLeaderboard` |
 | `BiliBiliScraper` | BiliBili search and video info | `api.bilibili.com` | Optional cookie | `examples/bilibili.example.ts` | Cookie may unlock authenticated stream access |
+| `BinanceScraper` | Binance public market data | Binance Spot market data API | No | `examples/binance.example.ts` | Market data only: ticker, average price, klines, order book; no trading/account endpoints |
+| `BimasIslamScraper` | Indonesian Kemenag prayer time regions and schedules | `bimasislam.kemenag.go.id` | No | `examples/bimas-islam.example.ts` | `getProvinces`, `getCities`, `getPrayerTimes` |
 | `BMKGScraper` | Indonesian earthquake and weather feeds | `data.bmkg.go.id`, `nowcasting.bmkg.go.id` | No | `examples/bmkg.example.ts` | Autogempa, gempa dirasakan, nowcasting, forecast |
+| `CaiScraper` | Character.AI search, chat, and voice details | `neo.character.ai` API & WebSockets | Yes (Token required) | `examples/cai.example.ts` | `searchCharacters`, `chat`, `getVoice` |
 | `CapCutScraper` | Resolve CapCut template video URL | `capdownloader.com/wp-json/aio-dl/video-data/` | No | `examples/capcut.example.ts` | Mocked in example runner |
+| `ClinicalTrialsScraper` | ClinicalTrials.gov public study search and detail lookup | ClinicalTrials.gov API v2 | No | `examples/clinical-trials.example.ts` | `searchStudies`, `getStudy`; public registry metadata only |
+| `CodashopScraper` | Game nickname verification | `order-sg.codashop.com` | No | `examples/codashop.example.ts` | Supported game names are validated in the scraper |
+| `CrossrefScraper` | Scholarly work and journal metadata | Crossref REST API | No | `examples/crossref.example.ts` | `searchWorks`, `getWorkByDoi`, `searchJournals`; optional polite `mailto` option |
 | `CuacaScraper` | Indonesian weather by location/coordinate | BMKG weather APIs | Optional API key for warnings | `examples/cuaca.example.ts` | Reads optional `BMKG_WARNING_API_KEY` in example |
+| `DetikNewsScraper` | Detik news search | `detik.com` search HTML | No | `examples/detik-news.example.ts` | `search(query, resultType)` |
+| `DonghubScraper` | Drama search and detail pages | `donghub.vip` | No | `examples/donghub.example.ts` | `search`, `getDetail` |
+| `DownrScraper` | Resolve public video download options | `downr.org` Netlify functions | No | `examples/downr.example.ts` | `getVideo(url)`; use only authorized public media URLs |
 | `DrakorScraper` | Korean drama search/list/detail | `drakorkita30.kita.baby` | No | `examples/drakor.example.ts` | `search`, `detail`, `ongoing`, `getAll` |
 | `DramaboxScraper` | Dramabox search | `dramabox.com` | No | `examples/dramabox.example.ts` | `search(query)` |
+| `EcbScraper` | ECB euro foreign exchange reference rates | ECB eurofxref XML feed | No | `examples/ecb.example.ts` | `getDailyReferenceRates`, `convertFromEuro`; no investment advice |
+| `FacebookScraper` | Facebook public profile/post metadata and image download | `facebook.com`, `mbasic.facebook.com` | No credential option in current scraper | `examples/facebook.example.ts` | Public pages only; no session extraction |
 | `FaceswapScraper` | Face-swap image processing | `api.lovefaceswap.com` | No | `examples/faceswap.example.ts` | Mocked in example runner |
+| `FaceswapV2Scraper` | Face-swap image processing via URL inputs | `supawork.ai` headshot API | No | `examples/faceswap-v2.example.ts` | `swap(targetImageUrl, targetFaceUrl)` |
+| `ForeignNewsScraper` | RSS news feeds | BBC, CNBC Indonesia, CNN Indonesia, Antara, Republika | No | `examples/foreign-news.example.ts` | `getBbcNews`, `getCnbcNews`, `getCnnNews`, `getAntaraNews`, `getRepublikaNews` |
+| `GDriveScraper` | Public Google Drive file metadata and direct download URL | `drive.google.com/uc` | No for public files | `examples/gdrive.example.ts` | `getFileInfo(fileIdOrUrl)` |
+| `GenshinImpactScraper` | Genshin official manga and Hoyowiki entries | `genshin.hoyoverse.com`, `sg-public-api.hoyolab.com` | No | `examples/genshinimpact.example.ts` | `getMangaChapters`, `getWikiCategories`, `getWikiEntries` |
+| `GoqrScraper` | QR code image generation | `api.qrserver.com` | No | `examples/goqr.example.ts` | Returns image `Buffer` |
+| `GutendexScraper` | Project Gutenberg book metadata search | Gutendex JSON API | No | `examples/gutendex.example.ts` | `searchBooks`, `getBook`; metadata only, no bulk book downloads |
+| `HackerNewsScraper` | Hacker News stories, items, and users | Hacker News Firebase API | No | `examples/hacker-news.example.ts` | `getTopStories`, `getNewStories`, `getBestStories`, `getItem`, `getUser` |
 | `HokInfoScraper` | Honor of Kings character info | Fandom MediaWiki parse API | No | `examples/hok-info.example.ts` | Uses `api.php?action=parse` |
 | `HtmlToJpgScraper` | HTML file to JPG conversion | `api.freeconvert.com` | No credential in code | `examples/html-to-jpg.example.ts` | File-based conversion; skipped in runner without fixture |
 | `IkiruMangaScraper` | Manga search | `02.ikiru.wtf` | No | `examples/ikiru-manga.example.ts` | Mock server fallback available |
 | `ImageScraper` | Safebooru image search | `safebooru.org` | No | `examples/image.example.ts` | Current site type: `safebooru` |
+| `ImgflipScraper` | Meme templates and caption generation | `api.imgflip.com` | Credentials required for `captionMeme` only | `examples/imgflip.example.ts` | `getMemes`, `captionMeme` |
 | `ImgUpscalerScraper` | Image upscaling | `get1.imglarger.com` | No | `examples/img-upscaler.example.ts` | Mocked in example runner |
 | `InstagramScraper` | Profile, feed, post, download | `instagram.com` web/API endpoints | Internal defaults; custom cookie supported | `examples/instagram.example.ts` | Use only legitimate session cookies |
+| `KemendagScraper` | Indonesian basic goods dataset discovery | `data.go.id` CKAN API | No | `examples/kemendag.example.ts` | `getBapokDatasets()` |
 | `KomikindoScraper` | Manga search/detail | `komikindo.ch` | No | `examples/komikindo.example.ts` | `search`, `getDetail` |
+| `KompasNewsScraper` | Kompas news search | `search.kompas.com` | No | `examples/kompas-news.example.ts` | `search(query)` |
+| `LikeeScraper` | Likee public video metadata | `likeedownloader.com/process` | No | `examples/likee.example.ts` | `getInfo(url)` |
 | `LyricsScraper` | Lyrics search | `lrclib.net` JSON API | No | `examples/lyrics.example.ts` | Replaced blocked HTML scraping with API integration |
-| `MConverterScraper` | File conversion helpers | `mconverter.eu` | No | `examples/mconverter.example.ts` | `getTargets`, `convert`, `convertBuffer` |
 | `McAddonScraper` | Minecraft addon search/detail | `mmcreviews.com` | No | `examples/mcaddon.example.ts` | `search`, `getDetail`, `getAddon` |
+| `MConverterScraper` | File conversion helpers | `mconverter.eu` | No | `examples/mconverter.example.ts` | `getTargets`, `convert`, `convertBuffer` |
 | `MediafireScraper` | Resolve Mediafire download link | Mediafire public HTML page | No | `examples/mediafire.example.ts` | Default example uses active 10MB test file |
 | `ModAndroidScraper` | Android APK/mod search aggregations | `an1.com`, `modyolo.com`, `aptoide.com`, `uptodown.com` | No | `examples/mod-android.example.ts` | `android1`, `modyolo`, `aptoide`, `uptodown`, `searchAll` |
+| `MyAnimeListScraper` | Anime search and top anime via Jikan | `api.jikan.moe/v4` | No | `examples/myanimelist.example.ts` | `search`, `getTopAnime` |
+| `NanoBananaScraper` | AI image edit workflow | `app.live3d.io` | No | `examples/nanobanana.example.ts` | Buffer upload, job creation, and polling |
 | `OcrScraper` | OCR image scan | `newocr.com` | No | `examples/ocr.example.ts` | File/buffer based |
+| `OpenAlexScraper` | Scholarly works, authors, and institutions | OpenAlex API | No | `examples/openalex.example.ts` | `searchWorks`, `getWork`, `searchAuthors`, `searchInstitutions`; optional polite `mailto` option |
+| `OpenMeteoScraper` | Weather forecast and current weather by coordinate | Open-Meteo forecast API | No | `examples/open-meteo.example.ts` | `getForecast`, `getCurrentWeather`; validates coordinates and forecast bounds |
+| `OpenLibraryScraper` | Open Library book, work, author, and ISBN lookup | Open Library JSON APIs | No | `examples/open-library.example.ts` | `searchBooks`, `getWork`, `getAuthor`, `getByIsbn` |
+| `OsvScraper` | Open Source Vulnerabilities package and vulnerability lookup | OSV API v1 | No | `examples/osv.example.ts` | `queryPackage`, `getVulnerability`; public advisory metadata |
+| `PackagistScraper` | PHP Composer package metadata and search | Packagist search API and Composer metadata API | No | `examples/packagist.example.ts` | `searchPackages`, `getPackage` |
 | `PhotoAiScraper` | Photo AI upload/status | `photoai.imglarger.com` | No | `examples/photo-ai.example.ts` | Mocked in example runner |
 | `PinterestScraper` | Pin search, detail, download | `pinterest.com` | Internal defaults; custom cookie supported | `examples/pinterest.example.ts` | Supports `credentials` option |
+| `PixivScraper` | Pixiv artwork metadata | `pixiv.net/ajax/illust` | Internal defaults; custom credentials supported | `examples/pixiv.example.ts` | `getIllust(illustId)` |
 | `PlayStoreScraper` | Google Play app search | `play.google.com` | No | `examples/playstore.example.ts` | `search(query, limit)` |
+| `PubgMobileScraper` | PUBG Mobile announcement list | `pubgmobile.com` server-rendered news path | No | `examples/pubgmobile.example.ts` | `getNews()` |
+| `PypiScraper` | Python package and release metadata | PyPI JSON API | No | `examples/pypi.example.ts` | `getPackage`, `getRelease` |
+| `RemovebgScraper` | Background removal | Official `remove.bg` API | Yes (API key) | `examples/removebg.example.ts` | Prefer constructor credentials with `apiKey` |
 | `ResepScraper` | Recipe search | `cookpad.com` | No | `examples/resep.example.ts` | Returns recipe items |
+| `RestCountriesScraper` | Country lookup by name/code and bounded country lists | REST Countries v3.1 API | No | `examples/restcountries.example.ts` | `getAll(fields)` requires explicit fields to keep payloads bounded |
+| `RorScraper` | Research organization registry search and lookup | ROR REST API | No | `examples/ror.example.ts` | `searchOrganizations`, `getOrganization` |
 | `SakuraNovelScraper` | Novel search/detail/chapter | `sakuranovel.id` | No | `examples/sakura-novel.example.ts` | Mock server fallback available |
+| `SfileScraper` | Sfile public file metadata | `sfile.mobi`, `sfile.co` | No | Not yet available | `getMetadata(url)` |
+| `SoundcloudScraper` | SoundCloud track metadata | `soundcloud.com`, `api-v2.soundcloud.com` | No | `examples/soundcloud.example.ts` | Extracts or falls back to a client ID |
 | `SpotifyScraper` | Spotify track, album, playlist, search, downloads | Spotify Web API and Accounts API | Internal defaults; custom client credentials supported | `examples/spotify.example.ts` | Partial custom credentials are rejected |
 | `StalkScraper` | npm package metadata lookup | `registry.npmjs.org` | No | `examples/stalk.example.ts` | Default example query is `axios` |
+| `TeraboxScraper` | TeraBox public share listing | `terabox.com/share/list` | Internal defaults; custom credentials supported | `examples/terabox.example.ts` | `getShareList(url)` |
+| `TextReplaceScraper` | Replace text in images | `imgupscaler.ai`, `magiceraser.org` APIs | No | `examples/text-replace.example.ts` | Buffer upload, job creation, and polling |
 | `ThreadsScraper` | Threads profile, post, search, download | `threads.net` web/API endpoints | Internal defaults; custom cookie supported | `examples/threads.example.ts` | Use only legitimate session cookies |
+| `TiktokScraper` | TikTok video, user, post, and search metadata | `tiktok.com` web/API endpoints | Internal defaults; custom credentials supported | `examples/tiktok.example.ts` | `getVideo`, `getUser`, `getUserPosts`, `searchVideos`, `searchUsers` |
 | `TopAnimeScraper` | MyAnimeList top anime list | `myanimelist.net` | No | `examples/top-anime.example.ts` | `getTopAnime(limit)` |
 | `TwitterScraper` | Tweet, profile, search, timeline, downloads | `x.com/i/api/graphql` | Cookie/CSRF commonly required | `examples/twitter.example.ts` | Supports `credentials` option |
 | `UguuScraper` | Temporary file upload | `uguu.se/upload` | No | `examples/uguu.example.ts` | `upload(buffer, filename)` |
+| `UnblurVideoScraper` | Video enhancement workflow | `api.unblurimage.ai` | No | `examples/unblur-video.example.ts` | Buffer upload, OSS PUT, job creation, and polling |
+| `UnsplashScraper` | Unsplash photo search | `unsplash.com/napi/search/photos` | No | `examples/unsplash.example.ts` | `searchPhotos(query, page, perPage)` |
+| `UnwatermarkScraper` | Image watermark/text/logo restoration workflow | `api.unwatermark.ai` | No | `examples/unwatermark.example.ts` | Buffer upload, job creation, and polling |
 | `UpscalerScraper` | Image enhancement | `aienhancer.ai` | No | `examples/upscaler.example.ts` | Rejects remote URL string input |
+| `UpscalerV3Scraper` | Image upscale, background removal, style, and deblur actions | `imageupscaler.com/wp-admin/admin-ajax.php` | No | `examples/upscaler-v3.example.ts` | Requires explicit `nonce` and `pid` parameters |
+| `UsgsEarthquakeScraper` | Earthquake feeds and bounded event queries | USGS earthquake GeoJSON feeds and event API | No | `examples/usgs-earthquake.example.ts` | `getSummary`, `queryEvents`; validates feed/query bounds |
 | `VideyScraper` | Video upload | `videy.co/api/upload` | No | `examples/videy.example.ts` | `upload`, `uploadBuffer` |
 | `WallpaperScraper` | Wallpaper search | `wallhaven.cc/api/v1/search` | No | `examples/wallpaper.example.ts` | Replaced blocked HTML scraping with API integration |
 | `Webp2Mp4Scraper` | WebP to MP4/PNG conversion | `ezgif.com` | No | `examples/webp2mp4.example.ts` | `toMp4`, `toPng` |
+| `WikipediaScraper` | Wikipedia search and page summary | MediaWiki and REST summary APIs | No | `examples/wikipedia.example.ts` | `search`, `getSummary` |
+| `WorldBankScraper` | World Bank country and indicator metadata/data | World Bank API v2 | No | `examples/world-bank.example.ts` | `searchCountries`, `getCountry`, `searchIndicators`, `getIndicator` |
 | `WwCharScraper` | Wuthering Waves character info | Fandom MediaWiki parse API | No | `examples/ww-char.example.ts` | Uses `api.php?action=parse` |
+| `XiaohongshuScraper` | Xiaohongshu note metadata | Xiaohongshu web endpoints | Internal defaults; custom credentials supported | `examples/xiaohongshu.example.ts` | `getNote(url)` |
 | `YouTubeScraper` | YouTube metadata, search, playlist, channel, downloads | `yt-dlp`, `play-dl`, YouTube pages | No cookie option exposed in current scraper options | `examples/youtube.example.ts` | Media download features need external tools |
 
+## Scraper Response Reference
+
+Every scraper method still uses the standard `WsperResponse<T>` envelope. The table below documents `response.data` for scraper exports that were missing from the previous `Available Scrapers` table or only mentioned in prose.
+
+| Scraper | Primary methods | Success `response.data` |
+| --- | --- | --- |
+| `AioScraper` | `download(url)` | `AioResult`: optional `title`, `thumbnail`, `duration`, `source`, and `medias[]` entries with `url`, `quality`, `type`, `ext`, optional `size`. |
+| `ArenaAiScraper` | `getCategories()`, `getLeaderboard(category)` | Categories return `{ date, fetched_at, leaderboards, errors }`; leaderboard returns `{ meta, models }` where each model has rank, model, vendor, license, score, confidence interval, and votes. |
+| `BimasIslamScraper` | `getProvinces()`, `getCities(provinceId)`, `getPrayerTimes(provinceId, cityId, month, year)` | Region methods return `{ id, name }[]`; prayer times return daily entries with `tanggal`, `imsak`, `subuh`, `terbit`, `dhuha`, `dzuhur`, `ashar`, `maghrib`, and `isya`. |
+| `BinanceScraper` | `getTicker24hr(symbol)`, `getAveragePrice(symbol)`, `getKlines(symbol, interval, options?)`, `getOrderBook(symbol, limit?)` | Public market data only. Ticker/average/order book/klines preserve price and quantity strings from Binance and include source timestamps/IDs where available. No trading or account endpoints are implemented. |
+| `ClinicalTrialsScraper` | `searchStudies(query, options?)`, `getStudy(nctId)` | Search returns `{ totalCount, nextPageToken, studies }`; each study includes NCT ID, title, status, date fields, conditions, interventions, phases, study type, and lead sponsor metadata. |
+| `CodashopScraper` | `checkNickname(game, id, zone?)` | `{ success, game, name, userId, zoneId? }`. |
+| `CrossrefScraper` | `searchWorks(query, options?)`, `getWorkByDoi(doi)`, `searchJournals(query, options?)` | Works include DOI, title/subtitle arrays, publisher, type, URL, issued date, subjects, authors, reference count, and score. Journals include title, publisher, ISSNs, DOI count, and subjects. |
+| `DetikNewsScraper` | `search(query, resultType?)` | `DetikNewsItem[]` with `title`, `url`, `imageUrl`, `category`, `description`, and `date`. |
+| `DonghubScraper` | `search(query)`, `getDetail(url)` | Search returns `{ query, results }`; detail returns drama metadata including `title`, `url`, `image`, `genres`, `synopsis`, and `episodeList`. |
+| `DownrScraper` | `getVideo(url)` | `DownrResult`: optional `url`, `title`, `author`, `duration`, `thumbnail`, and `medias[]` entries with URL, quality, size, extension, and media type. |
+| `EcbScraper` | `getDailyReferenceRates()`, `convertFromEuro(amount, currency)` | Daily rates return `{ date, base: "EUR", rates }`; conversion returns amount, target currency, rate, converted amount, and rate date from the ECB eurofxref XML feed. |
+| `FacebookScraper` | `getUserProfile(usernameOrId)`, `getPostDetail(url)`, `downloadImage(url)` | Profile data, post detail data, or `{ url, buffer?, title? }` for downloaded image data. |
+| `FaceswapV2Scraper` | `swap(targetImageUrl, targetFaceUrl)` | `{ url, requestId? }`. |
+| `ForeignNewsScraper` | `getBbcNews(region?)`, `getCnbcNews()`, `getCnnNews()`, `getAntaraNews()`, `getRepublikaNews()` | `ForeignNewsItem[]` with `title`, `description`, `link`, `pubDate`, and `guid`. |
+| `GDriveScraper` | `getFileInfo(fileIdOrUrl)` | `{ filename, size, mimeType, directDownloadUrl, requiresConfirm, confirmToken }`. |
+| `GenshinImpactScraper` | `getMangaChapters(lang?)`, `getWikiCategories()`, `getWikiEntries(menuId, pageNum?, pageSize?)` | Manga chapters, wiki categories, or wiki entries with IDs, names, icons, rarity, weapon type, and element where available. |
+| `GoqrScraper` | `generateQrCode(data, size?)` | `Buffer` containing the generated QR code image bytes. |
+| `GutendexScraper` | `searchBooks(query, options?)`, `getBook(id)` | Search returns `{ count, next, previous, results }`; each book includes ID, title, authors/translators, subjects, bookshelves, languages, copyright flag, media type, format URLs, and download count. |
+| `HackerNewsScraper` | `getTopStories(limit?)`, `getNewStories(limit?)`, `getBestStories(limit?)`, `getItem(id)`, `getUser(username)` | Story list methods return `HackerNewsItem[]` with IDs, titles, authors, scores, URLs, comment IDs, timestamps, and descendant counts. Item/user methods return a single item or user object, or `null` when the Firebase API returns no object. |
+| `ImgflipScraper` | `getMemes()`, `captionMeme(options)` | Meme templates return `ImgflipMemeTemplate[]`; captioning returns `{ url, page_url }`. |
+| `KemendagScraper` | `getBapokDatasets()` | `KemendagBapokItem[]` with title, resource ID, format, download URL, and last-modified timestamp. |
+| `KompasNewsScraper` | `search(query)` | `KompasNewsItem[]` with `title`, `url`, `imageUrl`, `category`, `date`, and `description`. |
+| `LikeeScraper` | `getInfo(url)` | `{ pageUrl, thumbnail?, title?, playUrl? }`. |
+| `MyAnimeListScraper` | `search(query, limit?)`, `getTopAnime()` | `MalAnimeItem[]` with MAL ID, titles, type, episode count, status, score, rating, synopsis, image URL, and genres. |
+| `NanoBananaScraper` | `edit(buffer, prompt)` | `{ resultUrl }`. |
+| `OpenAlexScraper` | `searchWorks(query, options?)`, `getWork(idOrDoi)`, `searchAuthors(query, options?)`, `searchInstitutions(query, options?)` | Search methods return `{ meta, results }`. Works include IDs, DOI, title/display name, publication year, type, citation count, and open access status. Authors and institutions include counts and affiliation/country metadata. |
+| `OpenMeteoScraper` | `getForecast(latitude, longitude, options?)`, `getCurrentWeather(latitude, longitude)` | `OpenMeteoForecast` with coordinates, timezone, elevation, `currentWeather`, and optional `hourly`/`daily` series records. Invalid coordinates and forecast ranges return validation responses. |
+| `OpenLibraryScraper` | `searchBooks(query, options?)`, `getWork(workKey)`, `getAuthor(authorKey)`, `getByIsbn(isbn)` | Search returns Open Library docs with title, author names, first publish year, ISBNs, language, subjects, and cover ID. Work, author, and ISBN methods return normalized metadata for the requested Open Library key. |
+| `OsvScraper` | `queryPackage({ ecosystem, name, version? })`, `getVulnerability(id)` | Query returns `{ vulns }`; vulnerabilities include OSV ID, summary/details, aliases, published/modified timestamps, database-specific metadata, and affected package/range/version entries. |
+| `PackagistScraper` | `searchPackages(query, options?)`, `getPackage(name)` | Search returns package summary rows with name, description, repository, downloads, and favorites. Package lookup returns normalized Composer versions with license, authors, requirements, source URL, and dist URL. |
+| `PixivScraper` | `getIllust(illustId)` | `PixivArtwork` with artwork ID, title, description, type, created date, image URLs, tags, user ID, and user name. |
+| `PubgMobileScraper` | `getNews()` | `PubgMobileNewsItem[]` with `title`, `date`, `url`, and `summary`. |
+| `PypiScraper` | `getPackage(name)`, `getRelease(name, version)` | PyPI package metadata with normalized `info`, release file maps, and distribution URLs. File records include filename, package type, Python version, URL, size, upload time, and digests. |
+| `RemovebgScraper` | `remove(buffer)` | `{ buffer }` where `buffer` contains the background-removed image bytes. |
+| `RestCountriesScraper` | `getByName(name, options?)`, `getByCode(code, options?)`, `getAll(fields)` | Country records with names, ISO codes, capital, region, population, flags, timezones, coordinates, and optional currencies, languages, maps, and status fields. `getAll` requires explicit fields to avoid unbounded responses. |
+| `RorScraper` | `searchOrganizations(query, options?)`, `getOrganization(id)` | ROR organization metadata with ID, name, status, types, country, links, aliases, and acronyms. `getOrganization` accepts a compact ROR ID or `https://ror.org/...` URL. |
+| `SfileScraper` | `getMetadata(url)` | `SfileMetadata` with optional filename, size, author, upload date, download count, plus `pageUrl`. |
+| `SoundcloudScraper` | `getTrack(url)` | `SoundcloudTrack` with track metadata, artwork, duration, genre, user info, stats, and optional stream URL. |
+| `TeraboxScraper` | `getShareList(url)` | `{ shareid, uk, list }`; each file includes `fsId`, filename, size, directory flag, category, path, and optional download URL. |
+| `TextReplaceScraper` | `replace(buffer, originalText, replaceText, fileName?)` | `{ outputUrl }`. |
+| `TiktokScraper` | `getVideo(url)`, `getVideoDirect(url)`, `getUser(username)`, `getUserPosts(username, count?)`, `searchVideos(query, count?)`, `searchUsers(query, count?)` | Video metadata, user metadata, post arrays, search result objects, or user search arrays depending on the method. |
+| `UnblurVideoScraper` | `enhance(buffer, resolution?, fileName?)` | `{ jobId, inputUrl?, outputUrl? }`. |
+| `UnsplashScraper` | `searchPhotos(query, page?, perPage?)` | `UnsplashPhotoItem[]` with IDs, descriptions, dimensions, likes, image URLs, and photographer info. |
+| `UnwatermarkScraper` | `restore(buffer)` | `{ jobId, inputUrl?, outputUrl? }`. |
+| `UpscalerV3Scraper` | `process(buffer, params)` | `{ output }`; `params` must include `functionType`, explicit `nonce`, and explicit `pid`. |
+| `UsgsEarthquakeScraper` | `getSummary(feed?)`, `queryEvents(options?)` | `UsgsEarthquakeSummary` with feed metadata, generated timestamp, count, and normalized event rows including magnitude, place, ISO timestamps, coordinates, depth, significance, alert/status, and source URLs. |
+| `WikipediaScraper` | `search(query, lang?)`, `getSummary(title, lang?)` | Search returns result rows with title, page ID, snippet, and timestamp; summary returns title, extracts, description, optional thumbnail URL, and page URL. |
+| `WorldBankScraper` | `searchCountries(query, options?)`, `getCountry(code)`, `searchIndicators(query, options?)`, `getIndicator(country, indicator, options?)` | Country and indicator searches return `{ pagination, items }`; indicator data returns `{ pagination, values }` with country, indicator, date, numeric value, unit, observation status, and decimal metadata. |
+| `XiaohongshuScraper` | `getNote(url)` | `XiaohongshuNote` with title, description, type, timestamp, user, images, and engagement stats. |
+
 ## What's New
 
-The latest scraper reliability pass fixed 12 failing scraper functions and their corresponding tests.
+The latest research implementation passes added sixteen public, no-credential scrapers from the roadmap:
+
+- `UsgsEarthquakeScraper` for USGS earthquake GeoJSON feeds and bounded event queries.
+- `OpenMeteoScraper` for Open-Meteo forecast/current weather by coordinate.
+- `RestCountriesScraper` for country lookup by name/code and bounded `getAll(fields)` lists.
+- `HackerNewsScraper` for Hacker News stories, items, and users through the public Firebase API.
+- `OpenLibraryScraper` for Open Library book search, work, author, and ISBN metadata.
+- `GutendexScraper` for Project Gutenberg metadata through the Gutendex JSON API.
+- `WorldBankScraper` for World Bank countries, indicators, and bounded indicator time-series.
+- `EcbScraper` for ECB daily euro reference rates and EUR conversion helpers.
+- `CrossrefScraper` for scholarly work and journal metadata through the Crossref REST API.
+- `OpenAlexScraper` for scholarly works, authors, and institutions.
+- `RorScraper` for research organization registry search and lookup.
+- `ClinicalTrialsScraper` for ClinicalTrials.gov public study search and detail lookup.
+- `PypiScraper` for Python package and release metadata through the PyPI JSON API.
+- `PackagistScraper` for PHP Composer package metadata and search.
+- `OsvScraper` for public OSV package vulnerability queries and vulnerability detail lookup.
+- `BinanceScraper` for Binance public market data only, with no account or trading endpoints.
+- Verification for the latest full pass: `npm run typecheck`, `npm run test`, and `npm run build`; full Vitest suite reported 132 test files passed and 441 tests passed.
+
+The previous scraper reliability pass fixed 12 failing scraper functions and their corresponding tests.
 
 - `LyricsScraper` now uses LRCLIB's public JSON API instead of a Cloudflare-protected Musixmatch HTML page.
 - `WallpaperScraper` now uses Wallhaven's public search API instead of a Cloudflare-protected Wallpaperflare HTML page.
@@ -537,6 +739,22 @@ npx tsx examples/wallpaper.example.ts cyberpunk
 npx tsx examples/stalk.example.ts axios
 npx tsx examples/mediafire.example.ts "https://www.mediafire.com/file/ipnyzofjcwri357/test-10mb.bin/file"
 npx tsx examples/upscaler.example.ts testassets/photo.jpg
+npx tsx examples/usgs-earthquake.example.ts all_day
+npx tsx examples/open-meteo.example.ts -6.2 106.8
+npx tsx examples/restcountries.example.ts Indonesia
+npx tsx examples/hacker-news.example.ts 10
+npx tsx examples/open-library.example.ts dune
+npx tsx examples/gutendex.example.ts alice
+npx tsx examples/world-bank.example.ts IDN SP.POP.TOTL
+npx tsx examples/ecb.example.ts 100 USD
+npx tsx examples/crossref.example.ts "open science"
+npx tsx examples/openalex.example.ts "open science"
+npx tsx examples/ror.example.ts "University of Indonesia"
+npx tsx examples/clinical-trials.example.ts diabetes
+npx tsx examples/pypi.example.ts requests
+npx tsx examples/packagist.example.ts monolog/monolog
+npx tsx examples/osv.example.ts PyPI requests
+npx tsx examples/binance.example.ts BTCUSDT
 ```
 
 For scrapers that support cookies or credentials, use only accounts and sessions you are authorized to access. Do not hardcode real cookies, tokens, client secrets, or API keys in source files.
@@ -571,6 +789,24 @@ npm run build
 npx tsx examples/alllexamp.ts
 ```
 
+### Live/e2e tests and credentials
+
+Unit tests run fully offline (mocks + local fixture servers) and never need
+real credentials. Tests that hit live platform APIs are guarded by
+`tests/helpers/credentials.ts` — when the required `WSPER_*` env variables are
+missing they are **skipped with a clear message** instead of failing:
+
+```bash
+# everything offline — live tests skip cleanly
+npm run test
+
+# include live Spotify tests
+WSPER_SPOTIFY_CLIENT_ID=... WSPER_SPOTIFY_CLIENT_SECRET=... npm run test:spotify
+
+# include live Instagram tests
+WSPER_INSTAGRAM_COOKIE=... npm run test:instagram
+```
+
 ## Project Structure
 
 ```txt
@@ -599,6 +835,57 @@ tests/
 dist/                      Build output only; do not edit manually
 ```
 
+## Credentials Configuration
+
+The library never ships personal credentials. Resolution order per scraper:
+
+1. **Constructor injection (recommended)** — pass `options.credentials` when creating a scraper.
+2. **Environment variables** — optional `WSPER_*` fallbacks read lazily by `src/core/credentials.ts`.
+3. **Nothing** — the scraper runs in public mode, or methods that require credentials fail with a clear error code (e.g. `CAI_CREDENTIALS_REQUIRED`, `SPOTIFY_CREDENTIALS_MISSING`).
+
+```ts
+const scraper = new InstagramScraper({
+  credentials: { cookie: process.env.WSPER_INSTAGRAM_COOKIE },
+  queue: {
+    concurrency: 1,
+    minDelayMs: 1500,
+    maxDelayMs: 4000,
+    timeoutMs: 30000,
+    retries: 2,
+  },
+});
+```
+
+Supported credential env variables (see `.env.example`, copy to `.env` for examples):
+
+| Variable | Platform |
+| --- | --- |
+| `WSPER_SPOTIFY_CLIENT_ID` / `WSPER_SPOTIFY_CLIENT_SECRET` | Spotify Web API |
+| `WSPER_SPOTIFY_CALLBACK_URL` / `WSPER_SPOTIFY_MARKET` | Spotify OAuth / market |
+| `WSPER_INSTAGRAM_COOKIE` | Instagram |
+| `WSPER_THREADS_COOKIE` | Threads |
+| `WSPER_TWITTER_COOKIE` | Twitter/X |
+| `WSPER_PINTEREST_COOKIE` | Pinterest |
+| `WSPER_TIKTOK_COOKIE` | TikTok |
+| `WSPER_FACEBOOK_COOKIE` | Facebook |
+| `WSPER_BILIBILI_COOKIE` | BiliBili |
+| `WSPER_CAI_TOKEN` | Character.AI |
+| `REMOVEBG_API_KEY` | remove.bg |
+
+For running the bundled examples with your own account, copy
+`examples/config/credentials.example.ts` to `examples/config/credentials.ts`
+(gitignored) and fill in your values. Examples run in two modes via
+`WSPER_EXAMPLE_MODE`:
+
+- `smoke` (default) — fast, few requests, local credentials are **not** loaded; pair with `WSPER_MOCK_BASE_URL` for fully offline runs.
+- `real` — loads `examples/config/credentials.ts` (or `.env`) and uses a polite queue. Use your own cookies/tokens only; you bear the platform ToS and rate-limit risk.
+
+> **Migration note:** `src/core/credentials.ts` previously contained baked-in
+> default cookies/tokens. These were removed; the exported `credentials` object
+> now resolves from `WSPER_*` env variables and is empty by default. If you
+> relied on the old defaults, inject credentials via constructor options or env.
+> Rotate any cookies/tokens that were ever committed.
+
 ## Environment Variables
 
 These variables appear in the repository:
@@ -612,7 +899,7 @@ These variables appear in the repository:
 | `BILI_COOKIE` | `examples/bilibili.example.ts` | No | Optional BiliBili cookie for authenticated stream access. |
 | `WSPER_COOKIE` | `tests/core/credentials.test.ts` | No | Test-only variable proving runtime credential resolution does not read env credentials automatically. |
 
-Credential configuration is constructor-based. The library does not rely on `.env` files for runtime credentials.
+Credential configuration is constructor-first with optional `WSPER_*` env fallbacks (see "Credentials Configuration" above). The library itself never reads `.env` files; only the examples helper (`examples/config/runtime.ts`) parses a local `.env` for convenience.
 
 ```ts
 import { WsperScraper } from "wsper-js";
@@ -623,6 +910,9 @@ const wsper = new WsperScraper({
     clientSecret: "your-client-secret",
   },
   credentials: {
+    cai: {
+      bearerToken: "<YOUR_TOKEN_HERE>",
+    },
     twitter: {
       cookie: "<YOUR_COOKIE_HERE>",
       csrfToken: "<YOUR_CSRF_TOKEN_HERE>",