@media-engine/providers 0.1.0 → 1.0.0

package/README.md

@@ -1,295 +1,202 @@
 # @media-engine/providers
 
-Provider package for Media Engine.
+**English** | [Русский](https://github.com/Yaneart/media-engine/blob/main/packages/providers/README.ru.md)
 
-This package contains no-token metadata provider factories such as KinoBD, Cinemeta, Shikimori, AniList, Wikidata, and local IMDb datasets, plus no-token KinoBD and FlixHQ streaming providers and a local experimental provider.
+Ready-to-use data sources for Media Engine.
 
-The package depends on `@media-engine/core` for provider contracts and normalized media types. Core must not import this package.
-
-## Install
+Install this package when you do not want to write your own provider adapters.
 
 ```bash
 npm install @media-engine/core @media-engine/providers
 ```
 
-Use this package from server-side or trusted application code. Do not expose provider tokens in browser bundles.
-
-Current structure:
-
-```txt
-src/
-  shared/
-  kinobd/
-  cinemeta/
-  shikimori/
-  anilist/
-  wikidata/
-  imdb-dataset/
-  experimental-streaming/
-  kinobd-streaming/
-  flixhq-streaming/
-  index.ts
-```
-
-No API keys or environment reads are stored in this package. Applications pass provider secrets from the outside.
-
-Streaming providers in this package return normalized embed/player access metadata. They do not make Media Engine a streaming service, do not host video, and do not extract direct video files by default. Review each upstream source's usage rules before enabling a provider in a product.
-
-## KinoBD Provider
-
-`kinobdProvider` creates a no-token metadata provider for movies and series through the public KinoBD API used by ReYohoho-style clients.
+## A small setup
 
 ```ts
 import { MediaEngine } from "@media-engine/core";
-import { kinobdProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  providers: [kinobdProvider()],
+import {
+  aniListProvider,
+  ddbbStreamingProvider,
+  flixHqStreamingProvider,
+  kinobdProvider,
+  kinobdStreamingProvider,
+  shikimoriProvider,
+  tvMazeProvider,
+} from "@media-engine/providers";
+
+const media = new MediaEngine({
+  providers: [kinobdProvider(), shikimoriProvider(), aniListProvider(), tvMazeProvider()],
 });
-```
-
-Supported data:
 
-- title search for movies and series;
-- IMDb and Kinopoisk ID lookup;
-- movie and series details;
-- posters, Kinopoisk ratings, IMDb ratings, genres, countries, and persons when available;
-- configurable `imageLimit` and `personLimit` for details payloads.
-
-KinoBD is the first no-token movie and series provider for the local API stand because it returns practical Russian metadata, Kinopoisk IDs, IMDb IDs, ratings, and posters without requiring a user token.
-
-## Cinemeta Provider
-
-`cinemetaProvider` creates a no-token metadata provider for movies and series through the public Cinemeta/Stremio metadata API.
-
-```ts
-import { MediaEngine } from "@media-engine/core";
-import { cinemetaProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  providers: [cinemetaProvider()],
-});
+const result = await media.search({ title: "One Piece" });
 ```
 
-Supported data:
+Add only the providers that make sense for your application. Media Engine will call compatible ones and merge matching answers.
 
-- title search for movies and series;
-- IMDb ID lookup;
-- movie and series details;
-- posters, backdrops, IMDb ratings, genres, cast, writers, and directors when available;
-- configurable `imageLimit`, `personLimit`, and `enrichSearchLimit`.
+## Available metadata sources
 
-Cinemeta is a secondary no-token movie and series provider. Its results are merged with KinoBD and Wikidata to improve metadata completeness without application secrets.
+- `kinobdProvider()` — localized movie and series data;
+- `cinemetaProvider()` — IMDb-linked movie and series data;
+- `shikimoriProvider()` — anime search and details;
+- `aniListProvider()` — international anime titles, popularity, and artwork;
+- `tvMazeProvider()` — fallback IMDb-backed series identities and localized aliases;
+- `wikidataProvider()` — fallback structured identity and metadata enrichment;
+- `imdbDatasetProvider()` — local IMDb data supplied as small in-memory TSV fixtures or through an application-owned indexed storage adapter.
 
-## Shikimori Provider
+None of these built-in providers needs your API key. TMDB IDs may appear in results, but this package does not call the TMDB API.
 
-`shikimoriProvider` creates a metadata provider for anime.
+TVmaze data is licensed under CC BY-SA. The provider keeps a TVmaze show URL in source attribution; applications using TVmaze data should preserve and render that link. See the [TVmaze API licensing terms](https://www.tvmaze.com/api#licensing).
 
-```ts
-import { MediaEngine } from "@media-engine/core";
-import { shikimoriProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  providers: [
-    shikimoriProvider({
-      userAgent: "MyApp/1.0.0",
-    }),
-  ],
-});
-```
+Wikidata fallback discovery loads at most three title-relevant entities through a selected-property query and caches entity/IMDb mappings for six hours in a 256-entry process-local LRU by default. `entityLimit` is bounded to 1–10, `cacheTtlMs` to 0–7 days, and `cacheMaxEntries` to 2–2048; a zero TTL disables this provider-local cache.
 
-Supported data:
+The backward-compatible IMDb TSV adapter parses the complete input into memory and is intended for small datasets and fixtures. Full-dataset integrations can inject the exported synchronous/asynchronous `ImdbDatasetStorage` contract, including a direct ID lookup and bounded normalized title search, without adding a database dependency for other package users.
 
-- title search for anime;
-- Shikimori ID lookup;
-- anime details;
-- posters, screenshots, ratings, genres, persons, episodes, and alternative titles;
-- Shikimori and MyAnimeList external IDs in normalized results.
+An included persisted adapter can stream plain or gzip IMDb TSV files into a versioned, atomically replaced SQLite/FTS index. `buildImdbDatasetSqliteIndex()` creates it and `openImdbDatasetSqliteStorage()` opens it read-only for `imdbDatasetProvider({ storage })`. This optional path lazily uses built-in `node:sqlite` and requires Node.js 22.13 or newer; importing the package or using the small in-memory adapter keeps the Node.js 20 baseline.
 
-The provider does not store API keys or read environment variables. Tests use mock `fetch` implementations and do not call the real Shikimori API.
+Expected upstream failures are reported as typed `ProviderError` values, and shared HTTP errors expose their originating status through `getProviderHttpStatus`. An untyped Cinemeta IMDb lookup returns `null` only after both movie and series candidates confirm absence; a temporary branch outage remains retryable unless the other branch returned usable details. AniList similarly distinguishes GraphQL rate limits and server outages from validation errors or malformed payloads, allowing Media Engine to avoid caching incomplete metadata as a healthy response.
 
-## AniList Provider
+Shared `fetchJson` calls stream at most 4 MiB by default before parsing and accept a positive `maxResponseBytes` override for provider-specific limits. A declared or chunked oversized body is cancelled and reported as the non-retryable `PROVIDER_RESPONSE_TOO_LARGE`; malformed JSON within the limit remains `PROVIDER_INVALID_RESPONSE`.
 
-`aniListProvider` adds public no-token anime search and details through AniList GraphQL. It complements Shikimori with English and international title aliases, AniList/MyAnimeList IDs, popularity counts, ratings, posters, and genres.
+Low-level adapters may supply a `ProviderHttpScheduler` through `FetchJsonOptions.scheduler` when
+they need deterministic control of retry and total-timeout timers, especially in tests. Normal
+provider calls omit it and use the platform timers.
 
-```ts
-import { aniListProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  providers: [aniListProvider()],
-});
-```
+## Player sources
 
-Public metadata requests do not require OAuth. The provider excludes adult results by default and can be configured with `includeAdult: true` when appropriate for the host application.
-
-## Wikidata Provider
-
-`wikidataProvider` creates a no-token metadata provider for basic movie and series search.
+- `kinobdStreamingProvider()` — movie, series, and anime player options;
+- `flixHqStreamingProvider()` — international movie and selected series-episode options;
+- `ddbbStreamingProvider()` — opt-in Kinopoisk/IMDb lookup through an independent DDBB player route;
+- `aniLibertyStreamingProvider()` — opt-in exact title/year anime lookup with direct HLS episodes;
+- `experimentalStreamingProvider()` — data configured by your own application, useful in tests and UI work.
 
 ```ts
-import { MediaEngine } from "@media-engine/core";
-import { wikidataProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  providers: [wikidataProvider()],
+const media = new MediaEngine({
+  streamingProviders: [
+    kinobdStreamingProvider(),
+    flixHqStreamingProvider(),
+  ],
 });
-```
-
-Supported data:
-
-- title search for movies and series;
-- IMDb ID lookup through Wikidata;
-- basic details with title, description, release date, image, and IMDb ID when present.
 
-Wikidata is a free metadata source whose results are merged with KinoBD and Cinemeta. It is less complete than a dedicated movie database and should be treated as one enrichment source rather than the only source.
-
-## IMDb Dataset Provider
-
-`imdbDatasetProvider` creates a local parser-backed provider for official IMDb non-commercial TSV datasets. It does not call imdb.com pages, does not use an unofficial API, and does not scrape.
-
-```ts
-import { imdbDatasetProvider } from "@media-engine/providers";
-
-const provider = imdbDatasetProvider({
-  titleBasicsTsv: titleBasicsFileContent,
-  titleRatingsTsv: titleRatingsFileContent,
+const result = await media.getAvailability({
+  type: "series",
+  title: "Game of Thrones",
+  seasonNumber: 1,
+  episodeNumber: 1,
 });
 ```
 
-Supported data:
-
-- title search for movies and series;
-- IMDb ID lookup;
-- basic details from `title.basics.tsv`;
-- IMDb ratings from `title.ratings.tsv` when provided.
-
-Before using this provider, download the official IMDb datasets and verify that your use complies with IMDb's non-commercial dataset terms.
-
-## Experimental Streaming Provider
+These are third-party player targets, not videos hosted by Media Engine. Availability depends on the upstream source and the user's environment.
 
-`experimentalStreamingProvider` creates a configured streaming provider for local architecture validation. It does not scrape websites and does not call a real streaming API. Applications pass already allowed embed or external player URLs from the outside.
+The repository API enables `ddbbStreamingProvider()` after its repeated reliability/diversity
+checkpoint; direct package consumers still choose their own provider list explicitly. It accepts
+only Kinopoisk or IMDb IDs, returns generic movie/series/anime embeds, and does not claim exact
+season/episode mapping. Its diversity-first mapping keeps one main option per returned player before
+adding unique translation URLs. Missing nullable players produce no result; confirmed 404/410 or
+stable deletion markers are removed, while transient validation failures remain `unknown`.
 
 ```ts
-import { experimentalStreamingProvider } from "@media-engine/providers";
-
-const provider = experimentalStreamingProvider({
-  name: "local-embed",
-  entries: [
-    {
-      type: "anime",
-      title: "Example Anime",
-      ids: {
-        shikimori: "20",
-      },
-      episodes: [
-        {
-          absoluteEpisodeNumber: 1,
-          options: [
-            {
-              id: "example-episode-1",
-              player: {
-                kind: "embed",
-                label: "Embed Player",
-              },
-              translation: {
-                title: "AniDUB",
-                type: "dub",
-                language: "ru",
-              },
-              quality: {
-                label: "720p",
-                height: 720,
-              },
-              access: {
-                url: "https://example.test/embed/episode-1",
-              },
-              availability: "available",
-            },
-          ],
-        },
-      ],
-    },
+const media = new MediaEngine({
+  streamingProviders: [
+    kinobdStreamingProvider(),
+    flixHqStreamingProvider(),
+    ddbbStreamingProvider(), // explicit opt-in
+    aniLibertyStreamingProvider(), // explicit opt-in
   ],
 });
 ```
 
-Supported behavior:
-
-- lookup by configured external IDs;
-- exact normalized title lookup when IDs are absent;
-- movie-level and episode-level stream options;
-- multiple player, translation, and quality options for one item or episode;
-- provider filtering through `StreamQuery.providers`.
-
-Use this provider only for experiments, tests, and UI wiring. A real provider such as Kodik should be implemented separately only after its API/embed usage rules are documented and allowed.
-
-## KinoBD Streaming Provider
-
-`kinobdStreamingProvider` creates a no-token streaming provider for ReYohoho-style iframe player availability. It calls KinoBD-style player endpoints and returns normalized `StreamOption` values with embed URLs, provider labels, translations, and quality labels.
+The repository API also enables `aniLibertyStreamingProvider()` after that checkpoint. Because
+AniLiberty does not publish MAL, AniList, or Shikimori IDs for releases, the adapter requires both
+title and year, accepts only one exact normalized match, and revalidates the loaded release before
+returning streams. It supports generic episode maps and exact `absoluteEpisodeNumber` queries, but
+does not guess season/episode mappings. Each safe first-party 480p/720p/1080p URL is returned as
+direct HLS; release geo and copyright blocks are preserved as normalized availability states.
+
+Live player validation removes an option only after HTTP 404/410 or a stable deletion marker. Rate limits, server errors, network failures, and validation timeouts keep the discovered option with `availability: "unknown"`, allowing the engine to expose the degradation and retry it instead of caching a transiently reduced result.
+
+KinoBD bounds one availability lookup to 24 child HTTP attempts by default and validates at most eight discovered players through three workers. Public tuning remains bounded (`childRequestLimit` up to 64, `playerValidationLimit` up to 16, and `playerValidationConcurrency` up to 4). Nested iframe validation starts only when the fixed provider deadline can still grant a full validation window. `onPlayerAudit` receives additive `metrics` for discovered and validated players, limit/budget skips, transient unknown results, confirmed removals, and consumed child requests.
+
+FlixHQ site navigation cannot leave its configured origin, including through redirects. External player and subtitle checks resolve every A/AAAA answer, reject private, local, reserved, multicast, or mixed public/private destinations, validate every bounded redirect hop, and pin the connection to the approved address. A custom provider `fetch` is an explicit trusted transport injection intended for controlled tests or self-hosted environments; it is responsible for equivalent network policy.
+
+DDBB caps its JSON response, output option count, live validation count, validation concurrency,
+validation body size, and per-player timeout. Its default transport applies the same hardened DNS,
+redirect, and connection-pinning policy to the DDBB endpoint and returned players. A custom `fetch`
+is the same explicit trusted test/self-hosted boundary used by the other streaming providers. Options
+beyond the bounded validation count remain visible with `availability: "unknown"`; they are never
+presented as successfully checked. Alloha is represented by one main option because voiceovers can
+be switched inside that player; if no main iframe exists, at most one safe translation URL becomes
+the entry point. Other player families keep their distinct translation options.
+
+AniLiberty bounds search candidates, release episodes, JSON bytes, retries, and total provider time
+through the shared engine/provider controls. Its default transport uses the hardened DNS, redirect,
+and connection-pinning policy for API calls. Direct HLS targets still pass the shared browser-facing
+output URL policy; playback network policy remains the consuming application's responsibility.
+
+Before built-in providers expose artwork, player, or subtitle URLs, one output policy accepts only HTTP(S) targets without credentials, raw control characters, or literal local/private/reserved addresses. Valid paths and CDN query parameters, including expiring signatures, are preserved. This browser-facing check does not replace DNS validation or an application-owned media proxy.
+
+## Torrent discovery sources
+
+`ytsTorrentProvider()` is an opt-in no-key movie source. It resolves an exact IMDb identity or one
+exact title/year match and returns normalized 720p/1080p/2160p-style magnet candidates with size,
+release metadata, and best-effort peer counts.
+
+`jacRedTorrentProvider()` is an opt-in no-key Russian/multilingual source for exact title/year
+movie, series, and anime lookup. It can filter a requested season, but intentionally returns no
+result for exact ordinary or absolute episode queries. Returned title, year, category, season,
+info hash, source URL, release metadata, and peer fields are bounded and revalidated. Each result
+also retains the bounded JacRed tracker ID as `catalogSource`; known values receive labels such as
+BitRu, RuTracker, RuTor, Kinozal, NNM-Club, and Knaben without implying a guaranteed audio
+language. Its
+`baseUrl` and `searchPath` are configurable because the live first-party route and published route
+currently differ.
+
+`bitsearchTorrentProvider()` is an opt-in no-key broad international source. It requires title and
+year, pins movie/TV/anime categories, and revalidates the exact release title, explicit year, media
+type, requested season, and requested ordinary or absolute episode before returning a candidate.
+It consumes only the documented bounded search response, remembers an exhausted anonymous quota
+from `X-RateLimit-*`, and emits one canonical magnet per validated info hash. The public anonymous
+tier is currently limited to 200 requests/day per IP, so applications should keep engine caching
+enabled and avoid speculative calls.
+
+`magnetzTorrentProvider()` is an opt-in no-key international meta-search source. It performs one
+bounded first-page search request, then revalidates exact title, year, season, and ordinary or
+absolute episode markers locally. It never fans out into per-result detail calls. The adapter emits
+one canonical magnet per validated info hash, preserves reported peer state and source attribution,
+and spaces request starts by one second because the live service has returned short burst 429s even
+while its rate-limit header still reported capacity.
 
 ```ts
-import { MediaEngine } from "@media-engine/core";
-import { kinobdStreamingProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  streamingProviders: [kinobdStreamingProvider()],
-});
-
-const availability = await engine.getAvailability({
-  type: "movie",
-  ids: {
-    kinopoisk: "258687",
+const media = new MediaEngine({
+  torrentProviders: [
+    ytsTorrentProvider(),
+    jacRedTorrentProvider(),
+    bitsearchTorrentProvider(),
+    magnetzTorrentProvider(),
+  ],
+  providerTimeouts: {
+    "yts-torrent": 15_000,
+    "jacred-torrent": 20_000,
+    "bitsearch-torrent": 15_000,
+    "magnetz-torrent": 15_000,
   },
 });
-```
-
-Supported behavior:
-
-- movie and series lookup through `/api/player/search` and `/playerdata`;
-- anime fallback lookup by resolving a Shikimori ID through Shikimori's public API and searching KinoBD players by title;
-- optional anime cache lookup through a configured `animeCacheBaseUrl` with `/cache_shiki` when an application has an allowed backend for it;
-- player aggregation for the known KinoBD/ReYohoho embeddable player list, including Collaps, Vibix, Alloha, Kodik, KinoTochka, FlixCDN, Ashdi, Turbo, VideoCDN, Bazon, UStore, Pleer, VideoSpider, Iframe, Moonwalk, HDVB, CDNMovies, Lookbase, Kholobok, VideoAPI, Voidboost, Videoseed, and VK keys when upstream returns them;
-- fallback to player iframe candidates from `/api/player/search` when `/playerdata` is unavailable or returns no usable iframe options;
-- normalized embed player options with translation and quality metadata;
-- best-effort translation type and language inference for labels such as Russian dubbing, Ukrainian dubbing, English subtitles, and known Russian voiceover teams including AlexFilm, HDrezka Studio, LE-Production, Shachiburi, LostFilm, AniDUB, AniLibria, and 2x2;
-- best-effort filtering of clearly broken player pages, including HTTP 404/410/5xx and known unavailable-player HTML markers; live player page validation is bounded through `playerValidationLimit` and `playerValidationTimeoutMs`;
-- noisy external-only or non-playback keys such as `ia`, `ext`, `netflix`, `nf`, `torrent`, `trailer`, `trailer_local`, and `youtube` are excluded from requests and filtered from upstream responses;
-- provider filtering through `StreamQuery.providers`.
-
-This provider does not use a Kodik API token and does not extract direct video files. It returns player/embed URLs for the application UI to render.
-
-For live source-filter audits, pass `onPlayerAudit`. The callback reports discovered and shown player labels plus filtered labels with stable reasons such as `provider_not_allowed`, `missing_iframe`, `known_broken_url`, and `player_validation_failed`. Callback failures are isolated from normal availability behavior.
-
-## FlixHQ Streaming Provider
-
-`flixHqStreamingProvider` discovers international movie embeds and explicitly requested series episodes without user credentials.
 
-```ts
-import { MediaEngine } from "@media-engine/core";
-import { flixHqStreamingProvider } from "@media-engine/providers";
-
-const engine = new MediaEngine({
-  streamingProviders: [flixHqStreamingProvider()],
-});
-
-const availability = await engine.getAvailability({
-  type: "series",
-  title: "House of the Dragon",
-  year: 2022,
-  seasonNumber: 2,
-  episodeNumber: 3,
+const torrents = await media.discoverTorrents({
+  type: "movie",
+  title: "Inception",
+  year: 2010,
+  ids: { imdb: "tt1375666" },
 });
 ```
 
-The provider validates discovered embeds with bounded requests and normalizes public `sub.info` subtitle tracks. If upstream explicitly returns an HLS or MP4 URL, it also normalizes the kind, advertised quality, and unambiguous expiry metadata. It does not reverse-engineer protected embed streams. Series lookup requires both season and episode numbers; anime is intentionally unsupported to avoid matching live-action adaptations.
-
-## Shared Utilities
+The provider does not download torrent metadata, contact trackers, join a swarm, or play video.
+The completed multi-source checkpoint kept repository API defaults empty: Bitsearch has a small
+anonymous daily quota, Magnetz exposes burst-sensitive limits, and JacRed still has an occasional
+long timeout tail. Applications can opt into any subset explicitly and monitor it with
+`pnpm smoke:torrents`; package consumers remain in control of their own request budget.
 
-`src/shared` contains provider-side helpers used by future concrete providers:
+Provider options, limitations, and safety rules are summarized in the [provider guide](https://github.com/Yaneart/media-engine/blob/main/docs/providers.md).
 
-- `fetchJson`;
-- `parseJsonResponse`;
-- `mapProviderHttpError`;
-- `mapHttpStatusToProviderErrorCode`.
+## License
 
-These helpers map HTTP, JSON parsing, network, timeout, and rate-limit failures into `ProviderError` from `@media-engine/core`. `fetchJson` retries retryable provider failures with a short backoff by default.
+MIT