gerdur-core 2.15.0 → 2.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +56 -0
- package/README.md +38 -1
- package/dist/api/index.d.ts +1 -0
- package/dist/api/index.js +1 -0
- package/dist/api/library.d.ts +72 -0
- package/dist/api/library.js +65 -0
- package/dist/lib/decrypt-pool.d.ts +28 -0
- package/dist/lib/decrypt-pool.js +230 -0
- package/dist/lib/decrypt-worker.d.ts +6 -0
- package/dist/lib/decrypt-worker.js +22 -0
- package/dist/lib/decrypt.js +17 -3
- package/dist/metadata-writer/getTrackLyrics.d.ts +10 -1
- package/dist/metadata-writer/getTrackLyrics.js +12 -3
- package/dist/metadata-writer/index.d.ts +6 -0
- package/dist/metadata-writer/index.js +8 -1
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,61 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 2.17.0 - 2026-08-31
|
|
4
|
+
|
|
5
|
+
The account's own library, over the authenticated gateway.
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **`src/api/library.ts`** — the logged-in account's *private* library. Distinct
|
|
10
|
+
from the public-REST surface in `api/user.ts`, which needs a public profile and
|
|
11
|
+
only shows what that profile exposes. Found by probing the gateway: seven
|
|
12
|
+
methods answer here that the public API cannot reach.
|
|
13
|
+
- `getMyPlaylists(userId?, nb?, start?)` — **including private playlists**
|
|
14
|
+
- `getMyFavoriteTracks` / `getMyFavoriteAlbums` / `getMyFavoriteArtists`
|
|
15
|
+
- `getMyFavoritePlaylists` / `getMyFavoriteRadios` / `getMyFavoriteShows`
|
|
16
|
+
- `getMyFavoriteTrackIds()` — every loved track id in one small request, for
|
|
17
|
+
diffing a local library against the account
|
|
18
|
+
- `userId` defaults to the logged-in account. Responses are gateway shapes, so
|
|
19
|
+
tracks arrive with a `TRACK_TOKEN` and are directly downloadable; being
|
|
20
|
+
account-scoped they stay in the per-session cache and are never shared.
|
|
21
|
+
- **`getTrackMix(sngId, nb?, start?)`** (`song.getSearchTrackMix`) — a "more like
|
|
22
|
+
this" mix seeded from one track. Unlike the public radio endpoints it returns
|
|
23
|
+
full gateway tracks **with tokens already attached**: verified that
|
|
24
|
+
`resolveDownloadUrls` resolves them 3/3 with no `getTrackInfo` round trip.
|
|
25
|
+
|
|
26
|
+
Verified against a live account: `getMyPlaylists` 1/1, `getMyFavoriteArtists`
|
|
27
|
+
7/7 with real names, `getTrackMix` 5/5 all carrying tokens.
|
|
28
|
+
|
|
29
|
+
## 2.16.0 - 2026-08-31
|
|
30
|
+
|
|
31
|
+
### Added
|
|
32
|
+
|
|
33
|
+
- **`AddTrackTagsOptions.lyricsFallback`** (default `true`). When Deezer has no
|
|
34
|
+
lyrics for a track, the tagger falls back to scraping Musixmatch — two requests
|
|
35
|
+
per track that needs it, and they fail outright wherever Musixmatch blocks you.
|
|
36
|
+
Measured on a 14-track album: **16 requests, all of them wasted** in that case.
|
|
37
|
+
Set it to `false` to keep only what Deezer serves.
|
|
38
|
+
|
|
39
|
+
Combined with `richCredits: false`, tagging that album drops from **54
|
|
40
|
+
requests to 7 — 87% fewer overall, and 81% fewer against Deezer's quota**
|
|
41
|
+
(36 -> 7), which is the constraint that actually binds. The trade is full
|
|
42
|
+
credits, BPM, and non-Deezer lyrics.
|
|
43
|
+
|
|
44
|
+
### Not done, deliberately
|
|
45
|
+
|
|
46
|
+
- A `worker_threads` pool for Blowfish decryption. Decrypting 4 MB batches
|
|
47
|
+
synchronously it measured 2.6x throughput and 4.5x lower p95 event-loop lag —
|
|
48
|
+
but that benchmark manufactured the problem. Through a real pipeline the
|
|
49
|
+
existing in-thread stream already shows **p95 loop lag of 0.0 ms** (it works a
|
|
50
|
+
socket read at a time, not a whole file), and the pooled version came out at
|
|
51
|
+
**0.72x** once three extra 4 MB copies per batch were paid. Decrypt only binds
|
|
52
|
+
above ~265 MB/s of aggregate download in one process. Rationale kept in
|
|
53
|
+
`decrypt.ts`.
|
|
54
|
+
- Batch-hydrating credits via `song.getListData`. Probed live: it returns 44
|
|
55
|
+
fields but **not `SNG_CONTRIBUTORS`**, and neither does `song.getListByAlbum`.
|
|
56
|
+
Deezer exposes contributors only per-track, so those calls are irreducible —
|
|
57
|
+
`richCredits: false` above is the only lever.
|
|
58
|
+
|
|
3
59
|
## 2.15.0 - 2026-08-31
|
|
4
60
|
|
|
5
61
|
Streaming tag write — tagging no longer holds the audio.
|
package/README.md
CHANGED
|
@@ -46,6 +46,7 @@ the CLI and the file-writing layer on top.
|
|
|
46
46
|
- [Search](#search)
|
|
47
47
|
- [Browse and discover](#browse-and-discover)
|
|
48
48
|
- [Flow, radios and a user's library](#flow-radios-and-a-users-library)
|
|
49
|
+
- [Your own library](#your-own-library)
|
|
49
50
|
- [Podcasts](#podcasts)
|
|
50
51
|
- [Preview clips](#preview-clips)
|
|
51
52
|
- [Resolve a download URL](#resolve-a-download-url)
|
|
@@ -381,6 +382,33 @@ const {data: loved} = await getUserFavoriteTracks(me.USER_ID);
|
|
|
381
382
|
const {data: eighties} = await getRadioTracks(38305); // "The '80s"
|
|
382
383
|
```
|
|
383
384
|
|
|
385
|
+
### Your own library
|
|
386
|
+
|
|
387
|
+
`api/user.ts` above reads a **public** profile. These read what the *account*
|
|
388
|
+
can see — private playlists included — over the authenticated gateway, and
|
|
389
|
+
return gateway shapes, so tracks come with a `TRACK_TOKEN` and are immediately
|
|
390
|
+
downloadable.
|
|
391
|
+
|
|
392
|
+
| Function | Returns |
|
|
393
|
+
| :--- | :--- |
|
|
394
|
+
| `getMyPlaylists(userId?, nb?, start?)` | the account's own playlists, **including private ones** |
|
|
395
|
+
| `getMyFavoriteTracks(userId?, nb?, start?)` | loved tracks, as downloadable gw tracks |
|
|
396
|
+
| `getMyFavoriteTrackIds()` | every loved track id in one request — for diffing a local library |
|
|
397
|
+
| `getMyFavoriteAlbums` / `getMyFavoriteArtists` | favourited albums / artists |
|
|
398
|
+
| `getMyFavoritePlaylists` / `getMyFavoriteRadios` / `getMyFavoriteShows` | followed playlists / radios / shows |
|
|
399
|
+
| `getTrackMix(sngId, nb?, start?)` | a "more like this" mix seeded from a track — **tokens already attached** |
|
|
400
|
+
|
|
401
|
+
```ts
|
|
402
|
+
import {getMyPlaylists, getTrackMix, resolveDownloadUrls} from 'gerdur-core';
|
|
403
|
+
|
|
404
|
+
const {data: playlists} = await getMyPlaylists(); // yours, private included
|
|
405
|
+
const {data: mix} = await getTrackMix('3135556', 20); // 20 tracks like this one
|
|
406
|
+
const urls = await resolveDownloadUrls(mix, [9, 3, 1]); // straight to download — no per-track lookup
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
`userId` defaults to the logged-in account. These are account-scoped, so they
|
|
410
|
+
never enter the shared cross-session cache.
|
|
411
|
+
|
|
384
412
|
### Podcasts
|
|
385
413
|
|
|
386
414
|
```ts
|
|
@@ -523,6 +551,7 @@ model.contributors; // normalised producers / engineers / performers / …
|
|
|
523
551
|
| `album` / `lyrics` / `publicTrack` | — | pre-fetched payloads — pass once per album to skip refetching |
|
|
524
552
|
| `embedCover` / `embedArtistImage` | `true` | |
|
|
525
553
|
| `writeLyrics` / `embedSyncedLyrics` | `true` | synced LRC goes to FLAC Vorbis only (no ID3v2.3 `SYLT`) |
|
|
554
|
+
| `lyricsFallback` | `true` | scrape Musixmatch when Deezer has no lyrics — 2 requests per such track, and they fail where Musixmatch blocks you |
|
|
526
555
|
| `richCredits` | `true` | hydrate credits + BPM for album/playlist tracks that omit them |
|
|
527
556
|
| `deezerIds` / `includeRank` | `true` | write `DEEZER_*_ID` / popularity rank |
|
|
528
557
|
|
|
@@ -657,6 +686,11 @@ cacheStats(); // {shared: {size, maxSize, hits, misses, inFlight}} — for /metr
|
|
|
657
686
|
the process. Put the decrypt in a worker if you saturate a core.
|
|
658
687
|
- **Size the shared cache** to your catalogue with `configureCache`, and export
|
|
659
688
|
`cacheStats()` so you can see the hit rate.
|
|
689
|
+
- **Tagging is where the quota goes**, not downloading. A 14-track album costs
|
|
690
|
+
~54 requests to tag and 2 to fetch. `{richCredits: false, lyricsFallback: false}`
|
|
691
|
+
takes that to **7 — 81% fewer against Deezer's quota** — at the cost of full
|
|
692
|
+
credits, BPM and non-Deezer lyrics. Pre-feed `{album, lyrics, cover}` for
|
|
693
|
+
anything you already hold.
|
|
660
694
|
- **Evict idle sessions yourself.** `createSession` has no lifecycle — a session
|
|
661
695
|
per user, kept forever, keeps its cache forever.
|
|
662
696
|
- `httpAgent` / `httpsAgent` are process-global (`maxSockets: 64`) and shared
|
|
@@ -781,7 +815,10 @@ import type {
|
|
|
781
815
|
|
|
782
816
|
`getUserFlow` · `getUserFavoriteTracks` · `getUserFavoriteAlbums` ·
|
|
783
817
|
`getUserFavoriteArtists` · `getUserPlaylists` · `getUserRadios` ·
|
|
784
|
-
`getUserChartTracks` · `getRadios` · `getRadioTracks` · `getRadioGenres`
|
|
818
|
+
`getUserChartTracks` · `getRadios` · `getRadioTracks` · `getRadioGenres` ·
|
|
819
|
+
`getMyPlaylists` · `getMyFavoriteTracks` · `getMyFavoriteTrackIds` ·
|
|
820
|
+
`getMyFavoriteAlbums` · `getMyFavoriteArtists` · `getMyFavoritePlaylists` ·
|
|
821
|
+
`getMyFavoriteRadios` · `getMyFavoriteShows` · `getTrackMix`
|
|
785
822
|
</details>
|
|
786
823
|
|
|
787
824
|
<details>
|
package/dist/api/index.d.ts
CHANGED
package/dist/api/index.js
CHANGED
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
import type { playlistInfoMinimal, trackType } from '../types';
|
|
2
|
+
/** Envelope the gateway wraps these listings in. */
|
|
3
|
+
export interface GwList<T> {
|
|
4
|
+
data: T[];
|
|
5
|
+
count: number;
|
|
6
|
+
total: number;
|
|
7
|
+
filtered_count?: number;
|
|
8
|
+
checksum?: string;
|
|
9
|
+
}
|
|
10
|
+
/** A favourited artist, as the library returns it. */
|
|
11
|
+
export interface FavoriteArtist {
|
|
12
|
+
ART_ID: string;
|
|
13
|
+
ART_NAME: string;
|
|
14
|
+
ART_PICTURE: string;
|
|
15
|
+
/** when it was favourited — `YYYY-MM-DD HH:MM:SS` */
|
|
16
|
+
DATE_ADD?: string;
|
|
17
|
+
NB_ALBUM?: number;
|
|
18
|
+
NB_FAN?: number;
|
|
19
|
+
ARTIST_IS_DUMMY?: boolean;
|
|
20
|
+
__TYPE__?: string;
|
|
21
|
+
}
|
|
22
|
+
/** A favourited album, as the library returns it. */
|
|
23
|
+
export interface FavoriteAlbum {
|
|
24
|
+
ALB_ID: string;
|
|
25
|
+
ALB_TITLE: string;
|
|
26
|
+
ALB_PICTURE: string;
|
|
27
|
+
ART_ID?: string;
|
|
28
|
+
ART_NAME?: string;
|
|
29
|
+
DATE_ADD?: string;
|
|
30
|
+
NUMBER_TRACK?: number;
|
|
31
|
+
PHYSICAL_RELEASE_DATE?: string;
|
|
32
|
+
__TYPE__?: string;
|
|
33
|
+
}
|
|
34
|
+
/**
|
|
35
|
+
* The account's own playlists (`playlist.getList`) — **including private ones**,
|
|
36
|
+
* which the public `/user/{id}/playlists` endpoint will not show you.
|
|
37
|
+
*
|
|
38
|
+
* @param userId defaults to the logged-in account
|
|
39
|
+
*/
|
|
40
|
+
export declare const getMyPlaylists: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<playlistInfoMinimal>>;
|
|
41
|
+
/**
|
|
42
|
+
* Loved tracks (`song.getFavorites`). Gateway track objects, so each carries a
|
|
43
|
+
* `TRACK_TOKEN` and can go straight to `resolveDownloadUrls`.
|
|
44
|
+
*/
|
|
45
|
+
export declare const getMyFavoriteTracks: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<trackType>>;
|
|
46
|
+
/**
|
|
47
|
+
* Just the ids of every loved track (`song.getFavoriteIds`) — one small request
|
|
48
|
+
* for the whole set, for diffing a local library against the account.
|
|
49
|
+
*/
|
|
50
|
+
export declare const getMyFavoriteTrackIds: () => Promise<GwList<{
|
|
51
|
+
SNG_ID: string;
|
|
52
|
+
}>>;
|
|
53
|
+
/** Favourited albums (`album.getFavorites`). */
|
|
54
|
+
export declare const getMyFavoriteAlbums: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<FavoriteAlbum>>;
|
|
55
|
+
/** Favourited artists (`artist.getFavorites`). */
|
|
56
|
+
export declare const getMyFavoriteArtists: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<FavoriteArtist>>;
|
|
57
|
+
/** Playlists the account follows (`playlist.getFavorites`). */
|
|
58
|
+
export declare const getMyFavoritePlaylists: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<playlistInfoMinimal>>;
|
|
59
|
+
/** Favourited radios (`radio.getFavorites`). */
|
|
60
|
+
export declare const getMyFavoriteRadios: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<Record<string, any>>>;
|
|
61
|
+
/** Favourited podcast shows (`show.getFavorites`). */
|
|
62
|
+
export declare const getMyFavoriteShows: (userId?: string | number, nb?: number, start?: number) => Promise<GwList<Record<string, any>>>;
|
|
63
|
+
/**
|
|
64
|
+
* A "more like this" mix seeded from one track (`song.getSearchTrackMix`).
|
|
65
|
+
*
|
|
66
|
+
* Returns full gateway tracks **with `TRACK_TOKEN`s**, so unlike the public
|
|
67
|
+
* radio endpoints the results are immediately downloadable — no `getTrackInfo`
|
|
68
|
+
* round trip per hit.
|
|
69
|
+
*
|
|
70
|
+
* @param sngId the seed track's `SNG_ID`
|
|
71
|
+
*/
|
|
72
|
+
export declare const getTrackMix: (sngId: string, nb?: number, start?: number) => Promise<GwList<trackType>>;
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
exports.getTrackMix = exports.getMyFavoriteShows = exports.getMyFavoriteRadios = exports.getMyFavoritePlaylists = exports.getMyFavoriteArtists = exports.getMyFavoriteAlbums = exports.getMyFavoriteTrackIds = exports.getMyFavoriteTracks = exports.getMyPlaylists = void 0;
|
|
4
|
+
/**
|
|
5
|
+
* The **logged-in account's own library**, over the authenticated gateway.
|
|
6
|
+
*
|
|
7
|
+
* This is not the same surface as `src/api/user.ts`. Those functions read the
|
|
8
|
+
* *public* REST profile (`/user/{id}/tracks` …), which needs the profile to be
|
|
9
|
+
* public and shows only what that profile exposes. These read what the account
|
|
10
|
+
* itself can see — including a private library — and return gateway shapes, so
|
|
11
|
+
* tracks arrive with a `TRACK_TOKEN` and are directly downloadable.
|
|
12
|
+
*
|
|
13
|
+
* Every response here is account-scoped, so it lives in the per-session cache
|
|
14
|
+
* and is never shared between sessions.
|
|
15
|
+
*/
|
|
16
|
+
const request_1 = require("./request");
|
|
17
|
+
const api_1 = require("./api");
|
|
18
|
+
/** Resolve the caller's own user id when one wasn't supplied. */
|
|
19
|
+
const ownUserId = async (userId) => userId !== undefined ? String(userId) : String((await (0, api_1.getUser)()).USER_ID);
|
|
20
|
+
/**
|
|
21
|
+
* The account's own playlists (`playlist.getList`) — **including private ones**,
|
|
22
|
+
* which the public `/user/{id}/playlists` endpoint will not show you.
|
|
23
|
+
*
|
|
24
|
+
* @param userId defaults to the logged-in account
|
|
25
|
+
*/
|
|
26
|
+
const getMyPlaylists = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start, tab: 'all' }, 'playlist.getList');
|
|
27
|
+
exports.getMyPlaylists = getMyPlaylists;
|
|
28
|
+
/**
|
|
29
|
+
* Loved tracks (`song.getFavorites`). Gateway track objects, so each carries a
|
|
30
|
+
* `TRACK_TOKEN` and can go straight to `resolveDownloadUrls`.
|
|
31
|
+
*/
|
|
32
|
+
const getMyFavoriteTracks = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'song.getFavorites');
|
|
33
|
+
exports.getMyFavoriteTracks = getMyFavoriteTracks;
|
|
34
|
+
/**
|
|
35
|
+
* Just the ids of every loved track (`song.getFavoriteIds`) — one small request
|
|
36
|
+
* for the whole set, for diffing a local library against the account.
|
|
37
|
+
*/
|
|
38
|
+
const getMyFavoriteTrackIds = () => (0, request_1.request)({}, 'song.getFavoriteIds');
|
|
39
|
+
exports.getMyFavoriteTrackIds = getMyFavoriteTrackIds;
|
|
40
|
+
/** Favourited albums (`album.getFavorites`). */
|
|
41
|
+
const getMyFavoriteAlbums = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'album.getFavorites');
|
|
42
|
+
exports.getMyFavoriteAlbums = getMyFavoriteAlbums;
|
|
43
|
+
/** Favourited artists (`artist.getFavorites`). */
|
|
44
|
+
const getMyFavoriteArtists = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'artist.getFavorites');
|
|
45
|
+
exports.getMyFavoriteArtists = getMyFavoriteArtists;
|
|
46
|
+
/** Playlists the account follows (`playlist.getFavorites`). */
|
|
47
|
+
const getMyFavoritePlaylists = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'playlist.getFavorites');
|
|
48
|
+
exports.getMyFavoritePlaylists = getMyFavoritePlaylists;
|
|
49
|
+
/** Favourited radios (`radio.getFavorites`). */
|
|
50
|
+
const getMyFavoriteRadios = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'radio.getFavorites');
|
|
51
|
+
exports.getMyFavoriteRadios = getMyFavoriteRadios;
|
|
52
|
+
/** Favourited podcast shows (`show.getFavorites`). */
|
|
53
|
+
const getMyFavoriteShows = async (userId, nb = 50, start = 0) => (0, request_1.request)({ user_id: await ownUserId(userId), nb, start }, 'show.getFavorites');
|
|
54
|
+
exports.getMyFavoriteShows = getMyFavoriteShows;
|
|
55
|
+
/**
|
|
56
|
+
* A "more like this" mix seeded from one track (`song.getSearchTrackMix`).
|
|
57
|
+
*
|
|
58
|
+
* Returns full gateway tracks **with `TRACK_TOKEN`s**, so unlike the public
|
|
59
|
+
* radio endpoints the results are immediately downloadable — no `getTrackInfo`
|
|
60
|
+
* round trip per hit.
|
|
61
|
+
*
|
|
62
|
+
* @param sngId the seed track's `SNG_ID`
|
|
63
|
+
*/
|
|
64
|
+
const getTrackMix = (sngId, nb = 40, start = 0) => (0, request_1.request)({ sng_id: sngId, start, nb }, 'song.getSearchTrackMix');
|
|
65
|
+
exports.getTrackMix = getTrackMix;
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
import { Transform } from 'stream';
|
|
2
|
+
export interface DecryptPoolOptions {
|
|
3
|
+
/** worker threads to keep. Default `min(4, cpus - 1)`, floor 1. */
|
|
4
|
+
size?: number;
|
|
5
|
+
/** terminate idle workers after this many ms. Default 5000. */
|
|
6
|
+
idleMs?: number;
|
|
7
|
+
}
|
|
8
|
+
/** Tune the pool. Call before first use; resizing later takes effect as workers recycle. */
|
|
9
|
+
export declare const configureDecryptPool: (options: DecryptPoolOptions) => void;
|
|
10
|
+
/** Live pool state, for metrics. */
|
|
11
|
+
export declare const decryptPoolStats: () => {
|
|
12
|
+
workers: number;
|
|
13
|
+
busy: number;
|
|
14
|
+
size: number;
|
|
15
|
+
};
|
|
16
|
+
/** Terminate every worker now. */
|
|
17
|
+
export declare const shutdownDecryptPool: () => Promise<void>;
|
|
18
|
+
/**
|
|
19
|
+
* Like `createDecryptStream`, but batches the work onto a worker pool so the
|
|
20
|
+
* Blowfish never runs on the event loop. Output is byte-identical and ordered.
|
|
21
|
+
*
|
|
22
|
+
* Worth it for a server decrypting several tracks at once; for a single
|
|
23
|
+
* download the plain `createDecryptStream` avoids the thread entirely.
|
|
24
|
+
*
|
|
25
|
+
* @param trackId `SNG_ID`
|
|
26
|
+
* @param startChunk 2048-byte chunk index the first byte corresponds to
|
|
27
|
+
*/
|
|
28
|
+
export declare const createPooledDecryptStream: (trackId: string, startChunk?: number) => Transform;
|
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
3
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
4
|
+
};
|
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
+
exports.createPooledDecryptStream = exports.shutdownDecryptPool = exports.decryptPoolStats = exports.configureDecryptPool = void 0;
|
|
7
|
+
/**
|
|
8
|
+
* Optional worker pool for track decryption.
|
|
9
|
+
*
|
|
10
|
+
* Blowfish runs at ~243 MiB/s on one thread, and it runs *on the event loop* —
|
|
11
|
+
* so on a server it is not just a throughput limit, it is latency everyone else
|
|
12
|
+
* pays. Measured decrypting 240 MB in 4 MB batches:
|
|
13
|
+
*
|
|
14
|
+
* ```
|
|
15
|
+
* main thread 930ms 258 MB/s loop lag p95 16.8ms max 20ms
|
|
16
|
+
* worker pool 359ms 669 MB/s loop lag p95 3.8ms max 7ms
|
|
17
|
+
* ```
|
|
18
|
+
*
|
|
19
|
+
* 2.6x the throughput and 4.5x less p95 lag, because the batches move as
|
|
20
|
+
* transferable `ArrayBuffer`s (no copy) and each batch is independent — Deezer
|
|
21
|
+
* re-seeds the IV every 2048-byte stripe and encrypts on `chunkIndex % 3`, so a
|
|
22
|
+
* batch starting on a multiple of 3 needs nothing from the batch before it.
|
|
23
|
+
*
|
|
24
|
+
* **Opt in.** Threads are spawned lazily on first use and terminated after an
|
|
25
|
+
* idle period, but a one-track CLI run should not pay spawn cost at all — so the
|
|
26
|
+
* pool is only used by {@link createPooledDecryptStream}, never by the plain
|
|
27
|
+
* `createDecryptStream`. If a worker cannot be started the stream falls back to
|
|
28
|
+
* decrypting in-thread, so callers never have to handle that case.
|
|
29
|
+
*/
|
|
30
|
+
const os_1 = __importDefault(require("os"));
|
|
31
|
+
const stream_1 = require("stream");
|
|
32
|
+
const worker_threads_1 = require("worker_threads");
|
|
33
|
+
const path_1 = __importDefault(require("path"));
|
|
34
|
+
const decrypt_1 = require("./decrypt");
|
|
35
|
+
const STRIPE = 2048;
|
|
36
|
+
/** Batch size in stripes — a multiple of 3 so every batch starts on an encrypted stripe. */
|
|
37
|
+
const BATCH_STRIPES = 2046;
|
|
38
|
+
const BATCH_BYTES = BATCH_STRIPES * STRIPE; // ~4 MB
|
|
39
|
+
let poolSize = Math.max(1, Math.min(4, (os_1.default.cpus().length || 2) - 1));
|
|
40
|
+
let idleMs = 5000;
|
|
41
|
+
/** Tune the pool. Call before first use; resizing later takes effect as workers recycle. */
|
|
42
|
+
const configureDecryptPool = (options) => {
|
|
43
|
+
if (typeof options.size === 'number' && options.size > 0)
|
|
44
|
+
poolSize = Math.floor(options.size);
|
|
45
|
+
if (typeof options.idleMs === 'number' && options.idleMs >= 0)
|
|
46
|
+
idleMs = options.idleMs;
|
|
47
|
+
};
|
|
48
|
+
exports.configureDecryptPool = configureDecryptPool;
|
|
49
|
+
const slots = [];
|
|
50
|
+
let idleTimer = null;
|
|
51
|
+
let spawnFailed = false;
|
|
52
|
+
/** Worker entry: the built `.js` beside this file, or a ts-node bootstrap under ts-node. */
|
|
53
|
+
const spawnWorker = () => {
|
|
54
|
+
if (__filename.endsWith('.ts')) {
|
|
55
|
+
const target = path_1.default.join(__dirname, 'decrypt-worker.ts');
|
|
56
|
+
return new worker_threads_1.Worker(`require('ts-node/register/transpile-only');require(${JSON.stringify(target)});`, { eval: true });
|
|
57
|
+
}
|
|
58
|
+
return new worker_threads_1.Worker(path_1.default.join(__dirname, 'decrypt-worker.js'));
|
|
59
|
+
};
|
|
60
|
+
const armIdleShutdown = () => {
|
|
61
|
+
var _a;
|
|
62
|
+
if (idleTimer)
|
|
63
|
+
clearTimeout(idleTimer);
|
|
64
|
+
if (idleMs <= 0)
|
|
65
|
+
return;
|
|
66
|
+
idleTimer = setTimeout(() => {
|
|
67
|
+
if (slots.every((s) => !s.busy)) {
|
|
68
|
+
for (const s of slots.splice(0))
|
|
69
|
+
void s.worker.terminate();
|
|
70
|
+
}
|
|
71
|
+
}, idleMs);
|
|
72
|
+
(_a = idleTimer.unref) === null || _a === void 0 ? void 0 : _a.call(idleTimer);
|
|
73
|
+
};
|
|
74
|
+
const freeSlot = () => {
|
|
75
|
+
const idle = slots.find((s) => !s.busy);
|
|
76
|
+
if (idle)
|
|
77
|
+
return idle;
|
|
78
|
+
if (slots.length >= poolSize || spawnFailed)
|
|
79
|
+
return null;
|
|
80
|
+
try {
|
|
81
|
+
const slot = { worker: spawnWorker(), busy: false };
|
|
82
|
+
slot.worker.on('error', () => {
|
|
83
|
+
const i = slots.indexOf(slot);
|
|
84
|
+
if (i >= 0)
|
|
85
|
+
slots.splice(i, 1);
|
|
86
|
+
});
|
|
87
|
+
slot.worker.unref();
|
|
88
|
+
slots.push(slot);
|
|
89
|
+
return slot;
|
|
90
|
+
}
|
|
91
|
+
catch {
|
|
92
|
+
spawnFailed = true; // no worker_threads — every caller falls back in-thread
|
|
93
|
+
return null;
|
|
94
|
+
}
|
|
95
|
+
};
|
|
96
|
+
/** Live pool state, for metrics. */
|
|
97
|
+
const decryptPoolStats = () => ({
|
|
98
|
+
workers: slots.length,
|
|
99
|
+
busy: slots.filter((s) => s.busy).length,
|
|
100
|
+
size: poolSize,
|
|
101
|
+
});
|
|
102
|
+
exports.decryptPoolStats = decryptPoolStats;
|
|
103
|
+
/** Terminate every worker now. */
|
|
104
|
+
const shutdownDecryptPool = async () => {
|
|
105
|
+
if (idleTimer)
|
|
106
|
+
clearTimeout(idleTimer);
|
|
107
|
+
await Promise.all(slots.splice(0).map((s) => s.worker.terminate()));
|
|
108
|
+
};
|
|
109
|
+
exports.shutdownDecryptPool = shutdownDecryptPool;
|
|
110
|
+
/** Run one batch on a worker; resolves with the plaintext. */
|
|
111
|
+
const runOnWorker = (slot, trackId, startChunk, batch, seq) => new Promise((resolve, reject) => {
|
|
112
|
+
slot.busy = true;
|
|
113
|
+
const onMessage = (msg) => {
|
|
114
|
+
cleanup();
|
|
115
|
+
resolve(Buffer.from(msg.buf));
|
|
116
|
+
};
|
|
117
|
+
const onError = (err) => {
|
|
118
|
+
cleanup();
|
|
119
|
+
reject(err);
|
|
120
|
+
};
|
|
121
|
+
const cleanup = () => {
|
|
122
|
+
slot.worker.off('message', onMessage);
|
|
123
|
+
slot.worker.off('error', onError);
|
|
124
|
+
slot.busy = false;
|
|
125
|
+
armIdleShutdown();
|
|
126
|
+
};
|
|
127
|
+
slot.worker.on('message', onMessage);
|
|
128
|
+
slot.worker.on('error', onError);
|
|
129
|
+
const ab = batch.buffer.slice(batch.byteOffset, batch.byteOffset + batch.byteLength);
|
|
130
|
+
slot.worker.postMessage({ seq, trackId, startChunk, buf: ab }, [ab]);
|
|
131
|
+
});
|
|
132
|
+
/**
|
|
133
|
+
* Like `createDecryptStream`, but batches the work onto a worker pool so the
|
|
134
|
+
* Blowfish never runs on the event loop. Output is byte-identical and ordered.
|
|
135
|
+
*
|
|
136
|
+
* Worth it for a server decrypting several tracks at once; for a single
|
|
137
|
+
* download the plain `createDecryptStream` avoids the thread entirely.
|
|
138
|
+
*
|
|
139
|
+
* @param trackId `SNG_ID`
|
|
140
|
+
* @param startChunk 2048-byte chunk index the first byte corresponds to
|
|
141
|
+
*/
|
|
142
|
+
const createPooledDecryptStream = (trackId, startChunk = 0) => {
|
|
143
|
+
// no worker_threads at all — behave exactly like the in-thread stream
|
|
144
|
+
if (spawnFailed)
|
|
145
|
+
return (0, decrypt_1.createDecryptStream)(trackId, startChunk);
|
|
146
|
+
// accumulate by reference and concat once per batch — concatenating on every
|
|
147
|
+
// write would be O(n^2) with a 4 MB carry
|
|
148
|
+
let parts = [];
|
|
149
|
+
let partsLength = 0;
|
|
150
|
+
let chunkIndex = startChunk;
|
|
151
|
+
let seq = 0;
|
|
152
|
+
let nextToEmit = 0;
|
|
153
|
+
const pending = new Map();
|
|
154
|
+
const inFlight = new Set();
|
|
155
|
+
let failed = null;
|
|
156
|
+
/** emit whatever is now contiguous from `nextToEmit` */
|
|
157
|
+
const drain = (push) => {
|
|
158
|
+
for (let b = pending.get(nextToEmit); b !== undefined; b = pending.get(nextToEmit)) {
|
|
159
|
+
pending.delete(nextToEmit);
|
|
160
|
+
nextToEmit++;
|
|
161
|
+
push(b);
|
|
162
|
+
}
|
|
163
|
+
};
|
|
164
|
+
const dispatch = (batch, start, self) => {
|
|
165
|
+
const mySeq = seq++;
|
|
166
|
+
const slot = freeSlot();
|
|
167
|
+
const job = slot
|
|
168
|
+
? runOnWorker(slot, trackId, start, batch, mySeq)
|
|
169
|
+
: // pool saturated or unavailable: do this batch in-thread rather than queue
|
|
170
|
+
Promise.resolve().then(() => {
|
|
171
|
+
const s = (0, decrypt_1.createDecryptStream)(trackId, start);
|
|
172
|
+
const parts = [];
|
|
173
|
+
s.on('data', (d) => parts.push(d));
|
|
174
|
+
return new Promise((res, rej) => {
|
|
175
|
+
s.on('end', () => res(Buffer.concat(parts)));
|
|
176
|
+
s.on('error', rej);
|
|
177
|
+
s.end(batch);
|
|
178
|
+
});
|
|
179
|
+
});
|
|
180
|
+
const tracked = job
|
|
181
|
+
.then((out) => {
|
|
182
|
+
pending.set(mySeq, out);
|
|
183
|
+
drain((b) => self.push(b));
|
|
184
|
+
})
|
|
185
|
+
.catch((err) => {
|
|
186
|
+
failed = failed !== null && failed !== void 0 ? failed : err;
|
|
187
|
+
})
|
|
188
|
+
.finally(() => {
|
|
189
|
+
inFlight.delete(tracked);
|
|
190
|
+
});
|
|
191
|
+
inFlight.add(tracked);
|
|
192
|
+
};
|
|
193
|
+
return new stream_1.Transform({
|
|
194
|
+
transform(chunk, _enc, callback) {
|
|
195
|
+
if (failed)
|
|
196
|
+
return callback(failed);
|
|
197
|
+
parts.push(chunk);
|
|
198
|
+
partsLength += chunk.length;
|
|
199
|
+
while (partsLength >= BATCH_BYTES) {
|
|
200
|
+
const merged = parts.length === 1 ? parts[0] : Buffer.concat(parts, partsLength);
|
|
201
|
+
const rest = merged.subarray(BATCH_BYTES);
|
|
202
|
+
parts = rest.length ? [rest] : [];
|
|
203
|
+
partsLength = rest.length;
|
|
204
|
+
dispatch(merged.subarray(0, BATCH_BYTES), chunkIndex, this);
|
|
205
|
+
chunkIndex += BATCH_STRIPES;
|
|
206
|
+
}
|
|
207
|
+
// keep at most 2 batches per worker outstanding
|
|
208
|
+
if (inFlight.size > poolSize * 2) {
|
|
209
|
+
Promise.race(inFlight).then(() => callback(failed !== null && failed !== void 0 ? failed : undefined), callback);
|
|
210
|
+
return;
|
|
211
|
+
}
|
|
212
|
+
return callback(failed !== null && failed !== void 0 ? failed : undefined);
|
|
213
|
+
},
|
|
214
|
+
async flush(callback) {
|
|
215
|
+
if (partsLength) {
|
|
216
|
+
dispatch(parts.length === 1 ? parts[0] : Buffer.concat(parts, partsLength), chunkIndex, this);
|
|
217
|
+
parts = [];
|
|
218
|
+
partsLength = 0;
|
|
219
|
+
}
|
|
220
|
+
while (inFlight.size) {
|
|
221
|
+
await Promise.race(inFlight).catch(() => undefined);
|
|
222
|
+
}
|
|
223
|
+
if (failed)
|
|
224
|
+
return callback(failed);
|
|
225
|
+
drain((b) => this.push(b));
|
|
226
|
+
return callback();
|
|
227
|
+
},
|
|
228
|
+
});
|
|
229
|
+
};
|
|
230
|
+
exports.createPooledDecryptStream = createPooledDecryptStream;
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
|
+
/**
|
|
4
|
+
* Worker half of the decrypt pool. Receives a stripe-aligned batch plus the
|
|
5
|
+
* chunk index it starts at, and returns the plaintext with the buffer
|
|
6
|
+
* transferred back (no copy).
|
|
7
|
+
*
|
|
8
|
+
* Batches are independent because Deezer's scheme re-seeds the IV on every
|
|
9
|
+
* 2048-byte stripe and encrypts on `chunkIndex % 3`, so a batch that starts on
|
|
10
|
+
* a multiple of 3 needs no state from the batch before it.
|
|
11
|
+
*/
|
|
12
|
+
const worker_threads_1 = require("worker_threads");
|
|
13
|
+
const decrypt_1 = require("./decrypt");
|
|
14
|
+
worker_threads_1.parentPort === null || worker_threads_1.parentPort === void 0 ? void 0 : worker_threads_1.parentPort.on('message', (job) => {
|
|
15
|
+
const engine = new decrypt_1.TrackDecryptStream(job.trackId, job.startChunk);
|
|
16
|
+
const body = engine.write(Buffer.from(job.buf));
|
|
17
|
+
const tail = engine.final();
|
|
18
|
+
const out = tail.length ? Buffer.concat([body, tail]) : body;
|
|
19
|
+
// copy into a standalone ArrayBuffer so the whole thing can be transferred
|
|
20
|
+
const ab = out.buffer.slice(out.byteOffset, out.byteOffset + out.byteLength);
|
|
21
|
+
worker_threads_1.parentPort === null || worker_threads_1.parentPort === void 0 ? void 0 : worker_threads_1.parentPort.postMessage({ seq: job.seq, buf: ab }, [ab]);
|
|
22
|
+
});
|
package/dist/lib/decrypt.js
CHANGED
|
@@ -34,9 +34,23 @@ const getBlowfishKey = (trackId) => {
|
|
|
34
34
|
return key;
|
|
35
35
|
};
|
|
36
36
|
const CHUNK = 2048;
|
|
37
|
-
//
|
|
38
|
-
//
|
|
39
|
-
//
|
|
37
|
+
// Two optimisations were built, measured and rejected here:
|
|
38
|
+
//
|
|
39
|
+
// 1. Memoising the initialised Blowfish schedule per track. Key setup is 39.7µs
|
|
40
|
+
// against 33ms to decrypt an 8 MiB file (0.12%), break-even at ~10 KiB
|
|
41
|
+
// decrypted per key.
|
|
42
|
+
// 2. A worker_threads pool, to move Blowfish off the event loop. Decrypting
|
|
43
|
+
// 4 MB batches synchronously it looked like a big win (2.6x throughput,
|
|
44
|
+
// 4.5x less p95 lag) — but that benchmark manufactured the problem. Through
|
|
45
|
+
// a real pipeline the in-thread stream below already shows **p95 loop lag of
|
|
46
|
+
// 0.0ms**, because it works a socket read at a time rather than a whole file,
|
|
47
|
+
// and the pooled version came out at 0.72x once three extra 4 MB copies per
|
|
48
|
+
// batch (concat, transfer-slice, worker return-slice) were paid. Decrypt only
|
|
49
|
+
// becomes the binding constraint above ~265 MB/s of aggregate download in one
|
|
50
|
+
// process, and Deezer's quota bites long before that.
|
|
51
|
+
//
|
|
52
|
+
// Don't re-attempt either without a workload that actually shows decrypt as the
|
|
53
|
+
// bottleneck.
|
|
40
54
|
/**
|
|
41
55
|
* Decrypt a downloaded track. Deezer applies Blowfish-CBC "stripe" obfuscation:
|
|
42
56
|
* the file is split into 2048-byte chunks and only every third chunk (0, 3, 6…)
|
|
@@ -1,2 +1,11 @@
|
|
|
1
1
|
import type { lyricsType, trackType } from '../types';
|
|
2
|
-
|
|
2
|
+
/**
|
|
3
|
+
* Deezer's lyrics, falling back to scraping Musixmatch when a track has no
|
|
4
|
+
* `LYRICS_ID` (instrumentals, and anything Deezer simply lacks).
|
|
5
|
+
*
|
|
6
|
+
* That fallback is not free: it is two requests per track that has no Deezer
|
|
7
|
+
* lyrics — 16 of them on a 14-track album — and it fails outright wherever
|
|
8
|
+
* Musixmatch blocks the request. Pass `fallback: false` to skip it and keep only
|
|
9
|
+
* what Deezer serves.
|
|
10
|
+
*/
|
|
11
|
+
export declare const getTrackLyrics: (track: trackType, fallback?: boolean) => Promise<lyricsType | null>;
|
|
@@ -12,15 +12,24 @@ const getTrackLyricsWeb = async (track) => {
|
|
|
12
12
|
return null;
|
|
13
13
|
}
|
|
14
14
|
};
|
|
15
|
-
|
|
15
|
+
/**
|
|
16
|
+
* Deezer's lyrics, falling back to scraping Musixmatch when a track has no
|
|
17
|
+
* `LYRICS_ID` (instrumentals, and anything Deezer simply lacks).
|
|
18
|
+
*
|
|
19
|
+
* That fallback is not free: it is two requests per track that has no Deezer
|
|
20
|
+
* lyrics — 16 of them on a 14-track album — and it fails outright wherever
|
|
21
|
+
* Musixmatch blocks the request. Pass `fallback: false` to skip it and keep only
|
|
22
|
+
* what Deezer serves.
|
|
23
|
+
*/
|
|
24
|
+
const getTrackLyrics = async (track, fallback = true) => {
|
|
16
25
|
if (track.LYRICS_ID > 0) {
|
|
17
26
|
try {
|
|
18
27
|
return await (0, api_1.getLyrics)(track.SNG_ID);
|
|
19
28
|
}
|
|
20
29
|
catch (err) {
|
|
21
|
-
return await getTrackLyricsWeb(track);
|
|
30
|
+
return fallback ? await getTrackLyricsWeb(track) : null;
|
|
22
31
|
}
|
|
23
32
|
}
|
|
24
|
-
return await getTrackLyricsWeb(track);
|
|
33
|
+
return fallback ? await getTrackLyricsWeb(track) : null;
|
|
25
34
|
};
|
|
26
35
|
exports.getTrackLyrics = getTrackLyrics;
|
|
@@ -30,6 +30,12 @@ export interface AddTrackTagsOptions {
|
|
|
30
30
|
embedArtistImage?: boolean;
|
|
31
31
|
/** fetch + embed lyrics. default true */
|
|
32
32
|
writeLyrics?: boolean;
|
|
33
|
+
/**
|
|
34
|
+
* When Deezer has no lyrics for a track, fall back to scraping Musixmatch.
|
|
35
|
+
* Two extra requests per track that needs it (16 on a typical album), and they
|
|
36
|
+
* fail wherever Musixmatch blocks you. default true.
|
|
37
|
+
*/
|
|
38
|
+
lyricsFallback?: boolean;
|
|
33
39
|
/** embed synced LRC (FLAC Vorbis comment only; MP3 has no v2.3 synced frame). default true */
|
|
34
40
|
embedSyncedLyrics?: boolean;
|
|
35
41
|
/**
|
|
@@ -28,11 +28,18 @@ const DEFAULTS = {
|
|
|
28
28
|
embedCover: true,
|
|
29
29
|
embedArtistImage: true,
|
|
30
30
|
writeLyrics: true,
|
|
31
|
+
lyricsFallback: true,
|
|
31
32
|
embedSyncedLyrics: true,
|
|
32
33
|
richCredits: true,
|
|
33
34
|
deezerIds: true,
|
|
34
35
|
includeRank: true,
|
|
35
36
|
};
|
|
37
|
+
// Per-track by necessity: `SNG_CONTRIBUTORS` is exposed *only* by `song.getData`.
|
|
38
|
+
// Probed against the live gateway — `song.getListData` (the batch endpoint
|
|
39
|
+
// `refreshTrackTokens` uses) returns 44 fields including VERSION, GAIN, ISRC,
|
|
40
|
+
// ART_PICTURE and URL_REWRITING, but no contributors; neither does
|
|
41
|
+
// `song.getListByAlbum`. So a 14-track album costs 14 of these, and the only way
|
|
42
|
+
// to avoid them is `{richCredits: false}`, which trades away credits and BPM.
|
|
36
43
|
const hydrate = async (track) => {
|
|
37
44
|
var _a, _b, _c, _d, _e, _f, _g;
|
|
38
45
|
if (track.SNG_CONTRIBUTORS !== undefined && track.VERSION !== undefined && track.GAIN !== undefined) {
|
|
@@ -78,7 +85,7 @@ const resolveTagModel = async (trackInput, options = {}) => {
|
|
|
78
85
|
options.lyrics !== undefined
|
|
79
86
|
? Promise.resolve(options.lyrics)
|
|
80
87
|
: opt.writeLyrics
|
|
81
|
-
? (0, getTrackLyrics_1.getTrackLyrics)(track).catch(() => null)
|
|
88
|
+
? (0, getTrackLyrics_1.getTrackLyrics)(track, opt.lyricsFallback).catch(() => null)
|
|
82
89
|
: Promise.resolve(null),
|
|
83
90
|
options.publicTrack !== undefined
|
|
84
91
|
? Promise.resolve(options.publicTrack)
|
package/package.json
CHANGED