@doitian/dsh-music 0.1.2 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +206 -21
- package/lib/cache.js +156 -0
- package/lib/dj.js +443 -67
- package/lib/index.js +39 -5
- package/lib/likes.js +169 -0
- package/lib/netease.js +130 -0
- package/lib/panel.html +316 -41
- package/lib/router.js +203 -18
- package/lib/session.js +109 -18
- package/lib/state.js +55 -0
- package/package.json +5 -3
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
NetEase Cloud Music ([music.163.com](https://music.163.com)) playback and an AI DJ, inside DeepSeek Harness.
|
|
4
4
|
|
|
5
|
-
A **Music** entry appears in the DSH sidebar. It opens a player — your queue, synced lyrics, an `<audio>` element that streams through the host, QR sign-in, and a continuous AI DJ. The same player is also exposed to the agent through seven tools, so it can search, queue, and curate from a conversation.
|
|
5
|
+
A **Music** entry appears in the DSH sidebar. It opens a player — your queue, synced lyrics, an `<audio>` element that streams through the host, QR sign-in, NetEase likes, and a continuous AI DJ. The same player is also exposed to the agent through seven tools, so it can search, queue, and curate from a conversation.
|
|
6
6
|
|
|
7
7
|
```
|
|
8
8
|
┌──────────────────────────────────────────────────────────────────┐
|
|
@@ -144,15 +144,162 @@ Two implementation details worth knowing:
|
|
|
144
144
|
`/music/health` reports both: `audio.preferred` and `audio.last` (what the last
|
|
145
145
|
stream actually served, including `downgraded`).
|
|
146
146
|
|
|
147
|
+
## Likes and taste
|
|
148
|
+
|
|
149
|
+
A track holds one of three taste levels — **liked**, **unrated**, or
|
|
150
|
+
**disliked** — and the player draws whichever it is:
|
|
151
|
+
|
|
152
|
+
| Control | Level it sets |
|
|
153
|
+
|---|---|
|
|
154
|
+
| the **♡ / ♥** in a queue row, or the heart in the transport row | **liked** — the track goes into the account's 我喜欢的音乐 playlist on NetEase. Clicking the filled heart takes the like back. |
|
|
155
|
+
| the **✕** in the transport row | **disliked** — local, and fed to the DJ, which stops picking the track. Clicking the filled ✕ clears it. |
|
|
156
|
+
| the **✕** on a queue row | **disliked** as well. Taking a track out of the queue is a judgement about the track, not just about the list — without recording it the DJ re-derives the same track from the same similarity and charts within a batch or two. |
|
|
157
|
+
| neither | unrated. |
|
|
158
|
+
|
|
159
|
+
A like is a **NetEase** like, not a plugin flag, and the two halves of what that
|
|
160
|
+
needs are both plain endpoints:
|
|
161
|
+
|
|
162
|
+
- `POST /api/song/like` — one of the few *write* endpoints the web API still
|
|
163
|
+
serves unencrypted, so no `weapi` client is needed. It takes the direction
|
|
164
|
+
rather than toggling: the page always says which level it wants, so a stale
|
|
165
|
+
poll cannot turn a click into the wrong one, and disliking a liked track is one
|
|
166
|
+
request instead of two racing ones.
|
|
167
|
+
- `GET /api/song/like/check` — which of a batch of ids are liked. The answer is
|
|
168
|
+
cached per track (5 minutes), so a whole queue's hearts cost one request, and
|
|
169
|
+
only tracks with no fresh answer ever cost another.
|
|
170
|
+
|
|
171
|
+
Four properties are worth knowing, because they are what makes a heart mean
|
|
172
|
+
something:
|
|
173
|
+
|
|
174
|
+
- **An unknown track is not an unliked one.** A track nobody has asked about has
|
|
175
|
+
no answer, and renders as unrated until NetEase gives one.
|
|
176
|
+
- **A refusal is not an answer.** An anonymous session answers `code 301` for
|
|
177
|
+
every id, and a delisted track answers `400` with NetEase's own reason
|
|
178
|
+
(`下架歌曲无法收藏` for 晴天, for instance). Both leave the heart empty and say
|
|
179
|
+
why in the toast; a refused check is retried later instead of being cached as
|
|
180
|
+
"not liked", so signing in mid-session fills the hearts without a restart.
|
|
181
|
+
- **The account is the record.** `feedback.likes` in `session.json` is a *copy*
|
|
182
|
+
of the account's likes, kept for the DJ's taste digest. It is reconciled with
|
|
183
|
+
the account at startup — one read of the liked playlist — so a like made on
|
|
184
|
+
the phone counts, and a like an earlier build recorded locally (because it had
|
|
185
|
+
nowhere to send it) does not linger as a phantom.
|
|
186
|
+
- **A dislike is local.** The plain web API has no dislike endpoint, so this is
|
|
187
|
+
where the plugin's own taste memory is the only record. Liking and disliking
|
|
188
|
+
are the same three levels rather than two flags: one replaces the other, and
|
|
189
|
+
disliking a liked track removes it on NetEase as well.
|
|
190
|
+
- **Removing a queued track is a dislike, once.** The row's ✕ goes through the
|
|
191
|
+
queue endpoint, whose removal *is* the judgement: the row leaves the list and
|
|
192
|
+
an unrated track is recorded as disliked in the same request, so the DJ does
|
|
193
|
+
not derive it again from the same similarity. The confirm names the removal —
|
|
194
|
+
the judgement is the host's side of it. A track that already carries a level
|
|
195
|
+
keeps it, because removing is often just queue housekeeping (clearing out what
|
|
196
|
+
has already been heard) and rewriting a like into a dislike would be worse
|
|
197
|
+
than missing the signal. Removing the track that is playing also advances
|
|
198
|
+
playback, which a removal otherwise would not.
|
|
199
|
+
|
|
200
|
+
### Skips
|
|
201
|
+
|
|
202
|
+
A skip is a fourth, softer signal, and nobody has to press anything for it:
|
|
203
|
+
**moving to the next track before 30 seconds, or before a quarter of the track
|
|
204
|
+
when that is longer**, records one — from the transport row or the agent's
|
|
205
|
+
`music_control next`; jumping to a queue row does not count. It is local, like
|
|
206
|
+
a dislike.
|
|
207
|
+
|
|
208
|
+
What does *not* count is as deliberate as what does: going back, a track ending
|
|
209
|
+
or failing on its own, single-repeat mode (which stays on the track), and a
|
|
210
|
+
track that never reported a position — paused, blocked by autoplay, or still
|
|
211
|
+
loading — because a track nobody heard was not rejected. Nor does the next
|
|
212
|
+
that the transport row's **✕** presses after a dislike: the dislike is already
|
|
213
|
+
the record, and counting it as a skip too would weigh an early dislike more
|
|
214
|
+
than a later one.
|
|
215
|
+
|
|
216
|
+
The DJ reads skips at two grains:
|
|
217
|
+
|
|
218
|
+
- **Per track.** One skip is a penalty, since it may have been the wrong
|
|
219
|
+
moment rather than the wrong song. A second takes the track out of the pool
|
|
220
|
+
for good, like a dislike.
|
|
221
|
+
- **Per artist.** The skip marks the play it cut short in the history, and that
|
|
222
|
+
play stops counting toward the artist's affinity — otherwise abandoning an
|
|
223
|
+
artist's songs would keep making them a "favourite". Skipped plays then count
|
|
224
|
+
against the artist instead, and the model is shown the recently skipped
|
|
225
|
+
tracks and the artists skipped more than once.
|
|
226
|
+
|
|
147
227
|
## The AI DJ
|
|
148
228
|
|
|
149
229
|
The DJ keeps the queue stocked. When the queue drops below
|
|
150
230
|
`djAutoExtendBelow`, it gathers candidates and appends a batch:
|
|
151
231
|
|
|
152
|
-
- similar songs for the current and
|
|
232
|
+
- similar songs (`simiSong`) for the current track and recent plays the
|
|
233
|
+
listener heard through — never a disliked or skipped one, and topped up from
|
|
234
|
+
the likes when the history is thin,
|
|
235
|
+
- personal FM (私人FM), two calls of three tracks, when signed in,
|
|
236
|
+
- up to six liked tracks not played in the last three days, marked `[liked]`
|
|
237
|
+
for the model,
|
|
153
238
|
- the daily recommendations and the anonymous new-song feed,
|
|
154
|
-
- the
|
|
155
|
-
-
|
|
239
|
+
- the four general charts — 飙升榜, 新歌榜, 原创榜, 热歌榜 — sampled together,
|
|
240
|
+
- for a mood brief, its best-loved playlists, and a keyword song search.
|
|
241
|
+
|
|
242
|
+
Personal FM stands in for heartbeat mode (心动模式), which would be the better
|
|
243
|
+
personalised source but answers `code 500` on the plain web API for every seed;
|
|
244
|
+
it needs the `weapi` transport this plugin does not implement. `simiSong`
|
|
245
|
+
answers five tracks whatever `limit` asks for.
|
|
246
|
+
|
|
247
|
+
**A brief finds music through playlists.** A song search only matches titles
|
|
248
|
+
and lyrics, so "rainy afternoon jazz" mostly finds library tracks literally
|
|
249
|
+
named *Rainy Afternoon Jazz*. Listeners name and tag playlists by mood (雨天,
|
|
250
|
+
爵士, 深夜…), so the brief searches playlists, takes the most played among
|
|
251
|
+
the six most relevant (skipping any under ten tracks), and samples across
|
|
252
|
+
them.
|
|
253
|
+
|
|
254
|
+
**The model rewrites the brief first.** Playlists are named in Chinese, so an
|
|
255
|
+
English brief searched as written matches badly — live, "90s cantopop" found
|
|
256
|
+
Notorious B.I.G. and Eminem. When a model route is available, one call turns
|
|
257
|
+
the brief into up to three Chinese search terms the way listeners name
|
|
258
|
+
playlists (`90年代 粤语金曲`, `港乐 经典`, `粤语老歌`), and the same brief then
|
|
259
|
+
found 张国荣, 刘德华 and 许冠杰. Each term gets its own playlist, so one broad
|
|
260
|
+
term cannot take every slot, and the song search uses the first.
|
|
261
|
+
|
|
262
|
+
The rewrite is made **once per brief, not per plan**, and kept for a week: a
|
|
263
|
+
brief means the same thing next week. It uses the curation route and the same
|
|
264
|
+
session identity. Without a model, or when the rewrite answers nothing usable,
|
|
265
|
+
the brief is searched as written — the plan never fails over it — and the
|
|
266
|
+
rewrite is tried again after half an hour rather than on every plan. The terms
|
|
267
|
+
used are reported as `searchedAs` on the plan and on `dj.lastPlan`, and the
|
|
268
|
+
`music_dj` result says *Brief searched as: …*.
|
|
269
|
+
|
|
270
|
+
### Caching, sampling and pacing
|
|
271
|
+
|
|
272
|
+
Most sources change on the scale of hours or days, so each is cached for
|
|
273
|
+
about as long as it stays the same, per account where it is personal:
|
|
274
|
+
|
|
275
|
+
| Source | Kept for |
|
|
276
|
+
|---|---|
|
|
277
|
+
| similar songs (per seed), track details (per id) | 24 h |
|
|
278
|
+
| a playlist's track list | 12 h |
|
|
279
|
+
| a brief's rewrite into search terms | 7 days (30 min after a failed one) |
|
|
280
|
+
| the charts, a brief's searches | 6 h |
|
|
281
|
+
| the daily recommendations | 3 h |
|
|
282
|
+
| the new-song feed | 1 h |
|
|
283
|
+
| personal FM | never — every call answers a different three |
|
|
284
|
+
|
|
285
|
+
**Every plan samples.** Each source contributes a random handful up to its
|
|
286
|
+
quota — similar 14, daily 8, charts 8, new songs 6, likes 6, FM 6, and for a
|
|
287
|
+
brief 15 from its playlists plus 8 from the song search — drawn only from
|
|
288
|
+
tracks still eligible, and kept in the source's own order. Taking the first
|
|
289
|
+
few would serve the same tracks until the source changed; sampling a cached
|
|
290
|
+
list yields a different handful every plan for free. It also reaches past
|
|
291
|
+
what NetEase sends in full: for a playlist the account does not own, only
|
|
292
|
+
the first 20 tracks come with details, but every id does, so the DJ samples
|
|
293
|
+
ids from the whole list and looks up just the ones it chose. The quotas also
|
|
294
|
+
keep any one source from crowding the others out of the pool.
|
|
295
|
+
|
|
296
|
+
**Requests are paced, not fired as a burst.** Each request starts a random
|
|
297
|
+
0.25–0.75 s after the previous one, so a plan reads like someone clicking
|
|
298
|
+
through the app rather than a crawler. Only the DJ's own probing is paced;
|
|
299
|
+
searches and plays you ask for go out at once. Live, a first plan with a
|
|
300
|
+
brief sends 14 requests over about 7 s; the next one, mostly from cache,
|
|
301
|
+
sends 2. The DJ stocks the queue before it runs dry, so the wait is not
|
|
302
|
+
heard.
|
|
156
303
|
|
|
157
304
|
Then one of two tiers chooses:
|
|
158
305
|
|
|
@@ -161,9 +308,15 @@ Then one of two tiers chooses:
|
|
|
161
308
|
(recently played, favourite artists, likes, dislikes) goes to
|
|
162
309
|
`ctx.llm.stream()`, and the model returns the picks and a one-line vibe. This
|
|
163
310
|
is the tier that reasons about a mood.
|
|
164
|
-
- **Heuristic tier** — always available: candidates are scored by
|
|
165
|
-
affinity,
|
|
166
|
-
|
|
311
|
+
- **Heuristic tier** — always available: candidates are scored by mood-brief
|
|
312
|
+
match and artist affinity, then interleaved so consecutive tracks do not
|
|
313
|
+
share an artist. See [Heuristic tier](#heuristic-tier).
|
|
314
|
+
|
|
315
|
+
Either way, the pool never holds a track that is disliked, recently played, or
|
|
316
|
+
**already in the queue** — a batch is appended, so a pick the queue already
|
|
317
|
+
holds would be dropped and the batch would come up short. The model is also
|
|
318
|
+
told which track its picks will follow, and is held to the count it was asked
|
|
319
|
+
for.
|
|
167
320
|
|
|
168
321
|
**Every model call carries a session identity.** A leaf call cannot set headers
|
|
169
322
|
— `GenerateOptions` has no `headers` field — so `sessionId` is the only identity
|
|
@@ -315,7 +468,28 @@ batch; it never leaves the queue empty.
|
|
|
315
468
|
|
|
316
469
|
### Heuristic tier
|
|
317
470
|
|
|
318
|
-
Always available, used whenever the model tier is unavailable
|
|
471
|
+
Always available, used whenever the model tier is unavailable. Each candidate
|
|
472
|
+
scores:
|
|
473
|
+
|
|
474
|
+
| Signal | Weight |
|
|
475
|
+
|---|---|
|
|
476
|
+
| matches the mood brief (keyword search) | +3.0 |
|
|
477
|
+
| from personal FM | +1.0 |
|
|
478
|
+
| similar to the current or recent tracks | +0.6 |
|
|
479
|
+
| a rested like | +0.5 |
|
|
480
|
+
| per artist among the listener's most played (skipped plays excluded) | +2.2 |
|
|
481
|
+
| per artist shared with the current track | +1.4 |
|
|
482
|
+
| per artist play count in the last 60 plays, skips excluded | +0.15 each, capped at +1.0 |
|
|
483
|
+
| per artist skip among the last 150 plays | −0.6 each, capped at −2.4 |
|
|
484
|
+
| skipped once (twice excludes it) | −1.5 |
|
|
485
|
+
| VIP-only | −0.4 |
|
|
486
|
+
| jitter, so repeat plans differ | 0–0.8 |
|
|
487
|
+
|
|
488
|
+
The brief outweighs a favourite artist on purpose: it is the listener's explicit
|
|
489
|
+
ask, and without that weight a chart hit by a favourite would outrank every
|
|
490
|
+
track that matches it. The ranked list is then interleaved starting from the
|
|
491
|
+
track the batch is appended after, so no two adjacent tracks share an artist —
|
|
492
|
+
including the seam between the old queue and the new batch.
|
|
319
493
|
|
|
320
494
|
## Configuration
|
|
321
495
|
|
|
@@ -366,6 +540,8 @@ browser (DSH web GUI, http://127.0.0.1:<port>)
|
|
|
366
540
|
├─ lib/session.js cookie store + QR login state machine
|
|
367
541
|
├─ lib/state.js queue, cursor, transport revisions
|
|
368
542
|
├─ lib/dj.js candidate pool + model/heuristic tiers
|
|
543
|
+
├─ lib/cache.js source cache, pool sampling, request pacing
|
|
544
|
+
├─ lib/likes.js the account's like state, cached
|
|
369
545
|
└─ lib/router.js JSON API, HTML, Range-capable audio proxy
|
|
370
546
|
```
|
|
371
547
|
|
|
@@ -398,8 +574,10 @@ Four design notes:
|
|
|
398
574
|
|
|
399
575
|
- The `/music` route prefix is registered on the bare HTTP server, which owns no
|
|
400
576
|
authentication of its own. Anyone who can reach the port can browse the
|
|
401
|
-
library
|
|
402
|
-
|
|
577
|
+
library, stream audio, and **change the account's likes** — `POST
|
|
578
|
+
/music/api/taste` is the panel's own control, so it is a real write to the
|
|
579
|
+
NetEase account. The route is loopback-only in the shipped composition. It
|
|
580
|
+
exposes no credentials — only search results, the queue, the like state, and
|
|
403
581
|
audio bytes.
|
|
404
582
|
- The NetEase cookie lives in `$DSH_HOME/music/session.json` in plain text, the
|
|
405
583
|
same posture as the rest of the profile's session data. `music_login` with
|
|
@@ -424,6 +602,7 @@ when something looks wrong:
|
|
|
424
602
|
"music_dj", "music_now_playing", "music_login"],
|
|
425
603
|
"dj": { "enabled": false, "model": null, "lastRoute": null,
|
|
426
604
|
"modelError": null, "lastPlanAt": null, "error": null },
|
|
605
|
+
"likes": { "cached": 12, "pending": 0, "error": null },
|
|
427
606
|
"dataDir": "C:\\Users\\…\\.dsh\\music",
|
|
428
607
|
"session": { "authenticated": true, "nickname": "…", "vip": true },
|
|
429
608
|
"uptimeMs": 123456
|
|
@@ -433,7 +612,10 @@ when something looks wrong:
|
|
|
433
612
|
`tools` is the point: the HTTP route is registered before the tools, so a tool
|
|
434
613
|
that failed to register would otherwise leave a route that answers normally.
|
|
435
614
|
Seeing all seven names is what proves startup completed. `panelContract` is the
|
|
436
|
-
page/engine boundary the browser half checks before it starts playback.
|
|
615
|
+
page/engine boundary the browser half checks before it starts playback. `likes`
|
|
616
|
+
is the cache behind the hearts: `cached` is how many answers it holds, and
|
|
617
|
+
`error` names why a check produced none — an anonymous session never asks, so
|
|
618
|
+
both stay empty.
|
|
437
619
|
|
|
438
620
|
### If the browser blocks autoplay
|
|
439
621
|
|
|
@@ -469,8 +651,8 @@ down first. A restart brings both halves back into agreement.
|
|
|
469
651
|
|
|
470
652
|
```powershell
|
|
471
653
|
npm run check # node --check on every module
|
|
472
|
-
npm test #
|
|
473
|
-
npm run test:live #
|
|
654
|
+
npm test # 154 deterministic tests: pure, like state, DJ, browser half
|
|
655
|
+
npm run test:live # 34 integration tests against the live NetEase API
|
|
474
656
|
npm run test:all # both
|
|
475
657
|
```
|
|
476
658
|
|
|
@@ -486,10 +668,12 @@ network cases inside it.
|
|
|
486
668
|
Or run one file directly:
|
|
487
669
|
|
|
488
670
|
```powershell
|
|
489
|
-
node test/netease.test.mjs #
|
|
490
|
-
node test/
|
|
491
|
-
node test/
|
|
492
|
-
node test/
|
|
671
|
+
node test/netease.test.mjs # 34 pure: normalisation, quality ladder, likes, cookies, taste, skips, player state
|
|
672
|
+
node test/cache.test.mjs # 10 source cache: lifetimes, shared loads, sampling, pacing
|
|
673
|
+
node test/likes.test.mjs # 12 like-state cache: what counts as an answer, refusals, batching, writes
|
|
674
|
+
node test/dj.test.mjs # 60 AI DJ: model call identity, route resolution, failure reporting, curation, skips, pool sources, caching, brief rewriting, queue invariants
|
|
675
|
+
node test/client.test.mjs # 38 browser half: the engine against a fake DOM, and the page it pairs with
|
|
676
|
+
node test/host.test.mjs # 34 integration: routes, streaming, curation, quality, taste, skips
|
|
493
677
|
```
|
|
494
678
|
|
|
495
679
|
The DJ tests drive `ctx.llm.stream()` with a stub that emits the documented
|
|
@@ -522,7 +706,7 @@ poll, and a track change replaces the pane. Two more cover the pane's shape: it
|
|
|
522
706
|
is capped to a few lines, collapses to its header on demand, and remembers that
|
|
523
707
|
choice across loads — while still fetching the lines, so expanding is instant.
|
|
524
708
|
|
|
525
|
-
`npm test` runs the
|
|
709
|
+
`npm test` runs the five deterministic files in sequence (`npm run test:all`
|
|
526
710
|
adds the live one), deliberately **not** `node --test <dir>`: the directory form forks one child process per file, which
|
|
527
711
|
is blocked in sandboxed environments.
|
|
528
712
|
|
|
@@ -581,9 +765,10 @@ workflow, tag, and commit — `npm view @doitian/dsh-music@<version> dist.attest
|
|
|
581
765
|
a signed-in VIP account — 周杰伦's 晴天 (`id 186016`) is the canonical example,
|
|
582
766
|
because it belongs to a digital album. The plugin surfaces NetEase's own
|
|
583
767
|
refusal (`403` plus a reason) rather than retrying.
|
|
584
|
-
- **No `weapi`/`eapi` encryption**, so
|
|
585
|
-
|
|
586
|
-
|
|
768
|
+
- **No `weapi`/`eapi` encryption**, so scrobbling and playlist writes are not
|
|
769
|
+
implemented. Liking a track needs none of it — see
|
|
770
|
+
[Likes and taste](#likes-and-taste) — but a *dislike* stays local, because
|
|
771
|
+
there is no plain endpoint for one.
|
|
587
772
|
- **`apiPrefix` must stay `music`** unless `BASE` in `lib/client.js` is changed
|
|
588
773
|
to match.
|
|
589
774
|
- **The DJ's model tier is covered against stub contracts, not a live
|
package/lib/cache.js
ADDED
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* How the DJ reads its sources: a time-bounded cache, the sampling it draws
|
|
3
|
+
* its pool with, and the pacing its requests are sent at.
|
|
4
|
+
*
|
|
5
|
+
* Most of the DJ's sources change on the scale of hours or days — a chart
|
|
6
|
+
* daily, similarity effectively never — so fetching each one on every plan
|
|
7
|
+
* spends requests to learn nothing. The cache keeps each source for as long as
|
|
8
|
+
* it plausibly stays the same, and the DJ samples from what it holds, so a
|
|
9
|
+
* long cached list still yields a different handful each time. What does go
|
|
10
|
+
* out is paced like a person browsing, not fired as a burst.
|
|
11
|
+
*
|
|
12
|
+
* @module dsh-music/cache
|
|
13
|
+
*/
|
|
14
|
+
|
|
15
|
+
/** Entries kept at most; the oldest are dropped first. */
|
|
16
|
+
const MAX_ENTRIES = 2_000;
|
|
17
|
+
|
|
18
|
+
export class TtlCache {
|
|
19
|
+
/**
|
|
20
|
+
* @param {object} [options]
|
|
21
|
+
* @param {() => number} [options.now] clock, injectable for tests.
|
|
22
|
+
* @param {number} [options.maxEntries]
|
|
23
|
+
*/
|
|
24
|
+
constructor({ now = Date.now, maxEntries = MAX_ENTRIES } = {}) {
|
|
25
|
+
this.now = now;
|
|
26
|
+
this.maxEntries = maxEntries;
|
|
27
|
+
/** @type {Map<string, {value: unknown, expiresAt: number}>} */
|
|
28
|
+
this.entries = new Map();
|
|
29
|
+
}
|
|
30
|
+
|
|
31
|
+
/** The live value for `key`, or `undefined` when absent or expired. */
|
|
32
|
+
peek(key) {
|
|
33
|
+
const entry = this.entries.get(key);
|
|
34
|
+
if (!entry) return undefined;
|
|
35
|
+
if (entry.expiresAt <= this.now()) {
|
|
36
|
+
this.entries.delete(key);
|
|
37
|
+
return undefined;
|
|
38
|
+
}
|
|
39
|
+
return entry.value;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
set(key, value, ttlMs) {
|
|
43
|
+
this.entries.delete(key);
|
|
44
|
+
this.entries.set(key, { value, expiresAt: this.now() + ttlMs });
|
|
45
|
+
while (this.entries.size > this.maxEntries) this.entries.delete(this.entries.keys().next().value);
|
|
46
|
+
return value;
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
/**
|
|
50
|
+
* The cached value for `key`, loading it when absent or expired.
|
|
51
|
+
*
|
|
52
|
+
* The pending load is what gets cached, so two plans racing for the same
|
|
53
|
+
* source share one request. A failed load is forgotten rather than cached:
|
|
54
|
+
* the next plan should try again, not inherit the outage for the whole TTL.
|
|
55
|
+
*
|
|
56
|
+
* @template T
|
|
57
|
+
* @param {string} key
|
|
58
|
+
* @param {number} ttlMs
|
|
59
|
+
* @param {() => Promise<T>} load
|
|
60
|
+
* @returns {Promise<T>}
|
|
61
|
+
*/
|
|
62
|
+
get(key, ttlMs, load) {
|
|
63
|
+
const cached = this.peek(key);
|
|
64
|
+
if (cached !== undefined) return cached;
|
|
65
|
+
const pending = Promise.resolve().then(load);
|
|
66
|
+
this.set(key, pending, ttlMs);
|
|
67
|
+
pending.catch(() => {
|
|
68
|
+
if (this.entries.get(key)?.value === pending) this.entries.delete(key);
|
|
69
|
+
});
|
|
70
|
+
return pending;
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
clear() {
|
|
74
|
+
this.entries.clear();
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
get size() {
|
|
78
|
+
return this.entries.size;
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
/**
|
|
83
|
+
* A random `count` of `items`, kept in their original order.
|
|
84
|
+
*
|
|
85
|
+
* Random so a cached list still yields a different handful each plan — taking
|
|
86
|
+
* the first `count` would serve the same tracks until the source changes. In
|
|
87
|
+
* order, because a source's order carries meaning (chart rank, search
|
|
88
|
+
* relevance), and the pool keeps it.
|
|
89
|
+
*
|
|
90
|
+
* @template T
|
|
91
|
+
* @param {T[]} items
|
|
92
|
+
* @param {number} count
|
|
93
|
+
* @param {() => number} [random]
|
|
94
|
+
* @returns {T[]}
|
|
95
|
+
*/
|
|
96
|
+
export function sampleInOrder(items, count, random = Math.random) {
|
|
97
|
+
if (count >= items.length) return [...items];
|
|
98
|
+
if (count <= 0) return [];
|
|
99
|
+
const indices = items.map((_, index) => index);
|
|
100
|
+
for (let index = 0; index < count; index += 1) {
|
|
101
|
+
const swap = index + Math.floor(random() * (indices.length - index));
|
|
102
|
+
[indices[index], indices[swap]] = [indices[swap], indices[index]];
|
|
103
|
+
}
|
|
104
|
+
return indices
|
|
105
|
+
.slice(0, count)
|
|
106
|
+
.sort((a, b) => a - b)
|
|
107
|
+
.map((index) => items[index]);
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
/**
|
|
111
|
+
* Spaces out request starts by a randomised gap.
|
|
112
|
+
*
|
|
113
|
+
* A plan touches up to a dozen sources, and firing them at once is a crawler's
|
|
114
|
+
* signature: a tight burst at a machine-regular rhythm. Starts are spaced
|
|
115
|
+
* instead, each by its own random gap, so the traffic reads like someone
|
|
116
|
+
* clicking through the app. Only starts are spaced — a request already sent may
|
|
117
|
+
* still be in flight when the next starts, as on any page that is loading.
|
|
118
|
+
*/
|
|
119
|
+
export class Pacer {
|
|
120
|
+
/**
|
|
121
|
+
* @param {object} [options]
|
|
122
|
+
* @param {number} [options.minGapMs] the shortest gap between two starts.
|
|
123
|
+
* @param {number} [options.jitterMs] the random span added to each gap.
|
|
124
|
+
* @param {() => number} [options.now]
|
|
125
|
+
* @param {(ms: number) => Promise<void>} [options.sleep]
|
|
126
|
+
* @param {() => number} [options.random]
|
|
127
|
+
*/
|
|
128
|
+
constructor({
|
|
129
|
+
minGapMs = 250,
|
|
130
|
+
jitterMs = 500,
|
|
131
|
+
now = Date.now,
|
|
132
|
+
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
|
|
133
|
+
random = Math.random,
|
|
134
|
+
} = {}) {
|
|
135
|
+
this.minGapMs = minGapMs;
|
|
136
|
+
this.jitterMs = jitterMs;
|
|
137
|
+
this.now = now;
|
|
138
|
+
this.sleep = sleep;
|
|
139
|
+
this.random = random;
|
|
140
|
+
/** The earliest the next request may start. */
|
|
141
|
+
this.nextAt = 0;
|
|
142
|
+
}
|
|
143
|
+
|
|
144
|
+
/**
|
|
145
|
+
* Run `task` at the next free start slot.
|
|
146
|
+
* @template T
|
|
147
|
+
* @param {() => Promise<T>} task
|
|
148
|
+
* @returns {Promise<T>}
|
|
149
|
+
*/
|
|
150
|
+
run(task) {
|
|
151
|
+
const at = Math.max(this.now(), this.nextAt);
|
|
152
|
+
this.nextAt = at + this.minGapMs + this.random() * this.jitterMs;
|
|
153
|
+
const wait = at - this.now();
|
|
154
|
+
return (wait > 0 ? this.sleep(wait) : Promise.resolve()).then(task);
|
|
155
|
+
}
|
|
156
|
+
}
|