@torrent-tv/proxy 2.83.3 → 2.83.5

package/CHANGELOG.md

@@ -1,3 +1,35 @@
+## 2.83.5
+- **Fix**: ONE OWNER OF THE FACT "THIS PROXY HAS PIECE N", AND IT IS THE STORE. The disk tier drops a piece once every reader is past it — correctly, and that is what bounds the spill — while the library kept a second copy of the same fact in its completion bitfield and nothing reconciled them. A read then concluded the piece was had, asked for it, was told it was absent, and failed; nor was it ever fetched again, because the library does not download what it believes it owns. Field 2026-09-12: a film played 80 seconds (1920 frames, and the picture never moved again); the encoder ran on to 725 s, so some 565 spilled pieces fell behind every read head and were dropped, piece 0 among them; the encoder lost its input and restarted, which re-opens the input at byte 0; and `/stream` answered `0 of 2363497962 bytes: Piece 0 is verified but absent from the store` to every read for the next 92 minutes while the browser retried one segment against a healthy transport at 4 ms.
+- **Fix**: So the store announces when it can no longer produce a piece AT ALL — not resident, not on disk, not inside a file held whole — and the pool withdraws the library's claim. The eviction's own bargain, written in its comment as "a seek back re-downloads it", is true for the first time. The three cases are one rule rather than a list of call sites: the disk tier says only that IT has lost a piece, and the store decides whether that is a loss at all, so a piece dropped as a duplicate of an assembled file says nothing and a piece still resident says nothing.
+- **Chore**: The withdrawal is a plain function over plain values (`services/download/withdraw-claim.js`) and holds no pool, no store and no client, so it can be exercised with an object literal. It deliberately does not re-select the piece: every torrent is added with `deselect: true`, which makes the library skip that, and the download set keeps its one owner.
+- **Chore**: `withdrawn=` joins the piece store's own line, so the bargain is checkable by number instead of by belief.
+- **Fix**: A READ NOW SURVIVES A PIECE WITHDRAWN UNDER IT. An empty answer is a wait, not a failure: the claim has just gone, so the piece is being fetched again, and the read goes back one step and takes the whole ordinary path for it — declare it, steer it onto the fastest holders, wait. Once per piece; a second emptiness says the bytes are not coming and is reported as that rather than as "verified but absent", which was never true.
+- **Fix**: THE DELAY AFTER A LOST INPUT NOW BINDS. It existed — doubling from 2 s to 15 s — and it was timed against the DEAD RUN, which the plan never consults: a run whose input has gone is not alive, so the plan read the stretch it held as free and placed another there at once, which died the same way. Field: 2432 ffmpeg starts in 23 minutes, one every 0.57 s, for 61 minutes, against a delay that had reached its ceiling long before. The wait lives beside the decision it governs now, per output, and the plan places nothing there until it is over — unless something on that output is still producing, which is proof the input can be read.
+- **Fix**: The wait is lifted only by proof that the input was there, which is a segment having come out of it. Any other ending nearly did it, and that is wrong: at the moment of failure several runs end at once, and one ending `gone` with nothing produced would have lifted the wait the one beside it had just set.
+- **Chore**: The two figures and the timer leave `hls-session-manager.js` for the encoding layer; the file is 28 lines shorter.
+- **Fix**: AN ESTABLISHED FACT IS SAID ONCE, THEN WITH DECREASING FREQUENCY. One absent piece produced about 55 log lines a second: 235 000 lines in 92 minutes, of which 68.8 % were byte-identical repeats — measured, not estimated. Repeats are matched verbatim, the whole line, which catches nearly all of that and cannot merge two different statements; normalising numbers would catch a little more and would also merge the memory series, which exists precisely to catch a runaway. A held-back line is said again at doubling intervals up to a minute, and says how many were held back and over what span, because the rate is the fact.
+- **Fix**: The log file may now grow to 1 GB before it turns over, from 32 MB. At the old size that one failure turned the file over twice inside the session it belonged to — 159 000 lines covering 38 minutes, then 76 385 covering 23 — so 61 minutes of 92 survived and the second rotation overwrote the turn holding the onset. The disk it was bounded for had 91.4 GB free.
+- **Chore**: KNOWN CONSEQUENCE, stated rather than discovered later: a torrent whose pieces have been dropped is no longer `done`, so a film watched to the end is not assembled into a whole file. That is the honest state — we do not have those bytes — and the previous behaviour was worse than it looked: the assembly reads the file through the store, so it would have waited for pieces the library wrongly believed it had.
+
+## 2.83.4
+- **Chore**: WHAT IS IN A TORRENT IS NOW A THING RATHER THAN A RECOMPUTATION. `services/torrent/Contents.js` holds one torrent's list of files, its pictures with the sound, subtitles and images that belong to each, the order a person reads them in, and what belongs to nothing. The pairing rules were already there as functions; what did not exist was anywhere to keep their answer, so the warm-up on one path and the playback plan on another worked out the same grouping over the same list several times per opened film. Checked against the survey collection: 134 torrents, 44 of them with more than one picture, 1357 pictures and 421 files paired to one, not a single file paired to two — and all 1357 groupings agree with the per-picture call they replace, so nothing about the pairing changes.
+- **Chore**: It holds only what the torrent says about itself. Nothing about a FILM: a poster or a title is an answer from a third party about an identity read from a file's own bytes, keyed by the picture's file, and it must never delay playback. No reference to another layer either — not a container, not a priority map, not a viewer; what leaves is plain indices into the torrent's own list.
+- **Chore**: Telling a real episode from an extra was measured and deliberately not built. Over those 134 torrents the words that would say so (`sample`, `trailer`, `extra`, `bonus`, `preview`, `making`) matched four files and all four were ordinary titles; the ratio of a video to the median of its own torrent runs continuously from 0.00 to 1.00 with no gap, because a collection of short clips is made of short clips. There is nothing to derive a rule from, and an invented one would decide the order in which a pool owner's bandwidth is spent.
+- **Fix**: THE MAP NOW SAYS WHEN NOBODY WANTS A FILE ANY MORE. It could make only one of its two statements: a file whose viewers had all gone was deleted from the orchestrator's own memory and published nowhere, so the bands it had stated on the swarm's behalf stood until the torrent itself was removed. Read from the demand register — which is what decides what is fetched — a film nobody had watched for an hour was indistinguishable from one being watched now. A file with nobody on it is now published with nothing in it, and so is a file whose session has gone, once.
+- **Fix**: A map with nothing in it is answered before the file's length, its duration or the torrent's list are looked at. All three are what the ordinary path needs to turn seconds into bytes, and all three are on their way out at exactly the moment a departure is said, so waiting on them would make the departure depend on facts that are already gone.
+- **Fix**: An empty map cannot bring a torrent back. It is said when the last viewer of a file leaves, which is also when the torrent may be going, and the worker's ordinary lookup rebuilds a dead handle from its recipe — so asking that way would have re-added a torrent in order to be told that nothing is wanted of it.
+- **Fix**: THE TWO ENDS OF A FILE ARE THE FILE'S, NOT A READ'S. A container keeps its directory at one end or the other — `ftyp` and an EBML header at the front, the `moov` at the very back of an MP4 that was not written for streaming — and nothing can be read of such a file until they arrive. They were wanted by nobody: the prefetch that fetches them is an ordinary read, and a read gives up what it stated the moment it finishes, so the ends of an open film were held by nothing at all once the codec probe was done. They are stated by the file now, urgent while the playback plan is waiting for them and kept afterwards at the level of something nobody is waiting for, and they are given up when the map says nobody wants that file any more.
+- **Chore**: The size of that claim is not chosen and cannot be: one byte at each end, and since the piece is what a swarm delivers, one byte at each end IS one piece at each end. The 256 KB and 2 MB the prefetch reads are its own affair, and a directory bigger than the piece it starts in is asked for by the read that needs it.
+- **New**: A torrent whose episode nobody has chosen yet warms its FIRST item — that picture and the sound and subtitles beside it — instead of nothing. One item, not twenty: a pack's worth of edges would spend the pool owner's bandwidth on nineteen files nobody opened, while one item is two pieces and is also the likeliest pick. It is stated as something nobody is waiting for, so on a proxy serving somebody else it costs nothing until their own film has everything it needs.
+- **Fix**: NOTHING COUNTS READERS ANY MORE. The pool asked how many claims a torrent's files had, and that count decided whether it kept its swarm, whether it was removed, whether it was evicted under the disk cap, whether it could be seeded for and whether a complete file could be written out. It was a second bookkeeping of a fact the demand register already holds, and it was wrong in both directions: a torrent being opened has no readers, and a viewer who has left can leave one behind. One question is asked instead — is anything wanted of this torrent — and it is answered by what has been stated.
+- **Fix**: A swarm is let go on a DEPARTURE and never on a beginning. A torrent nothing has ever been wanted of is one being opened, its metadata just landed and its file list on its way to the person choosing, and taking its swarm away there is the 2.83.1 failure down to the 741 connections. The rule is two facts and a table, separated from the doing so it can be exercised without a live client.
+- **Fix**: A torrent is on the idle clock from the moment it exists. The only thing that ever started that clock was a reader letting go, so a file list fetched and never played was held for the life of the process.
+- **Chore**: `acquireFile`, its release, the two worker commands behind it and the module that kept the claims are gone, with all seven call sites: the stream route, the transcode-session route, the subtitle controller, the read inside the worker, and the session manager's hold on its source. The session manager is 65 lines shorter for it.
+- **New**: WHAT IS IN A TORRENT IS ANSWERED HERE, AND NOWHERE ELSE. `GET /api/sources/:key/files` returns the files in the order a person reads them — by folder, then by name, with runs of digits compared as numbers — each saying what it is, together with the pictures and what belongs to each of them by index. The browser used to decide all of that itself, twice: a list of video extensions in its torrent parser and a second, shorter pair inside its picker. The three had already diverged — measured 2026-09-12, `.dat` was offered there as video and not counted here, which also decides whether a sidecar whose name matches nothing can belong to the only video present. That one disagreement is settled by union: `.dat` is a Video CD's MPEG-1 stream, and a file offered and found unplayable costs one probe, while a film never offered cannot be reached at all.
+- **Chore**: The paths leave here already relative to the torrent's root, so the rule for stripping the torrent's own name lives in one place instead of two that can disagree.
+- **Fix**: A warm-up can no longer take the urgency away from a playback plan that somebody is waiting for. Both ask for the same two ends of a file, and the warm-up fires off the files beside the picture without awaiting them, so it can arrive second and restate the same claimants at its own lower level — putting a person on a loading screen behind a film somebody else is watching. The statement is raise-only; the caller entitled to put them back down is the read that has just finished, and it says so.
+- **Fix**: A departure could be noticed by nobody. The background fill states a claim per file and withdraws it once nothing else wants that file, but the withdrawal happened inside a pass whose result nothing acted on — so a torrent everybody had left could be held "wanted" by the fill's own claim at the moment of the departure and then never re-examined: in its swarm, with no idle clock running.
+
 ## 2.83.3
 
 - **Fix**: LEAVING A SWARM IS NOW A CONSEQUENCE OF A DEPARTURE, not something a pass over every torrent decides. 2.83.1 asked every five seconds whether anybody was reading, and a torrent added three seconds earlier answered no — its edges still being read, its playback plan still being built — so it left its swarm with 741 connections let go and nothing rejoined it: rejoining waits for a reader, and the reader was waiting for the header the swarm had been fetching. 2.83.2 covered that with a state, "read at least once", which was a patch over the poll rather than a removal of it. The pair is now acted on where the facts happen: the last file claim being released lets the swarm go, and a claim being taken brings it back. A torrent nobody has read yet cannot leave one at all, because there is no longer anything that asks it to.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@torrent-tv/proxy",
-  "version": "2.83.3",
+  "version": "2.83.5",
   "description": "Torrent proxy client that exposes webseed-like HTTP stream endpoint.",
   "license": "GPL-3.0-or-later",
   "publishConfig": {

package/research/handover-reader-claims-removal-2026-09-12.md

@@ -0,0 +1,166 @@
+# Handover: the priority map is the only thing that says a file is wanted
+
+Written 2026-09-12, after the instruction that reversed the work in the tree.
+
+## The instruction, in the user's own terms
+
+The priority map answers ONE question: how much is this stretch of this file
+wanted. Nothing anywhere asks WHO is reading a torrent. There are no reader
+counters. Everything built around counting readers comes out.
+
+The map is rebuilt when viewers come and go, so it already shows the current
+state of affairs. A second bookkeeping of the same thing is what has to go.
+
+## What is in the working tree right now, and why it is all wrong
+
+HEAD is `79e802a`. On top of it, UNCOMMITTED:
+
+| file | what was done to it |
+|---|---|
+| `services/torrent/ReaderClaims.js` | NEW, untracked. The whole class is the mistake. |
+| `services/torrent-pool.js` | `fileUsageByTorrent` replaced by `this.readers = new ReaderClaims()`; a listener on it calls `rejoinSwarm`, `leaveSwarm` and `#scheduleIdleRemoval`; `torrentsForUploadPolicy` changed from taking a usage map to taking an `isRead` predicate. |
+| `services/torrent-worker/worker.js` | `warmActiveFiles` (`:842`) and `keepWholeFiles` (`:1024`) read `pool.readers.of(torrent)`. |
+| `test/swarm-follows-readers.test.js` | the announcement test rewritten around `ReaderClaims`. |
+| `test/upload-hurry.test.js` | rewritten around the `isRead` predicate. |
+
+All of it counts readers. All of it comes out. Two further facts about this
+tree, both of which the next agent needs:
+
+1. A bulk text replacement earlier in that session DESTROYED three methods of
+   `TorrentPool` — `#scheduleIdleRemoval`, `#cancelIdleRemoval` and
+   `#forgetDeadTorrent` — which were then called but not defined. They were
+   restored BY HAND from HEAD and adapted, not diffed line by line against the
+   original. One log line was changed deliberately: the idle-timer message no
+   longer prints a reader count. **Diff the restored bodies against
+   `git show HEAD:services/torrent-pool.js` before trusting them**, or discard
+   the tree and start from HEAD.
+2. Discarding the uncommitted changes is probably the cheapest start, since
+   every one of them is being reversed. That is a destructive action and is the
+   USER'S CALL — ask before running anything that throws work away.
+
+## What already exists and answers the question without counting anybody
+
+Read these before designing. Line numbers are of the working tree.
+
+**The map is built from viewers and published.** `services/priority/PriorityOrchestrator.js`
+builds one map per film+file, and a second per output, from where the viewers
+are. `services/priority/PriorityMap.js` holds the scale and the words for
+reading it — `isAtAWatchingViewer`, `isBehindEverybody`, `isNobodyComingNow`.
+
+**The map reaches the torrent thread as a command.** `Command.PRIORITY_MAP` in
+`services/torrent-worker/worker.js:539` calls `pool.applyPriorityMap(torrent,
+fileIndex, zones, durationSeconds)`.
+
+**`applyPriorityMap` (`services/torrent-pool.js:1191`) is where seconds become
+bytes**, and it writes the result into the demand register:
+`demandFor(torrent).register.state({ claimant, fileIndex, byteStart, byteEnd,
+urgency })`, with the claimant named `${MAP_CLAIMANT}:${fileIndex}:${index}`
+(`:1257`). It WITHDRAWS its own stale claimants for that file afterwards
+(`:1266-1273`), so a map that shrinks does shrink the register. **Whether a map
+that wants NOTHING leaves the register empty for that file is not established —
+check it, do not assume it.**
+
+**The register is therefore the existing statement of what is wanted.**
+`services/demand/DemandRegister.js` offers `size`, `files()`, `windows()`,
+`at(urgency)`, `union(fileIndex)`, `mostUrgent()`, `withdraw(claimant)`,
+`clear()`. `services/download/registry.js` offers `demandFor(torrent)`,
+`hasUnmetDemand(torrent)`, `forgetTorrent(torrent)`, `reconcileAll()`.
+
+## THE TRAP — read this before writing a line
+
+**Not every claimant in the register is a viewer, and the ones that are not are
+exactly what a naive rule kills.**
+
+Measured claimants other than the map:
+
+- `services/torrent-worker/piece-reader.js:670` — a read that is STOPPED on a
+  piece right now, stated at the BLOCKED level. Only a read can say this; the
+  map deliberately never states BLOCKED (see the comment at
+  `torrent-pool.js:1204`).
+- `services/torrent-pool.js:1147`, `:1159`, `:1162` — the background fill and
+  the tail.
+
+And several reads exist that no viewer has yet asked for: the codec probe
+reading the file's head and tail, the keyframe table, the subtitle cluster
+walk, a sidecar soundtrack being fetched whole.
+
+**This is not hypothetical.** On 2026-09-11 a rule of exactly this shape shipped
+as proxy 2.83.1: a torrent nobody had stated anything about left its swarm. A
+torrent that has just been added has stated nothing yet, because its edges are
+still being read and its playback plan is still being built. Field record:
+added 20:57:17, first peer 20:57:18, out of the swarm 20:57:21 with 741
+connections let go, and nothing ever rejoined it — rejoining waited for a
+reader, and the reader was waiting for the header the swarm had been fetching.
+`peers=0 connected of 0 known (94 queued, 5 tracker(s) offered up to 683
+seeders)` for as long as the viewer would wait. Playback never started.
+
+So "the map wants nothing, therefore let the swarm go" is WRONG as written. The
+honest question is whether ANYTHING in the register wants bytes of this torrent
+— the map's zones and every other claimant together — and that question has to
+be asked of the register, which already knows, rather than of a counter.
+
+## What has to change, and what each site should ask instead
+
+Each of these currently asks "who is reading". Work out, per site, what it
+actually needs from the register.
+
+1. **`TorrentPool.acquireFile`** (`torrent-pool.js:1770`) and its release
+   function. Callers: `routes/stream/get.js:211`,
+   `routes/api/transcode-sessions/post.js:47`,
+   `services/controllers/SubtitleController.js:54`,
+   `services/torrent-worker/worker.js:251` and `:366`,
+   `services/torrent-worker/pool-adapter.js:88`,
+   `services/torrent-worker/client.js:327`. Note it also does two things that
+   are NOT counting: it cancels a pending idle removal and stamps
+   `#lastAccess`, which the disk-cap LRU reads. Decide where those two belong
+   before deleting the method.
+2. **Leaving and rejoining the swarm.** `leaveSwarm` and `rejoinSwarm` are
+   exported from `torrent-pool.js:465` and `:497`. Today the trigger is the
+   first claim taken and the last released. It becomes a fact of the register.
+3. **Idle removal.** `#scheduleIdleRemoval` fires on the last reader leaving and
+   re-checks for readers when the timer fires.
+4. **The upload policy.** `torrentsForUploadPolicy` (`torrent-pool.js:349`) sets
+   `torrent.hasActiveReader` and `torrent.hasUnmetDemand`. The second already
+   comes from the register (`hasUnmetDemand`); the first is the counter.
+5. **`#reportStalledDownloads`** (`:1311`) and **`#adjustUploadLimit`** (`:1354`)
+   skip a torrent with no readers.
+6. **The disk-cap LRU** (`:1411`) picks torrents with no readers.
+7. **`#removeTorrent`** (`:1886`) prints a refcount for diagnostics.
+8. **`warmActiveFiles`** (`worker.js:842`) walks "the files being read" to keep
+   subtitle cues current. `register.files()` says which files anything wants.
+9. **`keepWholeFiles`** (`worker.js:1024`) refuses to write a completed file out
+   while it is being read, so that the write does not take the disk from a
+   viewer. This one genuinely needs "is anybody reading THIS file right now" and
+   is the site most likely to need a different answer from the register than a
+   plain "is it wanted".
+
+## Rules that hold whatever the design turns out to be
+
+- **Plan first, approval before any edit.** No exceptions, including when told
+  to get on with it.
+- **Never add a line to `services/hls-session-manager.js`.** It is being taken
+  apart. Deletion only; `git diff --numstat` must show more removed than added.
+- **All code, comments and documents in the repositories are English.**
+- **Do not run the whole test suite.** Run the tests of the files changed.
+  Relevant here: `test/swarm-follows-readers.test.js`, `test/upload-hurry.test.js`,
+  `test/file-claims.test.js`, `test/background-fill.test.js`,
+  `test/priority-map-download.test.js`, `test/read-window.test.js`,
+  `test/piece-reader.test.js`, `test/dead-torrent-handle.test.js`.
+- **A CHANGELOG entry is part of the change**, written at the version
+  `npm run patch` will produce — the current `package.json` version plus one
+  patch. Never edit the version in `package.json` by hand.
+- **Bumping the proxy means bumping the addon**, `ha-addon/torrent_tv_proxy/config.yaml`,
+  with its own CHANGELOG entry, and the proxy is published FIRST.
+- **Publishing is pre-authorised.** Destructive actions are not.
+- Refer to work by what it is, never by a list number.
+
+## State of the tree as measured, 2026-09-12
+
+- `node --check` passes on the three touched source files.
+- Biome is clean over `services/`, `test/` and `routes/` — 282 files.
+- Tests of the changed files: 131 of 131 pass.
+- Nothing has been verified in the field; the released proxy is 2.83.3 with
+  addon 0.74.0 on the host.
+
+None of that argues for keeping the work. It is recorded so that a failure
+after the removal is not mistaken for a failure that was already there.

package/research/piece-withdrawn-but-still-claimed-2026-09-12.md

@@ -0,0 +1,175 @@
+# A film played 80 seconds and then stood still for 92 minutes — 2026-09-12
+
+Proxy 2.83.4, addon 0.74.1, one viewer, `Reacher.S04E07.1080p.rus.LostFilm.TV.mkv`
+(2 363 497 962 bytes). Reported by the user as "the film only managed to play
+one minute nineteen seconds".
+
+Every claim below is tagged by where it comes from: **[log]** read out of a
+captured log, **[code]** read out of the source, **[measured]** computed from
+the captured logs, **[derived]** reasoned from two of the above.
+
+## 1. What the viewer saw
+
+**[log, browser]** Every one of the 1112 `decode` readings in the captured
+window is identical: `t=80.0s size=1920x1080 frames=1920 dropped=0
+readyState=2`. 1920 frames at 24 fps is 80.0 s, so the picture advanced 80
+seconds and then never moved again. The page stayed open until 18:52.
+
+**[log, browser]** From then on the player asked for one segment and was
+refused, about once a minute for 92 minutes:
+
+```
+18:30:47 non-fatal: fragLoadError currentTime=79.97 msReadyState=open
+         frag=segment-00008.mp4 sn=8 action=5 error=Error: HTTP Error 503 …
+18:41:15 fatal:     fragLoadError … sn=8 … HTTP Error 503
+18:41:17 recovered fatal networkError, resuming at 80.0s
+```
+
+`msReadyState=open` throughout, so this is **not** the closed-MediaSource
+failure of item 86. **[log]** The transport was healthy the whole time —
+`rtt=4ms`, `state=connected ice=connected`, every counter advancing — so it is
+not item 10 either.
+
+## 2. What the proxy answered, and why
+
+**[code]** 503 in `routes/transcode/session-file/get.js:143` is the
+`warming-up` branch: the segment is still being produced. It was never produced.
+
+**[log]** 239 times:
+
+```
+transcode 17ec5953 holding segment-00008.mp4: the file is not on disk
+  (runs from #8, viewer at #8, encoder alive, index #8, produced 0.0s at 2.23x
+   — the encoder has not made it)
+[hold] segment-00008.mp4 warming-up after 60400ms (the map wants it now, rank 69 of 69)
+```
+
+**[log]** The encoder could not open its input, 5712 times:
+
+```
+encode-run #8..#247 lost its input ([http @ …] Error reading HTTP response: End of file)
+Error opening input file http://127.0.0.1:9090/stream?sourceKey=94dda59f…   (8514×)
+Error opening input files: End of file                                       (5712×)
+```
+
+**[log]** And `/stream` said exactly what was wrong, 2884 times:
+
+```
+stream: read of "Reacher.S04E07.1080p.rus.LostFilm.TV.mkv" bytes 0-2363497961
+  ended after 0 of 2363497962 bytes: Piece 0 is verified but absent from the store.
+```
+
+Zero bytes of 2.36 GB. ffmpeg reads an empty body as the end of the file.
+
+## 3. The cause
+
+**[log]** The torrent believed it had the whole file:
+
+```
+[stats] 94dda59f 5d276be5 peers=54 connected … file=100.0% header=8388608/8388608B
+```
+
+**[log]** The store did not:
+
+```
+piece-store "Reacher.S04E07…": resident=27/42 (108MB of 168MB allowed)
+  committed=108MB blocks=27 on-disk=26MB pinned=0 spilled=7
+  reads=5932 (73.8% from memory) spills=571 revivals=41
+```
+
+**[measured]** 571 spills at a 4 MB piece is about 2284 MB written to disk;
+26 MB remained. So roughly 565 pieces had been removed. **[log]** It was not
+the disk cap: `disk: 93634MB free; spilled pieces 26MB of 46656MB` — the
+ceiling never came near binding, and 93 GB were free.
+
+**[code]** That leaves exactly one remover: `PieceDiskStore.forgetBehind`,
+called from `SharedPieceStore.reviseSpillCeiling`, which drops every piece
+lying behind the earliest read head. **[log]** The encoder had reached
+`proxyProcessed=725.643`, twelve minutes of film, so piece 0 — the start of the
+file and its Matroska header — was far behind every reader and was dropped.
+That is the rule working as designed.
+
+**[code]** `piece-reader.js:151` then asks `torrent.bitfield.get(index)`. The
+library still answered yes, because it had downloaded and hashed that piece
+once. **Nothing ever told it otherwise:** `bitfield.set` does not occur
+anywhere in `services/piece-store/`, `services/torrent-pool.js` or
+`services/torrent-worker/`. So the read concluded the piece was had, asked the
+store, was told it was absent, and threw.
+
+**[derived]** And it was never fetched again, because the library does not
+download a piece it believes it owns. The comment on `forgetBehind` states the
+bargain in as many words — "a seek back re-downloads it, which is the same
+bargain this tier makes whenever it drops a piece for room" — and that bargain
+had never once been honoured.
+
+**[derived]** The trigger is ordinary: an encoder restart re-opens its input at
+byte 0 (`bytes 0-2363497961` in the log), which is precisely the region
+`forgetBehind` removes first.
+
+This is the "un-have problem" the parent `CLAUDE.md` names in the Level 2 disk
+design — "mark the piece incomplete in the completion store → it re-downloads on
+the next read". The eviction shipped; the un-have did not.
+
+## 4. Two amplifiers, each a defect of its own
+
+### The delay after a lost input did not bind
+
+**[code]** `hls-session-manager.js` timed a retry against the DEAD RUN, doubling
+2 s → 15 s. **[code]** `EncodeRun.isAlive` is false in `RETRY_WAIT`, and the
+orchestrator plans from live runs only — so the plan saw the stretch as free and
+placed a new run at once, on every event.
+
+**[measured]** 2432 `restart for` lines in 23 minutes: one every 0.57 s, for 61
+minutes, while the delay printed `retrying in 15s`. The comment beside that
+timer predicted this exactly and the code did not prevent it.
+
+### The log destroyed the record of its own failure
+
+**[measured]** 76 385 lines in 23 minutes and 159 000 in the previous 38 — about
+55 lines a second, of which **52 567 of 76 385 (68.8 %) are byte-identical
+repeats**. **[code]** `MAX_FILE_BYTES` was 32 MiB with one previous turn kept, so
+the file turned over twice inside the session: 61 minutes of 92 survived, and the
+onset — everything before 17:51 — was overwritten by the second rotation and is
+gone. **[log]** `/data` had 91.4 GB free.
+
+## 5. What was built (proxy 2.83.5)
+
+1. the store announces when it can no longer produce a piece at all — not
+   resident, not on disk, not in a file held whole — and the pool withdraws the
+   library's claim with `_markUnverified`. The disk tier states only its own
+   loss; the store decides whether that is a loss;
+2. a read whose piece is withdrawn under it goes back one step and takes the
+   whole ordinary path again instead of throwing. Once per piece;
+3. the delay lives in the orchestrator, per output, and the plan places nothing
+   there until it is over — unless something on that output is still producing.
+   It is lifted only by a run that actually produced;
+4. an established fact is logged once, then at doubling intervals to a minute,
+   saying how many were held back. Matched verbatim; digits are deliberately not
+   normalised, because that would merge the memory series;
+5. the log file may reach 1 GB before turning over.
+
+## 6. What this does NOT establish
+
+1. **Nothing here has been seen in the field.** The next session is the proof.
+   What should appear: `withdrawn=` climbing on the piece-store line while reads
+   keep succeeding, `its input was not there (attempt N) — placing nothing for
+   Xs`, and `[said N more time(s) …]` instead of a flood. What should disappear:
+   `Piece N is verified but absent from the store`.
+2. **The onset was not read.** Both log turns begin with the failure already
+   established, so which piece failed FIRST, and whether the encoder's first
+   input loss had the same cause, is not known. The mechanism is established;
+   its first instance is not.
+3. **A consequence, stated rather than discovered later:** a torrent whose
+   pieces have been withdrawn is no longer `done`, so a film watched to the end
+   is not assembled into a whole file. That is the honest state — those bytes
+   are not here — and the previous behaviour was worse than it looked, since the
+   assembly reads the file through the store and would have waited for pieces
+   the library wrongly believed it had. What is NOT measured is how often a film
+   now fails to be assembled, and whether keeping the pieces of a nearly-finished
+   file would be worth the disk.
+4. **The re-download cost is not measured.** A seek back past the read heads now
+   waits for pieces to be fetched again instead of failing. How long that takes
+   on a real swarm, and whether the viewer notices, needs a session.
+5. **`peers=54 connected of -4599 known (…, -1128 queued)`** — the negative
+   figures come from WebTorrent's own `_peersLength`/`_numQueued`, read for the
+   log only. Not investigated, not touched.

package/research/priority-map-is-the-truth-2026-09-12.md

@@ -0,0 +1,225 @@
+# The priority map is the only statement of what is wanted
+
+Measurements and decisions behind the work listed in the roadmap as "the
+priority map is the only source of truth about what is wanted". Written
+2026-09-12, extended as the steps land.
+
+Everything below is tagged by where it comes from: READ (from the source),
+MEASURED (a number produced here), DECIDED (agreed with the user), or REFUSED
+(considered and not built, with the reason).
+
+## Why 2.83.1 stopped playback — READ
+
+The map is built by `PriorityOrchestrator.publishFor`, and its only input is the
+live SESSIONS (`hls-session-manager.js` hands it `sessionGroups`). A session
+exists once a file has been chosen and its playback plan built. The torrent,
+however, is added on `POST /api/sources/:key/warm`, while the person is still
+reading the list of episodes.
+
+So there is an interval in which the map cannot say anything at all, and in it
+"the map wants nothing" is indistinguishable from "there is nobody to ask yet".
+The field record of 2026-09-11 lies entirely inside that interval: added
+20:57:17, first peer 20:57:18, out of the swarm 20:57:21 with 741 connections
+let go.
+
+## The same fault with the opposite sign — READ
+
+When the last viewer of a file leaves, `publishFor` deletes that file's map from
+its own memory and publishes NOTHING:
+
+```js
+for (const key of [...this.#maps.keys()]) {
+  if (!byFile.has(key)) { this.#maps.delete(key); this.#last.delete(key); }
+}
+```
+
+`applyPriorityMap` withdraws a file's claimants only when it is CALLED, so the
+demand register keeps that file's zones for the life of the torrent — nothing
+else removes them but `forgetTorrent`, which runs when the torrent is removed
+altogether.
+
+Both ends are therefore broken: today the map can say neither "wanted" nor "not
+wanted". That is what has to be fixed before the register can be believed by
+anybody.
+
+## The register alone cannot be asked — READ
+
+A read states a claim only while it is STOPPED on a missing piece
+(`piece-reader.js:765`, `Urgency.BLOCKED`) and withdraws it when the read ends
+(`:1052`). So an empty register routinely means "at this instant everybody has
+what they need", not "nobody wants this". A swarm rule built on the register
+alone would release and rejoin repeatedly. The stable statement is the map,
+which holds while people are there.
+
+## No viewer-to-torrent link is needed — DECIDED
+
+A session carries `sourceKey`, and the map is addressed by source and file. So
+"is anybody here for this source" is already answered by any file of it having a
+non-empty map. A second holder of that same relation would be the two-owners
+fault this work removes, in a new place.
+
+## What is in a torrent, as one statement — MEASURED
+
+`services/torrent/Contents.js`, validated against the survey collection in
+`Dropbox/trn` on 2026-09-12 (137 files, 134 parsed):
+
+| | |
+|---|---|
+| torrents parsed | 134 |
+| of them with more than one picture | 44 |
+| items (pictures with what belongs to them) | 1357 |
+| files paired to a picture | 421 |
+| files paired to more than one picture | 0 |
+| largest torrent | 265 pictures |
+| items whose grouping differs from the per-picture call it replaces | 0 |
+
+The last row is what says this is a move rather than a change: every one of the
+1357 groupings agrees with `matchSidecarFiles` asked the old way, one picture at
+a time.
+
+The largest figure matters to the step that follows: a pack of 265 pictures is
+why the edges are claimed one item at a time rather than for the whole list —
+two pieces outstanding at a time, and the order is what bounds the cost.
+
+## Telling an episode from an extra — REFUSED
+
+Proposed, measured, and not built.
+
+**Words.** `sample|trailer|extras?|bonus|preview|teaser|making|interview|
+deleted|featurette` as whole tokens against every video path of the 134
+torrents: **four matches, all four false**. Three are the word "making" inside
+ordinary episode titles ("Making cash with her pussy", "Perfect pleasure making
+pussy") and their sizes are 0.84 to 2.35 of their torrent's median, i.e. they
+are the content. Not one real `Sample/` file exists in the collection, so the
+rule cannot even be validated here — that is a property of this collection, not
+a claim that extras do not exist.
+
+**Size.** The ratio of each video to the median of the videos in its own torrent
+runs continuously from 0.0000 to 1.0 with no gap anywhere: 53 videos sit under a
+quarter of their median, and they are short clips that ARE what the torrent is
+for (`REX1080_<name> - 720P_4000K_….mp4`, 6-23 MB, in collections of them). Two
+shapes are genuinely not content — `preview1..5.mp4` in one torrent, and
+`VIDEO_TS/VIDEO_TS.VOB` plus `VTS_01_0.VOB`, which are a DVD's menu and a
+structure question rather than a size one.
+
+So there is nothing here to derive a threshold from, and an invented one would
+decide the order in which a stranger's bandwidth is spent. Every picture is an
+item. The survey script is `scratchpad`-only and is reproduced by walking the
+collection with `parse-torrent` and the two functions named above.
+
+## Two classifications of one torrent, and they have already diverged — MEASURED
+
+The browser classifies the file list itself (`public/domain/torrent-parser.js`,
+`VIDEO_EXTENSIONS`, `classifyMediaFiles`, `orderForDisplay`) and the proxy does
+it again (`services/torrent/files.js`). Compared 2026-09-12: the browser offers
+`.dat` as video and the proxy does not count it, every other extension agreeing.
+On such a torrent the person is offered a file the proxy does not consider a
+picture, and `countVideoFiles` answers 0 — which also changes the one-picture
+relaxation in the pairing.
+
+One fact, one owner: the proxy owns the composition and the file-list route
+returns it. The cost is one round trip for a `.torrent` the browser parsed
+locally; the proxy holds those same bytes from registration, so it can answer
+without the swarm.
+
+## What the pool asks now, and the two things it no longer counts — DECIDED
+
+One question: **is anything wanted of this torrent**, answered from the register
+(`isWanted`). It is true from the moment a torrent is opened, because a torrent
+with no file list yet cannot be stated about and is being fetched precisely
+because somebody asked for it; and it becomes false exactly when the last
+viewer leaves, because that is when the map is published with nothing in it.
+
+Two facts decide the swarm, and they are separated from the doing
+(`swarmDecisionFor`) because this is the rule that has failed twice in the
+field:
+
+| wanted | ever wanted | swarm | idle clock |
+|---|---|---|---|
+| yes | — | take | stopped |
+| no | yes | let go | running |
+| no | no | leave alone | running |
+
+The last row is the 2.83.1 failure as a rule rather than as a mechanism: a
+torrent nobody has ever wanted is one being OPENED, and taking its swarm away
+there destroys the connections it will need a minute later.
+
+**Leaving on `done` was considered and NOT built.** A complete torrent can want
+nothing from a swarm, and the field case of 596 connections was exactly that.
+But a piece can be lost after the fact — the disk tier discards pieces under
+its own cap — and rejoining would then need a trigger nothing states today, so
+the failure mode is a read that waits for ever. The complete-file case is
+already answered from two directions: the upload policy refuses to seed when
+nothing anybody asked for is missing, and `keepWholeFiles` destroys a torrent
+whose every file has been written out whole.
+
+**What is accepted as narrowed, and stated so it is not discovered as a
+surprise.** A read that is flowing states nothing — it is reading bytes that
+are present — so a torrent is no longer held by the mere existence of a read.
+Three cases were checked: a `/stream` read belongs to a session whose map is
+live; the subtitle walk reads only what is downloaded; the background fill of a
+sidecar is covered by that file's ends. What is left is the disk cap evicting a
+torrent whose read is in flight during a moment when nothing is stated for it.
+The eviction order is the mitigation rather than a guard: candidates are sorted
+by when they were last WANTED, so a torrent an encoder is reading sorts last
+among them.
+
+**A torrent is on the idle clock from the moment it exists.** Until now the only
+thing that ever started that clock was a reader letting go, so a file list
+fetched and never played was held for the life of the process.
+
+## What the review of this work caught — READ
+
+Two defects, both found by reading the finished code rather than by a test, and
+both fixed with a check apiece.
+
+1. **A warm-up could take the urgency away from a plan somebody was waiting
+   for.** The two ends of a file are stated by whoever asks for them, and there
+   are two such callers: the playback plan, which a person is watching a loading
+   screen for, and the warm-up, which by its whole purpose nobody is waiting
+   for. The warm-up fires off the files beside the picture without awaiting
+   them, so it can arrive second — and it restated the same claimants at its own
+   lower level. The statement is raise-only now; the one caller entitled to put
+   them back down is the read that has just finished, which says so.
+2. **A departure could be noticed by nobody.** The background fill states a
+   claim of its own per file and withdraws it once nothing else wants that file
+   — but the withdrawal happened inside a pass that nothing acted on, so a
+   torrent everybody had left could be held "wanted" by the fill's own claim at
+   the moment of the departure and then never re-examined: in its swarm, with no
+   idle clock running. The pass acts on what it just said now.
+
+## Where the shared statement lives — DECIDED, then REVERSED the same day
+
+It was first published as a package, `@torrent-tv/torrent-contents`, so that
+both runtimes could read the same three files: the proxy states what to fetch
+from them, and the browser showed the list of episodes the instant a `.torrent`
+was dropped, before a proxy had been chosen.
+
+**Withdrawn within the hour, on the user's question: is the package better than
+moving it all to the proxy, when the torrent has to go there after the choice
+anyway.** It is not. The package bought one property — the instant list — with
+three permanent costs: a third artifact to version and publish, two copies at
+runtime that drift by version (a test was written for that drift, which is an
+admission the risk is real), and three wiring points that break the page at load
+with no error anybody sees (a second test was written for those).
+
+The answer is the route that already lists a source's files. For a magnet
+nothing changes at all — that is the path it has always taken — and the dropped
+`.torrent` now takes the same one. What leaves the browser with it is not only
+the classification but the whole notion of deciding anything from a name; what
+stays is what only the browser can see (the trackers and the web seeds in the
+file it holds) and how a name is SHOWN.
+
+**The cost is answered by connecting earlier, which is the user's own point.**
+Everything a viewer does needs a proxy, and choosing one and connecting to it
+used to begin only after a file had been picked. It begins when the page loads
+now, silently and with its failure swallowed, and `#acquireTransport` is
+joinable — so whatever the viewer does next joins that attempt instead of
+starting a second. Verified in a browser 2026-09-12: the page opens, the connect
+runs on its own (`public-only connect failed …; trying local path` on a dev
+server with no proxies registered), nothing is shown for it, and the page's own
+adapter turns the proxy's answer into the three lists with the order preserved,
+the torrent's own numbers kept and the release's repeated part taken off the
+names.
+
+`@torrent-tv/torrent-contents@1.0.0` is deprecated on npm with the reason.