dsh-read-animation 0.0.0-stage → 0.1.0

package/CHANGELOG.md

@@ -0,0 +1,103 @@
+# Changelog
+
+## 0.1.0 — unreleased
+
+Initial version.
+
+### Fixed after an adversarial audit
+
+Four independent reviews (blast radius, plugin code, packaging and lifecycle, then
+an adversarial pass over every high-severity finding) ran before this was
+published. What they found, and what changed:
+
+- **The frame sheet's `variantId` was not a content digest.** It was a hand-rolled
+  `sha256:filmanim…` string, and the DeepSeek adapter validates persisted ids
+  against `/^sha256:[0-9a-f]{64}$/` when it loads its upload index — then swallows
+  the failure into an *empty* index. The effect was not a visible error: it
+  discarded every other session's cached file id and re-uploaded every image.
+  Reproduced against the real exported index class (plugin id: written 1, reloaded
+  0; conforming id: 1/1). It is now the SHA-256 of the bytes being returned, which
+  also stops one on-disk cache entry from aliasing two different byte streams.
+- **The degraded path was worse than the missing feature.** The store reaches the
+  frame it re-encodes through its own `readImage` call, which this plugin wraps, so
+  falling back to the store's request read could hand the model the *whole
+  animation* where it expects one frame. The request path now suspends the
+  animation substitution around that call, and reads its own tiles through the
+  unwrapped read. Pinned by an integration check that breaks a stored frame and
+  asserts the fallback is exactly one frame.
+- **No decoded-pixel ceiling on the `read_animation` path.** `limitInputPixels`
+  was disabled with nothing in its place: a 60 KB file declaring a 9000x9000
+  canvas was refused by the deployment's own attachment store and decoded here
+  anyway, at about 310 MiB. Both the probe and the frame extraction now carry the
+  deployment's `imageLimits.maxImagePixels`.
+- **The sheet cache ignored the size the caller asked for**, so a caller with an
+  8 KiB budget could be served a 28 KB sheet built for a 4 MiB one. The target is
+  now part of the cache key, and a tile is never larger than a single frame would
+  have been for that caller.
+- **The measured header wrap could still be defeated.** A floor of 16 characters
+  kept the loop from reaching a budget that fit, so on a narrow canvas it gave up
+  and returned ink wider than the sheet. The floor is now low enough to be
+  satisfiable, and a canvas too narrow for legible text gets no header at all.
+- **The canvas was composed before it was shrunk**, building a 6156x6306 (about
+  111 MiB) image to throw away. Tile size and grid are settled first.
+- **Torn state files could poison the next record.** A crash part way through an
+  append left a line with no newline, so the following record concatenated onto it
+  and became unreadable too. A torn tail is now detected and compacted away.
+- **The store's bound applied to memory but not to what was read back**, and the
+  log was never compacted. Both fixed; the file is rewritten when it grows past
+  twice the entries worth keeping.
+- **Teardown was last-writer-wins.** Each install captured whatever was on the
+  method, so disposing one left another's patch removed and could reinstall a
+  disposed closure as a permanent own property. Three wrappers now share one
+  installer that resolves its delegate per call, removes cleanly (deleting an
+  inherited property rather than leaving a shadow behind), and reports whether it
+  actually took effect — the previous identity check compared a value that proxy
+  reads always re-wrap, so it could never detect a failed install.
+- **`npm test` did not run the invariant checks.** The no-`@deepseek-ai`-import
+  rule and the bundle/manifest consistency check lived only in the integration
+  harness, which prints SKIP and exits 0 without `DSH_HARNESS_MODULES`. Both now
+  run in `npm test`, which is also the `prepublishOnly` gate.
+- **`files[]` omitted `scripts/`** while `package.json` advertised two npm scripts
+  that live there, so the published package's own scripts were broken.
+
+- **A pasted animation keeps playing in the UI, and the message is left alone.**
+  The browser already uploads an animated GIF/WebP byte-for-byte and labelled as
+  such; the Host flattens it to a single
+  frame during prompt admission and keeps no copy of the source. The plugin wraps
+  `ctx.attachments.admitPromptContent` to capture the frames and the original file
+  at that moment, and `ctx.attachments.readImage` so the UI is served the animation
+  itself (the browser animates whatever bytes that one read returns). What you send
+  is what you send: the chat message is unchanged.
+- **The model now receives a labelled sheet of the frames.** The delivery step is
+  `ctx.attachments.readImageRequest(ref, target, signal)`, the method each adapter
+  calls to build its own copy of one image. It answers with exactly one image, so
+  the frames are composed into a grid chosen to be as square as the sheet budget
+  allows — every tile at its native size, with the frame index, duration and start
+  time printed under each, and the animation's own facts in a wrapped header. The
+  sheet is built once per animation and cached.
+- Rewriting the request from an `llm/stream` listener is **not** used and would not
+  work: a loop-built request arrives deep-frozen ("listeners read it, never rewrite
+  it"), so that earlier attempt silently did nothing.
+- Captured frames and the original are persisted per `$DSH_HOME` in
+  `read-animation/v1/pasted.jsonl`, keyed by the message's attachment id, so a
+  session resumed in a new process still displays — and still delivers — an
+  animation pasted before the restart.
+- A frame's index and start time ride in its attachment display name
+  (`x.gif#frame16@1960ms`), which the harness renders beside the picture once the
+  frames are delivered — so the timing never has to be repeated as message text.
+  (An earlier version of this feature put a header plus one caption per frame into
+  the user's own message; consecutive text parts are concatenated by the UI with
+  no separator, so a one-image message became an unreadable wall.)
+- `read_animation` tool: decodes an animated GIF or animated WebP, samples frames
+  across the timeline (or by explicit index / instant), commits each frame to the
+  attachment store, and returns a header plus one captioned image per frame.
+- Refuses animated PNG by name instead of silently returning its first frame,
+  which is what the host image library does with one.
+- Refuses static images and points at `read_image`; answers a single-frame GIF or
+  WebP with a note that it is not animated.
+- Discloses decode-time substitutions in the model-facing text: zero frame
+  delays treated as 100 ms, missing delays, transparency flattened onto white.
+- The package imports no `@deepseek-ai/*` module at runtime, so it loads both as
+  an installed package and as a `link:`ed checkout. The compiled schemas ship as
+  literals beside their authoring specs, and the integration harness keeps the
+  two in step.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 NeoHotaru
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,321 @@
-# Temporary Holding Version
+# dsh-read-animation
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+A DeepSeek Harness plugin that lets the agent **actually see an animated GIF**.
+
+`read_image` on a GIF gives the model one picture — the first frame — and no
+signal that the other frames existed. This plugin decodes every frame and hands
+the model a labelled sequence of real frames sampled across the animation's
+timeline.
+
+```sh
+dsh plugin --profile web add dsh-read-animation
+```
+
+---
+
+## The problem
+
+DeepSeek Harness normalizes every stored image to a single-frame 8-bit sRGB
+raster before it reaches a model, and it does this to animations too.
+`@deepseek-ai/dsh-attachment-local` excludes `image/gif` from its pass-through
+path and re-encodes through a pipeline that never sets `animated` or `page`, so
+`sharp` returns page 0 and the rest are dropped on the floor.
+
+Measured, on this machine, against an 8-frame GIF:
+
+```
+metadata:  {"format":"gif","width":240,"height":160,"pages":8,
+            "delay":[80,80,80,80,80,80,80,80],"loop":0}
+default -> 240x160  pages=1     ← what read_image delivers
+animated -> 240x1280 pages=1    ← all eight frames, when asked for
+```
+
+No layer above the read step can repair this. By the time a vision plugin, an
+image router, or the LLM adapter sees the image it is already a single frame —
+which is why the growing ecosystem of DSH vision plugins
+([`dsh-multimodal`](https://github.com/SCT192221/dsh-multimodal),
+[`dsh-mmroute`](https://github.com/jmxsxwyzjdwl/dsh-mmroute),
+[`dsh-vision-mix`](https://github.com/haiziyao/dsh-vision-mix),
+[`dsh-tool-plus`](https://github.com/xiaoso456/dsh-tool-plus), …) all say they
+"support PNG/JPEG/WebP/GIF" without any of them showing you motion. The
+animation is still intact in exactly one place: the file. So the tool lives
+there.
+
+Animated PNG has the same disease with a worse prognosis: libvips 8.18.7 does
+not read `acTL`/`fcTL`/`fdAT` at all, reports an APNG as a one-page PNG, and
+silently discards every frame after the first. This plugin detects that case from
+the chunk stream and refuses it by name instead of returning frame 0.
+
+## Two ways in, both handled
+
+**Pasting into the chat box.** That path is already clean on the browser side: the
+paste handler hands the clipboard's own `File` straight to the upload path, and
+that path encodes `image/gif` and `image/webp` unchanged (`imageMediaType()`
+accepts both and returns them as-is) — no canvas, no `drawImage`, no `toBlob`.
+**The bytes reach the Host whole.** It is the Host that destroys them:
+`admitPromptContent` runs every image through the single-frame normalizer and
+keeps no copy of the source.
+
+`admitPromptContent` is therefore the last moment the original bytes exist. The
+plugin wraps it, decodes the animation and stores the frames — then **returns the
+original result untouched**:
+
+```
+you paste 1 animation
+   ├─ the message: unchanged — the picture you sent, not one word added
+   ├─ the UI: the animation itself, played from the original bytes
+   └─ the model: one sheet of its frames, each labelled with index and time
+```
+
+**What you send is what you send.** Nothing is added to the message you wrote,
+and it does not turn into eight stills.
+
+**And it keeps playing in the UI.** The Web UI has exactly one way to display a
+durable image: `session.readAttachment(id)` → the Host's
+`ctx.attachments.readImage(ref)` →
+`URL.createObjectURL(new Blob([bytes], { type: attachment.mediaType }))`. Both the
+bytes and the media type come from that one host call, and the browser animates
+whatever it is handed. So the plugin serves the original at that **display read** —
+no client plugin, no media route, no second authorization path. The single frame
+in the store is only a lossy artifact of a normalizer that cannot represent
+motion; handing back the file the person actually pasted is the more truthful
+answer to "read this stored image", and the other two callers of that layer (ACP
+image blocks, session-log export) are better with an animation too.
+
+Each captured frame carries its index and start time in its attachment display
+name (`x.gif#frame16@1960ms`), which the harness renders to the model as
+`Image "<name>" …` — so the timing is already attached to the frames, waiting
+for the delivery step below.
+
+**The display path and the model's are independent**: the UI is served the
+original bytes by `readImage`, while the model's copy comes from
+`readImageRequest`, which composes the sheet.
+
+> **Where the frames are delivered**: `readImageRequest(ref, target, signal)` is
+> the method each adapter calls to build its own copy of one image
+> (`dsh-llm-deepseek` and `dsh-llm-pi-ai` both do), so it is the seam that reaches
+> the model. It answers with exactly one image, so the frames are composed into one
+> labelled sheet: a 4x2 grid for eight samples of a phone-shaped animation, every
+> tile at its native size, with the frame index, duration and start time printed
+> under each. Nothing is re-derived per request — the same stored attachment always
+> produces the same sheet — and the sheet is built once and cached.
+>
+> The route that looks obvious, rewriting the request from an `llm/stream`
+> listener, is closed by design: a loop-built request "arrives deep-frozen
+> (mutation throws): its content is a pure function of the session log", so
+> listeners may read it but never rewrite it. An earlier version of this plugin
+> tried exactly that and silently did nothing, because the probe behind it used a
+> hand-built request, which is the one case that documentation exempts.
+
+**Files.** Animations in the workspace, or attached as files, still exist on
+disk byte-for-byte; the `read_animation` tool below reads those.
+
+## What it does
+
+`read_animation(path, …)` returns one text header plus one captioned image per
+sampled frame:
+
+```
+<path>D:\work\dance.gif</path>
+<type>animation</type>
+<content>
+image/gif animation, 480x270 px, 24 frames, 1200 ms total, ~20 fps, loops forever.
+Attached below are 6 frames sampled evenly in time from this animation. They are
+consecutive moments of the SAME animation, in time order, not separate images;
+each is preceded by its frame index and start time.
+Only frames 0, 4, 9, 13, 18, 23 of 24 were sampled, so any motion between two
+attached frames is not visible. Re-read with explicit "frames" indices to inspect
+a specific moment, or a higher "max_frames" to sample more densely.
+</content>
+```
+
+…then, for each frame:
+
+```
+frame 9 of 24 (sample 3) · starts at 450 ms · displayed for 50 ms
+[ image ]
+```
+
+Everything the model needs to reason about *motion* is in that text: how long the
+animation runs, how fast it plays, whether it loops, which frames it is looking
+at, and — stated outright, because it is the easiest thing to get wrong — that
+the gaps are gaps.
+
+## Install
+
+```sh
+dsh plugin --profile web add dsh-read-animation
+```
+
+Replace `web` with your profile name. **Not on npm yet** — until it is, install
+straight from the repository:
+
+```sh
+dsh plugin --profile web add github:NeoHotaru/dsh-read-animation
+```
+
+That route is verified end to end: it installs, `dsh plugin add` registers the
+bundle, and the `sharp` native binary resolves from the package's own
+`node_modules`. (A git spec copies the whole repository, tests and fixtures
+included, which is why the npm package above is the nicer install once it
+exists.)
+
+Or from a checkout:
+
+```sh
+cd dsh-read-animation
+npm install                                     # fetches sharp
+dsh plugin --profile web add link:$PWD
+```
+
+The same spec works in the Web sidebar's Plugins page, and in Settings the plugin
+list is read-only. Restart the running instance afterwards: plugin rows take
+effect on the next process, and a `link:`ed checkout is loaded once at startup.
+
+## Usage
+
+The model reaches for it on its own for a `.gif`, because the tool description
+says to. You can also ask directly:
+
+> what happens in this GIF? — `animations/loading-spinner.gif`
+
+Arguments:
+
+| argument | meaning |
+|---|---|
+| `file_path` | the animation to read (required) |
+| `frames` | explicit 0-based indices, in the order given — up to 16 |
+| `at` | millisecond instants to sample instead of an even spread |
+| `max_frames` | how many frames to sample (default 8, maximum 16) |
+| `max_edge` | long-edge size of each returned frame in px (default 640) |
+
+A useful loop, when the first read is not enough detail:
+
+```
+read_animation { file_path: "dance.gif" }              → 8 frames across the timeline
+read_animation { file_path: "dance.gif",
+                 frames: [11, 12, 13], max_edge: 1024 } → those three, full size
+```
+
+## Configuration
+
+Optional, in the plugin row of your profile's `cordis.patch.yml`:
+
+```yaml
+- id: read-animation
+  name: 'dsh-read-animation'
+  config:
+    maxFrames: 8          # default frames per call, 1..16
+    maxEdge: 640          # default long-edge cap, 64..2048
+    background: white     # "white" flattens transparency, "keep" preserves it
+    maxInputBytes: 67108864   # refuse animations larger than this
+    expandPastedAnimations: true   # a pasted animation keeps playing in the UI and reaches the model as a frame sheet
+```
+
+## What it will not do
+
+- **Video.** MP4/WebM need ffmpeg and a different decode path. Use
+  [`dsh-video-tools`](https://github.com/PiedPiper911/dsh-video-tools).
+- **APNG.** Not decodable by the host image library; refused with an explanation
+  (see above).
+- **Read a static image.** It says so and points at `read_image`. A single-frame
+  GIF or WebP is answered, with a note that it is not animated.
+- **Guess.** When the sampler returns fewer frames than you asked for, the header
+  says which ones and why.
+
+## How it behaves under the hood
+
+Frames are committed through `ctx.attachments.saveImages`, the same store every
+other harness image goes through. So frames are content-addressed (two identical
+frames in one GIF cost one attachment; re-reading a file reuses what is already
+stored), durable across a session replay, and subject to the deployment's own
+image limits. Frames are resized to the target edge and re-encoded as PNG before
+storage, so the harness's normalizer has nothing left to do to them.
+
+Reading goes through `ctx.fs.readBytes` and emits `fs/observed`, so the sandbox
+and the read-before-write policy apply exactly as they do to `read_image`. The
+tool refuses to run when the resolved model route does not declare image input,
+for the same reason `read_image` does.
+
+**A pasted animation** is handled in three steps, none of which touch your
+message: **capture** at admission (`ctx.attachments.admitPromptContent` is the last
+moment the original bytes exist), **display** at `ctx.attachments.readImage` (the
+one read the UI makes, served the original so it plays), and **delivery** at
+`ctx.attachments.readImageRequest` (the model's copy, composed into one labelled
+sheet). All three are **monkey-patches of core service methods**, so they do
+nothing at all the moment anything is not as expected — a missing method, a failed
+assignment, a decode that throws, an original that cannot be read back, a batch
+with no room. The caller always receives exactly what it would have received
+anyway. Captured frames and the original file are recorded in
+`$DSH_HOME/read-animation/v1/pasted.jsonl`, keyed by the attachment id the message
+carries, so a session resumed in a new process still displays — and still expands
+— an animation pasted before the restart. Delivery swaps references only — at that
+layer an image is still `{type: 'image', attachment}` — and never touches bytes.
+
+### Design notes for contributors
+
+Two decisions are worth knowing before you change anything.
+
+**Frames are extracted one at a time with `sharp(buf, { page: i })`, not by
+decoding the animation as one tall strip and slicing it.** Both were measured on
+a 600×400, 60-frame GIF: the strip path costs 54.9 MiB resident and is *slower*
+here (228 ms for 8 frames at a 512 px target, because it resizes all 60 frames)
+than per-frame extraction (182 ms, one frame resident at a time). The correctness
+claim this rests on — that `{ page: i }` is byte-identical to frame `i` of the
+composited strip — is pinned by a test.
+
+**The package imports no `@deepseek-ai/*` at runtime.** The harness's own tools
+build their definitions with `defineTool`, which compiles a schema spec and wraps
+the body with argument validation. That is right for a plugin installed *as a
+package*, where pnpm places it under `$DSH_HOME/profiles/<name>/node_modules/`
+and `@deepseek-ai/*` resolves by walking up to the profile's hoisted tree. It is
+wrong for a plugin *linked from a working copy*: a `link:`ed package loads
+through its real path, so its imports resolve upward from the checkout, where no
+`@deepseek-ai/*` exists — measured, by importing the entry with `node_modules`
+absent (`ERR_MODULE_NOT_FOUND: Cannot find package '@deepseek-ai/dsh-tools'`).
+`lib/tool.js` therefore ships the compiled JSON Schema as literals beside the
+authoring spec, and the integration harness asserts the real `defineTool` still
+derives exactly those literals from that spec. Edit the spec, run
+`npm run derive-schema`, paste the result.
+
+## Development
+
+```sh
+npm install
+npm test                 # decoder, sampling, envelope, wrapping, store, layout, arguments
+npm run test:integration # real dsh-fs, dsh-attachments, and dsh-tools, in-process
+npm run fixtures         # regenerate test/fixtures (needs Python 3 + Pillow)
+```
+
+`npm test` needs nothing but Node and the installed `sharp`, and it is the gate
+`npm publish` runs. It also holds the two invariants that used to live only in the
+integration harness — the no-`@deepseek-ai`-import rule and the bundle's
+consistency with `package.json` — because that harness prints SKIP and exits 0
+when `DSH_HARNESS_MODULES` is unset, so "the integration suite is green" could
+otherwise mean nothing had been checked.
+
+`npm run test:integration` boots the genuine `dsh-fs-local`,
+`dsh-attachment-local`, `dsh-system-prompt`, and `dsh-tools` plugins in-process,
+registers the tool, and executes it. It exists to prove the one thing no unit
+test can reach and the whole design depends on: that a single tool result
+carrying *several* image blocks survives the real registry and the real
+attachment store. It also drives the pasted-animation path end to end — a real
+`llm.stream` dispatch, a deep-frozen request, and the request image the adapter
+receives — because three separate bugs in this plugin were invisible to
+everything that did not go through that path. Only the `llm` service and the
+agent object are stubbed, both because they need a live model session. Point it
+at your harness install:
+
+```sh
+DSH_HARNESS_MODULES="$(npm root -g)/@deepseek-ai/dsh/node_modules" npm run test:integration
+```
+
+On a machine that has no `sharp` reachable from the checkout,
+`DSH_READ_ANIMATION_SHARP=<path to a loadable sharp>` overrides resolution.
+
+The fixtures are committed, so the suite runs without Python; `make-fixtures.py`
+regenerates them and documents what each one is for.
+
+## License
+
+MIT. See [THIRD_PARTY.md](THIRD_PARTY.md) for sharp and libvips.