@phamkhachoabk/dsh-local-vision-describe 0.1.0

package/README.md

@@ -0,0 +1,69 @@
+# Local Vision Describe
+
+Tell a text-only model what an image *shows*, not just what it *says*.
+
+[Local Vision](../local-vision) reads the text in an attached image. That is
+the whole answer for a document or a code screenshot, and no answer at all for
+a diagram, a chart, a UI mockup, or a photo — OCR returns nothing and the model
+is left to assume the image was empty.
+
+This bundle adds the other half: a vision-language model running on your own
+machine describes each image, and that description is added to the conversation
+beside the recognized text. **The image itself is still never sent to the
+model.**
+
+## Install
+
+```
+dsh plugin --profile web add @phamkhachoabk/dsh-local-vision-describe
+python3 -m pip install mlx-vlm
+```
+
+The first command pulls in Local Vision itself, so it is the only plugin
+command needed on a fresh machine. Restart the harness afterwards: bundle
+layers are composed at boot.
+
+The second is the part this bundle cannot do for you. MLX needs a Python
+environment, and the checkpoint (about 2.5 GB) downloads itself on the plugin's
+first start. Until both exist, the provider reports itself unusable and Local
+Vision keeps working with OCR alone.
+
+**Apple Silicon only** for now. MLX is built on Metal, so there is no Linux or
+Intel build. The seam is a registry, so a provider for another platform is one
+new package and one row in this bundle's patch — nothing else changes.
+
+## What it costs
+
+Seconds per new image, not milliseconds. Results are cached by attachment and
+prompt, so an image already described in the conversation is free on every
+later turn.
+
+Turn it off without uninstalling with the `describeImages` setting under
+`local-vision`; OCR then behaves exactly as before.
+
+## What the model sees
+
+```
+<local-vision-ocr attachment="sha256:a1b2c3d4" engine="apple-vision-documents" confidence="0.94">
+…recognized text…
+</local-vision-ocr>
+
+<local-vision-description attachment="sha256:a1b2c3d4" engine="mlx-vlm">
+A dark terminal window with a red error banner across the top third…
+</local-vision-description>
+```
+
+Two blocks, because they answer different questions and fail independently: an
+image whose text could not be read can still be described, and the other way
+round. Both are labelled as user-provided content rather than instructions.
+
+## Packages
+
+| Package | Role |
+| --- | --- |
+| `@phamkhachoabk/dsh-vision-describe` | Service Definition — `ctx.visionDescribe`, the engine contract, the ladder |
+| `@phamkhachoabk/dsh-vision-describe-mlx` | Service Provider — a persistent MLX vision-language model |
+| `@phamkhachoabk/dsh-local-vision-describe` | Bundle — the one name you install |
+
+The consumer is Local Vision's own `local-vision-context` row, which reads
+`ctx.visionDescribe` only when a provider is installed.

package/cordis.patch.yml

@@ -0,0 +1,20 @@
+# The Local Vision Describe bundle: the two rows that add whole-image
+# understanding on top of the Local Vision bundle it depends on. Rows are
+# addressed by id, so a profile's own cordis.patch.yml can reconfigure or
+# disable either of them individually — disabling `vision-describe-mlx` alone
+# keeps `ctx.visionDescribe` available for another provider to fill.
+#
+# The consumer row is not repeated here: `local-vision-context` comes from the
+# Local Vision bundle, and reads `ctx.visionDescribe` only when it is present.
+
+- insert:
+    # Service Definition: the ctx.visionDescribe registry, the engine contract,
+    # and the fallback ladder. Carries the prompt every request resolves through.
+    - id: vision-describe
+      name: '@phamkhachoabk/dsh-vision-describe'
+
+    # Service Provider: a vision-language model held in memory by MLX, on
+    # Apple Silicon. Needs python3 with mlx-vlm installed; the checkpoint
+    # downloads itself on first use.
+    - id: vision-describe-mlx
+      name: '@phamkhachoabk/dsh-vision-describe-mlx'

package/package.json

@@ -0,0 +1,31 @@
+{
+  "name": "@phamkhachoabk/dsh-local-vision-describe",
+  "version": "0.1.0",
+  "description": "Local Vision image descriptions for DeepSeek Harness: a vision-language model on your own machine tells a text-only model what each image shows, beyond the text OCR can read",
+  "type": "module",
+  "exports": {
+    "./cordis.patch.yml": "./cordis.patch.yml",
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "cordis.patch.yml",
+    "README.md"
+  ],
+  "license": "MIT",
+  "publishConfig": {
+    "access": "public"
+  },
+  "dsh": {
+    "bundle": {
+      "patch": "./cordis.patch.yml"
+    }
+  },
+  "dependencies": {
+    "@phamkhachoabk/dsh-local-vision": "^0.3.0",
+    "@phamkhachoabk/dsh-vision-describe": "^0.1.0",
+    "@phamkhachoabk/dsh-vision-describe-mlx": "^0.1.0"
+  },
+  "peerDependencies": {
+    "@deepseek-ai/cordis": "^4.0.2"
+  }
+}