dshm-compat 0.0.0-stage → 0.1.1

package/CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+## 0.1.1
+
+- Tool results are now lossless JSON: `undefined`-valued fields are stripped at
+  the tool boundary. This fixes `compat_analyze` / `compat_plan` / `compat_scan`
+  failing with `value is not lossless JSON` on DSHM.
+- Local tarball installation: `compat_install` (and `compat_convert` with
+  `install:true`) unpacks a `.tgz` into the owning DSH profile's `node_modules`,
+  records the dependency, and appends the bundle. This works around DSHM's host
+  installer rejecting local paths, `file:` and `http://` specs. A restart is
+  required, and the result says `restartRequired: true`.
+- Local install preserves the tarball's file modes (needed for native worker
+  binaries such as `dsh-ops-worker`).
+- Docs updated with the DSHM `pnpm view` limitation and the local install path.
+
+## 0.1.0
+
+- Initial analyzer, S0/S1 converter, dependency preflight/vendoring, guest
+  probe/provision, install queue, and dsh-ops 0.3.x recipe.

package/INSTALL.md

@@ -0,0 +1,68 @@
+# Installing dshm-compat
+
+The artifact is a normal DSH bundle package: `dshm-compat-0.1.0.tgz`.
+It is host-only, has no runtime dependencies, and adds eight agent tools.
+
+## A. DSHM (the tablet)
+
+### A1. npm or GitHub
+
+Publish the package to npm (remove `"private": true` from `package.json`
+first), or push this directory to a GitHub repository.
+
+On DSHM builds without a shell, the official **Settings → Plugins → Add plugin**
+input calls `pnpm view` before installing and fails with
+`Command failed with EACCES: pnpm view ... spawn pnpm EACCES`. Use one of these
+instead:
+
+- ask the DSHM Agent to run `pnpm add dshm-compat@0.1.1` in its bash tool (the
+  bundled pnpm shim routes that command to the host install queue); or
+- install from a GitHub repository URL; or
+- after any working install, use the plugin's own `compat_install` tool for
+  npm/GitHub specs.
+
+The plugin's local converted tarballs are installed by `compat_install` itself:
+it unpacks them into `profiles/<profile>/node_modules/<name>`, records the
+dependency and bundle, and asks for a DSHM restart. The host install queue does
+not accept local paths, `file:` or `http://` specs.
+
+### A2. HTTPS tarball
+
+Put `dshm-compat-0.1.0.tgz` on any HTTPS host (GitHub Release, Gitee Release,
+object storage) and paste the `.tgz` URL into the same Add plugin input. DSHM's
+installer accepts tarball specs; if the current market UI refuses a plain URL,
+use A1.
+
+### A3. hdc / local path
+
+If `hdc` is available and the app can read the path:
+
+```sh
+hdc file send dshm-compat-0.1.0.tgz /data/storage/el2/base/haps/entry/files/dshm-compat-0.1.0.tgz
+```
+
+Then enter the absolute path (or `file:/...`) in the Add plugin input. This
+depends on the DSHM build's local-file support; A1 is safer.
+
+## B. Desktop DSH (for testing only)
+
+```powershell
+& 'D:\安装\dsh-desktop\resources\runtime\cli\bin\dsh.cmd' plugin --profile desktop add 'D:\dsh\dsh-26-09-30\dshm-compat\dshm-compat-0.1.0.tgz'
+```
+
+If the profile reports `ERR_PNPM_UNEXPECTED_VIRTUAL_STORE`, its `node_modules`
+was migrated from another DSH home; repair that profile first or use a fresh
+profile.
+
+## C. After installation
+
+- The bundle row is `dshm-compat`; enable it if it is disabled.
+- Ask the agent to run `compat_status`; it should return the data directory and
+  empty report list.
+- Then run `compat_analyze`, `compat_plan`, `compat_convert`, `compat_install`.
+- No market input interception is included; the tools are the entry point.
+
+## D. Uninstall
+
+Remove `dshm-compat` from the plugin market/plugin manager. Its data lives under
+`$DSH_HOME/dshm-compat` and is not removed automatically.

package/README.md

@@ -1,3 +1,91 @@
-# Temporary Holding Version
+# dshm-compat
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+Analyze DSH plugins that cannot install or run on HarmonyOS DSHM, then convert
+them through the least invasive strategy that can work.
+
+## Current phase
+
+0.1.1 fixes DSHM tool-result validation and adds local-tarball installation.
+Phase 1 ships the analyzer, the S0/S1 converter, the dsh-ops recipe, guest
+probing/provisioning, the DSHM install-queue transport, and a profile-local
+fallback for artifacts the host queue rejects:
+
+- `compat_analyze` downloads an npm/GitHub/local package and classifies it.
+- `compat_plan` returns the concrete conversion plan without changing anything.
+- `compat_scan` triages up to 50 specs in parallel and returns a strategy table.
+- `compat_convert` rewrites an S0 package's manifest (foreign `os`/`cpu` gates,
+  install lifecycle scripts, client platform), vendors missing pure-JS runtime
+  dependencies into a flat `node_modules`, validates the resulting tarball, and
+  repacks it as a DSHM tarball; for S1 it removes a native dependency that the
+  DSHM core already provides. For `dsh-ops` it applies the 0.3.x recipe once an
+  AArch64 Linux worker is supplied via `workerPath`; without one it returns
+  `needs-native-build` with the exact cross-build command.
+- `compat_guest_probe` reports the Alpine guest kernel/packages/tools.
+- `compat_guest_provision` installs a reviewed allowlist of Alpine packages in
+  the guest (`confirm:true` required).
+- `compat_status` lists stored reports and install attempts.
+- `compat_install` installs an npm/GitHub spec through the host queue. Local
+  `.tgz` paths (and `file:`/`link:`) are unpacked directly into the owning DSH
+  profile and its bundle list, then need a DSHM restart; this is required
+  because the DSHM host queue rejects local specs.
+- `compat_convert` with `install:true` uses the same local fallback for its
+  converted tarball.
+
+The converter deliberately stops at the manifest/repack/vendor boundary:
+source transpilation and native builds are later phases. A report saying
+`S2`/`S3`/`S4` is returned as-is rather than being reported as a successful
+conversion.
+
+See [architecture](docs/ARCHITECTURE.md) and [conversion recipes](docs/RECIPES.md).
+
+## CLI
+
+The same converter runs on a PC without installing the DSH plugin:
+
+```sh
+node bin/dshm-compat.mjs analyze dsh-m
+node bin/dshm-compat.mjs plan dsh-ops
+node bin/dshm-compat.mjs convert dsh-cache-billing --no-vendor
+node bin/dshm-compat.mjs convert dsh-ops --worker ./dsh-ops-worker
+node bin/dshm-compat.mjs scan dsh-m dsh-ops some-plugin
+```
+
+`convert` writes the tarball under `$DSH_HOME/dshm-compat/builds` (override with
+`DSHM_COMPAT_DIR`).
+
+## Test
+
+```sh
+node test/run.mjs
+```
+
+The harness builds synthetic tarballs under `_tmp`, uses a fake DSHM core, and
+never touches a real DSH profile.
+
+Strategy codes:
+
+| Code | Meaning |
+|---|---|
+| `S0` | pure JS/TS; code-level conversion on the host |
+| `S1` | known native dependency with an OHOS/JS substitute |
+| `S2` | native binary or shell command; guest Alpine sidecar |
+| `S3` | source-available native; WASM or remote build |
+| `S4` | unsupported, e.g. a win32-only binary with no source |
+
+## Install
+
+Install the package directory as a DSH bundle in the target profile. On DSHM,
+the package can be packed and submitted through the install queue once the
+converter phase is in place.
+
+## Data
+
+`$DSH_HOME/dshm-compat/`
+
+- `reports/<fingerprint>.json` — full analyzer report.
+- `state.json` — bounded report/install index.
+
+## Safety
+
+The analyzer never executes install scripts. `compat_install` runs third-party
+code only after the caller passes `confirm:true`.

package/bin/dshm-compat.mjs

@@ -0,0 +1,83 @@
+#!/usr/bin/env node
+/**
+ * Standalone PC-side entry for dshm-compat.
+ *
+ * The same modules run inside DSHM, but a desktop checkout can convert a
+ * package and hand the tarball to the device. No dependencies are required.
+ */
+import { analyze } from '../lib/analyze.js'
+import { scan } from '../lib/batch.js'
+import { convert } from '../lib/convert.js'
+import { buildPlan } from '../lib/plan.js'
+import { status } from '../lib/state.js'
+
+const USAGE = `dshm-compat <command> [args]
+
+commands:
+  analyze <spec>                 classify one npm/GitHub/local package
+  plan <spec>                    print the conversion plan
+  convert <spec> [--no-vendor] [--worker <path>]
+                                write a converted tarball
+  scan <spec> [<spec> ...]       classify several packages
+  status                         show the local compatibility data directory
+
+spec: npm name[@version], https://github.com/owner/repo, or a local .tgz path
+`
+
+const args = process.argv.slice(2)
+const command = args[0]
+const rest = args.slice(1)
+
+/** @param {string} message */
+function fail(message) {
+  console.error(message)
+  process.exitCode = 1
+}
+
+if (command === undefined || command === '--help' || command === '-h') {
+  console.log(USAGE)
+} else if (command === 'analyze') {
+  if (rest[0] === undefined) fail('analyze requires a spec')
+  else {
+    try {
+      const report = await analyze(rest[0], { textScan: true })
+      console.log(JSON.stringify(report, null, 2))
+    } catch (error) { fail(String(error?.message ?? error)) }
+  }
+} else if (command === 'plan') {
+  if (rest[0] === undefined) fail('plan requires a spec')
+  else {
+    try {
+      const report = await analyze(rest[0], { textScan: true })
+      console.log(JSON.stringify(buildPlan(report), null, 2))
+    } catch (error) { fail(String(error?.message ?? error)) }
+  }
+} else if (command === 'convert') {
+  if (rest[0] === undefined) fail('convert requires a spec')
+  else {
+    try {
+      const vendor = !rest.includes('--no-vendor')
+      const workerAt = rest.indexOf('--worker')
+      const workerPath = workerAt >= 0 ? rest[workerAt + 1] : undefined
+      const report = await analyze(rest[0], { textScan: false })
+      const result = await convert(rest[0], { report, vendor, workerPath })
+      console.log(JSON.stringify(result, null, 2))
+      if (result.status !== 'ok') process.exitCode = 1
+    } catch (error) { fail(String(error?.message ?? error)) }
+  }
+} else if (command === 'scan') {
+  if (rest.length === 0) fail('scan requires at least one spec')
+  else {
+    try {
+      const result = await scan(rest)
+      console.log(JSON.stringify({
+        counts: result.counts,
+        items: result.items.map(({ report: _report, ...item }) => item),
+      }, null, 2))
+    } catch (error) { fail(String(error?.message ?? error)) }
+  }
+} else if (command === 'status') {
+  console.log(JSON.stringify(status(), null, 2))
+} else {
+  fail(`unknown command: ${command}\n\n${USAGE}`)
+}

package/cordis.patch.yml

@@ -0,0 +1,6 @@
+# dshm-compat: one host row. The row registers the compatibility analyzer and
+# (in later phases) conversion/install tools. It never changes other rows.
+- insert:
+    - id: dshm-compat
+      name: dshm-compat
+      config: {}

package/docs/ARCHITECTURE.md

@@ -0,0 +1,92 @@
+# dshm-compat architecture
+
+`dshm-compat` is a DSH plugin that classifies and converts plugins that cannot
+install or run on HarmonyOS DSHM. It is deliberately layered: a conversion that
+would require a different runtime is reported as such, not disguised as a
+manifest edit.
+
+## DSHM constraints that shape the design
+
+1. **No spawnable shell on the host.** The app sandbox rejects `execve` of
+   bundled tools. `dshm-vm-bash` redirects `bash -c` into an Alpine QEMU guest.
+2. **No host `dlopen` of profile native code.** `dshm-vm-bash`'s path notes say
+   `dlopen` only admits `/data/storage/el1/bundle/libs/arm64`; a `.node` placed
+   in a profile or plugin directory does not load. Native code must run in the
+   guest, or be WASM/pure JS.
+3. **The install path is a queue.** `$DSH_HOME/install-queue/<id>.req` is
+   consumed by the host's in-process installer, which writes `<id>.done` or
+   `<id>.fail` JSON. There is no package manager process to call.
+4. **The guest is a full Alpine 3.22 aarch64 userland** with 9p access to the
+   sandbox, QGA command execution, `apk`, and network through slirp. It is the
+   universal fallback for native code and shell commands.
+
+## Strategy layers
+
+| Code | Layer | What runs where | Conversion |
+|---|---|---|---|
+| `S0` | code | host | rewrite manifest, repack; dependency vendoring and source builds later |
+| `S1` | dependency | host | remove a native dependency that the DSHM core already provides as an OHOS port |
+| `S2` | environment | Alpine guest | provision Node/npm/build tools, run or build the native artifact, proxy tools |
+| `S3` | instruction | host/guest/remote | compile the addon to WASM with an N-API shim, or build/run remotely |
+| `S4` | none | nowhere | report the blocking artifact honestly |
+
+## Pipeline
+
+```
+spec ──► resolve (npm / GitHub / local .tgz)
+      ──► extract (ustar/pax, no dependency)
+      ──► analyze (manifest, native files, scripts, platform/Electron scan, core inventory)
+      ──► plan (S0..S4 + concrete steps)
+      ──► convert (S0/S1 now; S2+ is a plan)
+      ──► pack (ustar writer)
+      ──► install queue (DSHM only)
+```
+
+## Modules
+
+| Module | Responsibility |
+|---|---|
+| `lib/fetch.js` | resolve npm specs, GitHub repos, local archives; bounded downloads |
+| `lib/tar.js` | ustar/pax extraction with traversal and size guards |
+| `lib/analyze.js` | classification evidence, strategy, confidence, next action |
+| `lib/core.js` | locate `$DSH_HOME/cores/<version>/node_modules` and compare runtime dependencies |
+| `lib/rules.js` | exact recipes for known packages (`dsh-ops`) |
+| `lib/convert.js` | S0 manifest rewrite/repack and vendoring; S1 native-dependency removal |
+| `lib/vendor.js` | flat `node_modules` vendoring for missing pure-JS dependencies |
+| `lib/validate.js` | post-conversion manifest/entry/dependency validation |
+| `lib/plan.js` | user-visible conversion plan for every strategy |
+| `lib/guest.js` | read-only Alpine guest probe through `subprocess` |
+| `lib/install.js` | DSHM install-queue producer with polling and cancellation |
+| `lib/state.js` | bounded report/install index and atomic writes |
+| `lib/pack.js` | dependency-free ustar/gzip writer |
+
+## Tool surface
+
+| Tool | Side effects |
+|---|---|
+| `compat_analyze` | network download; writes a report under `$DSH_HOME/dshm-compat` |
+| `compat_plan` | network download; writes a report; no package changes |
+| `compat_scan` | network downloads; writes one report per successful spec |
+| `compat_convert` | writes a converted tarball; optionally submits it to the install queue |
+| `compat_guest_probe` | read-only guest command |
+| `compat_guest_provision` | installs allowlisted Alpine packages in the guest |
+| `compat_install` | writes one install-queue request; runs third-party code |
+| `compat_status` | read-only |
+
+## Phase status
+
+- **Phase 1 (done):** analyzer, batch triage, S0 manifest converter + flat
+  dependency vendoring + output validation, S1 manifest substitution,
+  dependency preflight, guest probe/provisioning, worker execution primitive,
+  dsh-ops 0.3.x recipe (external AArch64 worker), install queue, offline tests.
+- **Phase 2 (next):** on-device guest verification, dsh-ops worker build, nested
+  version-conflict vendoring, sidecar RPC.
+- **Phase 3:** WASM/remote fallback, market UI entry, compatibility registry
+  updates.
+
+## Non-goals
+
+- Translating a win32-only binary into OHOS code.
+- Loading profile `.node` files in the host.
+- Running arbitrary `postinstall` scripts on the host.
+- Claiming S2/S3/S4 conversions succeeded when only a plan exists.

package/docs/RECIPES.md

@@ -0,0 +1,140 @@
+# Conversion recipes
+
+These are the concrete recipes behind `S1`/`S2` plans. They are intentionally
+written as patches to the upstream package, not as claims that the current
+converter executes them.
+
+## `dsh-ops` (Rust native worker)
+
+Upstream: `T-Auto/dsh-ops`. The core is `native/` (Rust) and the published
+worker is `runtime/<target>/dsh-ops-worker`.
+
+### 1. Support matrix
+
+`lib/ptc/runtime.js` hard-codes:
+
+```js
+const PLATFORMS = ['win32-x64', 'linux-x64', 'darwin-arm64']
+```
+
+`resolveWorker()` checks this matrix **before** considering
+`DSH_OPS_WORKER_BIN`, so a binary path override alone cannot unblock DSHM.
+
+Patch:
+
+- add `linux-arm64` to `PLATFORMS`;
+- map an OHOS `process.platform` to `linux-arm64` before the matrix check;
+- keep `workerName()` returning `dsh-ops-worker` on non-win32;
+- look for `runtime/linux-arm64/dsh-ops-worker`.
+
+Also update:
+
+- `package.json` `os`/`cpu` (remove the gates or add `openharmony`/`arm64`);
+- `dsh-plugin.json` `platforms`, `architectures`, and `targets` for each
+  `x-components` entry.
+
+### 2. Build the worker
+
+From a Linux build host or CI:
+
+```sh
+rustup target add aarch64-unknown-linux-musl
+# either
+cross build --release --locked --manifest-path native/Cargo.toml --target aarch64-unknown-linux-musl
+# or
+cargo zigbuild --release --locked --manifest-path native/Cargo.toml --target aarch64-unknown-linux-musl
+```
+
+The Windows/macOS metadata crates are target-gated in `native/Cargo.toml`; the
+portable dependencies are `cap-std`, `cap-fs-ext`, `fs2`, `rustix`, `sha2`,
+`serde`, `serde_json`, `uuid`, `regex`.
+
+Stage the result as `runtime/linux-arm64/dsh-ops-worker` with a matching
+`provenance.json` (`version`, `target`, `executable`, `sha256`).
+
+### 3. Spawn through the guest
+
+`lib/file/worker.js` and `lib/ptc/worker.js` call:
+
+```js
+subprocess.spawn({ argv: [file], ... })
+```
+
+`dshm-vm-bash` only redirects `bash -c`, so patch both call sites to:
+
+```js
+argv: ['bash', '-c', 'exec "$0"', file]
+```
+
+The worker path is under the profile's 9p share, so the guest sees the same
+absolute path. If exec from 9p fails on a device, copy the worker to
+`/tmp/dsh-ops-worker` once (hash-keyed) and execute that copy.
+
+### 4. Shell component
+
+`lib/shells/resolve.js` treats every non-win32 host as POSIX and probes the
+**host** for `/bin/bash`. DSHM's host has no bash. Patch the DSHM branch to
+report the guest's `/bin/bash` as available; `ops_bash` then spawns
+`bash -c`, which `dshm-vm-bash` routes into the guest.
+
+Disable the PowerShell rung: the pwsh payload is win32-only.
+
+### 5. PTC vs file
+
+`lib/ptc/worker.js` calls `sandbox.confine()` for non-`danger-full-access`
+policies. If DSHM's sandbox provider cannot confine the worker, PTC tools fail
+closed. The `file` component uses policy checks but no `sandbox.confine`, so
+port `ops_*` file tools first and treat PTC as a later optimization.
+
+### 6. Verification
+
+Run on the device:
+
+- `ops_stat` on a workspace file;
+- `ops_grep` over a small tree;
+- `ops_write_file` then `ops_stat` (check the new revision);
+- `ops_copy` / `ops_move` (9p rename/atomicity);
+- a failure case with a read-only policy.
+
+Watch for `cap-std`/`rustix` failures on 9p (`openat2`, `statx`, `renameat2`,
+`flock`). If those fail, the fallback is a JS reimplementation of the worker
+protocol rather than a guest port.
+
+## Common native dependencies
+
+| Dependency | DSHM route |
+|---|---|
+| `sharp` | DSHM core ships an OpenHarmony port; remove the plugin's dependency so the core resolves it (S1) |
+| `koffi` | same as `sharp` |
+| `node-pty` | same as `sharp`; DSHM's terminal backend already uses it |
+| `better-sqlite3` / `sqlite3` | replace calls with `node:sqlite` (Node 24) or run in the guest |
+| `canvas` | no core port; guest or WASM |
+| `esbuild` | `esbuild-wasm` on the host or `esbuild` in the guest |
+| `bcrypt` | `bcryptjs` |
+| `argon2` | `hash-wasm` |
+
+## Electron API shim
+
+For host plugins that import `electron` only for platform glue:
+
+| Electron | DSHM equivalent |
+|---|---|
+| `app.getPath(...)` | DSH home/paths service |
+| `shell.openExternal` / `openPath` | `open-in-app` |
+| `dialog.showOpenDialog` | directory picker |
+| `clipboard` | ArkWeb clipboard API |
+| `ipcMain` / `ipcRenderer` | DSH client connection / Cordis events |
+| `BrowserWindow` | not shimmed; reimplement as a DSH client plugin or run remotely |
+
+## Shell commands
+
+A plugin that calls `child_process` directly will not be redirected by
+`dshm-vm-bash`. Two options:
+
+1. Rewrite the import in the bundled host code to DSH's `subprocess` service
+   (which routes `bash -c` to the guest).
+2. Run the whole plugin host half in the guest and proxy its tools over RPC.
+
+The guest has `apk`; ffmpeg, git, and Python can be installed offline by
+downloading the Alpine packages on the host, placing them in the 9p share, and
+running `apk add --no-network`.

package/icon.svg

@@ -0,0 +1,6 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" role="img" aria-label="DSHM compatibility">
+  <rect x="3" y="3" width="58" height="58" rx="14" fill="#1f2937"/>
+  <path d="M18 22h12l5 6-5 6H18l5-6-5-6Z" fill="#60a5fa"/>
+  <path d="M34 30h12l-5 6 5 6H34l-5-6 5-6Z" fill="#f59e0b" opacity=".95"/>
+  <path d="M14 45h36" stroke="#e5e7eb" stroke-width="3" stroke-linecap="round"/>
+</svg>