@harusame64/desktop-touch-mcp 0.15.3 → 0.15.5

package/README.ja.md

@@ -10,12 +10,14 @@ Claude がデスクトップを直接見て、直接操作する。
 マウス・キーボード・スクリーンショット・Windows UI Automation・Chrome DevTools Protocol・ターミナル・SmartScroll・Reactive Perception Graph を統合した 57 のツールを提供する MCP サーバーです。
 
 > *v0.15: Rust ネイティブエンジンにより**平均 82 倍高速化** — UIA フォーカス取得 2ms、SSE2 SIMD 画像差分 13〜15 倍速。設定不要：エンジンは自動ロード、不在時は PowerShell に透過フォールバック。*
+> *v0.15.5: **固定リリース検証** — npm ランチャーは対応する GitHub Release tag だけを取得し、Windows runtime zip を検証してから展開します。*
 
 ---
 
 ## 特徴
 
 - **⚡ 高性能 Rust ネイティブコア** — UIA ブリッジと画像差分エンジンを Rust (`napi-rs` + `windows-rs`) で実装し、ネイティブ `.node` アドオンとしてロード。専用 MTA スレッドからの直接 COM 呼び出しにより PowerShell プロセス起動を排除 — `getFocusedElement` は **2ms**（160 倍高速）、`getUiElements` はバッチ型 BFS アルゴリズムでクロスプロセス RPC を最小化し **約 100ms** で完了。画像差分は **SSE2 SIMD** で 13〜15 倍のスループット。ネイティブエンジンが利用不可の場合、全関数が PowerShell に透過フォールバック — 設定不要。
+- **🎯 Set-of-Marks (SoM) ビジュアルフォールバック** — ゲーム・RDP・非対応 Electron アプリで UIA が完全に機能しない場合でも、`screenshot(detail="text")` が Hybrid Non-CDP パイプラインを自動起動。Rust 画像前処理 → Windows OCR → クラスタリング → 赤い枠線 + 番号バッジ（`[1]`、`[2]`…）付き PNG 画像を生成し、`clickAt` 座標付きの要素リストを返します。CDP 不要。
 - **LLM ネイティブ設計** — 人間の操作を模倣するのではなく、「LLM がいかにコンテキストを消費せず高速に動けるか」を前提に設計。`run_macro` による複数操作の一括実行（API 往復の削減）と、**MPEG P-frame 方式のレイヤー差分** (`diffMode`) を組み合わせることで、無駄な画像転送や推論ループを極限まで削ぎ落とす。
 - **Reactive Perception Graph** — ウィンドウやブラウザタブに `lensId` を登録し、以後の action tool に渡すだけで、操作前の安全 guard と操作後の `post.perception` フィードバックを受け取れます。`screenshot` / `get_context` の反復を減らし、別ウィンドウへの誤入力や古い座標クリックを防ぎます。
 - **日本語/CJK 完全対応** — ウィンドウタイトル取得に Win32 `GetWindowTextW` を使用。nut-js の文字化けを回避。IME バイパス入力にも対応。
@@ -49,7 +51,9 @@ Claude がデスクトップを直接見て、直接操作する。
 npx -y @harusame64/desktop-touch-mcp
 ```
 
-npm ランチャーは初回起動時に GitHub Releases から最新の `desktop-touch-mcp-windows.zip` をダウンロードし、`%USERPROFILE%\.desktop-touch-mcp` に展開してキャッシュします。以後は新しい GitHub Release が出ていない限り、キャッシュ済みの実体を再利用します。
+npm ランチャーは npm package version に厳密に対応する runtime だけを取得します。`X.Y.Z` を実行した場合は GitHub Release `vX.Y.Z` のみを参照し、`desktop-touch-mcp-windows.zip` をダウンロードして SHA256 を検証できた場合にだけ `%USERPROFILE%\.desktop-touch-mcp` へ展開します。検証済みキャッシュは次回以降も再利用されます。
+
+キャッシュの保存先は `DESKTOP_TOUCH_MCP_HOME` で変更できます。
 
 ### Claude CLI への登録
 
@@ -90,7 +94,11 @@ cd desktop-touch-mcp
 npm install
 ```
 
-`npm install` 実行時に `prepare` スクリプトが TypeScript を `dist/` にコンパイルします。別途 `npm run build` は不要です。
+`npm install` 後にビルドを実行してください。
+
+```bash
+npm run build
+```
 
 ローカルチェックアウトを使う場合は、ビルド済みのサーバーを直接登録します。
 

package/README.md

@@ -1,6 +1,6 @@
 # desktop-touch-mcp
 
-[![desktop-touch-mcp MCP server](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp/badges/score.svg)](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
+[![desktop-touch-mcp MCP server](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp/badges/card.svg)](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
 
 [日本語](README.ja.md)
 
@@ -9,12 +9,14 @@
 An MCP server that gives Claude eyes and hands on Windows — 57 tools covering screenshots, mouse, keyboard, Windows UI Automation, Chrome DevTools Protocol, clipboard, desktop notifications, SmartScroll, and a Reactive Perception Graph for safe multi-step automation, designed from the ground up for LLM efficiency.
 
 > *v0.15: **82× average speedup** via Rust native engine — UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15× native speed. Zero-config: the engine auto-loads when present, with transparent PowerShell fallback.*
+> *v0.15.5: **Pinned release verification** — the npm launcher now fetches only the matching GitHub Release tag and verifies the Windows runtime zip before extraction.*
 
 ---
 
 ## Features
 
 - **⚡ High-performance Rust Native Core** — The UIA bridge and image-diff engine are written in Rust (`napi-rs` + `windows-rs`) and loaded as a native `.node` addon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning — `getFocusedElement` completes in **2 ms** (160× faster), and `getUiElements` returns full trees in **~100 ms** with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use **SSE2 SIMD** for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.
+- **🎯 Set-of-Marks (SoM) visual fallback** — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind. `screenshot(detail="text")` automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges (`[1]`, `[2]`…). Two parallel representations returned: a visual PNG for spatial orientation and a semantic `elements[]` list with `clickAt` coords — no CDP required.
 - **LLM-native design** — Built around how LLMs think, not how humans click. `run_macro` batches multiple operations into a single API call; `diffMode` sends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.
 - **Reactive Perception Graph** — Register a `lensId` for a window or browser tab, pass it to action tools, and get guard-checked `post.perception` feedback after each action. It reduces repeated `screenshot` / `get_context` calls and prevents wrong-window typing or stale-coordinate clicks.
 - **Full CJK support** — Uses Win32 `GetWindowTextW` for window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.
@@ -48,7 +50,9 @@ An MCP server that gives Claude eyes and hands on Windows — 57 tools covering
 npx -y @harusame64/desktop-touch-mcp
 ```
 
-The npm launcher downloads the latest `desktop-touch-mcp-windows.zip` from GitHub Releases on first run and caches it under `%USERPROFILE%\.desktop-touch-mcp`. Later runs reuse the cached release unless a newer GitHub Release is available.
+The npm launcher resolves runtime strictly by npm package version. For package `X.Y.Z`, it fetches only GitHub Release tag `vX.Y.Z`, downloads `desktop-touch-mcp-windows.zip`, verifies its SHA256 digest, and only then expands it under `%USERPROFILE%\.desktop-touch-mcp`. Verified cached releases are reused on later runs.
+
+Set `DESKTOP_TOUCH_MCP_HOME` to override the cache root directory.
 
 ### Register with Claude CLI
 
@@ -90,7 +94,11 @@ cd desktop-touch-mcp
 npm install
 ```
 
-`npm install` runs the `prepare` script, which compiles TypeScript to `dist/`. No separate build step is required.
+Build after install:
+
+```bash
+npm run build
+```
 
 For a local checkout, register the built server directly:
 
@@ -118,7 +126,7 @@ For a local checkout, register the built server directly:
 ### Screenshot (5)
 | Tool | Description |
 |---|---|
-| `screenshot` | Main capture. Supports `detail`, `dotByDot`, `dotByDotMaxDimension`, `grayscale`, `region` sub-crop, `diffMode` |
+| `screenshot` | Main capture. Supports `detail`, `dotByDot`, `dotByDotMaxDimension`, `grayscale`, `region` sub-crop, `diffMode`. `detail="text"` auto-activates the SoM pipeline when UIA is blind (games, RDP, custom Electron) |
 | `screenshot_background` | Capture a background window without focusing it (PrintWindow API) |
 | `screenshot_ocr` | Windows.Media.Ocr on a window; returns word-level text + screen clickAt coords |
 | `get_screen_info` | Monitor layout, DPI, cursor position |

package/bin/launcher.js

@@ -1,7 +1,7 @@
 #!/usr/bin/env node
 
 import { execFile, spawn } from "node:child_process";
-import { createWriteStream, existsSync } from "node:fs";
+import { createReadStream, createWriteStream, existsSync } from "node:fs";
 import {
   mkdir,
   mkdtemp,
@@ -11,13 +11,22 @@ import {
   rm,
   writeFile,
 } from "node:fs/promises";
+import { createHash } from "node:crypto";
 import os from "node:os";
 import path from "node:path";
 import { Readable } from "node:stream";
 import { pipeline } from "node:stream/promises";
 
-const REPO_API_URL = "https://api.github.com/repos/Harusame64/desktop-touch-mcp/releases/latest";
+const PACKAGE_VERSION = "0.15.5";
+const RELEASE_TAG = `v${PACKAGE_VERSION}`;
+const REPO_API_URL = `https://api.github.com/repos/Harusame64/desktop-touch-mcp/releases/tags/${RELEASE_TAG}`;
 const ASSET_NAME = "desktop-touch-mcp-windows.zip";
+const RELEASE_METADATA_FILE = ".desktop-touch-release.json";
+const RELEASE_MANIFEST = {
+  tagName: "v0.15.5",
+  assetName: ASSET_NAME,
+  sha256: "0150c5b1b2bfda9b39f2401bab49bf9fd02e5db1f3915300490ea7a93603f402",
+};
 const CACHE_ROOT = process.env.DESKTOP_TOUCH_MCP_HOME
   ? path.resolve(process.env.DESKTOP_TOUCH_MCP_HOME)
   : path.join(os.homedir(), ".desktop-touch-mcp");
@@ -42,33 +51,83 @@ function releaseDirForTag(tagName) {
   return path.join(RELEASES_DIR, tagToDirName(tagName));
 }
 
-async function isInstalled(releaseDir) {
-  return existsSync(path.join(releaseDir, "dist", "index.js"));
+function releaseMetadataPath(releaseDir) {
+  return path.join(releaseDir, RELEASE_METADATA_FILE);
 }
 
-async function readCurrentRelease() {
+function expectedReleaseSpec() {
+  if (RELEASE_MANIFEST.tagName !== RELEASE_TAG) {
+    throw new Error(
+      `Release manifest mismatch: PACKAGE_VERSION=${PACKAGE_VERSION}, manifest=${RELEASE_MANIFEST.tagName}`
+    );
+  }
+  if (!RELEASE_MANIFEST.sha256 || RELEASE_MANIFEST.assetName !== ASSET_NAME) {
+    throw new Error(`Missing release manifest for ${RELEASE_TAG}`);
+  }
+  if (!/^[a-f0-9]{64}$/i.test(RELEASE_MANIFEST.sha256)) {
+    throw new Error(`Invalid release SHA256 manifest for ${RELEASE_TAG}`);
+  }
+  return {
+    tagName: RELEASE_MANIFEST.tagName,
+    assetName: RELEASE_MANIFEST.assetName,
+    sha256: String(RELEASE_MANIFEST.sha256).toLowerCase(),
+  };
+}
+
+async function readReleaseMetadata(releaseDir) {
+  try {
+    const raw = await readFile(releaseMetadataPath(releaseDir), "utf8");
+    return JSON.parse(raw);
+  } catch {
+    return null;
+  }
+}
+
+async function isInstalled(releaseDir, expected) {
+  if (!existsSync(path.join(releaseDir, "dist", "index.js"))) return false;
+  const metadata = await readReleaseMetadata(releaseDir);
+  if (!metadata) return false;
+  return (
+    metadata.tagName === expected.tagName &&
+    metadata.assetName === expected.assetName &&
+    String(metadata.sha256 || "").toLowerCase() === expected.sha256
+  );
+}
+
+async function readCurrentRelease(expected) {
   try {
     const raw = await readFile(CURRENT_FILE, "utf8");
     const parsed = JSON.parse(raw);
     if (typeof parsed?.tagName !== "string") return null;
+    if (parsed.tagName !== expected.tagName) return null;
+    if (parsed.assetName !== expected.assetName) return null;
+    if (String(parsed.sha256 || "").toLowerCase() !== expected.sha256) return null;
     const releaseDir = releaseDirForTag(parsed.tagName);
-    if (!(await isInstalled(releaseDir))) return null;
+    if (!(await isInstalled(releaseDir, expected))) return null;
     return { tagName: parsed.tagName, releaseDir };
   } catch {
     return null;
   }
 }
 
-async function writeCurrentRelease(tagName) {
+async function writeReleaseMetadata(releaseDir, expected) {
+  await writeFile(
+    releaseMetadataPath(releaseDir),
+    `${JSON.stringify({ ...expected, updatedAt: new Date().toISOString() }, null, 2)}\n`,
+    "utf8"
+  );
+}
+
+async function writeCurrentRelease(expected) {
   await mkdir(CACHE_ROOT, { recursive: true });
   await writeFile(
     CURRENT_FILE,
-    `${JSON.stringify({ tagName, updatedAt: new Date().toISOString() }, null, 2)}\n`,
+    `${JSON.stringify({ ...expected, updatedAt: new Date().toISOString() }, null, 2)}\n`,
     "utf8"
   );
 }
 
-async function fetchLatestRelease() {
+async function fetchReleaseByTag(expected) {
   const response = await fetch(REPO_API_URL, {
     headers: {
       "Accept": "application/vnd.github+json",
@@ -77,7 +136,7 @@ async function fetchLatestRelease() {
   });
 
   if (!response.ok) {
-    throw new Error(`GitHub Releases API returned ${response.status} ${response.statusText}`);
+    throw new Error(`GitHub Releases API returned ${response.status} ${response.statusText} for ${expected.tagName}`);
   }
 
   const release = await response.json();
@@ -86,13 +145,16 @@ async function fetchLatestRelease() {
     : undefined;
 
   if (!release.tag_name || !asset?.browser_download_url) {
-    throw new Error(`Latest release does not contain ${ASSET_NAME}`);
+    throw new Error(`Release ${expected.tagName} does not contain ${ASSET_NAME}`);
   }
 
   const tagName = String(release.tag_name);
   if (!/^v\d+\.\d+\.\d+$/.test(tagName)) {
     throw new Error(`Unexpected tag format: ${tagName}`);
   }
+  if (tagName !== expected.tagName) {
+    throw new Error(`Unexpected tag: expected ${expected.tagName}, got ${tagName}`);
+  }
 
   return {
     tagName,
@@ -100,6 +162,25 @@ async function fetchLatestRelease() {
   };
 }
 
+async function sha256File(filePath) {
+  const hash = createHash("sha256");
+  await new Promise((resolve, reject) => {
+    const stream = createReadStream(filePath);
+    stream.on("error", reject);
+    stream.on("data", (chunk) => hash.update(chunk));
+    stream.on("end", resolve);
+  });
+  return hash.digest("hex").toLowerCase();
+}
+
+async function verifySha256(filePath, expectedSha256) {
+  const actual = await sha256File(filePath);
+  const expected = String(expectedSha256).toLowerCase();
+  if (actual !== expected) {
+    throw new Error(`SHA256 mismatch for ${ASSET_NAME}: expected ${expected}, got ${actual}`);
+  }
+}
+
 async function downloadFile(url, destination) {
   const response = await fetch(url, {
     headers: {
@@ -144,19 +225,19 @@ async function expandZip(zipPath, destination) {
 }
 
 async function findExtractedRoot(extractDir) {
-  if (await isInstalled(extractDir)) return extractDir;
+  if (existsSync(path.join(extractDir, "dist", "index.js"))) return extractDir;
 
   const entries = await readdir(extractDir, { withFileTypes: true });
   for (const entry of entries) {
     if (!entry.isDirectory()) continue;
     const candidate = path.join(extractDir, entry.name);
-    if (await isInstalled(candidate)) return candidate;
+    if (existsSync(path.join(candidate, "dist", "index.js"))) return candidate;
   }
 
   throw new Error("Release zip did not contain dist/index.js");
 }
 
-async function installRelease(release) {
+async function installRelease(release, expected) {
   await mkdir(RELEASES_DIR, { recursive: true });
 
   const targetDir = releaseDirForTag(release.tagName);
@@ -167,13 +248,15 @@ async function installRelease(release) {
   try {
     log(`Downloading ${ASSET_NAME} from ${release.tagName}`);
     await downloadFile(release.assetUrl, zipPath);
+    await verifySha256(zipPath, expected.sha256);
     await mkdir(extractDir, { recursive: true });
     await expandZip(zipPath, extractDir);
 
     const extractedRoot = await findExtractedRoot(extractDir);
     await rm(targetDir, { recursive: true, force: true });
     await rename(extractedRoot, targetDir);
-    await writeCurrentRelease(release.tagName);
+    await writeReleaseMetadata(targetDir, expected);
+    await writeCurrentRelease(expected);
     log(`Installed ${release.tagName} to ${targetDir}`);
     return targetDir;
   } finally {
@@ -182,26 +265,21 @@ async function installRelease(release) {
 }
 
 async function ensureRelease() {
-  const current = await readCurrentRelease();
-
-  let latest;
-  try {
-    latest = await fetchLatestRelease();
-  } catch (error) {
-    if (current) {
-      log(`Could not check GitHub Releases; using cached ${current.tagName}`);
-      return current.releaseDir;
-    }
-    throw error;
+  const expected = expectedReleaseSpec();
+  const targetDir = releaseDirForTag(expected.tagName);
+  if (await isInstalled(targetDir, expected)) {
+    await writeCurrentRelease(expected);
+    return targetDir;
   }
 
-  const targetDir = releaseDirForTag(latest.tagName);
-  if (await isInstalled(targetDir)) {
-    await writeCurrentRelease(latest.tagName);
-    return targetDir;
+  const current = await readCurrentRelease(expected);
+  if (current) {
+    return current.releaseDir;
   }
 
-  return installRelease(latest);
+  const release = await fetchReleaseByTag(expected);
+
+  return installRelease(release, expected);
 }
 
 function launchServer(releaseDir) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@harusame64/desktop-touch-mcp",
-  "version": "0.15.3",
+  "version": "0.15.5",
   "mcpName": "io.github.Harusame64/desktop-touch-mcp",
   "description": "LLM-native Windows computer-use MCP server with 56 tools for screenshots, UIA, mouse/keyboard, Chrome CDP, terminal, SmartScroll, and perception guards",
   "engines": {
@@ -41,14 +41,20 @@
   "scripts": {
     "build": "tsc",
     "sync-version": "node scripts/sync-version.mjs",
-    "version": "npm run sync-version && git add src/version.ts",
+    "check:launcher-manifest": "node scripts/check-launcher-manifest.mjs",
+    "version": "npm run sync-version && git add src/version.ts bin/launcher.js",
     "prepare": "tsc",
     "start": "node dist/index.js",
     "dev": "tsc --watch",
     "test": "vitest run",
     "test:capture": "node scripts/test-capture.mjs",
-    "test:e2e": "vitest run tests/e2e",
-    "test:headed": "HEADED=1 vitest run tests/e2e",
+    "test:unit": "vitest run --project=unit",
+    "test:e2e": "vitest run --project=e2e",
+    "test:headed": "HEADED=1 vitest run --project=e2e",
+    "test:e2e:browser": "vitest run --project=e2e \"tests/e2e/browser-*.test.ts\"",
+    "test:e2e:window": "vitest run --project=e2e tests/e2e/dock-window.test.ts tests/e2e/dock-auto.test.ts tests/e2e/focus-integrity.test.ts tests/e2e/force-focus.test.ts tests/e2e/screenshot-electron.test.ts tests/e2e/ui-elements-cache.test.ts",
+    "test:e2e:input": "vitest run --project=e2e tests/e2e/keyboard-focus-lost.test.ts tests/e2e/mouse-focus-lost.test.ts tests/e2e/terminal.test.ts",
+    "test:e2e:perception": "vitest run --project=e2e tests/e2e/perception-mvp.test.ts tests/e2e/rich-narration-edge.test.ts",
     "test:watch": "vitest",
     "generate:stub-catalog": "node scripts/generate-stub-tool-catalog.mjs",
     "check:stub-catalog": "node scripts/generate-stub-tool-catalog.mjs && git diff --exit-code src/stub-tool-catalog.ts",