@clowder-ai/companion 0.1.0-alpha.2 → 0.1.0-alpha.20

package/NOTICE

@@ -1,4 +1,12 @@
-Companion imagery and theme tokens originate in Clowder AI (MIT).
-See source-lock.json for the exact public source revision and content hashes.
+The original companion atlases and theme tokens originate in Clowder AI (MIT).
+See source-lock.json for their exact public source revision and content hashes.
+The legacy ragdoll-v1 and yarn-ball stills are AI-assisted derivatives of the
+listed public reference, restoring the complete cat silhouette and transparent
+background. Their reference and resulting hashes are recorded separately.
+The living Xianxian layers and clips are F317 assets: CVO-generated source video,
+keyframes by codex-astra, and deterministic extraction/alignment by opus55.
+source-lock.json pins the 20 bundled distributable files, their geometry file,
+and the source package manifest digest. Source video and private workspace paths
+are not bundled.
 The conversation transport and controls continue the F317 companion implementation
 by the Clowder AI contributors. No private runtime data or pet metadata is included.

package/README.md

@@ -0,0 +1,59 @@
+# Companion surface
+
+At rest the desktop shows only your selected cat. Click the cat for voice or text;
+right-click for history, screen sharing, household access, sound and hiding.
+Dragging moves the cat and folds its controls. Escape folds a panel without
+ending the conversation or clearing its draft.
+
+Voice requires an explicit **语音聊** click. An active microphone badge remains
+visible and ends the call in one click. Screen sharing has its own explicit
+picker and termination badge. There is no microphone action on a double-click,
+page load, body tap or drag.
+
+When a compatible Host advertises receive-only audio, **只听** opens the same
+conversation with playback but never requests or toggles a microphone. Older
+Hosts omit that capability, so the control stays hidden and ordinary voice keeps
+the exact legacy connect request. Text remains available independently.
+
+Typing works with voice off. The Host uses the existing owner conversation,
+configured duty cat, ordinary message routing and idempotency. The small history
+view reads a bounded recent subset; **完整聊天** opens the canonical conversation.
+No second transcript database, credentials or selectable Host identity live here.
+When the Host supplies a valid per-message identity snapshot, each history row
+keeps its real author while separately showing the partner, Live carrier and
+deep cat saved with that message. Legacy rows omit this context instead of
+borrowing today's selected companion.
+
+## Host integration
+
+This surface requires the additive desktop bridge **1.3.0**. The Host computes
+placement within the display work area, keeps the cat as its stable anchor, and
+owns bounded drag gestures and transparent-area hit testing. Older 1.0-only
+Hosts reject this manifest before installation. Media admission and capture
+remain in the trusted Host; this package receives neither streams nor provider
+SDP. Hiding releases media; the existing Clowder **聊聊** entry restores the cat.
+
+Xianxian uses one layered sitting body or one VP9 clip at a time. Expiring Host
+facts with task, result and event identities select tool work, workspace fetch,
+screen reading and applied-result delivery. Native travel is independent: the
+renderer observes the actual desktop window coordinates, so a running clip can
+appear only after the Host window really moves. Dragging suppresses that
+observer and freezes a visibly held candidate instead of renaming the running
+clip. Pounce requires an explicit play gesture by default; optional automatic
+play is bounded by idle and cooldown timers. Reduced motion keeps semantic
+poses static; a compatible Host also revokes its native movement lease.
+Other cats retain their own skins. The v3 living assets and their hashes are
+pinned in `source-lock.json`; the installable renderer carries all 21 assets
+itself, including the mid-edge `peek_fade` variant.
+
+The Host keeps a small transparent margin left of the pet hit target so the
+pounce's tail is visible without turning empty space into a click target.
+
+## Verification
+
+Run the repository's contract and SDK builds, then `pnpm --filter
+@clowder-ai/companion test` and `pnpm --filter @clowder-ai/companion lint`.
+`node packages/companion/scripts/preview.mjs` serves the accepted interaction
+spike at `/spike/` and the actual built renderer with an explicitly simulated
+bridge at `/previous`. The latter covers visual states and text controls; it
+does not prove native movement, real capture or model delivery.

package/manifest.json

@@ -1,6 +1,6 @@
 {
   "pluginId": "official.companion",
-  "version": "0.1.0-alpha.2",
+  "version": "0.1.0-alpha.20",
   "contractVersion": "0.1.0",
   "name": "猫猫球",
   "icon": {
@@ -37,14 +37,14 @@
       "type": "desktop-window",
       "id": "companion",
       "role": "companion",
-      "bridgeVersion": "1.0.0",
+      "bridgeVersion": "1.3.0",
       "surface": {
         "entrypoint": "renderer/index.html",
-        "integrity": "sha256-yVHMFz5zEuDTwQO/pgNRScotBqRzGd9o9wIIhadJDNQ="
+        "integrity": "sha256-jaXXDHzwGVDNnFoOhrj5nvqAXMlf02Jw8CeC7Xay538="
       },
       "presentation": {
-        "width": 330,
-        "height": 350,
+        "width": 300,
+        "height": 270,
         "frame": false,
         "transparent": true,
         "alwaysOnTop": true,

package/package.json

@@ -1,22 +1,35 @@
 {
   "name": "@clowder-ai/companion",
-  "version": "0.1.0-alpha.2",
+  "version": "0.1.0-alpha.20",
   "description": "The desktop voice body for your Clowder companion",
   "license": "MIT",
   "type": "module",
-  "files": ["renderer", "plugin.yaml", "manifest.json", "LICENSE", "NOTICE", "source-lock.json"],
-  "exports": { "./manifest": "./manifest.json" },
-  "engines": { "node": ">=22.12.0" },
+  "files": [
+    "renderer",
+    "plugin.yaml",
+    "manifest.json",
+    "LICENSE",
+    "NOTICE",
+    "source-lock.json"
+  ],
+  "exports": {
+    "./manifest": "./manifest.json"
+  },
+  "engines": {
+    "node": ">=22.12.0"
+  },
   "scripts": {
     "build": "node scripts/build.mjs",
     "test": "node --test test/*.test.mjs",
+    "test:browser": "pnpm build && node --test test/panel-layout.browser.mjs",
     "typecheck": "node scripts/check.mjs",
     "lint": "node scripts/check.mjs",
     "prepack": "pnpm build"
   },
   "devDependencies": {
-    "@clowder-ai/plugin-contract": "0.1.0-beta.16",
-    "@clowder-ai/plugin-sdk": "0.1.0-beta.11"
+    "@clowder-ai/plugin-contract": "0.1.0-beta.24",
+    "@clowder-ai/plugin-sdk": "0.1.0-beta.20",
+    "playwright": "1.61.1"
   },
   "repository": {
     "type": "git",

package/plugin.yaml

@@ -1,6 +1,6 @@
 {
   "pluginId": "official.companion",
-  "version": "0.1.0-alpha.2",
+  "version": "0.1.0-alpha.20",
   "contractVersion": "0.1.0",
   "name": "猫猫球",
   "icon": {
@@ -37,14 +37,14 @@
       "type": "desktop-window",
       "id": "companion",
       "role": "companion",
-      "bridgeVersion": "1.0.0",
+      "bridgeVersion": "1.3.0",
       "surface": {
         "entrypoint": "renderer/index.html",
-        "integrity": "sha256-yVHMFz5zEuDTwQO/pgNRScotBqRzGd9o9wIIhadJDNQ="
+        "integrity": "sha256-jaXXDHzwGVDNnFoOhrj5nvqAXMlf02Jw8CeC7Xay538="
       },
       "presentation": {
-        "width": 330,
-        "height": 350,
+        "width": 300,
+        "height": 270,
         "frame": false,
         "transparent": true,
         "alwaysOnTop": true,

package/renderer/call-transcript-view.mjs

@@ -0,0 +1,238 @@
+const MAX_LIVE_ROWS = 8;
+
+/**
+ * Projection of the Host-bound transcript for one trusted call.
+ *
+ * Durable rows are keyed only by Host messageId. Ephemeral audio is keyed by
+ * provider item/turn identity and is discarded when the Host projects the
+ * matching durable source. Text equality is deliberately never used.
+ */
+export class CallTranscriptView {
+  constructor(container, { onUnread = () => {} } = {}) {
+    this.container = container;
+    this.onUnread = onUnread;
+    this.durableRows = new Map();
+    this.liveRows = new Map();
+    this.liveSequence = 0;
+    this.optimisticIds = new Set();
+    this.unread = 0;
+    this.initialized = false;
+    container.addEventListener?.('scroll', () => {
+      if (this.nearBottom()) this.setUnread(0);
+    });
+  }
+
+  setCall(callId) {
+    if (this.callId === callId) return;
+    this.callId = callId ?? null;
+    this.scope = undefined;
+    this.durableRows.clear();
+    this.liveRows.clear();
+    this.liveSequence = 0;
+    this.optimisticIds.clear();
+    this.initialized = false;
+    this.container.replaceChildren();
+    this.container.dataset.callId = this.callId ?? '';
+    this.container.dataset.hasMore = 'false';
+    this.setUnread(0);
+  }
+
+  reset() { this.setCall(null); }
+
+  nearBottom() {
+    return this.container.scrollHeight - this.container.scrollTop - this.container.clientHeight < 32;
+  }
+
+  scrollToBottom() {
+    this.container.scrollTop = this.container.scrollHeight;
+    this.setUnread(0);
+  }
+
+  setUnread(count) {
+    if (this.unread === count) return;
+    this.unread = count;
+    this.onUnread(count);
+  }
+
+  visibleAnchor() {
+    const viewport = this.container.getBoundingClientRect?.();
+    if (!viewport) return null;
+    for (const [id, row] of this.durableRows) {
+      const rect = row.getBoundingClientRect?.();
+      if (rect && rect.bottom > viewport.top && rect.top < viewport.bottom) {
+        return { id, offset: rect.top - viewport.top };
+      }
+    }
+    return null;
+  }
+
+  createRow() {
+    return this.container.ownerDocument.createElement('p');
+  }
+
+  updateDurable(row, value) {
+    row.className = `call-transcript-row ${value.source.kind} ${value.role}`;
+    row.textContent = value.text;
+    row.dataset.messageId = value.messageId;
+    row.dataset.source = value.source.kind;
+    row.dataset.role = value.role;
+    delete row.dataset.live;
+    delete row.dataset.itemId;
+    delete row.dataset.turnId;
+    delete row.dataset.optimistic;
+    if (value.source.kind === 'voice') {
+      row.dataset.itemId = value.source.nativeItemId;
+      if (value.source.nativeTurnId) row.dataset.turnId = value.source.nativeTurnId;
+      delete row.dataset.clientMessageId;
+    } else {
+      row.dataset.clientMessageId = value.source.clientMessageId;
+    }
+    row.setAttribute?.('aria-label', value.source.kind === 'typed'
+      ? `你打字说：${value.text}`
+      : value.role === 'assistant' ? `猫猫：${value.text}` : `语音转写：${value.text}`);
+    return row;
+  }
+
+  removeReconciledLive(source) {
+    if (source.kind !== 'voice') return false;
+    let removed = false;
+    for (const [key, row] of this.liveRows) {
+      if ((row.dataset.itemId && row.dataset.itemId === source.nativeItemId)
+        || (source.nativeTurnId && row.dataset.turnId === source.nativeTurnId)) {
+        this.liveRows.delete(key);
+        removed = true;
+      }
+    }
+    return removed;
+  }
+
+  render() {
+    this.container.replaceChildren(...this.durableRows.values(), ...this.liveRows.values());
+  }
+
+  load(reply) {
+    if (!this.callId || reply?.kind !== 'transcript' || reply.scope?.callId !== this.callId) return false;
+    const pinned = this.nearBottom();
+    const readingPosition = this.container.scrollTop;
+    const anchor = pinned ? null : this.visibleAnchor();
+    const previous = this.durableRows;
+    const previousIds = new Set(previous.keys());
+    const next = new Map();
+    let added = 0;
+
+    for (const value of reply.rows) {
+      const reconciledLive = this.removeReconciledLive(value.source);
+      const row = previous.get(value.messageId) ?? this.createRow();
+      this.updateDurable(row, value);
+      this.optimisticIds.delete(value.messageId);
+      next.set(value.messageId, row);
+      if (this.initialized && !previousIds.has(value.messageId) && !reconciledLive) added += 1;
+    }
+    // An accepted delivery already has a durable Host messageId. A transcript
+    // read can race the MessageStore projection, so keep that exact row until
+    // the same id appears; never fall back to matching its text.
+    for (const id of this.optimisticIds) {
+      const row = previous.get(id);
+      if (row && !next.has(id)) next.set(id, row);
+    }
+
+    this.scope = reply.scope;
+    this.durableRows = next;
+    this.container.dataset.callId = reply.scope.callId;
+    this.container.dataset.hasMore = String(reply.hasMore);
+    this.render();
+
+    if (pinned) {
+      this.container.scrollTop = this.container.scrollHeight;
+      this.setUnread(0);
+    } else if (anchor) {
+      const row = this.durableRows.get(anchor.id) ?? this.durableRows.values().next().value;
+      const rect = row?.getBoundingClientRect?.();
+      const viewport = this.container.getBoundingClientRect?.();
+      this.container.scrollTop = rect && viewport
+        ? this.container.scrollTop + rect.top - viewport.top - anchor.offset
+        : readingPosition;
+      if (added) this.setUnread(this.unread + added);
+    } else {
+      this.container.scrollTop = readingPosition;
+      if (added) this.setUnread(this.unread + added);
+    }
+    this.initialized = true;
+    return true;
+  }
+
+  liveKey(event) {
+    if (event.itemId) return `item:${event.itemId}`;
+    if (event.turnId) return `turn:${event.turnId}`;
+    return `ephemeral:${++this.liveSequence}`;
+  }
+
+  appendLive(event) {
+    if (!this.callId || event?.type !== 'transcript' || event.callId !== this.callId) return false;
+    const pinned = this.nearBottom();
+    const key = this.liveKey(event);
+    let row = this.liveRows.get(key);
+    const isNew = !row;
+    if (!row) {
+      row = this.createRow();
+      row.className = `call-transcript-row voice ${event.role}`;
+      row.dataset.live = 'true';
+      row.dataset.source = 'voice';
+      row.dataset.role = event.role;
+      if (event.itemId) row.dataset.itemId = event.itemId;
+      if (event.turnId) row.dataset.turnId = event.turnId;
+      this.liveRows.set(key, row);
+      while (this.liveRows.size > MAX_LIVE_ROWS) this.liveRows.delete(this.liveRows.keys().next().value);
+    }
+    row.textContent = `${row.textContent}${event.text}`.slice(-16000);
+    row.setAttribute?.('aria-label', `${event.role === 'assistant' ? '猫猫' : '语音转写'}，实时：${row.textContent}`);
+    this.render();
+    if (pinned) {
+      this.container.scrollTop = this.container.scrollHeight;
+      this.setUnread(0);
+    } else if (isNew) this.setUnread(this.unread + 1);
+    return true;
+  }
+
+  appendTyped({ receipt, text }) {
+    if (!this.callId || receipt?.delivery !== 'accepted' || receipt.callId !== this.callId) return false;
+    const pinned = this.nearBottom();
+    let row = this.durableRows.get(receipt.messageId);
+    const isNew = !row;
+    if (!row) {
+      row = this.createRow();
+      this.durableRows.set(receipt.messageId, row);
+    }
+    this.updateDurable(row, {
+      messageId: receipt.messageId,
+      role: 'user',
+      text,
+      source: { kind: 'typed', clientMessageId: receipt.clientMessageId, callId: receipt.callId },
+    });
+    row.dataset.optimistic = 'true';
+    this.optimisticIds.add(receipt.messageId);
+    while (this.optimisticIds.size > MAX_LIVE_ROWS) {
+      const oldest = this.optimisticIds.values().next().value;
+      this.optimisticIds.delete(oldest);
+      if (this.durableRows.get(oldest)?.dataset.optimistic === 'true') this.durableRows.delete(oldest);
+    }
+    this.render();
+    if (pinned) {
+      this.container.scrollTop = this.container.scrollHeight;
+      this.setUnread(0);
+    } else if (isNew) this.setUnread(this.unread + 1);
+    return true;
+  }
+
+  finish(event) {
+    if (!this.callId || event?.type !== 'turn-done' || event.callId !== this.callId) return false;
+    for (const row of this.liveRows.values()) {
+      const matchesTurn = event.turnId && row.dataset.turnId === event.turnId;
+      const matchesUnkeyedRole = !event.turnId && event.role && !row.dataset.itemId
+        && !row.dataset.turnId && row.dataset.role === event.role;
+      if (matchesTurn || matchesUnkeyedRole) row.dataset.complete = 'true';
+    }
+    this.render();
+    return true;
+  }
+}

package/renderer/client.mjs

@@ -19,7 +19,7 @@ export function createCompanionClient(bridge) {
     return {
         state: () => invoke({ kind: 'state' }, 'state'),
         prepare: () => invoke({ kind: 'prepare' }, 'state'),
-        connectAudio: () => invoke({ kind: 'audio.connect' }, 'ok'),
+        connectAudio: (mode) => invoke(mode === undefined ? { kind: 'audio.connect' } : { kind: 'audio.connect', mode }, 'ok'),
         closeAudio: () => invoke({ kind: 'audio.close' }, 'ok'),
         muteMicrophone: (muted) => invoke({ kind: 'audio.microphone', muted }, 'ok'),
         muteSpeaker: (muted) => invoke({ kind: 'audio.speaker', muted }, 'ok'),
@@ -36,7 +36,19 @@ export function createCompanionClient(bridge) {
         screenFrame: (selectionId, frame) => invoke({ kind: 'screen.frame', selectionId, frame }, 'ok'),
         screenClose: () => invoke({ kind: 'screen.close' }, 'ok'),
         openConversation: () => invoke({ kind: 'conversation.open' }, 'navigation'),
+        readDecisions: (offset = 0, limit = 20) => invoke({ kind: 'decisions.read', offset, limit }, 'decisions'),
+        openDecision: (variantRef, target) => invoke({ kind: 'decision.open', variantRef, target }, 'navigation'),
+        inspectF221: (proposalId) => invoke({ kind: 'f221.inspect', proposalId }, 'decision-trial'),
         resize: (expanded) => invoke({ kind: 'view.resize', expanded }, 'ok'),
+        layout: (panel, width, height) => invoke({ kind: 'view.layout', panel, width, height }, 'layout'),
+        drag: (phase) => invoke({ kind: 'view.drag', phase }, 'ok'),
+        hide: () => invoke({ kind: 'view.hide' }, 'ok'),
+        resetPosition: () => invoke({ kind: 'view.reset' }, 'ok'),
+        readSettings: () => invoke({ kind: 'settings.read' }, 'settings'),
+        updateSetting: (field, value) => invoke({ kind: 'settings.update', field, value }, 'settings-update'),
+        disableCompanion: () => invoke({ kind: 'companion.disable' }, 'companion-lifecycle'),
+        readConversation: () => invoke({ kind: 'conversation.read' }, 'conversation'),
+        readTranscript: () => invoke({ kind: 'transcript.read' }, 'transcript'),
         subscribe: (listener) => bridge.subscribe(listener),
     };
 }

package/renderer/companion-identity.mjs

@@ -0,0 +1,42 @@
+const named = value => typeof value === 'string' && value.trim().length > 0;
+const actor = value => value && named(value.catId) && named(value.displayName);
+const model = value => value === null || (named(value) && value.length <= 160);
+const snapshotActor = value => actor(value) && value.catId.length <= 160 && value.displayName.length <= 160;
+
+function validSnapshot(value) {
+  return value?.v === 1 && value.name === '猫猫球'
+    && snapshotActor(value.partner) && named(value.partner.skin) && value.partner.skin.length <= 160
+    && snapshotActor(value.live) && value.live.transport === 'gpt_live_v3' && model(value.live.verifiedModel)
+    && snapshotActor(value.deep) && value.deep.catId === value.partner.catId && model(value.deep.verifiedModel);
+}
+
+function display(snapshot, partnerLabel) {
+  return {
+    title: snapshot.name,
+    companionKnown: true,
+    partnerLabel,
+    avatarCatId: snapshot.partner.catId,
+    skin: snapshot.partner.skin,
+    liveLabel: `Live 快端：${snapshot.live.displayName} · ${snapshot.live.verifiedModel ?? '型号未核实'}`,
+    deepLabel: `深思端：${snapshot.deep.displayName} · ${snapshot.deep.verifiedModel ?? '型号未核实'}`,
+  };
+}
+
+/** The current face comes from the Host's selected duty, never the background carrier. */
+export function currentCompanionIdentity(state) {
+  if (!actor(state?.duty) || !actor(state?.carrier) || !named(state.skin)
+    || state.liveTransport?.kind !== 'gpt_live_v3' || !model(state.liveTransport.verifiedModel)) return null;
+  const snapshot = {
+    v: 1, name: '猫猫球',
+    partner: { ...state.duty, skin: state.skin },
+    live: { ...state.carrier, transport: state.liveTransport.kind, verifiedModel: state.liveTransport.verifiedModel },
+    deep: { ...state.duty, verifiedModel: null },
+  };
+  return display(snapshot, `${snapshot.partner.displayName}陪伴中`);
+}
+
+/** A missing old snapshot never borrows today's preference. Keep the message's real author visible. */
+export function historicalCompanionIdentity(snapshot, authorLabel) {
+  if (!validSnapshot(snapshot)) return { authorLabel, companionKnown: false };
+  return { ...display(snapshot, `当时由${snapshot.partner.displayName}陪伴`), authorLabel };
+}