@forwardimpact/outpost 3.0.3

package/templates/.claude/skills/sync-teams/SKILL.md

@@ -0,0 +1,177 @@
+---
+name: sync-teams
+description: Sync recent Microsoft Teams chat messages into ~/.cache/fit/outpost/teams_chat/ as markdown files by reading the Teams IndexedDB cache from disk. Use on a schedule or when the user asks to sync their Teams chats. Requires macOS with the Teams desktop app installed.
+compatibility: Requires macOS with Microsoft Teams desktop app (com.microsoft.teams2) installed
+---
+
+# Sync Teams
+
+Sync recent Microsoft Teams chat messages into
+`~/.cache/fit/outpost/teams_chat/` as markdown files. This is an automated data
+pipeline skill — it ingests chat data that other skills (like
+`extract-entities`) consume downstream.
+
+This skill reads the Teams IndexedDB cache directly from disk — no browser
+automation, no API tokens, no network access needed. The Teams desktop app
+(which uses Edge WebView2) stores conversations and messages in a LevelDB-backed
+IndexedDB at a known location. This skill parses those files, deserializes the
+V8-encoded records, and writes markdown.
+
+## Trigger
+
+Run this skill on a schedule (every 15 minutes) or when the user asks to sync
+their Teams chats.
+
+## Prerequisites
+
+- macOS with the Microsoft Teams desktop app installed (`com.microsoft.teams2`)
+- `snappyjs` npm package installed (`npm install snappyjs`)
+- Node.js 20+ (uses `node:v8` built-in for deserialization)
+
+## Inputs
+
+- `~/Library/Containers/com.microsoft.teams2/Data/Library/Application Support/Microsoft/MSTeams/EBWebView/WV2Profile_tfw/IndexedDB/https_teams.microsoft.com_0.indexeddb.leveldb/`
+  — Teams IndexedDB (LevelDB on disk)
+- `~/.cache/fit/outpost/state/teams_last_sync` — ISO timestamp of last sync
+- `~/.cache/fit/outpost/state/teams_chat_index.tsv` — index of known chats
+
+## Outputs
+
+- `~/.cache/fit/outpost/teams_chat/{slug}.md` — one markdown file per chat
+  (overwritten each sync with current state)
+- `~/.cache/fit/outpost/state/teams_last_sync` — updated with sync timestamp
+- `~/.cache/fit/outpost/state/teams_chat_index.tsv` — updated chat index
+
+---
+
+## Implementation
+
+Run the sync as a single Node.js script:
+
+    node scripts/sync.mjs [--days N]
+
+- `--days N` — only include messages from the last N days (default: 30)
+
+The script:
+
+1. Reads all LevelDB `.ldb` (SSTable) and `.log` (write-ahead log) files from
+   the Teams IndexedDB directory
+2. Decompresses Snappy-compressed blocks and deserializes V8-encoded values
+   using Node's built-in `v8.deserialize()`
+3. Extracts conversation records (with member lists, topics, chat type) and
+   message records (with sender names, HTML content, timestamps)
+4. Groups messages by conversation, filters by date window, and converts HTML
+   content to plain text
+5. Writes one markdown file per chat to `~/.cache/fit/outpost/teams_chat/`
+6. Updates sync state (timestamp and chat index)
+
+### Architecture
+
+Three modules, following the same pattern as `sync-apple-mail`:
+
+| Module                       | Purpose                                                                                                                     |
+| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
+| `scripts/leveldb-reader.mjs` | Parse LevelDB SSTable and WAL files. Handles Snappy decompression. No external dependencies except `snappyjs`.              |
+| `scripts/idb-reader.mjs`     | Chromium IndexedDB layer. Strips Blink envelope, calls `v8.deserialize()`, classifies records as conversations or messages. |
+| `scripts/sync.mjs`           | Main sync script. Reads data, groups by conversation, normalizes names, writes markdown and state.                          |
+
+### How It Works
+
+The Teams desktop app uses Edge WebView2 internally. WebView2 stores IndexedDB
+data in LevelDB (the same way Chrome does). The key databases are:
+
+- **conversation-manager** — stores conversation metadata: ID, type (Chat vs
+  Thread), members, topic, last message time
+- **replychain-manager** — stores actual messages: sender display name, HTML
+  content, timestamps, reactions, edit status
+
+Since LevelDB is an append-only format, files can be read while Teams is
+running. Newer `.ldb` files supersede older ones for the same records.
+
+### Name Resolution
+
+Teams conversation records don't store human-readable member names — only orgid
+identifiers. Display names are resolved from:
+
+1. **Conversation topic** (for named group chats)
+2. **Message sender names** (`imDisplayName` field) — for 1:1 chats, the chat is
+   named after the other participant(s)
+
+## Output Format
+
+Each `{slug}.md` file follows the same format as the previous browser-based
+implementation:
+
+**1:1 chat:**
+
+```markdown
+# Chat with {First Last}
+
+**Platform:** Microsoft Teams
+**Last Synced:** {YYYY-MM-DD}
+
+---
+
+### {Sender First Last}
+**Date:** {YYYY-MM-DD HH:MM:SS}
+
+{message content}
+
+---
+```
+
+**Group chat:**
+
+```markdown
+# Chat: {Topic} (group)
+
+**Platform:** Microsoft Teams
+**Type:** Group chat
+**Participants:** {Name1}, {Name2}, ...
+**Last Synced:** {YYYY-MM-DD}
+
+---
+
+### {Sender First Last}
+**Date:** {YYYY-MM-DD HH:MM:SS}
+
+{message content}
+```
+
+Key conventions:
+
+- Messages in **chronological order** (oldest first)
+- **Normalize names** from Teams format ("Last, First") to "First Last"
+- **Platform** line distinguishes Teams from email in downstream processing
+- **Plain text only** — HTML is stripped, mentions are preserved as plain text
+- Skip system messages (calls, member adds/removes, topic changes)
+
+## Error Handling
+
+- Teams app not installed → report and stop
+- IndexedDB directory missing → report and stop
+- LevelDB file parse error → skip that file, continue with others
+- V8 deserialization failure → skip that record, continue
+- Snappy decompression failure → skip that block, continue
+- Empty chat (no messages in window) → skip, don't write a file
+- Always update sync state, even on partial success
+
+## Constraints
+
+- **Read-only.** Never writes to the Teams IndexedDB or sends messages.
+- **Cache-dependent.** Only conversations cached locally by Teams are available.
+  This covers recently viewed chats, not full history.
+- **Both 1:1 and group chats** are synced (channels are excluded).
+- **No message limit per chat** — all cached messages within the `--days` window
+  are included.
+
+## Limitations
+
+- The IndexedDB is a **cache, not an archive**. Only conversations the user has
+  recently opened in Teams will have cached message data. Older conversations
+  that haven't been opened may have conversation metadata but no messages.
+- Clearing the Teams cache (a common troubleshooting step) will temporarily
+  remove all local data until Teams rebuilds it from the server.
+- Some V8-serialized records (~17% in testing) use formats that
+  `v8.deserialize()` cannot decode. These are silently skipped — they are
+  typically IndexedDB metadata, not conversation or message records.

package/templates/.claude/skills/sync-teams/scripts/idb-reader.mjs

@@ -0,0 +1,147 @@
+#!/usr/bin/env node
+/**
+ * Read Chromium IndexedDB records from a LevelDB-backed store.
+ *
+ * Chromium stores IndexedDB data in LevelDB with a specific key encoding
+ * (database ID, object store ID) and V8-serialized values wrapped in a Blink
+ * envelope. This module handles the key parsing and value deserialization.
+ *
+ * Exports: readIndexedDb(dir) → { conversations: Map, messages: Map }
+ */
+
+import v8 from "node:v8";
+import { readAllEntries } from "./leveldb-reader.mjs";
+
+// Chromium IndexedDB key prefix types (from indexed_db_leveldb_coding.h)
+// Key format: [database_id varint] [object_store_id varint] [key_type byte] [...]
+// We care about object store data records (key_type = 1) and database metadata
+
+function readIdbVarint(buf, offset) {
+  let result = 0;
+  let shift = 0;
+  let pos = offset;
+  while (pos < buf.length) {
+    const byte = buf[pos];
+    result |= (byte & 0x7f) << shift;
+    pos++;
+    if ((byte & 0x80) === 0) break;
+    shift += 7;
+    if (shift > 35) break;
+  }
+  return { value: result, bytesRead: pos - offset };
+}
+
+/**
+ * Try to deserialize from the second 0xFF marker within the first `limit` bytes.
+ * The Blink envelope has: [varint wire_size] [0xFF blink_ver] [envelope...] [0xFF v8_ver] [V8 data]
+ * We want the second 0xFF that starts valid V8 data.
+ */
+function deserializeFromSecondMarker(rawValue, limit) {
+  let ffCount = 0;
+  for (let i = 0; i < limit; i++) {
+    if (rawValue[i] !== 0xff) continue;
+    ffCount++;
+    if (ffCount >= 2) {
+      try {
+        return v8.deserialize(rawValue.subarray(i));
+      } catch {
+        // keep scanning
+      }
+    }
+  }
+  return null;
+}
+
+/**
+ * Fallback: try deserializing from every 0xFF position within `limit` bytes.
+ */
+function deserializeFromAnyMarker(rawValue, limit) {
+  for (let i = 0; i < limit; i++) {
+    if (rawValue[i] !== 0xff) continue;
+    try {
+      return v8.deserialize(rawValue.subarray(i));
+    } catch {
+      continue;
+    }
+  }
+  return null;
+}
+
+/**
+ * Try to deserialize a Chromium IndexedDB value.
+ * Values have a Blink envelope before the V8 payload.
+ * Scans for the V8 version tag (0xFF) and attempts deserialization.
+ */
+function tryDeserialize(rawValue) {
+  if (!rawValue || rawValue.length < 4) return null;
+
+  const headerLimit = Math.min(rawValue.length, 60);
+  const result = deserializeFromSecondMarker(rawValue, headerLimit);
+  if (result !== null) return result;
+
+  const fallbackLimit = Math.min(rawValue.length, 100);
+  return deserializeFromAnyMarker(rawValue, fallbackLimit);
+}
+
+/**
+ * Parse a Chromium IndexedDB key prefix to extract database and object store IDs.
+ * Returns null if the key doesn't look like an IndexedDB data record.
+ */
+function _parseKeyPrefix(key) {
+  if (key.length < 3) return null;
+  const db = readIdbVarint(key, 0);
+  if (db.bytesRead + 1 > key.length) return null;
+  const os = readIdbVarint(key, db.bytesRead);
+  return {
+    databaseId: db.value,
+    objectStoreId: os.value,
+    remaining: key.subarray(db.bytesRead + os.bytesRead),
+  };
+}
+
+/**
+ * Read all IndexedDB records from a Chromium LevelDB directory and return
+ * conversations and messages.
+ *
+ * @param {string} dir - path to the .indexeddb.leveldb directory
+ * @returns {{ conversations: object[], messages: object[] }}
+ */
+export function readIndexedDb(dir) {
+  // Use Maps so later entries (from newer .ldb files) overwrite older ones.
+  // LevelDB reads files in ascending order — newer compactions have higher
+  // numbers, so the last write for a given key is the most current.
+  const convMap = new Map();
+  const msgMap = new Map();
+
+  for (const entry of readAllEntries(dir)) {
+    const obj = tryDeserialize(entry.value);
+    if (!obj || typeof obj !== "object") continue;
+
+    if (isConversation(obj)) {
+      const id = obj.id;
+      if (id) convMap.set(id, obj);
+      continue;
+    }
+
+    if (obj.messageMap && obj.conversationId) {
+      const rcId = `${obj.conversationId}:${obj.replyChainId ?? ""}`;
+      msgMap.set(rcId, obj);
+    }
+  }
+
+  return {
+    conversations: [...convMap.values()],
+    messages: [...msgMap.values()],
+  };
+}
+
+function isConversation(obj) {
+  return (
+    obj.id &&
+    typeof obj.id === "string" &&
+    (obj.type === "Chat" || obj.type === "Thread") &&
+    (obj.members !== undefined ||
+      obj.threadProperties !== undefined ||
+      obj.lastMessageTimeUtc !== undefined)
+  );
+}

package/templates/.claude/skills/sync-teams/scripts/leveldb-reader.mjs

@@ -0,0 +1,288 @@
+#!/usr/bin/env node
+/**
+ * Read LevelDB SSTable (.ldb) and write-ahead log (.log) files.
+ *
+ * Parses the binary SSTable format directly — no LevelDB library needed.
+ * Handles Snappy-compressed blocks via snappyjs (pure JS, zero native deps).
+ *
+ * Exports a single generator function: readAllEntries(directory)
+ */
+
+import { readFileSync, readdirSync } from "node:fs";
+import { join } from "node:path";
+import SnappyJS from "snappyjs";
+
+const uncompress =
+  SnappyJS.uncompress ?? SnappyJS.default?.uncompress ?? SnappyJS;
+
+const LEVELDB_TABLE_MAGIC = 0xdb4775248b80fb57n;
+
+function readFixed32(buf, offset) {
+  return buf.readUInt32LE(offset);
+}
+
+function readFixed64(buf, offset) {
+  const lo = buf.readUInt32LE(offset);
+  const hi = buf.readUInt32LE(offset + 4);
+  return BigInt(hi) * 0x100000000n + BigInt(lo);
+}
+
+function readVarint(buf, offset) {
+  let result = 0;
+  let shift = 0;
+  let pos = offset;
+  while (pos < buf.length) {
+    const byte = buf[pos];
+    result |= (byte & 0x7f) << shift;
+    pos++;
+    if ((byte & 0x80) === 0) break;
+    shift += 7;
+    if (shift > 35) break;
+  }
+  return { value: result, bytesRead: pos - offset };
+}
+
+function decompressBlock(raw, compressionType) {
+  if (compressionType === 1) {
+    try {
+      return Buffer.from(uncompress(raw));
+    } catch {
+      return raw;
+    }
+  }
+  return raw;
+}
+
+function parseBlockEntries(blockData) {
+  const entries = [];
+  if (blockData.length < 4) return entries;
+
+  const numRestarts = readFixed32(blockData, blockData.length - 4);
+  const restartsOffset = blockData.length - 4 - numRestarts * 4;
+  if (restartsOffset < 0) return entries;
+
+  let pos = 0;
+  let prevKey = Buffer.alloc(0);
+
+  while (pos < restartsOffset) {
+    if (pos + 3 > blockData.length) break;
+
+    const shared = readVarint(blockData, pos);
+    pos += shared.bytesRead;
+    const nonShared = readVarint(blockData, pos);
+    pos += nonShared.bytesRead;
+    const valueLen = readVarint(blockData, pos);
+    pos += valueLen.bytesRead;
+
+    if (pos + nonShared.value + valueLen.value > blockData.length) break;
+
+    const keyDelta = blockData.subarray(pos, pos + nonShared.value);
+    pos += nonShared.value;
+
+    const key = Buffer.concat([prevKey.subarray(0, shared.value), keyDelta]);
+    prevKey = key;
+
+    const value = Buffer.from(blockData.subarray(pos, pos + valueLen.value));
+    pos += valueLen.value;
+
+    entries.push({ key, value });
+  }
+  return entries;
+}
+
+function readBlock(fileData, offset, size) {
+  if (offset + size + 5 > fileData.length) return null;
+  const raw = fileData.subarray(offset, offset + size);
+  const compressionType = fileData[offset + size];
+  return decompressBlock(raw, compressionType);
+}
+
+function parseFooter(fileData) {
+  const footer = fileData.subarray(fileData.length - 48);
+  const mi = readVarint(footer, 0);
+  readVarint(footer, mi.bytesRead);
+  let pos = mi.bytesRead + readVarint(footer, mi.bytesRead).bytesRead;
+  const idx = readVarint(footer, pos);
+  pos += idx.bytesRead;
+  const idx2 = readVarint(footer, pos);
+  return { indexOffset: idx.value, indexSize: idx2.value };
+}
+
+function* readSstEntries(filePath) {
+  let fileData;
+  try {
+    fileData = readFileSync(filePath);
+  } catch {
+    return;
+  }
+  if (fileData.length < 48) return;
+
+  const magic = readFixed64(fileData, fileData.length - 8);
+  if (magic !== LEVELDB_TABLE_MAGIC) return;
+
+  let footer;
+  try {
+    footer = parseFooter(fileData);
+  } catch {
+    return;
+  }
+
+  const indexBlock = readBlock(fileData, footer.indexOffset, footer.indexSize);
+  if (!indexBlock) return;
+
+  for (const indexEntry of parseBlockEntries(indexBlock)) {
+    const handle = readVarint(indexEntry.value, 0);
+    const handleSize = readVarint(indexEntry.value, handle.bytesRead);
+
+    const dataBlock = readBlock(fileData, handle.value, handleSize.value);
+    if (!dataBlock) continue;
+
+    yield* parseBlockEntries(dataBlock);
+  }
+}
+
+// LevelDB write-ahead log format: 32 KB blocks with record headers
+const LOG_BLOCK_SIZE = 32768;
+const LOG_HEADER_SIZE = 7; // checksum(4) + length(2) + type(1)
+
+/**
+ * Parse records from a single WAL block, yielding { payload, type } for each.
+ */
+function* parseLogBlock(fileData, blockStart, blockEnd) {
+  let offset = blockStart;
+  while (offset + LOG_HEADER_SIZE <= blockEnd) {
+    const length = fileData.readUInt16LE(offset + 4);
+    const type = fileData[offset + 6];
+    if (type === 0 || length === 0) break;
+
+    const payload = fileData.subarray(
+      offset + LOG_HEADER_SIZE,
+      offset + LOG_HEADER_SIZE + length,
+    );
+    yield { payload, type };
+    offset += LOG_HEADER_SIZE + length;
+  }
+}
+
+/**
+ * Accumulate record fragments across blocks.
+ * Type 1 = full, 2 = first, 3 = middle, 4 = last.
+ * Returns the updated pending buffer (or null).
+ */
+function handleFragment(type, payload, pendingRecord, emit) {
+  if (type === 1) {
+    emit(payload);
+    return null;
+  }
+  if (type === 2) return [payload];
+  if (type === 3) {
+    if (pendingRecord) pendingRecord.push(payload);
+    return pendingRecord;
+  }
+  if (type === 4 && pendingRecord) {
+    pendingRecord.push(payload);
+    emit(Buffer.concat(pendingRecord));
+    return null;
+  }
+  return pendingRecord;
+}
+
+function* readLogEntries(filePath) {
+  let fileData;
+  try {
+    fileData = readFileSync(filePath);
+  } catch {
+    return;
+  }
+
+  let pos = 0;
+  let pendingRecord = null;
+  const completed = [];
+
+  while (pos < fileData.length) {
+    const blockEnd = Math.min(pos + LOG_BLOCK_SIZE, fileData.length);
+
+    for (const { payload, type } of parseLogBlock(fileData, pos, blockEnd)) {
+      pendingRecord = handleFragment(type, payload, pendingRecord, (buf) =>
+        completed.push(buf),
+      );
+    }
+
+    for (const buf of completed) {
+      yield* parseWriteBatchEntries(buf);
+    }
+    completed.length = 0;
+
+    pos = blockEnd;
+  }
+}
+
+function* parseWriteBatchEntries(batchData) {
+  if (batchData.length < 12) return;
+
+  // WriteBatch header: sequence(8) + count(4)
+  const count = readFixed32(batchData, 8);
+  let pos = 12;
+
+  for (let i = 0; i < count && pos < batchData.length; i++) {
+    const tag = batchData[pos];
+    pos++;
+
+    if (tag === 1) {
+      // Put
+      const keyLen = readVarint(batchData, pos);
+      pos += keyLen.bytesRead;
+      if (pos + keyLen.value > batchData.length) break;
+      const key = Buffer.from(batchData.subarray(pos, pos + keyLen.value));
+      pos += keyLen.value;
+
+      const valLen = readVarint(batchData, pos);
+      pos += valLen.bytesRead;
+      if (pos + valLen.value > batchData.length) break;
+      const value = Buffer.from(batchData.subarray(pos, pos + valLen.value));
+      pos += valLen.value;
+
+      yield { key, value };
+    } else if (tag === 0) {
+      // Delete — skip the key
+      const keyLen = readVarint(batchData, pos);
+      pos += keyLen.bytesRead;
+      pos += keyLen.value;
+    } else {
+      break;
+    }
+  }
+}
+
+/**
+ * Read all key-value entries from a LevelDB directory.
+ * Yields { key: Buffer, value: Buffer } for each entry.
+ * Reads .ldb files (SSTables) and .log files (write-ahead log).
+ *
+ * @param {string} dir - path to the LevelDB directory
+ * @yields {{ key: Buffer, value: Buffer }}
+ */
+export function* readAllEntries(dir) {
+  let files;
+  try {
+    files = readdirSync(dir);
+  } catch {
+    return;
+  }
+
+  const ldbFiles = files
+    .filter((f) => f.endsWith(".ldb"))
+    .sort((a, b) => parseInt(a) - parseInt(b));
+
+  for (const file of ldbFiles) {
+    yield* readSstEntries(join(dir, file));
+  }
+
+  const logFiles = files
+    .filter((f) => f.endsWith(".log"))
+    .sort((a, b) => parseInt(a) - parseInt(b));
+
+  for (const file of logFiles) {
+    yield* readLogEntries(join(dir, file));
+  }
+}