decant-core 1.9.2 → 1.10.1

package/README.md

@@ -12,6 +12,12 @@ And every time one of those platforms changes its UI, seriously re-renders a mes
 
 `decant-core` provides reusable parsers, platform detection, and web article extraction so developers don't have to build and maintain the same fragile parsing layer over and over again.
 
+### Origin & Purpose
+
+Existing chat exporters often suffer from two major flaws: they break whenever platform DOMs update, and many route user conversations through third-party servers.
+
+`decant-core` was created to solve both at the foundation. Originally built to power local-first extensions like [AI Chat Exporter](https://ai-chat-exporter.covai.org/) and [Decant](https://decant.covai.org/), it decouples fragile platform parsing from presentation. By sharing this engine under AGPL-3.0, any browser extension, web clipper, archiver, or research tool can rely on a maintained, local-first extraction layer instead of reverse-engineering AI platforms in isolation.
+
 ```bash
 npm install decant-core
 ```
@@ -35,7 +41,7 @@ AI platforms don't expose stable public APIs for reading conversation history. N
 
 Maintaining that per-platform logic in every exporter is wasteful and fragile. `decant-core` centralizes it:
 
-- ✅ **10+ AI chat platform parsers** with normalized output — you get structured messages, models, metadata and Markdown, not DOM soup.
+- ✅ **18 AI chat platform parsers** with normalized output — you get structured messages, models, metadata and Markdown, not DOM soup.
 - ✅ **Web article extraction** — Mozilla Readability, Defuddle, and Article-Extractor run in parallel and arbitrate by content-quality scoring.
 - ✅ **Detection utilities** — tell an "AI chat page" apart from a "regular web page" before you decide which parser to run.
 - ✅ **Math & Markdown handling** — LaTeX normalization plus GFM tables/code fencing that survive round-trips into Obsidian, Logseq and Notion.
@@ -111,13 +117,34 @@ import { normalizeLatexMath } from "decant-core";
 
 ## Supported Platforms
 
-10+ AI chat platform parsers plus generic web article extraction:
-
-**ChatGPT · Claude · Google Gemini · Microsoft Copilot · Perplexity · DeepSeek · Qwen · Meta AI · Mistral (Le Chat) · Proton Lumo · Z.ai · Google AI Studio · NotebookLM · Google Search AI · Gemini Cloud Assist · Joyland · Chub**
+18 AI chat platform parsers plus generic web article extraction:
+
+| Platform                            | Parser                    | Extraction strategy                        |
+| :---------------------------------- | :------------------------ | :----------------------------------------- |
+| **ChatGPT**                         | `ChatGPTParser`           | DOM + internal API                         |
+| **Claude**                          | `ClaudeParser`            | DOM + internal API + React fiber           |
+| **Google Gemini**                   | `GeminiParser`            | DOM + batchexecute RPC                     |
+| **Microsoft Copilot**               | `CopilotParser`           | DOM (multi-domain)                         |
+| **Perplexity**                      | `PerplexityParser`        | Internal API + DOM fallback                |
+| **DeepSeek**                        | `DeepSeekParser`          | DOM + internal API (`fragments[]`)         |
+| **Qwen**                            | `QwenParser`              | DOM                                        |
+| **Meta AI**                         | `MetaParser`              | Internal API (GraphQL) + DOM fallback      |
+| **Mistral / Le Chat**               | `MistralParser`           | DOM                                        |
+| **Proton Lumo**                     | `LumoParser`              | DOM (API is E2E-encrypted, not readable)   |
+| **Z.ai**                            | `ZAiParser`               | Internal API (chat + batch) + DOM fallback |
+| **Grok**                            | `GrokParser`              | Internal API (response-node + load) + DOM  |
+| **Google AI Studio**                | `GoogleAIStudioParser`    | DOM                                        |
+| **NotebookLM**                      | `NotebookLMParser`        | DOM                                        |
+| **Google Search AI (AI Overviews)** | `GoogleSearchAIParser`    | DOM                                        |
+| **Gemini Cloud Assist**             | `GeminiCloudAssistParser` | DOM                                        |
+| **Joyland**                         | `JoylandParser`           | DOM                                        |
+| **Chub**                            | `ChubParser`              | DOM                                        |
+| **Generic Web Article**             | `ArticleParser`           | Readability + Defuddle + Article-Extractor |
 
 All parsers extend the base [`ChatParser`](ai/base.js) interface — a consistent `isAvailable(url)` +
-normalized `parse()` contract. For the extraction-strategy breakdown and maintenance model, see
-[SUPPORTED_PLATFORMS.md](SUPPORTED_PLATFORMS.md).
+normalized `parse()` contract. For the full extraction-strategy breakdown and maintenance model, see
+[SUPPORTED_PLATFORMS.md](SUPPORTED_PLATFORMS.md), also published as the
+[platform matrix](https://covai-labs.github.io/decant-core/platforms/) on the developer docs site.
 
 ---
 
@@ -127,6 +154,10 @@ normalized `parse()` contract. For the extraction-strategy breakdown and mainten
 
 That choice is deliberate. AI platforms change constantly, and parser fixes belong in a shared commons so the whole ecosystem benefits — not siloed in a proprietary fork. If you use `decant-core`, network-based deployments that serve modified versions must also offer the corresponding source. Please review [`LICENSE`](LICENSE) before incorporating it into your project.
 
+### Third-Party Test Fixtures Notice
+
+Sample test fixtures located in [`tests/fixtures/`](tests/fixtures/) consist of third-party DOM snapshots and API response excerpts retained solely for automated regression testing and platform interoperability under fair use principles. They are excluded from the project's AGPL-3.0 license. See [`tests/fixtures/README.md`](tests/fixtures/README.md) for details.
+
 ---
 
 ## Used by

package/ai/deepseek.js

@@ -17,6 +17,27 @@ function getUserToken() {
   }
 }
 
+export function extractDeepSeekMessageContent(msgNode) {
+  if (!msgNode) return "";
+  // Current API shape: content lives in fragments[] ({type, content}).
+  // REQUEST = user prompt, RESPONSE = final assistant answer.
+  // THINK / TOOL_* fragments are intermediate reasoning/tool output.
+  if (Array.isArray(msgNode.fragments) && msgNode.fragments.length > 0) {
+    const isUser = msgNode.role === "USER" || msgNode.role === "user";
+    const wanted = isUser ? "REQUEST" : "RESPONSE";
+    const picked = msgNode.fragments.filter((f) => f && f.type === wanted);
+    const fallback = picked.length > 0 ? picked : msgNode.fragments;
+    const text = fallback
+      .map((f) => (typeof f.content === "string" ? f.content : ""))
+      .join("\n\n")
+      .trim();
+    if (text) return text;
+  }
+  // Legacy shape: top-level content/text fields.
+  const legacy = msgNode.content || msgNode.text || "";
+  return typeof legacy === "string" ? legacy.trim() : "";
+}
+
 function getConversationId() {
   try {
     if (typeof window === "undefined" || !window.location) return null;
@@ -76,7 +97,7 @@ async function fetchDeepSeekConversation(sessionId, token) {
     .map((msgNode) => {
       const isUser = msgNode.role === "USER" || msgNode.role === "user";
       const role = isUser ? "User" : "DeepSeek";
-      const content = msgNode.content || msgNode.text || "";
+      const content = extractDeepSeekMessageContent(msgNode);
       return { role, content: content.trim() };
     })
     .filter((msg) => msg.content.length > 0);
@@ -130,12 +151,36 @@ export class DeepSeekParser extends ChatParser {
     const userSelector = ".fbb737a4";
     const assistantSelector = ".ds-markdown";
 
-    // We'll traverse the DOM to find these in order
-    const allElements = document.querySelectorAll(
-      `${userSelector}, ${assistantSelector}`,
+    // We'll traverse the DOM to find these in order. Nested matches
+    // (e.g. a .ds-markdown code fragment inside an assistant turn) are
+    // skipped so each turn is exported exactly once.
+    const allElements = Array.from(
+      document.querySelectorAll(`${userSelector}, ${assistantSelector}`),
     );
+    const outerElements = allElements.filter((el) => {
+      // Skip collapsible thinking-chain blocks: they live inside
+      // ds-think-content containers and would otherwise duplicate turns.
+      let ancestor = el.parentElement;
+      while (ancestor) {
+        const cls =
+          typeof ancestor.className === "string" ? ancestor.className : "";
+        if (/think/i.test(cls)) return false;
+        ancestor = ancestor.parentElement;
+      }
+      let parent = el.parentElement;
+      while (parent) {
+        if (
+          parent.matches &&
+          (parent.matches(userSelector) || parent.matches(assistantSelector))
+        ) {
+          return false;
+        }
+        parent = parent.parentElement;
+      }
+      return true;
+    });
 
-    allElements.forEach((el) => {
+    outerElements.forEach((el) => {
       let role = "Unknown";
       if (el.matches(userSelector)) {
         role = "User";

package/ai/grok.js

@@ -0,0 +1,216 @@
+import { ChatParser } from "./base.js";
+import { convertToMarkdown } from "../utils/html-to-markdown.js";
+
+export const GROK_BATCH_SIZE = 20;
+
+export function getGrokConversationId(url) {
+  if (!url || typeof url !== "string") return null;
+  return (
+    url.match(/grok\.com\/(?:chat|c|conversation)\/([a-f0-9-]+)/i)?.[1] ?? null
+  );
+}
+
+/**
+ * Order response-node entries into a single parent-linked chain.
+ * Response-nodes form a linear branch via parentResponseId; walk from
+ * the leaf (the id that is never a parent) back to the root and reverse.
+ */
+export function orderGrokNodes(responseNodes) {
+  if (!Array.isArray(responseNodes) || responseNodes.length === 0) return [];
+  const byId = new Map(responseNodes.map((n) => [n?.responseId, n]));
+  const parentIds = new Set(
+    responseNodes.map((n) => n?.parentResponseId).filter(Boolean),
+  );
+  let leaf = responseNodes.find((n) => n && !parentIds.has(n.responseId));
+  if (!leaf) leaf = responseNodes[responseNodes.length - 1];
+  const chain = [];
+  const seen = new Set();
+  let current = leaf;
+  while (current && current.responseId && !seen.has(current.responseId)) {
+    seen.add(current.responseId);
+    chain.unshift(current);
+    current = byId.get(current.parentResponseId) ?? null;
+  }
+  return chain;
+}
+
+export function formatGrokResponses(responses, orderedIds, fallbackTitle) {
+  const byId = new Map((responses || []).map((r) => [r?.responseId, r]));
+  const ids =
+    Array.isArray(orderedIds) && orderedIds.length > 0
+      ? orderedIds
+      : (responses || []).map((r) => r?.responseId).filter(Boolean);
+  const messages = [];
+  let title = fallbackTitle || "Grok Conversation";
+  for (const id of ids) {
+    const r = byId.get(id);
+    if (!r) continue;
+    const content = (r.message || "").trim();
+    if (!content) continue;
+    const role = r.sender === "human" ? "User" : "Grok";
+    const msg = { role, content };
+    if (r.createTime) msg.timestamp = r.createTime;
+    messages.push(msg);
+  }
+  return { messages, title };
+}
+
+export class GrokParser extends ChatParser {
+  name = "Grok";
+  isAvailable(url) {
+    return url.includes("grok.com");
+  }
+
+  async fetchConversationMeta(conversationId) {
+    const url =
+      `https://grok.com/rest/app-chat/conversations_v2/${conversationId}` +
+      `?includeWorkspaces=true&includeTaskResult=true`;
+    const response = await fetch(url, {
+      method: "GET",
+      credentials: "include",
+      headers: { Accept: "application/json" },
+    });
+    if (!response.ok) {
+      throw new Error(`Grok API request failed: ${response.status}`);
+    }
+    return response.json();
+  }
+
+  async fetchResponseNodes(conversationId) {
+    const url = `https://grok.com/rest/app-chat/conversations/${conversationId}/response-node`;
+    const response = await fetch(url, {
+      method: "GET",
+      credentials: "include",
+      headers: { Accept: "application/json" },
+    });
+    if (!response.ok) {
+      throw new Error(`Grok response-node request failed: ${response.status}`);
+    }
+    return response.json();
+  }
+
+  async fetchResponses(conversationId, responseIds) {
+    const url = `https://grok.com/rest/app-chat/conversations/${conversationId}/load-responses`;
+    const collected = [];
+    for (let i = 0; i < responseIds.length; i += GROK_BATCH_SIZE) {
+      const chunk = responseIds.slice(i, i + GROK_BATCH_SIZE);
+      const response = await fetch(url, {
+        method: "POST",
+        credentials: "include",
+        headers: {
+          Accept: "application/json",
+          "Content-Type": "application/json",
+        },
+        body: JSON.stringify({ responseIds: chunk }),
+      });
+      if (!response.ok) {
+        throw new Error(`Grok load-responses failed: ${response.status}`);
+      }
+      const json = await response.json();
+      if (Array.isArray(json?.responses)) collected.push(...json.responses);
+    }
+    return collected;
+  }
+
+  async parse(options = {}) {
+    const parserMode = options.parserMode || "prefer_api";
+    const currentUrl =
+      typeof window !== "undefined" && window.location
+        ? window.location.href || ""
+        : "";
+    const metadata = {
+      Source: "Grok",
+      Date: new Date().toLocaleString(),
+      Link: currentUrl,
+    };
+    const domTitle =
+      (document.title || "").replace(/\s*-\s*Grok\s*$/i, "").trim() ||
+      "Grok Conversation";
+
+    // Primary: internal API (response-node ordering + load-responses bodies)
+    if (parserMode !== "prefer_dom") {
+      try {
+        const conversationId = getGrokConversationId(currentUrl);
+        if (conversationId) {
+          const [meta, nodes] = await Promise.all([
+            this.fetchConversationMeta(conversationId).catch(() => null),
+            this.fetchResponseNodes(conversationId),
+          ]);
+          const ordered = orderGrokNodes(nodes?.responseNodes);
+          const orderedIds = ordered.map((n) => n.responseId);
+          if (orderedIds.length > 0) {
+            const responses = await this.fetchResponses(
+              conversationId,
+              orderedIds,
+            );
+            const title = meta?.conversation?.title?.trim() || domTitle;
+            const { messages } = formatGrokResponses(
+              responses,
+              orderedIds,
+              title,
+            );
+            if (messages.length > 0) {
+              return {
+                title,
+                messages,
+                url: currentUrl,
+                metadata: { ...metadata, Method: "API" },
+              };
+            }
+          }
+        }
+      } catch (e) {
+        console.warn(
+          "[AI Exporter] Grok API fetch failed, falling back to DOM:",
+          e,
+        );
+      }
+    }
+
+    // Secondary: DOM fallback
+    const userSelector = '[data-testid="user-message"]';
+    const assistantSelector = '[data-testid="assistant-message"]';
+    const candidates = Array.from(
+      document.querySelectorAll(`${userSelector}, ${assistantSelector}`),
+    );
+    // Keep only outermost matches to avoid nested bubbles duplicating turns.
+    const elements = candidates.filter((el) => {
+      let parent = el.parentElement;
+      while (parent) {
+        if (
+          parent.matches &&
+          (parent.matches(userSelector) || parent.matches(assistantSelector))
+        ) {
+          return false;
+        }
+        parent = parent.parentElement;
+      }
+      return true;
+    });
+
+    const messages = [];
+    for (const el of elements) {
+      const isUser = el.matches(userSelector);
+      const role = isUser ? "User" : "Grok";
+      const contentEl =
+        el.querySelector(".response-content-markdown") ||
+        el.querySelector(".message-bubble") ||
+        el;
+      const clone = contentEl.cloneNode(true);
+      clone
+        .querySelectorAll(
+          ".thinking-container, button, [data-testid='canvas-trigger']",
+        )
+        .forEach((n) => n.remove());
+      const text = convertToMarkdown(clone).trim();
+      if (text) messages.push({ role, content: text });
+    }
+
+    return {
+      title: domTitle,
+      messages,
+      url: currentUrl,
+      metadata: { ...metadata, Method: "DOM" },
+    };
+  }
+}

package/ai/index.js

@@ -27,6 +27,7 @@ export { GoogleSearchAIParser } from "./google_search_ai.js";
 export { GeminiCloudAssistParser } from "./gemini_cloud_assist.js";
 export { JoylandParser } from "./joyland.js";
 export { ChubParser } from "./chub.js";
+export { GrokParser } from "./grok.js";
 
 // Utilities
 export { convertToMarkdown, cleanMarkdown } from "../utils/html-to-markdown.js";

package/ai/meta.js

@@ -1,13 +1,361 @@
 import { ChatParser } from "./base.js";
 import { convertToMarkdown } from "../utils/html-to-markdown.js";
 
+// Observed Relay doc_ids for the conversation message list (Sept 2026).
+// These rotate when Meta redeploys the web client. The parser tries the
+// pinned IDs first, then resolves fresh ones from the page's own JS chunks,
+// and finally falls back to DOM extraction.
+export const META_MESSAGES_DOC_ID = "d6d27be74bdbe5bec25ad7685dc65ac4";
+export const META_MESSAGES_PAGE_DOC_ID = "bc523f4325577cba4ccbac4ffdec7ff4";
+export const META_PAGE_SIZE = 10;
+export const META_MAX_SCRIPT_CHUNKS = 25;
+export const META_MAX_CHUNK_CHARS = 5_000_000;
+export const META_MAX_DOCID_TRIALS = 8;
+
+// Markers that identify the conversation-messages Relay artifact inside a
+// JS chunk. Candidates are only trusted from chunks containing one of these.
+const META_QUERY_MARKERS = [
+  "latestBranchPath",
+  "GenAIMarkdownTextUXPrimitive",
+  "GenAIBotThinkingStatusPrimitive",
+];
+
+let cachedMetaDocIds = null;
+
+/** Test hook: clear the in-memory resolved doc_id cache. */
+export function __resetMetaDocIdCache() {
+  cachedMetaDocIds = null;
+}
+
+/**
+ * Extract persisted-query doc_ids from a JS chunk — but only when the chunk
+ * contains the conversation-messages query markers, so unrelated Relay
+ * artifacts (analytics, surveys) are never mistaken for message queries.
+ */
+export function extractMetaDocIdCandidates(jsText) {
+  if (typeof jsText !== "string") return [];
+  if (!META_QUERY_MARKERS.some((marker) => jsText.includes(marker))) return [];
+  const ids = new Set();
+  const patterns = [
+    /["']id["']\s*:\s*["']([a-f0-9]{32})["']/g,
+    /doc_id["']?\s*[:=]\s*["']([a-f0-9]{32})["']/g,
+  ];
+  for (const re of patterns) {
+    let match;
+    while ((match = re.exec(jsText)) !== null) ids.add(match[1]);
+  }
+  return [...ids];
+}
+
+/** Absolute URLs of the page's own script chunks (bounded). */
+export function getMetaScriptUrls(limit = META_MAX_SCRIPT_CHUNKS) {
+  try {
+    const urls = [];
+    const base =
+      typeof window !== "undefined" && window.location?.href
+        ? window.location.href
+        : "https://www.meta.ai/";
+    for (const script of document.querySelectorAll("script[src]")) {
+      const src = script.getAttribute("src");
+      if (!src || src.startsWith("data:") || src.startsWith("blob:")) continue;
+      try {
+        const absolute = new URL(src, base).href;
+        if (!urls.includes(absolute)) urls.push(absolute);
+      } catch {
+        // Ignore unresolvable src values
+      }
+      if (urls.length >= limit) break;
+    }
+    return urls;
+  } catch {
+    return [];
+  }
+}
+
+export function getMetaConversationId(url) {
+  if (!url || typeof url !== "string") return null;
+  return (
+    url.match(/meta\.ai\/(?:c|chat|prompt)\/([a-f0-9-]+)/i)?.[1] ??
+    url.match(/[?&]conversationId=([a-f0-9-]+)/i)?.[1] ??
+    null
+  );
+}
+
+export function extractMetaUserText(node) {
+  if (!node) return "";
+  for (const key of ["userContent", "content", "originalUserPrompt"]) {
+    const val = node[key];
+    if (typeof val === "string" && val.trim()) return val.trim();
+  }
+  return "";
+}
+
+export function extractMetaAssistantText(node) {
+  if (!node) return "";
+  const renderer = node.contentRenderer || {};
+  const message = renderer.message || {};
+  if (typeof message.content === "string" && message.content.trim()) {
+    return message.content.trim();
+  }
+  // Fallback: walk unified_response sections for markdown primitives.
+  const sections = renderer.unified_response?.sections;
+  if (Array.isArray(sections)) {
+    const parts = [];
+    for (const section of sections) {
+      const primitive = section?.view_model?.primitive;
+      if (
+        primitive &&
+        typeof primitive.text === "string" &&
+        primitive.text.trim()
+      ) {
+        parts.push(primitive.text.trim());
+      }
+    }
+    if (parts.length > 0) return parts.join("\n\n");
+  }
+  if (typeof node.content === "string" && node.content.trim()) {
+    return node.content.trim();
+  }
+  return "";
+}
+
+export function formatMetaEdges(edges) {
+  const seen = new Set();
+  const nodes = [];
+  for (const edge of edges || []) {
+    const cursor = edge?.cursor;
+    if (cursor) {
+      if (seen.has(cursor)) continue;
+      seen.add(cursor);
+    }
+    if (edge?.node) nodes.push(edge.node);
+  }
+  // Relay returns oldest-first across pages; order by branchPath as a guard.
+  nodes.sort((a, b) => {
+    const pa = parseInt(a?.branchPath ?? "0", 10);
+    const pb = parseInt(b?.branchPath ?? "0", 10);
+    if (Number.isNaN(pa) || Number.isNaN(pb)) return 0;
+    return pa - pb;
+  });
+  const messages = [];
+  for (const node of nodes) {
+    if (node.__typename === "UserMessage") {
+      const content = extractMetaUserText(node);
+      if (content) messages.push({ role: "User", content });
+    } else if (node.__typename === "AssistantMessage") {
+      const content = extractMetaAssistantText(node);
+      if (content) messages.push({ role: "Meta AI", content });
+    }
+  }
+  return messages;
+}
+
 export class MetaParser extends ChatParser {
   name = "Meta AI";
   isAvailable(url) {
     return url.includes("meta.ai");
   }
 
-  async parse() {
+  async fetchGraphQL(docId, variables) {
+    const response = await fetch("https://www.meta.ai/api/graphql", {
+      method: "POST",
+      credentials: "include",
+      headers: {
+        Accept: "application/json",
+        "Content-Type": "application/json",
+      },
+      body: JSON.stringify({ doc_id: docId, variables }),
+    });
+    if (!response.ok) {
+      throw new Error(`Meta GraphQL request failed: ${response.status}`);
+    }
+    return response.json();
+  }
+
+  async fetchConversation(conversationId, docIds = {}) {
+    const initialId = docIds.initial || META_MESSAGES_DOC_ID;
+    const pageId = docIds.page || META_MESSAGES_PAGE_DOC_ID;
+    const allEdges = [];
+    // First page (latest messages).
+    const first = await this.fetchGraphQL(initialId, {
+      conversationId,
+    });
+    const conv = first?.data?.conversation;
+    if (!conv || !conv.messages) {
+      throw new Error("Meta GraphQL response missing conversation.messages");
+    }
+    const title = conv.displayTitle || conv.title || "";
+    allEdges.push(...(conv.messages.edges || []));
+    // Walk backwards through history while older pages exist.
+    let pageInfo = conv.messages.pageInfo;
+    let guard = 0;
+    while (pageInfo?.hasPreviousPage && pageInfo?.startCursor && guard < 50) {
+      guard += 1;
+      const page = await this.fetchGraphQL(pageId, {
+        conversationId,
+        before: pageInfo.startCursor,
+        last: META_PAGE_SIZE,
+      });
+      const pageConv = page?.data?.conversation;
+      if (!pageConv?.messages) break;
+      allEdges.push(...(pageConv.messages.edges || []));
+      pageInfo = pageConv.messages.pageInfo;
+    }
+    return { title, edges: allEdges };
+  }
+
+  /** Verify a candidate doc_id returns a message list (self-validating). */
+  async trialMetaDocId(docId, variables) {
+    try {
+      const json = await this.fetchGraphQL(docId, variables);
+      const messages = json?.data?.conversation?.messages;
+      if (messages && Array.isArray(messages.edges)) return json;
+    } catch {
+      // Invalid/retired doc_id — caller tries the next candidate.
+    }
+    return null;
+  }
+
+  /**
+   * Resolve fresh doc_ids from the page's own JS chunks when the pinned IDs
+   * stop working. Every candidate is trial-verified against the live API, so
+   * wrong guesses cost one request and never corrupt the export. Results are
+   * cached in-memory for the session.
+   */
+  async resolveMetaDocIds(conversationId) {
+    if (cachedMetaDocIds) return cachedMetaDocIds;
+    const tried = new Set();
+    const candidates = [];
+    for (const chunkUrl of getMetaScriptUrls()) {
+      let jsText;
+      try {
+        const response = await fetch(chunkUrl, {
+          method: "GET",
+          credentials: "include",
+        });
+        if (!response.ok) continue;
+        jsText = await response.text();
+      } catch {
+        continue;
+      }
+      if (!jsText || jsText.length > META_MAX_CHUNK_CHARS) continue;
+      for (const id of extractMetaDocIdCandidates(jsText)) {
+        if (!tried.has(id)) {
+          tried.add(id);
+          candidates.push(id);
+        }
+      }
+      if (tried.size >= META_MAX_DOCID_TRIALS) break;
+    }
+
+    let initial = null;
+    let initialJson = null;
+    for (const id of candidates.slice(0, META_MAX_DOCID_TRIALS)) {
+      initialJson = await this.trialMetaDocId(id, { conversationId });
+      if (initialJson) {
+        initial = id;
+        break;
+      }
+    }
+    if (!initial) {
+      throw new Error("No working Meta doc_id found in page chunks");
+    }
+
+    // The pagination query may differ from the initial-page query: prefer the
+    // initial winner, then trial the remaining candidates with page vars.
+    let page = initial;
+    const pageInfo = initialJson.data.conversation.messages.pageInfo;
+    if (pageInfo?.hasPreviousPage && pageInfo?.startCursor) {
+      const pageVars = {
+        conversationId,
+        before: pageInfo.startCursor,
+        last: META_PAGE_SIZE,
+      };
+      if (!(await this.trialMetaDocId(initial, pageVars))) {
+        for (const id of candidates.slice(0, META_MAX_DOCID_TRIALS)) {
+          if (id === initial) continue;
+          if (await this.trialMetaDocId(id, pageVars)) {
+            page = id;
+            break;
+          }
+        }
+      }
+    }
+    cachedMetaDocIds = { initial, page };
+    return cachedMetaDocIds;
+  }
+
+  /** Pick the richest markdown candidate (thinking stubs are near-empty). */
+  pickAssistantContent(el) {
+    const candidates = Array.from(
+      el.querySelectorAll(".markdown-content, .ur-markdown, .prose"),
+    );
+    const pool = candidates.length > 0 ? candidates : [el];
+    let best = null;
+    let bestLen = -1;
+    for (const cand of pool) {
+      const len = (cand.textContent || "").trim().length;
+      if (len > bestLen) {
+        bestLen = len;
+        best = cand;
+      }
+    }
+    return best || el;
+  }
+
+  async parse(options = {}) {
+    const parserMode = options.parserMode || "prefer_api";
+    const currentUrl =
+      typeof window !== "undefined" && window.location
+        ? window.location.href || ""
+        : "";
+    const metadata = {
+      Source: "Meta AI",
+      Date: new Date().toLocaleString(),
+      Link: currentUrl,
+    };
+
+    // Primary: internal GraphQL API (same-session cookies required).
+    // Pinned doc_ids first; if Meta rotated them, resolve fresh ones from
+    // the page's own JS chunks before giving up to DOM extraction.
+    if (parserMode !== "prefer_dom") {
+      const conversationId = getMetaConversationId(currentUrl);
+      if (conversationId) {
+        const runApi = async (docIds) => {
+          const { title: apiTitle, edges } = await this.fetchConversation(
+            conversationId,
+            docIds,
+          );
+          const messages = formatMetaEdges(edges);
+          if (messages.length === 0) return null;
+          return {
+            title: apiTitle || "Meta AI Session",
+            messages,
+            url: currentUrl,
+            metadata: { ...metadata, Method: "API" },
+          };
+        };
+        try {
+          // Attempt 1: pinned doc_ids (fast path, no extra requests).
+          const pinned = await runApi();
+          if (pinned) return pinned;
+        } catch (e) {
+          console.warn("[AI Exporter] Meta pinned doc_ids failed:", e);
+          try {
+            // Attempt 2: resolve fresh doc_ids from the page's JS chunks.
+            const resolved = await runApi(
+              await this.resolveMetaDocIds(conversationId),
+            );
+            if (resolved) return resolved;
+          } catch (e2) {
+            console.warn("[AI Exporter] Meta doc_id resolution failed:", e2);
+          }
+        }
+        console.warn("[AI Exporter] Meta API exhausted, falling back to DOM");
+      }
+    }
+
+    // Secondary: DOM extraction fallback
     // Try to get the conversation title from the input field or the header button
     const titleInput = document.querySelector(
       'input[placeholder="Conversation title"]',
@@ -70,11 +418,10 @@ export class MetaParser extends ChatParser {
         }
       } else if (isAssistant) {
         role = "Meta AI";
-        // Assistant content is typically styled in markdown-content or prose
-        const contentEl =
-          el.querySelector(".markdown-content") ||
-          el.querySelector(".ur-markdown") ||
-          el.querySelector(".prose");
+        // Assistant turns nest several .markdown-content wrappers (a
+        // near-empty thinking stub plus the real answer). Convert the
+        // richest candidate so thinking stubs never shadow the answer.
+        const contentEl = this.pickAssistantContent(el);
         if (contentEl) {
           const clone = contentEl.cloneNode(true);
 
@@ -101,17 +448,13 @@ export class MetaParser extends ChatParser {
       }
     });
 
-    const currentUrl =
-      typeof window !== "undefined" && window.location
-        ? window.location.href || ""
-        : "";
-    const metadata = {
+    const domMetadata = {
       Source: "Meta AI",
       Date: new Date().toLocaleString(),
       Link: currentUrl,
       Method: "DOM",
     };
 
-    return { title, messages, url: currentUrl, metadata };
+    return { title, messages, url: currentUrl, metadata: domMetadata };
   }
 }

package/ai/mistral.js

@@ -8,13 +8,17 @@ export class MistralParser extends ChatParser {
   }
 
   async parse() {
-    // Extract Title
-    const titleElement = document.querySelector("span.truncate.text-sm");
-    let title = "Mistral Conversation";
-    if (titleElement && titleElement.innerText) {
-      title = titleElement.innerText.trim();
-    } else if (document.title) {
-      title = document.title.replace(" - Mistral", "").trim();
+    // Extract Title: document.title is the most reliable source; the
+    // sidebar truncate span may match unrelated UI ("Upgrade to Pro").
+    let title = (document.title || "")
+      .replace(/\s*-\s*Mistral\s*$/i, "")
+      .trim();
+    if (!title) {
+      const titleElement = document.querySelector(
+        "span.truncate.text-sm, [data-testid='conversation-title']",
+      );
+      title =
+        (titleElement?.textContent || "").trim() || "Mistral Conversation";
     }
 
     const messages = [];
@@ -28,21 +32,18 @@ export class MistralParser extends ChatParser {
       if (role === "user") {
         const contentEl =
           el.querySelector(".select-text") ||
-          el.querySelector(".whitespace-pre-wrap");
-        if (contentEl) {
-          messages.push({
-            role: "User",
-            content: contentEl.innerText.trim(),
-          });
+          el.querySelector(".whitespace-pre-wrap") ||
+          el;
+        const text = convertToMarkdown(contentEl).trim();
+        if (text) {
+          messages.push({ role: "User", content: text });
         }
       } else if (role === "assistant") {
-        const answerEl = el.querySelector('[data-message-part-type="answer"]');
-        if (answerEl) {
-          const markdown = convertToMarkdown(answerEl.innerHTML);
-          messages.push({
-            role: "Mistral",
-            content: markdown,
-          });
+        const answerEl =
+          el.querySelector('[data-message-part-type="answer"]') || el;
+        const markdown = convertToMarkdown(answerEl).trim();
+        if (markdown) {
+          messages.push({ role: "Mistral", content: markdown });
         }
       }
     }

package/ai/qwen.js

@@ -24,25 +24,44 @@ export class QwenParser extends ChatParser {
       if (element) {
         const text = element.textContent || element.value || element.innerText;
         if (text && text.trim() && text !== document.title) {
-          title = text.trim();
+          title = text.trim().replace(/\s+/g, " ");
           break;
         }
       }
     }
+    if (title === "Qwen Chat" && document.title) {
+      const docTitle = document.title.trim();
+      if (docTitle && docTitle.toLowerCase() !== "qwen studio") {
+        title = docTitle;
+      }
+    }
 
     const messages = [];
 
     // chat.qwen.ai uses specific class names
-    const chatMessages = document.querySelectorAll(".qwen-chat-message");
+    let chatMessages = document.querySelectorAll(".qwen-chat-message");
+    // Fallback for markup drift: user/assistant content blocks in order.
+    if (chatMessages.length === 0) {
+      chatMessages = document.querySelectorAll(
+        ".user-message-content, .qwen-markdown",
+      );
+    }
 
     chatMessages.forEach((message) => {
-      const isUser = message.classList.contains("qwen-chat-message-user");
+      // Fallback nodes are the content blocks themselves.
+      const isFallbackContent =
+        message.matches?.(".user-message-content, .qwen-markdown") ?? false;
+      const isUser = isFallbackContent
+        ? message.matches(".user-message-content")
+        : message.classList.contains("qwen-chat-message-user");
       const role = isUser ? "User" : "Qwen";
 
       let content = "";
       let attachments = [];
 
-      if (isUser) {
+      if (isFallbackContent) {
+        content = convertToMarkdown(message);
+      } else if (isUser) {
         // Extract attachments first
         const fileItems = message.querySelectorAll(
           ".index-module__file-message-document___OjWnc",

package/ai/z_ai.js

@@ -1,15 +1,138 @@
 import { ChatParser } from "./base.js";
 import { convertToMarkdown } from "../utils/html-to-markdown.js";
 
+export const ZAI_BATCH_SIZE = 20;
+
+export function getZaiChatId(url) {
+  if (!url || typeof url !== "string") return null;
+  return url.match(/chat\.z\.ai\/c\/([a-f0-9-]+)/i)?.[1] ?? null;
+}
+
+/** Walk the skeleton history (id -> {parentId}) from currentId to root. */
+export function orderZaiHistory(messagesById, currentId) {
+  if (!messagesById || typeof messagesById !== "object") return [];
+  const ordered = [];
+  const seen = new Set();
+  let current = currentId ?? null;
+  // Fallback: if no currentId, start from the node nobody parents.
+  if (!current || !messagesById[current]) {
+    const childIds = new Set(
+      Object.values(messagesById).flatMap((m) => m?.childrenIds || []),
+    );
+    const leaf = Object.keys(messagesById).find((id) => !childIds.has(id));
+    current = leaf ?? null;
+  }
+  while (current && messagesById[current] && !seen.has(current)) {
+    seen.add(current);
+    ordered.unshift(current);
+    current = messagesById[current].parentId ?? null;
+  }
+  return ordered;
+}
+
+export function formatZaiMessage(entry) {
+  if (!entry) return null;
+  const role = entry.role === "user" ? "User" : "Z.ai";
+  if (entry.role === "user") {
+    const content = (entry.content || "").trim();
+    return content ? { role, content } : null;
+  }
+  const blocks = Array.isArray(entry.content_blocks)
+    ? entry.content_blocks
+    : [];
+  const texts = blocks
+    .filter((b) => b && b.type === "text" && typeof b.content === "string")
+    .map((b) => b.content.trim())
+    .filter(Boolean);
+  // Fallback to legacy top-level content string when blocks are absent.
+  if (
+    texts.length === 0 &&
+    typeof entry.content === "string" &&
+    entry.content.trim()
+  ) {
+    texts.push(entry.content.trim());
+  }
+  if (texts.length === 0) return null;
+  let content = texts.join("\n\n");
+  const reasoning = blocks
+    .filter((b) => b && b.type === "reasoning" && typeof b.content === "string")
+    .map((b) => b.content.trim())
+    .filter(Boolean)
+    .join("\n\n");
+  if (reasoning) {
+    const quoted = reasoning
+      .split("\n")
+      .map((line) => `> ${line}`)
+      .join("\n");
+    content += `\n\n> 🧠 Thinking\n${quoted}`;
+  }
+  const msg = { role, content };
+  if (entry.timestamp) {
+    try {
+      msg.timestamp = new Date(entry.timestamp * 1000).toISOString();
+    } catch {
+      // Ignore timestamp formatting errors
+    }
+  }
+  return msg;
+}
+
 export class ZAiParser extends ChatParser {
   name = "Z.ai";
   isAvailable(url) {
     return url.includes("chat.z.ai");
   }
 
-  async parse() {
-    let title = "";
+  async fetchChatSkeleton(chatId) {
+    const url = `https://chat.z.ai/api/v1/chats/${chatId}`;
+    const response = await fetch(url, {
+      method: "GET",
+      credentials: "include",
+      headers: { Accept: "application/json" },
+    });
+    if (!response.ok) {
+      throw new Error(`Z.ai chat request failed: ${response.status}`);
+    }
+    return response.json();
+  }
+
+  async fetchMessageBatch(chatId, ids) {
+    const url = `https://chat.z.ai/api/v1/chats/${chatId}/messages/batch`;
+    const collected = {};
+    for (let i = 0; i < ids.length; i += ZAI_BATCH_SIZE) {
+      const chunk = ids.slice(i, i + ZAI_BATCH_SIZE);
+      const response = await fetch(url, {
+        method: "POST",
+        credentials: "include",
+        headers: {
+          Accept: "application/json",
+          "Content-Type": "application/json",
+        },
+        body: JSON.stringify({ ids: chunk }),
+      });
+      if (!response.ok) {
+        throw new Error(`Z.ai batch request failed: ${response.status}`);
+      }
+      const json = await response.json();
+      Object.assign(collected, json?.data || {});
+    }
+    return collected;
+  }
+
+  async parse(options = {}) {
+    const parserMode = options.parserMode || "prefer_api";
+    const currentUrl =
+      typeof window !== "undefined" && window.location
+        ? window.location.href || ""
+        : "";
+    const metadata = {
+      Source: "Z.ai",
+      Date: new Date().toLocaleString(),
+      Link: currentUrl,
+    };
+
     const titleEl = document.querySelector("title");
+    let title = "";
     if (titleEl) {
       title = titleEl.textContent.trim().replace(/\s+/g, " ");
     }
@@ -17,6 +140,43 @@ export class ZAiParser extends ChatParser {
       title = document.title.trim().replace(/\s+/g, " ");
     }
     title = title || "Z.ai Chat";
+
+    // Primary: internal API (chat skeleton for ordering + batched bodies)
+    if (parserMode !== "prefer_dom") {
+      try {
+        const chatId = getZaiChatId(currentUrl);
+        if (chatId) {
+          const skeleton = await this.fetchChatSkeleton(chatId);
+          const history = skeleton?.chat?.history || {};
+          const orderedIds = orderZaiHistory(
+            history.messages || {},
+            history.currentId,
+          );
+          if (orderedIds.length > 0) {
+            const bodies = await this.fetchMessageBatch(chatId, orderedIds);
+            const apiTitle = (skeleton?.title || "").trim() || title;
+            const apiMessages = orderedIds
+              .map((id) => formatZaiMessage(bodies[id]))
+              .filter(Boolean);
+            if (apiMessages.length > 0) {
+              return {
+                title: apiTitle,
+                messages: apiMessages,
+                url: currentUrl,
+                metadata: { ...metadata, Method: "API" },
+              };
+            }
+          }
+        }
+      } catch (e) {
+        console.warn(
+          "[AI Exporter] Z.ai API fetch failed, falling back to DOM:",
+          e,
+        );
+      }
+    }
+
+    // Secondary: DOM fallback
     const messages = [];
 
     // Selectors for z.ai messages
@@ -92,17 +252,13 @@ export class ZAiParser extends ChatParser {
       }
     });
 
-    const currentUrl =
-      typeof window !== "undefined" && window.location
-        ? window.location.href || ""
-        : "";
-    const metadata = {
+    const domMetadata = {
       Source: "Z.ai",
       Date: new Date().toLocaleString(),
       Link: currentUrl,
       Method: "DOM",
     };
 
-    return { title, messages, url: currentUrl, metadata };
+    return { title, messages, url: currentUrl, metadata: domMetadata };
   }
 }

package/detection/detect-platform.js

@@ -16,6 +16,7 @@ import { GoogleSearchAIParser } from "../ai/google_search_ai.js";
 import { GeminiCloudAssistParser } from "../ai/gemini_cloud_assist.js";
 import { JoylandParser } from "../ai/joyland.js";
 import { ChubParser } from "../ai/chub.js";
+import { GrokParser } from "../ai/grok.js";
 
 /**
  * Ordered list of parsers. First match wins.
@@ -39,6 +40,7 @@ export const parsers = [
   new GeminiCloudAssistParser(),
   new JoylandParser(),
   new ChubParser(),
+  new GrokParser(),
 ];
 
 /**

package/detection/domains.js

@@ -22,6 +22,7 @@ export const AI_CHAT_DOMAINS = [
   "joyland.ai",
   "chub.ai",
   "characterhub.org",
+  "grok.com",
 ];
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "decant-core",
-  "version": "1.9.2",
+  "version": "1.10.1",
   "description": "A shared web extraction layer for AI conversations and regular web pages.",
   "type": "module",
   "engines": {

package/utils/html-to-markdown.js

@@ -192,28 +192,63 @@ export function convertToMarkdown(htmlContent, options = {}) {
       registerMath(el, latex, isBlock, clone.ownerDocument);
     });
 
-    // 3. Process Google Search SGE LaTeX images with [data-xpm-latex]
-    clone.querySelectorAll("[data-xpm-latex]").forEach((el) => {
-      if (!clone.contains(el)) return;
-      const copyRoot = el.closest("[data-xpm-copy-root]");
-      if (!copyRoot) return;
-      const container = el.closest(".cPGBZb") || copyRoot;
-      if (!container.parentNode) return;
-
-      const latex = el.getAttribute("data-xpm-latex");
-
-      // Determine if it is block math
-      let isBlock = false;
-      const parent = container.parentNode;
-      if (parent) {
-        const parentText = parent.textContent
-          .replace(container.textContent, "")
-          .trim();
-        isBlock = parentText === "";
-      }
+    // 3. Process Google Search SGE LaTeX images with [data-xpm-latex] or fallback math roots
+    clone
+      .querySelectorAll(
+        "[data-xpm-latex], [data-xpm-copy-root][data-xpm-copy-text], [data-xpm-copy-root] img[alt]",
+      )
+      .forEach((el) => {
+        if (!clone.contains(el)) return;
+        const copyRoot = el.hasAttribute("data-xpm-copy-root")
+          ? el
+          : el.closest("[data-xpm-copy-root]");
+        if (!copyRoot) return;
+        const blockContainer = el.closest(".cPGBZb");
+        const inlineWrapper = el.closest(".mTEjhd") || el.closest(".dteT0b");
+        const container = blockContainer || inlineWrapper || copyRoot;
+        if (!container.parentNode) return;
+
+        const img = el.tagName === "IMG" ? el : copyRoot.querySelector("img");
+        const latex =
+          copyRoot.getAttribute("data-xpm-latex") ||
+          el.getAttribute("data-xpm-latex") ||
+          copyRoot.getAttribute("data-xpm-copy-text") ||
+          (img && img.getAttribute("data-xpm-latex")) ||
+          el.getAttribute("alt") ||
+          (img && img.getAttribute("alt")) ||
+          "";
+        if (!latex) return;
+
+        // Determine if block or inline based on DOM structure
+        let isBlock = false;
+        if (blockContainer) {
+          // Full block container (.cPGBZb)
+          isBlock = true;
+        } else if (inlineWrapper) {
+          // Inline math wrapper (.mTEjhd, .dteT0b)
+          isBlock = false;
+        } else {
+          const style = copyRoot.getAttribute("style") || "";
+          if (/display:\s*inline/i.test(style)) {
+            isBlock = false;
+          } else {
+            // Check enclosing block element (p, li, td, th, div) for surrounding text
+            const enclosingBlock = container.closest("p, li, td, th, div");
+            if (enclosingBlock) {
+              const cloneBlock = enclosingBlock.cloneNode(true);
+              const targetInClone =
+                cloneBlock
+                  .querySelector("[data-xpm-latex]")
+                  ?.closest("[data-xpm-copy-root]") ||
+                cloneBlock.querySelector("[data-xpm-latex]");
+              if (targetInClone) targetInClone.remove();
+              isBlock = cloneBlock.textContent.trim() === "";
+            }
+          }
+        }
 
-      registerMath(container, latex, isBlock, clone.ownerDocument);
-    });
+        registerMath(container, latex, isBlock, clone.ownerDocument);
+      });
 
     // 4. Process block display KaTeX (.katex-display)
     clone.querySelectorAll(".katex-display").forEach((el) => {