memex-mvp 0.5.3 → 0.6.0

package/HELP.md

@@ -227,12 +227,60 @@ Memex по дефолту сортирует по **релевантности**
 
 ---
 
+### 8. 🔗 Сохранение URL'ов в memex (Perplexity, статьи, AI-share'ы)
+
+Ты читаешь что-то — Perplexity research thread, длинную статью, GitHub-обсуждение, AI-chat share — и хочешь чтобы это жило в memex-памяти, искалось из любого AI-чата.
+
+**В любом MCP-агенте (Claude Code, Cursor, Cline, Continue, Zed):**
+
+```
+Сохрани https://perplexity.ai/share/<id> в memex
+Добавь эту статью в memex: https://example.com/great-post
+Захвати этот ChatGPT-разговор: https://chat.openai.com/share/<id>
+```
+
+**Что происходит за кулисами:**
+
+1. Агент сам делает fetch URL'a (через свой WebFetch)
+2. Если страница защищена Cloudflare (Perplexity, npm.com, Twitter, Medium…) — агент авто-retry через `r.jina.ai` proxy (бесплатный JS-runtime, обходит Cloudflare)
+3. Агент вызывает `memex_store_document(content, url, title)`
+4. Memex сохраняет содержимое как conversation с `source: "web"` — ищется через `memex_search` рядом с AI-чатами
+
+**Для Perplexity-thread'ов:** thread должен быть **PUBLIC**. В Perplexity: открой thread → Share → toggle "Public link" → скопируй новый URL → дай его агенту. URL из адресной строки браузера (`perplexity.ai/search/<id>`) — это **твой owner-URL, не shareable**.
+
+Если забудешь — memex детектит «private» в ответе и agent тебе явно скажет что делать.
+
+**Login-walled или paywalled контент не fetch'нется** (NYT subscription, твои приватные ChatGPT-чаты). Для них вставь контент руками:
+
+```
+Сохрани этот текст в memex (название: "..."): <вставь содержимое>
+```
+
+**Tag'и при сохранении** — для последующей фильтрации:
+
+```
+Сохрани https://... в memex, поставь теги "research" и "perplexity"
+```
+
+**Поиск:**
+
+```
+Найди в memex что Perplexity говорил про X на прошлой неделе
+```
+
+`memex_search` возвращает совпадения **и из AI-чатов, и из сохранённых URL'ов** в одном запросе — отсортировано по релевантности или дате.
+
+**Memex принципиально НЕ делает outbound network calls.** Fetcher живёт в твоём AI-агенте. Если он использует Jina для обхода Cloudflare — Jina видит URL (но НЕ остальной memex-корпус). Это выбор агента, не memex'a.
+
+---
+
 ## Какие MCP-tools агент может вызвать
 
 | Tool | Что делает |
 |---|---|
 | `memex_overview` | Снэпшот корпуса: источники, сколько сообщений, последние чаты, статус auto-capture |
 | `memex_search(query)` | Полнотекстовый поиск (FTS5) с recency boost'ом. Параметры: `project`, `source`, `chat`, `half_life_days`, `expand_match`, `sort` |
+| `memex_store_document(content, url?, title?)` | Сохранить внешний документ (web-страница, AI-chat share, paste) в memex. Агент сам делает fetch, memex хранит verbatim. Учит Jina-трюк для Cloudflare-страниц |
 | `memex_list_projects` | Список всех проектов с количеством разговоров |
 | `memex_list_conversations` | Список чатов отсортированных по recency |
 | `memex_get_conversation(id)` | Полный transcript одного чата |

package/README.md

@@ -48,6 +48,24 @@ Or use one-shot `sudo npm install -g memex-mvp`.
 npx memex-mvp install
 ```
 
+### Install via AI skill (Claude Code / OpenClaw)
+
+If you'd rather have an AI agent walk you through everything, drop the
+[install-memex skill](skills/install-memex/) into `~/.claude/skills/`:
+
+```sh
+mkdir -p ~/.claude/skills
+curl -fsSL https://raw.githubusercontent.com/parallelclaw/memex-mvp/main/skills/install-memex/SKILL.md \
+  -o ~/.claude/skills/install-memex/SKILL.md
+```
+
+Then in Claude Code (or any Skills-aware agent) just say:
+
+> install memex
+
+…or `/install-memex`. The agent handles `npm install`, MCP-config wiring,
+auto-capture daemon, and verification — ~2 minutes.
+
 ---
 
 ## Connect to your MCP client

package/README.ru.md

@@ -105,6 +105,22 @@ sudo npm install -g memex-mvp
 
 После установки `memex-sync install` поднимет фоновый daemon (`~/.memex/{inbox,data}/` создадутся автоматически при первом запуске).
 
+### Установка через AI-скилл (Claude Code / OpenClaw)
+
+Если хочешь чтобы агент сам всё сделал — закинь [install-memex skill](skills/install-memex/) в `~/.claude/skills/`:
+
+```bash
+mkdir -p ~/.claude/skills
+curl -fsSL https://raw.githubusercontent.com/parallelclaw/memex-mvp/main/skills/install-memex/SKILL.md \
+  -o ~/.claude/skills/install-memex/SKILL.md
+```
+
+Затем в Claude Code (или любом Skills-aware агенте) скажи:
+
+> установи memex
+
+…или `/install-memex`. Агент сам сделает `npm install`, пропишет MCP-config, поднимет daemon и проверит что всё работает — ~2 минуты.
+
 ### Подключение к Claude Code
 
 Сначала возьми **два абсолютных пути** в терминале:

package/lib/store-doc/canonicalize.js

@@ -0,0 +1,116 @@
+/**
+ * URL canonicalization for stable deduplication of stored web documents.
+ *
+ * Goal: two URLs that point to "the same document" should map to the same
+ * canonical form, so memex_store_document gives them the same conversation_id
+ * via sha256(canonical).
+ *
+ * What we normalize:
+ *   - Lowercase scheme + host
+ *   - Strip known tracking params (utm_*, fbclid, gclid, ref, mc_*, _ga, …)
+ *   - Drop the fragment (#anchor) — same document
+ *   - Normalize trailing slash on pathname
+ *
+ * What we DON'T normalize:
+ *   - Path case (some servers are case-sensitive)
+ *   - Non-tracking query params (?q= search, ?id= permalinks — meaningful)
+ *   - Port (rare in public URLs)
+ *
+ * If the input isn't a valid URL, we return the input unchanged. Callers
+ * should still hash the result for deduplication.
+ */
+
+// Well-known tracking-param families. Case-insensitive prefix match.
+const TRACKING_PREFIXES = [
+  'utm_',         // Google Analytics
+  'mc_',          // Mailchimp
+];
+const TRACKING_EXACT = new Set([
+  'fbclid',       // Facebook
+  'gclid',        // Google ads
+  'dclid',        // Google DoubleClick
+  'gbraid',       // Google
+  'wbraid',       // Google
+  'yclid',        // Yandex
+  'msclkid',      // Microsoft ads
+  'twclid',       // Twitter
+  'igshid',       // Instagram
+  'ref',          // generic referrer
+  'ref_source',
+  'ref_url',
+  'referrer',
+  'source',       // common referrer flag (NOT always tracking but very often)
+  '_ga',          // Google Analytics
+  '_gl',          // Google Analytics linker
+  'hsCtaTracking',
+  'hsenc',
+  'hsmi',
+  'mkt_tok',
+  'pk_campaign',
+  'pk_source',
+  'pk_medium',
+  'pk_keyword',
+  'pk_content',
+  'vero_id',
+  'vero_conv',
+]);
+
+function isTrackingParam(name) {
+  const lower = name.toLowerCase();
+  if (TRACKING_EXACT.has(lower)) return true;
+  for (const prefix of TRACKING_PREFIXES) {
+    if (lower.startsWith(prefix)) return true;
+  }
+  return false;
+}
+
+/**
+ * @param {string} rawUrl
+ * @returns {string} canonicalized URL (or the input unchanged if unparseable)
+ */
+export function canonicalize(rawUrl) {
+  if (typeof rawUrl !== 'string' || !rawUrl.trim()) return rawUrl;
+
+  let u;
+  try {
+    u = new URL(rawUrl.trim());
+  } catch (_) {
+    return rawUrl.trim();
+  }
+
+  // Lowercase scheme + host (URL parser already does that, but be explicit)
+  u.protocol = u.protocol.toLowerCase();
+  u.hostname = u.hostname.toLowerCase();
+
+  // Drop the fragment
+  u.hash = '';
+
+  // Strip tracking params
+  const cleanParams = new URLSearchParams();
+  for (const [k, v] of u.searchParams) {
+    if (!isTrackingParam(k)) cleanParams.append(k, v);
+  }
+  u.search = cleanParams.toString();
+
+  // Normalize trailing slash: drop trailing slash on non-root paths,
+  // so /foo and /foo/ are treated as the same document
+  if (u.pathname.length > 1 && u.pathname.endsWith('/')) {
+    u.pathname = u.pathname.replace(/\/+$/, '');
+  }
+
+  return u.toString();
+}
+
+/**
+ * Best-effort domain extraction for metadata (e.g. "perplexity.ai").
+ * Returns null for unparseable URLs.
+ */
+export function extractDomain(rawUrl) {
+  if (typeof rawUrl !== 'string') return null;
+  try {
+    const u = new URL(rawUrl);
+    return u.hostname.toLowerCase().replace(/^www\./, '');
+  } catch (_) {
+    return null;
+  }
+}

package/lib/store-doc/detect.js

@@ -0,0 +1,209 @@
+/**
+ * Pattern detection for memex_store_document.
+ *
+ * When the agent passes content to memex_store_document, memex sniffs it
+ * for known failure signatures (Cloudflare challenge, Perplexity-private,
+ * paywalls, …) and returns actionable warnings.
+ *
+ * Each detector returns either null or an object:
+ *   { type, blocking, message }
+ *
+ * `blocking: true` → memex returns stored:false to the agent. Use only for
+ * clear-cut failures where storing the content would pollute the corpus.
+ * `blocking: false` → memex stores the content but appends the warning so
+ * the agent can decide whether to surface it to the user.
+ *
+ * Patterns may grow over time as new failure modes appear in real use.
+ * Single-purpose regexes — order matters (more specific first).
+ */
+
+const CLOUDFLARE_PATTERNS = [
+  /Just a moment\.\.\./i,
+  /Verifying you are human/i,
+  /cf-turnstile/i,
+  /cf_chl_opt/i,
+  /__cf_bm/i,                      // Cloudflare's bot-management cookie name in challenge pages
+  /Checking your browser before accessing/i,
+];
+
+const PERPLEXITY_PRIVATE_PATTERNS = [
+  /This thread is private/i,
+  /Sign in if you are the owner of this thread/i,
+];
+
+const LOGIN_REQUIRED_PATTERNS = [
+  /Sign in to continue/i,
+  /Please log in to continue/i,
+  /Login required/i,
+  /<form[^>]+action=["']\/?(login|signin|sign-in|auth)/i,
+  /You must be logged in/i,
+];
+
+const PAYWALL_PATTERNS = [
+  /Subscribe to (read|continue)/i,
+  /Continue reading with subscription/i,
+  /\bpaywall\b/i,
+  /You've reached your free article limit/i,
+  /metered access/i,
+  /Become a (member|subscriber)/i,
+];
+
+/**
+ * Detect a Cloudflare bot-challenge page. Blocking.
+ */
+function detectCloudflareChallenge(content, _url) {
+  for (const re of CLOUDFLARE_PATTERNS) {
+    if (re.test(content)) {
+      return {
+        type: 'cloudflare-challenge',
+        blocking: true,
+        message:
+          'The content you passed looks like a Cloudflare bot-challenge page, not the actual content. The page is protected by Cloudflare and your direct fetch was blocked.\n\n' +
+          'Retry by prepending https://r.jina.ai/ to the URL:\n\n' +
+          '  WebFetch("https://r.jina.ai/" + originalUrl)\n\n' +
+          'Jina AI Reader runs a real browser server-side and bypasses Cloudflare. Returns clean markdown. Free for personal use, no API key needed.\n\n' +
+          'If Jina ALSO fails or returns "private/login required", the page needs authentication — Jina can\'t bypass that.',
+      };
+    }
+  }
+  return null;
+}
+
+/**
+ * Detect a private Perplexity thread. Blocking — Jina can't help here,
+ * the user needs to make the thread public first.
+ */
+function detectPerplexityPrivate(content, url) {
+  // Only flag if we have a URL hint that it's Perplexity, OR if the message
+  // text is unambiguously Perplexity's phrasing.
+  const isPerplexityUrl =
+    typeof url === 'string' && /perplexity\.ai/i.test(url);
+
+  let matched = false;
+  for (const re of PERPLEXITY_PRIVATE_PATTERNS) {
+    if (re.test(content)) {
+      matched = true;
+      break;
+    }
+  }
+  if (!matched) return null;
+  if (!isPerplexityUrl && !/perplexity/i.test(content)) {
+    // Same phrasing might appear on other sites — only act if we're confident
+    return null;
+  }
+
+  return {
+    type: 'perplexity-private',
+    blocking: true,
+    message:
+      'This Perplexity thread is marked private — even Jina Reader can\'t access it (this is an authentication wall, not Cloudflare bot protection).\n\n' +
+      'Tell the user: "To save this Perplexity thread to memex, you need to make it public first:\n' +
+      '  1. Open the thread in Perplexity\n' +
+      '  2. Click Share (top right)\n' +
+      '  3. Toggle \'Public link\' on\n' +
+      '  4. Copy the new shareable URL Perplexity shows\n' +
+      '  5. Send me THAT URL — it\'ll work"\n\n' +
+      'The URL in the user\'s address bar (perplexity.ai/search/<id>) is the owner\'s private URL, not the shareable one.',
+  };
+}
+
+/**
+ * Suspiciously short content from a URL that should be substantive.
+ * Non-blocking — we store it, but warn.
+ */
+function detectSuspiciouslySmall(content, url) {
+  const trimmed = (content || '').trim();
+  // Threshold: documents shorter than 200 chars are almost certainly noise
+  // (error pages, redirects, JS-only stubs). Pasted snippets can legitimately
+  // be that short, so only flag when we have a URL (suggesting a fetch was
+  // attempted) — pastes get a free pass.
+  if (!url) return null;
+  if (trimmed.length >= 200) return null;
+  return {
+    type: 'suspiciously-small',
+    blocking: false,
+    message:
+      `The content you passed is very short (${trimmed.length} chars). ` +
+      'The page might have been blocked, redirect-failed, or be JS-rendered with no SSR. ' +
+      'Stored as-is — consider verifying with the user that this is what they expected.',
+  };
+}
+
+/**
+ * Login required (form / prompt). Non-blocking but worth flagging.
+ */
+function detectLoginRequired(content, _url) {
+  for (const re of LOGIN_REQUIRED_PATTERNS) {
+    if (re.test(content)) {
+      return {
+        type: 'login-required',
+        blocking: false,
+        message:
+          'The page appears to require login (sign-in prompt / login form detected). ' +
+          'The content you stored may be a login page, not the actual content the user wanted. ' +
+          'Ask the user to paste the content manually if this isn\'t what they expected.',
+      };
+    }
+  }
+  return null;
+}
+
+/**
+ * Paywall / subscription-gated content. Non-blocking.
+ */
+function detectPaywalled(content, _url) {
+  for (const re of PAYWALL_PATTERNS) {
+    if (re.test(content)) {
+      return {
+        type: 'paywalled',
+        blocking: false,
+        message:
+          'The page appears to be paywalled (subscription/payment prompt detected). ' +
+          'The content stored may just be the teaser. ' +
+          'If the user has full access, they can paste the complete article manually.',
+      };
+    }
+  }
+  return null;
+}
+
+/**
+ * Returns array of warnings sorted with blocking warnings first.
+ * If the first warning is blocking, memex should refuse the store
+ * and return that warning to the agent.
+ *
+ * Detectors run in this order (more-specific first):
+ *   1. cloudflare-challenge  (blocking)
+ *   2. perplexity-private    (blocking)
+ *   3. suspiciously-small    (non-blocking)
+ *   4. login-required        (non-blocking)
+ *   5. paywalled             (non-blocking)
+ */
+export function detectIssues(content, url) {
+  const safeContent = typeof content === 'string' ? content : '';
+  const warnings = [];
+
+  // Blocking first — stop on first hit so we surface the most actionable.
+  const blocking =
+    detectCloudflareChallenge(safeContent, url) ||
+    detectPerplexityPrivate(safeContent, url);
+  if (blocking) {
+    warnings.push(blocking);
+    return warnings;
+  }
+
+  // Non-blocking — collect all that match.
+  for (const fn of [detectSuspiciouslySmall, detectLoginRequired, detectPaywalled]) {
+    const w = fn(safeContent, url);
+    if (w) warnings.push(w);
+  }
+
+  return warnings;
+}
+
+/**
+ * Convenience: is any warning blocking?
+ */
+export function isBlocked(warnings) {
+  return Array.isArray(warnings) && warnings.some((w) => w.blocking);
+}

package/lib/store-doc/extract-title.js

@@ -0,0 +1,116 @@
+/**
+ * Extract a title from fetched page content.
+ *
+ * Strategy (first hit wins):
+ *   1. Markdown H1 — `# Title text`  (Jina Reader's output starts with this)
+ *   2. HTML <title> — `<title>Page Title</title>`
+ *   3. HTML <h1>  — `<h1>Page Title</h1>`
+ *   4. First non-empty line if short enough to look like a title
+ *   5. URL slug fallback — last meaningful path segment, decoded
+ *   6. Domain fallback — just the domain name
+ *   7. "Untitled document"
+ *
+ * Returns a trimmed string up to MAX_LEN characters. Always returns a
+ * non-empty string (worst case "Untitled document").
+ */
+
+const MAX_LEN = 200;
+
+function trimTitle(s) {
+  if (!s) return '';
+  let t = String(s).replace(/\s+/g, ' ').trim();
+  if (t.length > MAX_LEN) t = t.slice(0, MAX_LEN).trim() + '…';
+  return t;
+}
+
+function fromMarkdownH1(content) {
+  // Markdown H1: line starts with single # then space, then text.
+  // Use \r? for cross-platform line endings. Stop at end-of-line.
+  const m = content.match(/^[ \t]*#[ \t]+([^\r\n]+?)[ \t]*$/m);
+  return m ? trimTitle(m[1]) : '';
+}
+
+function fromHtmlTitle(content) {
+  const m = content.match(/<title[^>]*>([^<]+)<\/title>/i);
+  return m ? trimTitle(decodeEntities(m[1])) : '';
+}
+
+function fromHtmlH1(content) {
+  // Inner text only — strip nested tags like <span>...</span>
+  const m = content.match(/<h1[^>]*>([\s\S]*?)<\/h1>/i);
+  if (!m) return '';
+  const inner = m[1].replace(/<[^>]+>/g, '');
+  return trimTitle(decodeEntities(inner));
+}
+
+function fromFirstLine(content) {
+  // First non-empty line, but only if it looks like a heading
+  // (short-ish, no markdown junk).
+  const lines = content.split(/\r?\n/);
+  for (const raw of lines) {
+    const line = raw.trim();
+    if (!line) continue;
+    // Skip leading markdown decorators / metadata
+    if (/^[#\-=*>|`]/.test(line)) continue;
+    if (line.length > 0 && line.length <= 120) {
+      return trimTitle(line);
+    }
+    // First substantive line is too long — give up on this strategy
+    break;
+  }
+  return '';
+}
+
+function fromUrlSlug(rawUrl) {
+  if (!rawUrl) return '';
+  try {
+    const u = new URL(rawUrl);
+    // Last meaningful path segment
+    const segs = u.pathname.split('/').filter(Boolean);
+    if (segs.length) {
+      const slug = decodeURIComponent(segs[segs.length - 1])
+        .replace(/[-_]+/g, ' ')
+        .replace(/\.(html?|md|pdf|txt)$/i, '')
+        .trim();
+      if (slug) return trimTitle(slug);
+    }
+    // No useful path — fall through to domain
+    return trimTitle(u.hostname.replace(/^www\./, ''));
+  } catch (_) {
+    return '';
+  }
+}
+
+// Minimal HTML-entity decode for &amp; &lt; &gt; &quot; &apos; &#39; &#nnn;
+function decodeEntities(s) {
+  if (!s) return s;
+  return String(s)
+    .replace(/&amp;/g, '&')
+    .replace(/&lt;/g, '<')
+    .replace(/&gt;/g, '>')
+    .replace(/&quot;/g, '"')
+    .replace(/&apos;/g, "'")
+    .replace(/&#39;/g, "'")
+    .replace(/&#x([0-9a-f]+);/gi, (_, hex) =>
+      String.fromCodePoint(parseInt(hex, 16))
+    )
+    .replace(/&#(\d+);/g, (_, dec) => String.fromCodePoint(parseInt(dec, 10)));
+}
+
+/**
+ * @param {string} content - fetched page content
+ * @param {string|null} url - source URL (used for slug fallback)
+ * @returns {string} a non-empty trimmed title
+ */
+export function extractTitle(content, url) {
+  const safe = typeof content === 'string' ? content : '';
+
+  return (
+    fromMarkdownH1(safe) ||
+    fromHtmlTitle(safe) ||
+    fromHtmlH1(safe) ||
+    fromFirstLine(safe) ||
+    fromUrlSlug(url) ||
+    'Untitled document'
+  );
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memex-mvp",
-  "version": "0.5.3",
+  "version": "0.6.0",
   "description": "Local-first MCP server for cross-agent AI memory. One SQLite + FTS5 corpus across Claude Code, Cowork, Cursor, Continue, Zed, Obsidian, and Telegram — passively captured, verbatim, searchable from any MCP-compatible client.",
   "type": "module",
   "main": "server.js",
@@ -16,6 +16,7 @@
     "ingest.js",
     "lib/",
     "bot/",
+    "skills/",
     "HELP.md",
     "README.md",
     "LICENSE"
@@ -25,7 +26,7 @@
     "sync": "node ingest.js",
     "ingest": "node ingest.js",
     "bot": "node bot/index.js",
-    "test": "node test/parser.test.js && node test/bot-inbox.test.js && node test/search-sort.test.js",
+    "test": "node test/parser.test.js && node test/bot-inbox.test.js && node test/search-sort.test.js && node test/store-document.test.js",
     "prepublishOnly": "npm test"
   },
   "engines": {