rssany 0.1.0 → 0.1.2

package/.env.example

@@ -33,7 +33,7 @@ SMTP_PASS=your-smtp-password
 # Resend API Key（driver=resend 时使用；https://resend.com/api-keys）
 RESEND_API_KEY=re_xxxxxxxxxxxxxxxxxxxx
 
-# ─── LLM（研究报告模式用）──────────────────────────────────────────────────
+# ─── LLM（解析 / Pipeline 等；也可在管理后台「设置 → LLM」写入 .rssany/config.json）──
 OPENAI_API_KEY=sk-...
 OPENAI_BASE_URL=https://api.openai.com/v1
 OPENAI_MODEL=gpt-4o-mini

package/README.md

@@ -5,13 +5,17 @@
 
 **RSSAny** 是一套自托管的订阅管线：列表 URL → **抓取与解析**（规则 / LLM）→ **正文提取**（自定义 / Readability / LLM）→ **upsert 去重** → 固定 **pipeline**（打标签、翻译等）→ 对外提供 `**/rss`** 等输出。
 
+## 界面预览
+
+![RSSAny Web 界面截图](images/ScreenShot_2026-04-10_153307_558.png)
+
 ---
 
 ## 功能概览
 
 - **统一订阅**：在 `.rssany/sources.json` 中配置网站列表、标准 RSS、IMAP 邮件等，由调度器按 `refresh` 策略拉取。
-- **可插拔信源**：`plugins/sources/` 与 `.rssany/plugins/sources/` 中的 **Site** 插件（`.rssany.js` / `.rssany.ts`），自定义列表解析与详情规则。
-- **正文补全**：可选 **enrich** 插件拉全文；无 enrich 时入库后仍会跑 pipeline。
+- **可插拔信源**：`app/plugins/builtin/` 与 `.rssany/plugins/` 中的 **Site** 插件（`.rssany.js` / `.rssany.ts`），自定义列表解析与详情规则。
+- **正文与解析**：在信源 `fetchItems`（及需要的 `ctx.extractItem` 等）内完成；入库后跑 pipeline。
 - **固定 pipeline**：`app/pipeline/` 中打标签、翻译等，由 `.rssany/config.json` 的 `pipeline.steps` 开关（**不是**用户目录下的 pipeline 插件）。
 - **LLM 辅助**：解析、提取、标签、翻译等可按配置走 OpenAI 兼容接口。
 - **站点登录**：需登录的站点通过 Puppeteer 管理 Cookie（与产品用户账号无关）。
@@ -56,7 +60,7 @@ pnpm run webui:install
   ```bash
    cp .env.example .env
   ```
-2. 信源与全局配置：首次启动会在 **`~/.rssany/`**（Windows：`%USERPROFILE%\.rssany\`）下自动从包内示例生成 `sources.json`、`config.json`（若已存在则不会覆盖）。也可手动复制仓库里的 `sources.example.json`、`config.examples.json`。
+2. 信源与全局配置：首次启动会在 **`~/.rssany/`**（Windows：`%USERPROFILE%\.rssany\`）下自动从包内 **`init/`** 目录中的默认数据复制生成 `sources.json`、`config.json`（若已存在则不会覆盖）。也可手动复制仓库里的 `init/sources.json`、`init/config.json`。
 3. （可选）LLM：在 `.env` 中设置 `OPENAI_API_KEY`、`OPENAI_BASE_URL`、`OPENAI_MODEL` 等。
 
 ### 运行
@@ -99,7 +103,7 @@ rssany
 
 重置数据（结束 `PORT` 监听进程并删除用户目录）：**`rssany reset`**（与仓库内 **`pnpm reset`** 相同逻辑；可在含 `.env` 的目录下执行以读取 `PORT` / `RSSANY_USER_DIR`）。
 
-用户数据在 **`~/.rssany/`**（Windows：`%USERPROFILE%\.rssany`），与工作目录无关。可选环境变量 **`RSSANY_USER_DIR`** 可指定其它路径。等价于 `node node_modules/rssany/dist/index.js`；CLI 名称为 `rssany`。内置 `plugins/`、`statics/`、`webui/build` 随包安装路径解析。
+用户数据在 **`~/.rssany/`**（Windows：`%USERPROFILE%\.rssany`），与工作目录无关。可选环境变量 **`RSSANY_USER_DIR`** 可指定其它路径。等价于 `node node_modules/rssany/dist/index.js`；CLI 名称为 `rssany`。内置 `app/plugins/builtin/`、`statics/`、`webui/build` 随包安装路径解析。
 
 ---
 
@@ -109,7 +113,6 @@ rssany
 sources.json / Site 插件
   → 调度器触发 fetchItems
   → upsertItems
-  → [可选] enrich 队列
   → pipeline（每条一次）
   → [可选] deliver.url POST（出站，非入站 API）
 ```
@@ -131,11 +134,7 @@ sources.json / Site 插件
 
 ### 信源插件（Site）
 
-放置于 `**plugins/sources/`** 或 `**.rssany/plugins/sources/**`，用户插件可与内置插件同 `id` 覆盖。最小约定包括 `id`、`listUrlPattern` 等（详见 `app/scraper/sources/web/site.ts`）。
-
-### Enrich 插件
-
-`**plugins/enrich/**`、`**.rssany/plugins/enrich/**`，按 enrich 管线加载。
+放置于 `**app/plugins/builtin/**` 或 `**.rssany/plugins/**`（扁平），用户插件可与内置插件同 `id` 覆盖。最小约定包括 `id`、`listUrlPattern` 等（详见 `app/scraper/sources/web/site.ts`）。
 
 ### Pipeline（固定代码）
 
@@ -175,7 +174,7 @@ sources.json / Site 插件
 
 ```
 ├── app/                 # 后端：路由、feeder、scraper、pipeline、mcp、db、auth…
-├── plugins/             # 内置信源 / enrich 等插件
+│   └── plugins/builtin/ # 内置信源 *.rssany.js
 └── webui/               # SvelteKit 前端
 
 ~/.rssany/               # 运行时用户数据（首次启动创建；或 RSSANY_USER_DIR）

package/{plugins/sources → app/plugins/builtin}/email.rssany.js

@@ -1,96 +1,92 @@
-// 内置 IMAP 邮件插件：匹配 imap://、imaps:// 协议 URL
-
-import { ImapFlow } from "imapflow";
-import { logger } from "../../app/core/logger/index.js";
-import { simpleParser } from "mailparser";
-import { createHash } from "node:crypto";
-
-function parseImapUrl(sourceId) {
-  const url = new URL(sourceId);
-  const host = url.hostname;
-  const port = url.port ? parseInt(url.port, 10) : 993;
-  const secure = url.protocol === "imaps:" || port === 993;
-  const user = decodeURIComponent(url.username);
-  const pass = decodeURIComponent(url.password);
-  const folder = decodeURIComponent(url.pathname.slice(1)) || "INBOX";
-  const limit = Math.max(1, parseInt(url.searchParams.get("limit") ?? "30", 10));
-  return { host, port, secure, user, pass, folder, limit };
-}
-
-function makeGuid(messageId, uid, host) {
-  const raw = messageId ?? `${uid}@${host}`;
-  return createHash("sha256").update(raw).digest("hex");
-}
-
-export default {
-  id: "__email__",
-  pattern: /^imaps?:\/\//,
-  priority: 0,
-  refreshInterval: "30min",
-  async fetchItems(sourceId, _ctx) {
-    const { host, port, secure, user, pass, folder, limit } = parseImapUrl(sourceId);
-    const client = new ImapFlow({
-      host,
-      port,
-      secure,
-      auth: { user, pass },
-      logger: false,
-    });
-
-    client.on("error", (err) => {
-      logger.error("source", "IMAP 连接异常", { err: err?.message, host, folder });
-    });
-
-    const items = [];
-    let connected = false;
-    try {
-      await client.connect();
-      connected = true;
-      const lock = await client.getMailboxLock(folder);
-      try {
-        const mailbox = client.mailbox;
-        if (mailbox === false) return [];
-        const total = mailbox.exists ?? 0;
-        if (total === 0) return [];
-        const start = Math.max(1, total - limit + 1);
-        for await (const msg of client.fetch(`${start}:*`, { source: true, envelope: true })) {
-          try {
-            if (msg.source === undefined || msg.envelope === undefined) continue;
-            const parsed = await simpleParser(msg.source);
-            const envelope = msg.envelope;
-            const guid = makeGuid(envelope.messageId, msg.uid, host);
-            const title = parsed.subject ?? envelope.subject ?? "(无主题)";
-            const fromAddr = envelope.from?.[0];
-            const authorRaw = fromAddr?.name || fromAddr?.address || undefined;
-            const author = authorRaw ? [authorRaw] : undefined;
-            const pubDate = parsed.date ?? envelope.date ?? new Date();
-            const link = `imap://${host}/${encodeURIComponent(folder)}#${msg.uid}`;
-            const htmlBody = typeof parsed.html === "string" ? parsed.html : undefined;
-            const textBody = typeof parsed.text === "string" ? parsed.text : undefined;
-            const content = htmlBody ?? (textBody ? `<pre>${textBody}</pre>` : undefined);
-            const summary = textBody?.slice(0, 300) || undefined;
-            items.push({ guid, title, link, pubDate, author, summary, content });
-          } catch (err) {
-            logger.warn("source", "解析单封邮件失败", { err: err?.message });
-          }
-        }
-      } finally {
-        lock.release();
-      }
-    } catch (err) {
-      logger.warn("source", "拉取 IMAP 邮件失败", { err: err?.message, host, folder });
-      return [];
-    } finally {
-      if (connected && client.usable) {
-        try {
-          await client.logout();
-        } catch (err) {
-          logger.warn("source", "IMAP 退出连接失败", { err: err?.message, host, folder });
-        }
-      } else {
-        client.close();
-      }
-    }
-    return items.sort((a, b) => b.pubDate.getTime() - a.pubDate.getTime());
-  },
-};
+// 内置 IMAP 邮件插件：匹配 imap://、imaps:// 协议 URL
+
+function parseImapUrl(sourceId) {
+  const url = new URL(sourceId);
+  const host = url.hostname;
+  const port = url.port ? parseInt(url.port, 10) : 993;
+  const secure = url.protocol === "imaps:" || port === 993;
+  const user = decodeURIComponent(url.username);
+  const pass = decodeURIComponent(url.password);
+  const folder = decodeURIComponent(url.pathname.slice(1)) || "INBOX";
+  const limit = Math.max(1, parseInt(url.searchParams.get("limit") ?? "30", 10));
+  return { host, port, secure, user, pass, folder, limit };
+}
+
+function makeGuid(messageId, uid, host, createHash) {
+  const raw = messageId ?? `${uid}@${host}`;
+  return createHash("sha256").update(raw).digest("hex");
+}
+
+export default {
+  id: "__email__",
+  pattern: /^imaps?:\/\//,
+  priority: 0,
+  refreshInterval: "30min",
+  async fetchItems(sourceId, ctx) {
+    const { deps } = ctx;
+    const { host, port, secure, user, pass, folder, limit } = parseImapUrl(sourceId);
+    const client = new deps.ImapFlow({
+      host,
+      port,
+      secure,
+      auth: { user, pass },
+      logger: false,
+    });
+
+    client.on("error", (err) => {
+      deps.logger.error("source", "IMAP 连接异常", { err: err?.message, host, folder });
+    });
+
+    const items = [];
+    let connected = false;
+    try {
+      await client.connect();
+      connected = true;
+      const lock = await client.getMailboxLock(folder);
+      try {
+        const mailbox = client.mailbox;
+        if (mailbox === false) return [];
+        const total = mailbox.exists ?? 0;
+        if (total === 0) return [];
+        const start = Math.max(1, total - limit + 1);
+        for await (const msg of client.fetch(`${start}:*`, { source: true, envelope: true })) {
+          try {
+            if (msg.source === undefined || msg.envelope === undefined) continue;
+            const parsed = await deps.simpleParser(msg.source);
+            const envelope = msg.envelope;
+            const guid = makeGuid(envelope.messageId, msg.uid, host, deps.createHash);
+            const title = parsed.subject ?? envelope.subject ?? "(无主题)";
+            const fromAddr = envelope.from?.[0];
+            const authorRaw = fromAddr?.name || fromAddr?.address || undefined;
+            const author = authorRaw ? [authorRaw] : undefined;
+            const pubDate = parsed.date ?? envelope.date ?? new Date();
+            const link = `imap://${host}/${encodeURIComponent(folder)}#${msg.uid}`;
+            const htmlBody = typeof parsed.html === "string" ? parsed.html : undefined;
+            const textBody = typeof parsed.text === "string" ? parsed.text : undefined;
+            const content = htmlBody ?? (textBody ? `<pre>${textBody}</pre>` : undefined);
+            const summary = textBody?.slice(0, 300) || undefined;
+            items.push({ guid, title, link, pubDate, author, summary, content });
+          } catch (err) {
+            deps.logger.warn("source", "解析单封邮件失败", { err: err?.message });
+          }
+        }
+      } finally {
+        lock.release();
+      }
+    } catch (err) {
+      deps.logger.warn("source", "拉取 IMAP 邮件失败", { err: err?.message, host, folder });
+      return [];
+    } finally {
+      if (connected && client.usable) {
+        try {
+          await client.logout();
+        } catch (err) {
+          deps.logger.warn("source", "IMAP 退出连接失败", { err: err?.message, host, folder });
+        }
+      } else {
+        client.close();
+      }
+    }
+    return items.sort((a, b) => b.pubDate.getTime() - a.pubDate.getTime());
+  },
+};

package/app/plugins/builtin/rss.rssany.js

@@ -0,0 +1,164 @@
+// 内置 RSS/Atom/JSON Feed：通过浏览器（Puppeteer）拉取 Feed URL，再用 rss-parser 解析；
+// 与站点插件一致走 Chrome，便于应对需浏览器环境或代理的场景；XML 使用 HTTP 响应原文（useHttpResponseBody）。
+
+const UA = "RssAny/1.0 (+https://github.com/joohw/rssany)";
+
+const IMAGE_TYPE_RE = /^image\//i;
+const IMAGE_EXT_IN_PATH_RE = /\.(jpg|jpeg|png|gif|webp|avif|svg)(\?|#|$)/i;
+
+function trimUrl(s) {
+  if (typeof s !== "string") return undefined;
+  const t = s.trim();
+  return t || undefined;
+}
+
+/** 从 rss-parser 条目上尽量取出配图 URL（入库用 imageUrl，与 Gateway 的 cover_img 对齐）。 */
+function extractItemImageUrl(item) {
+  const enc = item.enclosure;
+  if (enc && typeof enc.url === "string") {
+    const u = trimUrl(enc.url);
+    const t = typeof enc.type === "string" ? enc.type : "";
+    if (u && (IMAGE_TYPE_RE.test(t) || (!t && IMAGE_EXT_IN_PATH_RE.test(u)))) {
+      return u;
+    }
+  }
+
+  const itunesImg = item.itunes && typeof item.itunes.image === "string" ? item.itunes.image : undefined;
+  const fromItunes = trimUrl(itunesImg);
+  if (fromItunes) return fromItunes;
+
+  const thumbs = item.mediaThumbnail;
+  if (Array.isArray(thumbs) && thumbs[0]?.$) {
+    const u = trimUrl(thumbs[0].$.url ?? thumbs[0].$.href);
+    if (u) return u;
+  }
+
+  const mediaBlocks = item.mediaContent;
+  if (Array.isArray(mediaBlocks)) {
+    for (const block of mediaBlocks) {
+      const $ = block && block.$;
+      if (!$ || typeof $.url !== "string") continue;
+      const medium = $.medium;
+      const ctype = typeof $.type === "string" ? $.type : "";
+      if (medium === "image" || IMAGE_TYPE_RE.test(ctype)) {
+        const u = trimUrl($.url);
+        if (u) return u;
+      }
+    }
+  }
+
+  const atomLinks = item.atomLinks;
+  if (Array.isArray(atomLinks)) {
+    for (const l of atomLinks) {
+      const $ = l && l.$;
+      if (!$ || typeof $.href !== "string") continue;
+      const rel = String($.rel || "").toLowerCase();
+      const ctype = String($.type || "").toLowerCase();
+      if (rel === "enclosure" && ctype.startsWith("image/")) {
+        const u = trimUrl($.href);
+        if (u) return u;
+      }
+    }
+  }
+
+  const fromHtml =
+    firstImgSrcFromHtml(item.content) ||
+    firstImgSrcFromHtml(item.summary) ||
+    firstImgSrcFromHtml(item["content:encoded"]) ||
+    firstImgSrcFromHtml(item.contentSnippet);
+  if (fromHtml && /^https?:\/\//i.test(fromHtml)) {
+    return fromHtml;
+  }
+
+  return undefined;
+}
+
+function firstImgSrcFromHtml(html) {
+  if (typeof html !== "string" || !html) return undefined;
+  const m = html.match(/<img[^>]+src\s*=\s*["']([^"']+)["']/i);
+  return m ? trimUrl(m[1]) : undefined;
+}
+
+async function fetchFeedXml(url, ctx) {
+  const fetchHtml = ctx.fetchHtml;
+  if (typeof fetchHtml !== "function") {
+    throw new Error("RSS 插件需要 ctx.fetchHtml（请通过 feeder / buildSourceContext 调用）");
+  }
+  const { html } = await fetchHtml(url, {
+    waitMs: 800,
+    purify: false,
+    useHttpResponseBody: true,
+  });
+  return html;
+}
+
+export default {
+  id: "__rss__",
+  pattern: /^https?:\/\//,
+  match: looksLikeFeed,
+  priority: 20,
+  refreshInterval: "1h",
+  async fetchItems(sourceId, ctx) {
+    const { deps } = ctx;
+    const xml = await fetchFeedXml(sourceId, ctx);
+    const parser = new deps.RssParser({
+      timeout: 30_000,
+      headers: {
+        "User-Agent": UA,
+        Accept: "application/rss+xml,application/atom+xml,application/json,application/xml,text/xml,*/*",
+      },
+      customFields: {
+        item: [
+          ["media:thumbnail", "mediaThumbnail", { keepArray: true }],
+          ["media:content", "mediaContent", { keepArray: true }],
+          ["link", "atomLinks", { keepArray: true }],
+        ],
+      },
+    });
+    const feed = await parser.parseString(xml);
+    return (feed.items ?? []).map((item) => {
+      const link = item.link ?? item.guid ?? sourceId;
+      const guid = item.guid ?? deps.createHash("sha256").update(link).digest("hex");
+      const pubDate =
+        item.pubDate != null
+          ? new Date(item.pubDate)
+          : item.isoDate != null
+            ? new Date(item.isoDate)
+            : new Date();
+      const authorRaw =
+        typeof item.creator === "string" ? item.creator : typeof item.author === "string" ? item.author : undefined;
+      const author = authorRaw ? [authorRaw] : undefined;
+      const summary =
+        typeof item.summary === "string" ? item.summary : typeof item.contentSnippet === "string" ? item.contentSnippet : undefined;
+      const content =
+        typeof item.content === "string" ? item.content : typeof item["content:encoded"] === "string" ? item["content:encoded"] : undefined;
+      const imageUrl = extractItemImageUrl(item);
+      const base = {
+        guid,
+        title: item.title ?? "",
+        link,
+        pubDate,
+        author,
+        summary,
+        content,
+      };
+      if (!imageUrl) return base;
+      return { ...base, imageUrl, cover_img: imageUrl };
+    });
+  },
+};
+
+function looksLikeFeed(url) {
+  const lower = url.toLowerCase();
+  return (
+    lower.includes("/feed") ||
+    lower.includes("/rss") ||
+    lower.includes("/atom") ||
+    lower.endsWith(".xml") ||
+    lower.endsWith(".rss") ||
+    lower.endsWith(".atom") ||
+    lower.includes("format=rss") ||
+    lower.includes("format=atom") ||
+    lower.includes("output=rss")
+  );
+}

package/{plugins/templates → app/plugins}/site.rssany.js

@@ -1,5 +1,5 @@
 /**
- * Site 插件模板（管理页「添加插件」会复制到 `.rssany/plugins/sources/{id}.rssany.ts`）
+ * Site 插件模板（管理页「添加插件」会复制到 `.rssany/plugins/{id}.rssany.js`）
  * 修改 `id` 后请与文件名保持一致。
  *
  * 接口说明：app/scraper/sources/web/site.ts
@@ -7,20 +7,19 @@
 
 export default {
   id: "__PLUGIN_ID__",
-  listUrlPattern: "https://example.com/{segment}",
+  listUrlPattern: __LIST_URL_PATTERN__,
   refreshInterval: "1day",
 
-  /** sourceId 与订阅里 ref 一致；ctx 含 fetchHtml、extractItem 等 */
+  /** sourceId 与订阅里 ref 一致；ctx 含 fetchHtml、extractItem、deps（parseHtml 等） */
   async fetchItems(sourceId, ctx) {
     const { html, finalUrl } = await ctx.fetchHtml(sourceId, {
       waitMs: 2000,
       purify: true,
     });
-    void html;
+    const root = ctx.deps.parseHtml(html);
+    void root;
     void finalUrl;
-    // TODO: 解析列表页 HTML，产出 { title, link, summary?, pubDate? } 等 FeedItem
+    // TODO: 用 ctx.deps.parseHtml 解析列表页，产出 { title, link, summary?, pubDate? } 等 FeedItem
     return [];
   },
-
-  // enrichItem: async (item, ctx) => ctx.extractItem(item),
 };