sophhub 0.2.3 → 0.3.0

package/skills/sophnet-dailynews/src/SKILL.md

@@ -0,0 +1,179 @@
+---
+name: sophnet-dailynews
+description: Generate a daily, high-signal Markdown news report by scraping preset sources, filtering and deduplicating items. Use when the user asks for daily news, tech digest, today's links, or requests an automated curated report.
+---
+
+# Daily news report (Markdown)
+
+## Quick start
+
+1. Resolve the target date.
+   - Accept `YYYY-MM-DD` from the user.
+   - Default to local today.
+2. Read `sources.json` and `cache.json` **with explicit paths**.
+   - Always call the read tool with a `path` field (not `file_path`).
+   - Use these exact paths:
+     - `read(path="/Data/shutong.shan/clawd/skills/sophnet-dailynews/sources.json")`
+     - `read(path="/Data/shutong.shan/clawd/skills/sophnet-dailynews/cache.json")`
+3. Collect items in waves (Tier 1 → Tier 2 → Tier 3/browser) until the report has enough **high-quality** items.
+4. Write `NewsReport/YYYY-MM-DD-news-report.md`.
+5. Update `cache.json` (dedupe + historical stats).
+
+## Files in this skill
+
+- `sources.json`: tiers, batches, URLs, fetch method (`webfetch`/`browser`), extraction hint, enable/disable flags, and quality thresholds.
+- `cache.json`: last run metadata, per-source stats, URL/content dedupe caches, and per-day article history.
+
+## Common failure: `web_fetch` → `fetch failed`
+
+This usually means the fetch tool hit one of these:
+
+- a redirect it doesn't follow (302)
+- a method mismatch (some sites don't like `HEAD`)
+- bot protection / CDN quirks
+- response too large / slow for the tool limits
+
+This skill is tuned to avoid common failures:
+
+- Prefer **HN RSS** over HTML (`https://news.ycombinator.com/rss`).
+- Pin **HuggingFace Papers** to a concrete date URL to avoid redirects:
+  - `https://huggingface.co/papers/date/{{date}}`
+  - Replace `{{date}}` with the target date (`YYYY-MM-DD`) before fetching.
+
+## Output contract
+
+Produce exactly one Markdown file:
+
+- Path: `NewsReport/YYYY-MM-DD-news-report.md`
+- Target: `quality_thresholds.target_items` items (default 20)
+- Include only items with `quality_score >= quality_thresholds.min_score_to_include` (default 3)
+
+Each item must include:
+
+- `title`
+- `summary` (2–4 sentences, concrete and non-hypey)
+- `key_points` (max 3)
+- `url` (canonical if possible)
+- `source_id`
+- `keywords` (2–6)
+- `quality_score` (1–5, integer)
+
+## Collection workflow
+
+### 1) Initialize
+
+- Load `sources.json` → treat it as the source of truth.
+- Load `cache.json` → use it for dedupe and stats.
+- Create `NewsReport/` if missing.
+- If `NewsReport/YYYY-MM-DD-news-report.md` already exists, either:
+  - regenerate from scratch (preferred), or
+  - append only if explicitly requested.
+
+### 2) Fetch in waves (early stop)
+
+Follow the tier order in `sources.json`:
+
+- **Wave A (Tier 1 / batch_a + batch_b)**: high hit-rate sources first.
+- If you still have fewer than ~15 included items after filtering, continue.
+- **Wave B (Tier 2 / batch_a + batch_b)**: supplemental sources.
+- If you still have fewer than `target_items`, continue.
+- **Wave C (Tier 3 / browser sources)**: JS-rendered / blocked sources.
+
+Stop fetching when you have:
+
+- at least `target_items` included items, and
+- at least `quality_thresholds.early_stop_threshold` total candidates evaluated (default 25), or you have exhausted all enabled sources.
+
+### 3) Extract and normalize
+
+For each enabled source entry:
+
+- Fetch the page content using the configured `fetch_method`:
+  - `webfetch`: normal HTTP fetch.
+  - `browser`: render with a headless browser when available; otherwise skip with a recorded error.
+- Apply the `extract` hint from the source entry (examples: `top_10`, `latest_5`, `latest_issue`, `today_top_5`).
+- For each candidate item, normalize:
+  - clean title (no site suffix noise)
+  - canonical URL (strip tracking params when safe)
+  - short summary + key points
+
+**RSS-first rule (important):**
+
+- For RSS sources, **do not** `web_fetch` the item URLs.
+- Use the RSS item's title + description/summary as the source of truth.
+- If the RSS description is too thin to summarize, either:
+  - drop the item, or
+  - keep it with a one-sentence, non-speculative summary based only on the RSS description.
+- Only fetch item URLs if the source entry explicitly says `extract: fetch_items` (none do by default).
+
+If a `webfetch` attempt fails:
+
+- retry once (respect `fetch_config.webfetch.timeout_ms`)
+- then mark the source as failed and continue (do not abort the whole report)
+
+### 4) Filter, score, and dedupe
+
+Apply these rules in order:
+
+1. Reject obvious low-signal items (marketing fluff, generic science, job posts, thin announcements).
+2. Score each remaining item (`quality_score` 1–5) with a consistent rubric:
+   - 5: deeply useful + specific + actionable/insightful
+   - 4: strong signal, worth reading
+   - 3: decent, include only if you need more items
+   - 1–2: exclude
+3. Deduplicate:
+   - exact URL match (including `cache.json:url_cache`)
+   - near-duplicate title (treat ~80% similarity as duplicate; keep the higher-scored one)
+   - optional content hash match (when you have the full text)
+
+### 5) Select and sort
+
+- Keep only items with `quality_score >= min_score_to_include`.
+- Sort by `quality_score` descending.
+- Break ties by source credibility (Tier 1 > Tier 2 > Tier 3) and recency.
+- Take the top `target_items`.
+
+## Markdown template
+
+Use this structure:
+
+```markdown
+# Daily News Report (YYYY-MM-DD)
+
+> Sources used: N | Candidates evaluated: M | Included: K
+> Generated at: <local timestamp> | Skill: sophnet-dailynews
+
+---
+
+## 1. <Title>
+
+- **Summary**: ...
+- **Key Points**:
+  1. ...
+  2. ...
+  3. ...
+- **Source**: <source_id> — <url>
+- **Keywords**: `k1` `k2` `k3`
+- **Score**: 4/5
+```
+
+## Cache update rules (`cache.json`)
+
+Update these fields every run:
+
+- `last_run`: date, duration, items_collected, items_published, sources_used
+- `source_stats[source_id]`: total_fetches, success_count, avg_items_per_fetch, avg_quality_score, last_fetch, last_success
+- `url_cache.entries`: add included URLs (store timestamps; respect `_ttl_hours`)
+- `content_hashes.entries`: add hashes when available (respect `_ttl_hours`)
+- `article_history[YYYY-MM-DD]`: record the final included item list (at minimum: title + url + source_id + score)
+
+## Editing sources (`sources.json`)
+
+- Disable a flaky/low-quality source by setting `enabled: false` or moving it to `disabled`.
+- Prefer fixing extraction hints before adding new sources.
+- Keep Tier 1 small and high-signal; use Tier 2 for "fill".
+
+## Failure handling
+
+- If a source 403s on `webfetch`, try `browser` (if available) or skip and record the error.
+- If all enabled sources fail, still write a report header and an explicit "no items" section; do not silently succeed.

package/skills/sophnet-dailynews/src/cache.json

@@ -0,0 +1,151 @@
+{
+  "schema_version": "1.0",
+  "description": "Daily News Report 缓存文件，用于避免重复抓取和跟踪历史表现",
+
+  "last_run": {
+    "date": "2026-02-09",
+    "duration_seconds": 30,
+    "items_collected": 3,
+    "items_published": 3,
+    "sources_used": ["fs_blog", "hackernoon_pm"]
+  },
+
+  "source_stats": {
+    "_comment": "记录每个源的历史表现，用于动态调整优先级",
+    "hn": {
+      "total_fetches": 2,
+      "success_count": 0,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:19Z",
+      "last_success": null,
+      "last_error": "fetch failed"
+    },
+    "hf_papers": {
+      "total_fetches": 2,
+      "success_count": 1,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:19Z",
+      "last_success": "2026-02-09T04:44:19Z",
+      "last_error": null,
+      "note": "No new papers for 2026-02-09 date page"
+    },
+    "one_useful_thing": {
+      "total_fetches": 2,
+      "success_count": 1,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:24Z",
+      "last_success": "2026-02-09T04:44:24Z",
+      "last_error": null,
+      "note": "Landing page only, newsletter content not accessible"
+    },
+    "paul_graham": {
+      "total_fetches": 2,
+      "success_count": 1,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:26Z",
+      "last_success": "2026-02-09T04:44:26Z",
+      "last_error": null,
+      "note": "Minimal content returned from articles page"
+    },
+    "james_clear": {
+      "total_fetches": 2,
+      "success_count": 1,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:32Z",
+      "last_success": "2026-02-09T04:44:32Z",
+      "last_error": null,
+      "note": "Main site only, newsletter requires email signup"
+    },
+    "fs_blog": {
+      "total_fetches": 3,
+      "success_count": 3,
+      "avg_items_per_fetch": 1.0,
+      "avg_quality_score": 5.0,
+      "last_fetch": "2026-02-09T04:44:38Z",
+      "last_success": "2026-02-09T04:44:38Z",
+      "last_error": null
+    },
+    "hackernoon_pm": {
+      "total_fetches": 3,
+      "success_count": 3,
+      "avg_items_per_fetch": 1.0,
+      "avg_quality_score": 4.0,
+      "last_fetch": "2026-02-09T04:44:43Z",
+      "last_success": "2026-02-09T04:44:43Z",
+      "last_error": null,
+      "note": "Category page fetched, extracted 2 high-quality articles"
+    },
+    "scotthyoung": {
+      "total_fetches": 2,
+      "success_count": 1,
+      "avg_items_per_fetch": 0,
+      "avg_quality_score": 0,
+      "last_fetch": "2026-02-09T04:44:48Z",
+      "last_success": "2026-02-09T04:44:48Z",
+      "last_error": null,
+      "note": "Articles page only, book and course promo without article links"
+    }
+  },
+
+  "url_cache": {
+    "_comment": "已处理的 URL 缓存，避免重复收录",
+    "_ttl_hours": 168,
+    "entries": {
+      "https://fs.blog/brain-food/january-25-2026/": {
+        "timestamp": "2026-02-09T04:44:38Z",
+        "title": "The Spectrum of Independence"
+      },
+      "https://hackernoon.com/10-proven-ways-to-reduce-misalignment-between-stakeholders-in-product-teams": {
+        "timestamp": "2026-02-09T04:44:43Z",
+        "title": "10 Proven Ways to Reduce Misalignment Between Stakeholders in Product Teams"
+      },
+      "https://hackernoon.com/why-good-products-feel-broken": {
+        "timestamp": "2026-02-09T04:44:43Z",
+        "title": "Why Good Products Feel Broken"
+      }
+    }
+  },
+
+  "content_hashes": {
+    "_comment": "内容指纹，用于去重",
+    "_ttl_hours": 168,
+    "entries": {}
+  },
+
+  "article_history": {
+    "_comment": "已收录文章的简要记录",
+    "2026-02-07": [
+      {
+        "title": "The Spectrum of Independence",
+        "url": "https://fs.blog/brain-food/january-25-2026/",
+        "source_id": "fs_blog",
+        "score": 5
+      }
+    ],
+    "2026-02-09": [
+      {
+        "title": "The Spectrum of Independence",
+        "url": "https://fs.blog/brain-food/january-25-2026/",
+        "source_id": "fs_blog",
+        "score": 5
+      },
+      {
+        "title": "10 Proven Ways to Reduce Misalignment Between Stakeholders in Product Teams",
+        "url": "https://hackernoon.com/10-proven-ways-to-reduce-misalignment-between-stakeholders-in-product-teams",
+        "source_id": "hackernoon_pm",
+        "score": 4
+      },
+      {
+        "title": "Why Good Products Feel Broken",
+        "url": "https://hackernoon.com/why-good-products-feel-broken",
+        "source_id": "hackernoon_pm",
+        "score": 4
+      }
+    ]
+  }
+}

package/skills/sophnet-dailynews/src/sources.json

@@ -0,0 +1,230 @@
+{
+  "version": "2.2",
+  "last_updated": "2026-02-09",
+
+  "sources": {
+    "tier1": {
+      "description": "中国大陆可直连的高信号 RSS 源",
+      "batch_a": [
+        {
+          "id": "ithome",
+          "name": "IT之家",
+          "url": "https://www.ithome.com/rss/",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 4.2,
+          "success_rate": 0.95
+        },
+        {
+          "id": "kr36",
+          "name": "36氪",
+          "url": "https://36kr.com/feed",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 4.3,
+          "success_rate": 0.94
+        }
+      ],
+      "batch_b": [
+        {
+          "id": "jiqizhixin",
+          "name": "机器之心",
+          "url": "https://www.jiqizhixin.com/rss",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 4.4,
+          "success_rate": 0.92
+        },
+        {
+          "id": "leiphone",
+          "name": "雷峰网",
+          "url": "https://www.leiphone.com/feed",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 4.1,
+          "success_rate": 0.90
+        }
+      ]
+    },
+
+    "tier2": {
+      "description": "补充覆盖面（资讯/国际/生活方式）",
+      "batch_a": [
+        {
+          "id": "sspai",
+          "name": "少数派",
+          "url": "https://sspai.com/feed",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 4.0,
+          "success_rate": 0.90
+        },
+        {
+          "id": "chinadaily_china",
+          "name": "China Daily - China",
+          "url": "https://www.chinadaily.com.cn/rss/china_rss.xml",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 3.8,
+          "success_rate": 0.88
+        }
+      ],
+      "batch_b": [
+        {
+          "id": "chinadaily_world",
+          "name": "China Daily - World",
+          "url": "https://www.chinadaily.com.cn/rss/world_rss.xml",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 3.8,
+          "success_rate": 0.88
+        },
+        {
+          "id": "chinadaily_sports",
+          "name": "China Daily - Sports",
+          "url": "https://www.chinadaily.com.cn/rss/sports_rss.xml",
+          "fetch_method": "webfetch",
+          "extract": "top_10_rss",
+          "enabled": true,
+          "avg_quality": 3.6,
+          "success_rate": 0.88
+        }
+      ]
+    },
+
+    "tier3_browser": {
+      "description": "JS 渲染源（当前禁用，避免浏览器依赖）",
+      "sources": []
+    },
+
+    "disabled": {
+      "description": "已禁用的源（海外/反爬/提取不稳定）",
+      "sources": [
+        {
+          "id": "hn",
+          "name": "Hacker News",
+          "url": "https://news.ycombinator.com/rss",
+          "reason": "海外源在中国大陆不稳定/被阻断",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "hf_papers",
+          "name": "HuggingFace Papers",
+          "url": "https://huggingface.co/papers/date/{{date}}",
+          "reason": "海外源 + 页面提取不稳定",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "one_useful_thing",
+          "name": "One Useful Thing",
+          "url": "https://www.oneusefulthing.org",
+          "reason": "海外源 + Substack 结构影响提取",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "paul_graham",
+          "name": "Paul Graham Essays",
+          "url": "https://paulgraham.com/articles.html",
+          "reason": "海外源 + 提取不稳定",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "james_clear",
+          "name": "James Clear 3-2-1",
+          "url": "https://jamesclear.com/3-2-1",
+          "reason": "海外源 + newsletter 需要订阅",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "fs_blog",
+          "name": "Farnam Street Brain Food",
+          "url": "https://fs.blog/brain-food",
+          "reason": "海外源 + 提取不稳定",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "hackernoon_pm",
+          "name": "HackerNoon PM",
+          "url": "https://hackernoon.com/c/product-management",
+          "reason": "海外源 + 提取不稳定",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "scotthyoung",
+          "name": "Scott Young Blog",
+          "url": "https://scotthyoung.com/blog/articles",
+          "reason": "海外源 + 提取不稳定",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "producthunt",
+          "name": "Product Hunt",
+          "url": "https://www.producthunt.com",
+          "reason": "需要浏览器渲染",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "latent_space",
+          "name": "Latent Space",
+          "url": "https://www.latent.space",
+          "reason": "Substack 需要 JS 渲染",
+          "disabled_date": "2026-02-09"
+        },
+        {
+          "id": "tldr_ai",
+          "name": "TLDR AI",
+          "url": "https://tldr.tech/ai",
+          "reason": "订阅页面，无文章列表",
+          "disabled_date": "2026-01-21"
+        },
+        {
+          "id": "bensbites",
+          "name": "Ben's Bites",
+          "url": "https://bensbites.com/archive",
+          "reason": "需要登录/付费墙",
+          "disabled_date": "2026-01-21"
+        },
+        {
+          "id": "interconnects",
+          "name": "Interconnects AI",
+          "url": "https://interconnects.ai",
+          "reason": "内容提取失败，Substack 结构问题",
+          "disabled_date": "2026-01-21"
+        },
+        {
+          "id": "beehiiv_rss",
+          "name": "Beehiiv RSS feeds",
+          "url": "https://rss.beehiiv.com",
+          "reason": "RSS 抓取困难",
+          "disabled_date": "2026-01-21"
+        }
+      ]
+    }
+  },
+
+  "fetch_config": {
+    "webfetch": {
+      "timeout_ms": 30000,
+      "retry_count": 1,
+      "cache_ttl_minutes": 60
+    },
+    "browser": {
+      "timeout_ms": 45000,
+      "wait_for_selector": "article, .post, .item",
+      "screenshot_on_error": true
+    }
+  },
+
+  "quality_thresholds": {
+    "min_score_to_include": 3,
+    "target_items": 20,
+    "early_stop_threshold": 25
+  }
+}

package/skills/sophnet-schedule/skill.json

@@ -0,0 +1,20 @@
+{
+  "name": "sophnet-schedule",
+  "version": "1.0.0",
+  "types": [
+    "store"
+  ],
+  "displayName": "日程管理助手",
+  "description": "支持本地托底与 Google Calendar 接入的日程管理 skill",
+  "changelog": [
+    {
+      "version": "1.0.0",
+      "date": "2026-04-14",
+      "changes": [
+        "初次提交"
+      ]
+    }
+  ],
+  "createdAt": "2026-04-14",
+  "updatedAt": "2026-04-14"
+}