newscli-tool 1.0.0__tar.gz

newscli_tool-1.0.0/PKG-INFO

@@ -0,0 +1,184 @@
+Metadata-Version: 2.4
+Name: newscli-tool
+Version: 1.0.0
+Summary: Unified news aggregation CLI — 9 sources, 25+ modules, natural language DSL
+Author-email: kzclaw <kzclaw@users.noreply.github.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/kzclaw/news-cli
+Project-URL: Repository, https://github.com/kzclaw/news-cli
+Project-URL: Issues, https://github.com/kzclaw/news-cli/issues
+Keywords: news,hackernews,github,cli,aggregation,rss
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: requests>=2.28.0
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: lxml>=4.9.0
+
+# newscli
+
+**Unified news aggregation CLI** — pull from 9 sources in one command.
+
+```bash
+newscli get hackernews topstories 5
+newscli get github trending 10 language python
+newscli get all 15 json
+```
+
+[!["Build Status"](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://pypi.org/project/newscli-tool/)
+[!["PyPI Version"](https://img.shields.io/pypi/v/newscli-tool.svg)](https://pypi.org/project/newscli-tool/)
+[!["License"](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
+📄 [中文版](README_zh.md)
+
+---
+
+## Install
+
+```bash
+# From TestPyPI (works now, globally accessible)
+pip install --index-url https://test.pypi.org/simple/ newscli-tool
+
+# From GitHub (dev branch, latest)
+pip install git+https://github.com/kzclaw/news-cli.git
+
+# One-liner (any machine with curl + python3)
+curl -sSL https://raw.githubusercontent.com/kzclaw/news-cli/main/install.sh | bash
+```
+
+After install, `newscli` is available in your terminal globally.
+
+---
+
+## Features
+
+| Feature | Details |
+|---------|---------|
+| **9 sources** | Hacker News · GitHub Trending · Hugging Face · ZAKER · V2EX · Reddit · DEV.to · Lobsters · RSS |
+| **25+ modules** | Each source has multiple views — topstories, trending, ask, show, job, by language, by node, by subreddit… |
+| **Natural language DSL** | `get`, `list`, `看`, `拉` — no flags to remember |
+| **URL enrichment** | Auto-fetches og:description for every item with `summary = null` |
+| **Deduplication** | Cross-source Jaccard similarity, 70% threshold, keeps richer item |
+| **JSON output** | `NewsItem v1.0` schema — always 9 fields, `null` means "source doesn't have it" |
+
+---
+
+## Usage
+
+### Natural language mode
+
+```
+newscli get <source> <module> [limit] [json] [noenrich]
+```
+
+**Examples**
+
+```bash
+# Single source
+newscli get hackernews topstories 5
+newscli get github trending 10 language python
+newscli get v2ex node python 10
+newscli get reddit subreddit technology 10
+newscli get huggingface daily 5
+
+# Multi-source (AND)
+newscli get hackernews topstories 5 and github trending 10 and reddit subreddit programming 5
+
+# All sources
+newscli get all 15 json
+
+# JSON output (machine-readable)
+newscli get hackernews topstories 5 json
+newscli get all 3 json noenrich
+
+# List available sources / modules
+newscli list
+newscli list github
+newscli list v2ex
+```
+
+### Flag mode
+
+```bash
+python3 -m newscli --source hackernews:topstories --limit 5 --json
+python3 -m newscli --source "hackernews:topstories&github:trending" --limit 5
+```
+
+---
+
+## Source & Module Reference
+
+| Source | Modules |
+|--------|---------|
+| `hackernews` | `topstories` · `new` · `ask` · `show` · `jobs` |
+| `github` | `trending` · `trending-weekly` · `trending-monthly` · language parameter |
+| `huggingface` | `daily` · `weekly` · `monthly` |
+| `zaker` | `hot` · `news` · `search` · category parameter |
+| `v2ex` | `hot` · `latest` · `node:<name>` |
+| `reddit` | `subreddit:<name>` · `popular` · `hot` |
+| `devto` | `latest` · `top` · `tags:<tag>` |
+| `lobsters` | `newest` · `hot` · `top` · `upcoming` |
+| `rss` | `feed:<url>` — any RSS/Atom feed by URL |
+
+---
+
+## Output Schema
+
+Every item follows `NewsItem v1.0` — null means "source doesn't provide this", never faked:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `source` | `str` | Source name (e.g. `hackernews`) |
+| `module` | `str` | Sub-module (e.g. `topstories`) |
+| `title` | `str` | Item title |
+| `url` | `str` | Link to item |
+| `author` | `str\|null` | Author / submitter |
+| `published_at` | `datetime\|null` | Publication time |
+| `summary` | `str\|null` | Description or og:description |
+| `score` | `int\|null` | Score / points (if available) |
+| `comments` | `int\|null` | Comment count (if available) |
+
+---
+
+## Architecture
+
+```
+newscli/
+├── cli.py          # Dual-mode entry: flags + NL DSL
+├── aggregator.py   # ThreadPoolExecutor dispatcher + deduplication
+├── parser.py       # NL DSL parser (no AI — pure rules)
+├── enrich.py       # Concurrent og:description fetcher
+└── sources/
+    ├── base.py     # NewsSource ABC + NewsItem schema
+    ├── hackernews.py
+    ├── github.py
+    ├── huggingface.py
+    ├── zaker.py
+    ├── v2ex.py
+    ├── reddit.py
+    ├── devto.py
+    ├── lobsters.py
+    └── rss.py
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- `requests` · `beautifulsoup4` · `lxml` (installed automatically)
+
+---
+
+## License
+
+MIT · [kzclaw/news-cli](https://github.com/kzclaw/news-cli)

newscli_tool-1.0.0/README.md

@@ -0,0 +1,158 @@
+# newscli
+
+**Unified news aggregation CLI** — pull from 9 sources in one command.
+
+```bash
+newscli get hackernews topstories 5
+newscli get github trending 10 language python
+newscli get all 15 json
+```
+
+[!["Build Status"](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://pypi.org/project/newscli-tool/)
+[!["PyPI Version"](https://img.shields.io/pypi/v/newscli-tool.svg)](https://pypi.org/project/newscli-tool/)
+[!["License"](https://img.shields.io/badge/license-MIT-green.svg)](LICENSE)
+
+📄 [中文版](README_zh.md)
+
+---
+
+## Install
+
+```bash
+# From TestPyPI (works now, globally accessible)
+pip install --index-url https://test.pypi.org/simple/ newscli-tool
+
+# From GitHub (dev branch, latest)
+pip install git+https://github.com/kzclaw/news-cli.git
+
+# One-liner (any machine with curl + python3)
+curl -sSL https://raw.githubusercontent.com/kzclaw/news-cli/main/install.sh | bash
+```
+
+After install, `newscli` is available in your terminal globally.
+
+---
+
+## Features
+
+| Feature | Details |
+|---------|---------|
+| **9 sources** | Hacker News · GitHub Trending · Hugging Face · ZAKER · V2EX · Reddit · DEV.to · Lobsters · RSS |
+| **25+ modules** | Each source has multiple views — topstories, trending, ask, show, job, by language, by node, by subreddit… |
+| **Natural language DSL** | `get`, `list`, `看`, `拉` — no flags to remember |
+| **URL enrichment** | Auto-fetches og:description for every item with `summary = null` |
+| **Deduplication** | Cross-source Jaccard similarity, 70% threshold, keeps richer item |
+| **JSON output** | `NewsItem v1.0` schema — always 9 fields, `null` means "source doesn't have it" |
+
+---
+
+## Usage
+
+### Natural language mode
+
+```
+newscli get <source> <module> [limit] [json] [noenrich]
+```
+
+**Examples**
+
+```bash
+# Single source
+newscli get hackernews topstories 5
+newscli get github trending 10 language python
+newscli get v2ex node python 10
+newscli get reddit subreddit technology 10
+newscli get huggingface daily 5
+
+# Multi-source (AND)
+newscli get hackernews topstories 5 and github trending 10 and reddit subreddit programming 5
+
+# All sources
+newscli get all 15 json
+
+# JSON output (machine-readable)
+newscli get hackernews topstories 5 json
+newscli get all 3 json noenrich
+
+# List available sources / modules
+newscli list
+newscli list github
+newscli list v2ex
+```
+
+### Flag mode
+
+```bash
+python3 -m newscli --source hackernews:topstories --limit 5 --json
+python3 -m newscli --source "hackernews:topstories&github:trending" --limit 5
+```
+
+---
+
+## Source & Module Reference
+
+| Source | Modules |
+|--------|---------|
+| `hackernews` | `topstories` · `new` · `ask` · `show` · `jobs` |
+| `github` | `trending` · `trending-weekly` · `trending-monthly` · language parameter |
+| `huggingface` | `daily` · `weekly` · `monthly` |
+| `zaker` | `hot` · `news` · `search` · category parameter |
+| `v2ex` | `hot` · `latest` · `node:<name>` |
+| `reddit` | `subreddit:<name>` · `popular` · `hot` |
+| `devto` | `latest` · `top` · `tags:<tag>` |
+| `lobsters` | `newest` · `hot` · `top` · `upcoming` |
+| `rss` | `feed:<url>` — any RSS/Atom feed by URL |
+
+---
+
+## Output Schema
+
+Every item follows `NewsItem v1.0` — null means "source doesn't provide this", never faked:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `source` | `str` | Source name (e.g. `hackernews`) |
+| `module` | `str` | Sub-module (e.g. `topstories`) |
+| `title` | `str` | Item title |
+| `url` | `str` | Link to item |
+| `author` | `str\|null` | Author / submitter |
+| `published_at` | `datetime\|null` | Publication time |
+| `summary` | `str\|null` | Description or og:description |
+| `score` | `int\|null` | Score / points (if available) |
+| `comments` | `int\|null` | Comment count (if available) |
+
+---
+
+## Architecture
+
+```
+newscli/
+├── cli.py          # Dual-mode entry: flags + NL DSL
+├── aggregator.py   # ThreadPoolExecutor dispatcher + deduplication
+├── parser.py       # NL DSL parser (no AI — pure rules)
+├── enrich.py       # Concurrent og:description fetcher
+└── sources/
+    ├── base.py     # NewsSource ABC + NewsItem schema
+    ├── hackernews.py
+    ├── github.py
+    ├── huggingface.py
+    ├── zaker.py
+    ├── v2ex.py
+    ├── reddit.py
+    ├── devto.py
+    ├── lobsters.py
+    └── rss.py
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- `requests` · `beautifulsoup4` · `lxml` (installed automatically)
+
+---
+
+## License
+
+MIT · [kzclaw/news-cli](https://github.com/kzclaw/news-cli)

newscli_tool-1.0.0/newscli/__init__.py

@@ -0,0 +1,6 @@
+"""
+newscli/__init__.py — news-cli package entry
+"""
+
+__version__ = "1.0.0"
+__all__ = ["__version__"]

newscli_tool-1.0.0/newscli/aggregator.py

@@ -0,0 +1,263 @@
+"""
+aggregator.py — News Aggregator
+
+统一调度多个 source，支持：
+- 多 source 并行拉取（ThreadPoolExecutor）
+- 每个 source 的 module 子模块指定
+- source-specific 额外参数（category, language, node, subreddit 等）
+- 统一 JSON 输出（完整 schema）
+"""
+
+import concurrent.futures
+from dataclasses import asdict
+from typing import Optional
+from newscli.sources import REGISTRY, NewsItem, SourceError, rss as rss_module
+
+
+class NewsAggregator:
+    """多 source 统一聚合器"""
+
+    def __init__(
+        self,
+        sources: list[str] | None = None,
+        limit_per_source: int = 10,
+    ):
+        """
+        Args:
+            sources         : source 名称列表（如 ["hackernews", "github"]）
+                              None = 所有注册 source（不含 rss）
+            limit_per_source: 每个 source 最大拉取条数
+        """
+        self.sources = sources or list(REGISTRY.keys())
+        self.limit_per_source = limit_per_source
+
+    def fetch(
+        self,
+        source_filter: str | None = None,
+        limit: int | None = None,
+        keyword: str | None = None,
+        params: dict | None = None,
+        enrich: bool = True,  # 默认对 summary=null 的 item 拉取原文摘要
+    ) -> dict:
+        """
+        并行拉取所有 source，返回统一格式。
+
+        Args:
+            source_filter : 逗号分隔的 source:module 对列表
+                           如 "hackernews:topstories,github:trending,v2ex:latest"
+            limit         : 全局返回上限（None = 所有）
+            keyword       : 关键词过滤（所有 source 生效）
+            params        : source-specific 参数 dict
+                           格式：{"<source>": {"module": "...", "category": "..."}}
+
+        Returns:
+            {"ok": bool, "schema": str, "sources": {}, "items": [], "total": int, "errors": []}
+            items 每项 = NewsItem.to_dict()，即所有字段（含 None）
+        """
+        parsed = self._parse_filter(source_filter)
+        params = params or {}
+
+        # results key = source:module（如 "hackernews:topstories"）
+        results: dict[str, list[dict]] = {}
+        errors: list[str] = []
+
+        def _fetch_one(name: str, module: str | None, extra: dict):
+            """执行单次 fetch，返回 (key, items_list, errors_list)"""
+            # 唯一 key：不同 module 的同一 source 不会互相覆盖
+            key = f"{name}:{module}" if module else name
+
+            # RSS 特殊处理（不注册到 REGISTRY）
+            if name == "rss":
+                src = rss_module.RSSSource(
+                    source_key=extra.get("source_key"),
+                    feed_url=extra.get("url"),
+                )
+            else:
+                src_cls = REGISTRY.get(name)
+                if not src_cls:
+                    return key, [], [f"Unknown source: {name}"]
+                src = src_cls()
+
+            # global keyword filter applied at source.fetch() call
+            fetch_kwargs = {
+                "module": module,
+                "limit": self.limit_per_source,
+                "keyword": keyword,   # may be None = no filtering
+            }
+            # per-source params override (from merged extra)
+            for k in ("category", "language", "node", "subreddit",
+                      "start_time", "end_time", "url"):
+                if k in extra:
+                    fetch_kwargs[k] = extra[k]
+            # per-source keyword (params level) takes precedence over global
+            if "keyword" in extra:
+                fetch_kwargs["keyword"] = extra["keyword"]
+
+            try:
+                items = src.fetch(**fetch_kwargs)
+                return key, [item.to_dict() for item in items], []
+            except SourceError as e:
+                return key, [], [str(e)]
+            except Exception as e:
+                return key, [], [f"{name} unexpected error: {e}"]
+
+        # 构建 work items
+        work: list[tuple] = []
+        if parsed:
+            for name, module, extra in parsed:
+                # CLI params 覆盖 hard-coded extra
+                sp = params.get(name, {})
+                merged_extra = {**extra, **sp}
+                work.append((name, module, merged_extra))
+        else:
+            for name in self.sources:
+                sp = params.get(name, {})
+                work.append((name, None, sp))
+
+        # 并行执行
+        with concurrent.futures.ThreadPoolExecutor(max_workers=min(len(work), 8)) as ex:
+            futures = {ex.submit(_fetch_one, *w): w for w in work}
+            for fut in concurrent.futures.as_completed(futures):
+                key, items, errs = fut.result()
+                if items:
+                    results[key] = items
+                errors.extend(errs)
+
+        # 合并所有 items（按 source:module 分别存储，输出时全部扁平化）
+        all_items: list[dict] = []
+        for key, items in results.items():
+            all_items.extend(items)
+
+        # 跨 source 去重：标题相似度 ≥ 70%（仅多 source 时触发）
+        if len(results) > 1:
+            all_items = self._deduplicate(all_items)
+
+        # 可选：enrich — 对 summary=null 的 item 并发拉取原文 description
+        if enrich:
+            from .enrich import enrich_items
+            all_items = enrich_items(all_items)
+
+        if limit is not None:
+            all_items = all_items[:limit]
+
+        return {
+            "ok": True,
+            "schema": "NewsItem v1.0",
+            "sources": {k: len(v) for k, v in results.items()},
+            "items": all_items,
+            "total": len(all_items),
+            "errors": errors,
+        }
+
+    @staticmethod
+    def _deduplicate(items: list[dict]) -> list[dict]:
+        """
+        跨 source 去重。相似度阈值 70%（标题 normalized 后比对）。
+        保留第一条出现的item，移除后续相似项。
+        """
+        from urllib.parse import urlparse
+        def normalize_title(t: str) -> str:
+            """小写 + 去除标点 + strip()"""
+            import re
+            t = t.lower().strip()
+            t = re.sub(r'[^\w\s]', ' ', t)
+            t = re.sub(r'\s+', ' ', t).strip()
+            return t
+
+        def similarity(a: str, b: str) -> float:
+            """简单词集合 Jaccard 相似度"""
+            sa = set(a.split())
+            sb = set(b.split())
+            if not sa or not sb:
+                return 0.0
+            inter = len(sa & sb)
+            union = len(sa | sb)
+            return inter / union if union > 0 else 0.0
+
+        def item_key(item: dict) -> str:
+            domain = ""
+            if item.get("url"):
+                try:
+                    domain = urlparse(item["url"]).netloc
+                except Exception:
+                    pass
+            return f"{normalize_title(item['title'])}|{domain}"
+
+        seen: list[dict] = []
+        for item in items:
+            norm = normalize_title(item.get("title", ""))
+            if not norm:
+                seen.append(item)
+                continue
+            dup_idx = None
+            for i, s in enumerate(seen):
+                s_norm = normalize_title(s.get("title", ""))
+                # Different domain → different article, skip
+                s_domain = urlparse(s.get("url", "")).netloc or ""
+                item_domain = urlparse(item.get("url", "")).netloc or ""
+                if s_domain and item_domain and s_domain != item_domain:
+                    continue
+                if similarity(norm, s_norm) >= 0.70:
+                    dup_idx = i
+                    break
+            if dup_idx is not None:
+                # 保留信息更丰富的那个（非 null 字段数量多的）
+                existing = seen[dup_idx]
+                existing_nulls = sum(1 for k, v in existing.items() if v is None and k != 'extra')
+                item_nulls = sum(1 for k, v in item.items() if v is None and k != 'extra')
+                if item_nulls < existing_nulls:
+                    seen[dup_idx] = item
+            else:
+                seen.append(item)
+        return seen
+
+    @staticmethod
+    def _parse_filter(filter_str: str | None) -> list[tuple]:
+        """
+        解析 'source:module,source2:module2' → [(name, module, extra_dict), ...]
+
+        支持格式：
+          hackernews:topstories        → ("hackernews", "topstories", {})
+          v2ex:node:python             → ("v2ex", "node", {"node": "python"})
+          rss:bensbites                → ("rss", "bensbites", {"source_key": "bensbites"})
+          github:trending:language=Python → ("github", "trending", {"language": "Python"})
+          zaker:category:category=technology → ("zaker", "category", {"category": "technology"})
+          reddit:r/technology          → ("reddit", None, {"subreddit": "technology"})
+        """
+        if not filter_str:
+            return []
+        items = []
+        for part in filter_str.split("&"):
+            part = part.strip()
+            if not part:
+                continue
+            name, rest = part.split(":", 1) if ":" in part else (part, "")
+            name = name.strip()
+
+            extra = {}
+            tokens = rest.split(":") if rest else []
+            module = None
+            for token in tokens:
+                if "=" in token:
+                    k, v = token.split("=", 1)
+                    extra[k.strip()] = v.strip()
+                elif not module:
+                    module = token
+
+            # Reddit r/<subreddit> shorthand
+            if name == "reddit" and module and module.startswith("r/"):
+                extra["subreddit"] = module[2:]
+                module = None
+
+            # RSS source_key shorthand
+            if name == "rss" and module and "=" not in module:
+                extra["source_key"] = module
+
+            items.append((name, module, extra))
+        return items
+
+
+def fetch_all(keyword: str | None = None, limit: int | None = None) -> dict:
+    """快速入口：拉取所有注册 source"""
+    agg = NewsAggregator()
+    return agg.fetch(keyword=keyword, limit=limit)