cleanbookmarks 4.0.0__tar.gz

cleanbookmarks-4.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024-2026 holtwood
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

cleanbookmarks-4.0.0/PKG-INFO

@@ -0,0 +1,114 @@
+Metadata-Version: 2.4
+Name: cleanbookmarks
+Version: 4.0.0
+Summary: 离线书签清理与分类：规则优先 · LLM 可选
+Author: holtwood
+License-Expression: MIT
+Project-URL: Repository, https://github.com/build-workbench/bookmarks-cleaner
+Project-URL: Issues, https://github.com/build-workbench/bookmarks-cleaner/issues
+Keywords: bookmarks,classification,clean,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Internet :: WWW/HTTP :: Browsers
+Classifier: Topic :: Utilities
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: beautifulsoup4~=4.12.3
+Requires-Dist: lxml~=5.2.2
+Requires-Dist: pyyaml~=6.0.2
+Requires-Dist: chardet~=5.2.0
+Provides-Extra: llm
+Requires-Dist: requests~=2.32.3; extra == "llm"
+Provides-Extra: dev
+Requires-Dist: pytest~=8.2.2; extra == "dev"
+Dynamic: license-file
+
+# CleanBookmarks
+
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](#)
+
+**中文** | [English](https://github.com/build-workbench/bookmarks-cleaner/blob/main/README.en.md)
+
+书签太多太乱？一条命令帮你**去重、自动分类、整理导出**，全程本地运行。
+
+## 安装
+
+```bash
+pipx install cleanbookmarks
+```
+
+## 快速上手
+
+1. **在浏览器导出书签 HTML**
+   - Chrome / Edge：`书签管理器 → ⋮ → 导出书签`
+   - Firefox：`书签 → 管理书签 → 导入和备份 → 导出书签到 HTML`
+
+2. **运行分类**
+
+   ```bash
+   cleanbookmarks -i bookmarks.html -o output/
+   ```
+
+3. **导入回浏览器**：把 `output/` 下生成的 `*.html` 用浏览器的「导入书签」导回即可。同目录还有 `*.json`（结构化数据）和 `*.markdown`（分类报告）。
+
+![运行示例](https://raw.githubusercontent.com/build-workbench/bookmarks-cleaner/main/docs/screenshot.png)
+
+没有书签文件？下载仓库自带的示例试跑：
+
+```bash
+# 源码运行：直接使用仓库内示例
+cleanbookmarks -i examples/sample_bookmarks.html -o output/
+# pipx 安装：先下载示例（或任意浏览器导出的书签 HTML）
+curl -O https://raw.githubusercontent.com/build-workbench/bookmarks-cleaner/main/examples/sample_bookmarks.html
+cleanbookmarks -i sample_bookmarks.html -o output/
+```
+
+## 常用选项
+
+```bash
+cleanbookmarks -i a.html b.html -o output/ --workers 8   # 多个文件 + 并行
+cleanbookmarks -i "bookmarks/*.html" -o output/          # 支持 glob
+cleanbookmarks -i bookmarks.html -c config.local.json    # 自定义配置
+cleanbookmarks -i bookmarks.html --limit 20              # 先小批量试跑
+```
+
+默认配置开箱即用。想调整分类规则、置信度阈值、标题清理等，把默认配置复制为本地文件再修改，用 `-c` 指定：
+
+```bash
+# 源码运行：默认配置在 cleanbookmarks/resources/config.json
+cp cleanbookmarks/resources/config.json config.local.json
+# pipx 安装：先找到安装包内配置（pipx runpip cleanbookmarks show cleanbookmarks 可查路径）
+cleanbookmarks -i bookmarks.html -c config.local.json
+```
+
+完整参数见 `cleanbookmarks --help`。
+
+## LLM 分类（可选）
+
+默认全离线。若想让规则未命中的书签由 AI 兜底分类：
+
+```bash
+pip install "cleanbookmarks[llm]"
+```
+
+然后在 `config.local.json` 中开启：
+
+```json
+{ "llm": { "enable": true, "base_url": "https://api.openai.com", "model": "gpt-4o-mini", "api_key_env": "OPENAI_API_KEY" } }
+```
+
+设置 `OPENAI_API_KEY` 环境变量后重新运行即可。
+
+## 常见问题
+
+- **会误删吗？** 只在相同域名内判重，4 种策略（精确 URL、规范化 URL、标题+URL 相似度、标题相似度）任一命中才算重复，阈值保守。
+- **隐私？** 默认不发起任何网络请求；仅开启 LLM 后，书签标题/URL 才会发送给你配置的 API。
+- **支持中文书签吗？** 支持，分类词表含中英变体。
+- **导出文件怎么用？** Chrome / Edge / Firefox 均支持导入书签 HTML。

cleanbookmarks-4.0.0/README.md

@@ -0,0 +1,82 @@
+# CleanBookmarks
+
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue)](#)
+
+**中文** | [English](https://github.com/build-workbench/bookmarks-cleaner/blob/main/README.en.md)
+
+书签太多太乱？一条命令帮你**去重、自动分类、整理导出**，全程本地运行。
+
+## 安装
+
+```bash
+pipx install cleanbookmarks
+```
+
+## 快速上手
+
+1. **在浏览器导出书签 HTML**
+   - Chrome / Edge：`书签管理器 → ⋮ → 导出书签`
+   - Firefox：`书签 → 管理书签 → 导入和备份 → 导出书签到 HTML`
+
+2. **运行分类**
+
+   ```bash
+   cleanbookmarks -i bookmarks.html -o output/
+   ```
+
+3. **导入回浏览器**：把 `output/` 下生成的 `*.html` 用浏览器的「导入书签」导回即可。同目录还有 `*.json`（结构化数据）和 `*.markdown`（分类报告）。
+
+![运行示例](https://raw.githubusercontent.com/build-workbench/bookmarks-cleaner/main/docs/screenshot.png)
+
+没有书签文件？下载仓库自带的示例试跑：
+
+```bash
+# 源码运行：直接使用仓库内示例
+cleanbookmarks -i examples/sample_bookmarks.html -o output/
+# pipx 安装：先下载示例（或任意浏览器导出的书签 HTML）
+curl -O https://raw.githubusercontent.com/build-workbench/bookmarks-cleaner/main/examples/sample_bookmarks.html
+cleanbookmarks -i sample_bookmarks.html -o output/
+```
+
+## 常用选项
+
+```bash
+cleanbookmarks -i a.html b.html -o output/ --workers 8   # 多个文件 + 并行
+cleanbookmarks -i "bookmarks/*.html" -o output/          # 支持 glob
+cleanbookmarks -i bookmarks.html -c config.local.json    # 自定义配置
+cleanbookmarks -i bookmarks.html --limit 20              # 先小批量试跑
+```
+
+默认配置开箱即用。想调整分类规则、置信度阈值、标题清理等，把默认配置复制为本地文件再修改，用 `-c` 指定：
+
+```bash
+# 源码运行：默认配置在 cleanbookmarks/resources/config.json
+cp cleanbookmarks/resources/config.json config.local.json
+# pipx 安装：先找到安装包内配置（pipx runpip cleanbookmarks show cleanbookmarks 可查路径）
+cleanbookmarks -i bookmarks.html -c config.local.json
+```
+
+完整参数见 `cleanbookmarks --help`。
+
+## LLM 分类（可选）
+
+默认全离线。若想让规则未命中的书签由 AI 兜底分类：
+
+```bash
+pip install "cleanbookmarks[llm]"
+```
+
+然后在 `config.local.json` 中开启：
+
+```json
+{ "llm": { "enable": true, "base_url": "https://api.openai.com", "model": "gpt-4o-mini", "api_key_env": "OPENAI_API_KEY" } }
+```
+
+设置 `OPENAI_API_KEY` 环境变量后重新运行即可。
+
+## 常见问题
+
+- **会误删吗？** 只在相同域名内判重，4 种策略（精确 URL、规范化 URL、标题+URL 相似度、标题相似度）任一命中才算重复，阈值保守。
+- **隐私？** 默认不发起任何网络请求；仅开启 LLM 后，书签标题/URL 才会发送给你配置的 API。
+- **支持中文书签吗？** 支持，分类词表含中英变体。
+- **导出文件怎么用？** Chrome / Edge / Firefox 均支持导入书签 HTML。

cleanbookmarks-4.0.0/cleanbookmarks/__init__.py

@@ -0,0 +1,3 @@
+"""CleanBookmarks - 离线书签清理与分类 CLI"""
+
+__version__ = "4.0.0"

cleanbookmarks-4.0.0/cleanbookmarks/cache.py

@@ -0,0 +1,102 @@
+"""线程安全的 LRU 缓存"""
+
+from collections import OrderedDict
+from typing import Callable, Generic, Hashable, Optional, TypeVar
+import threading
+
+T = TypeVar("T")
+
+
+class CacheManager(Generic[T]):
+    """LRU 缓存管理器"""
+
+    def __init__(self, max_size: int = 1000, strategy: str = "lru", thread_safe: bool = True):
+        if strategy != "lru":
+            raise ValueError(f"不支持的淘汰策略: {strategy}，目前只支持 'lru'")
+        self.max_size = max_size
+        self.strategy = strategy
+        self._cache: OrderedDict[Hashable, T] = OrderedDict()
+        self._lock = threading.Lock() if thread_safe else None
+        self._stats = {"hits": 0, "misses": 0, "evictions": 0, "put_count": 0, "get_count": 0}
+
+    def get(self, key: Hashable) -> Optional[T]:
+        if self._lock:
+            with self._lock:
+                return self._get_unsafe(key)
+        return self._get_unsafe(key)
+
+    def _get_unsafe(self, key: Hashable) -> Optional[T]:
+        self._stats["get_count"] += 1
+        if key in self._cache:
+            self._cache.move_to_end(key)
+            self._stats["hits"] += 1
+            return self._cache[key]
+        self._stats["misses"] += 1
+        return None
+
+    def get_or_compute(self, key: Hashable, factory: Callable[[], T]) -> T:
+        if self._lock:
+            with self._lock:
+                value = self._get_unsafe(key)
+                if value is not None:
+                    return value
+                value = factory()
+                self._put_unsafe(key, value)
+                return value
+        value = self._get_unsafe(key)
+        if value is not None:
+            return value
+        value = factory()
+        self._put_unsafe(key, value)
+        return value
+
+    def put(self, key: Hashable, value: T) -> None:
+        if self._lock:
+            with self._lock:
+                self._put_unsafe(key, value)
+        else:
+            self._put_unsafe(key, value)
+
+    def _put_unsafe(self, key: Hashable, value: T) -> None:
+        self._stats["put_count"] += 1
+        if key in self._cache:
+            self._cache.move_to_end(key)
+            self._cache[key] = value
+        else:
+            self._cache[key] = value
+            if len(self._cache) > self.max_size:
+                self._cache.popitem(last=False)
+                self._stats["evictions"] += 1
+
+    def invalidate(self, key: Hashable) -> bool:
+        if self._lock:
+            with self._lock:
+                if key in self._cache:
+                    del self._cache[key]
+                    return True
+                return False
+        if key in self._cache:
+            del self._cache[key]
+            return True
+        return False
+
+    def clear(self) -> None:
+        if self._lock:
+            with self._lock:
+                self._cache.clear()
+        else:
+            self._cache.clear()
+
+    def get_stats(self) -> dict:
+        total = self._stats["hits"] + self._stats["misses"]
+        hit_rate = self._stats["hits"] / total if total > 0 else 0.0
+        return {**self._stats, "size": len(self._cache), "max_size": self.max_size, "hit_rate": hit_rate}
+
+    def __len__(self) -> int:
+        return len(self._cache)
+
+    def __contains__(self, key: Hashable) -> bool:
+        if self._lock:
+            with self._lock:
+                return key in self._cache
+        return key in self._cache

cleanbookmarks-4.0.0/cleanbookmarks/classifier.py

@@ -0,0 +1,282 @@
+"""书签分类器 - 规则优先 + LLM(可选) 两级级联"""
+
+from __future__ import annotations
+
+import hashlib
+import logging
+import threading
+from datetime import datetime
+from typing import Dict, Optional
+
+from cleanbookmarks.cache import CacheManager
+from cleanbookmarks.config import load_json_config, resolve_config_path
+from cleanbookmarks.models import BookmarkFeatures, ClassificationResult
+from cleanbookmarks.rules import RuleEngine
+from cleanbookmarks.text_utils import (
+    detect_language,
+    is_video_url,
+    normalize_category_config,
+)
+
+try:
+    from cleanbookmarks.llm import LLMClassifier
+except ImportError:
+    LLMClassifier = None  # type: ignore[assignment,misc]
+
+
+class BookmarkClassifier:
+    """书签分类器
+
+    两级级联：规则引擎给出确定性主分类，LLM(可选)在规则未命中时兜底、命中时补充子分类。
+    """
+
+    def __init__(
+        self,
+        config_path: Optional[str] = None,
+        config: Optional[Dict] = None,
+    ):
+        resolved_path, _ = resolve_config_path(config_path)
+        self.config_path = str(resolved_path)
+        self.logger = logging.getLogger(__name__)
+
+        if isinstance(config, dict):
+            normalized = normalize_category_config(config)
+            if not isinstance(normalized.get("category_rules"), dict) or not normalized.get("category_rules"):
+                raise ValueError("传入的 config 缺少有效的 category_rules")
+            self._config = normalized
+        else:
+            self._config = None
+        self._rule_engine: Optional[RuleEngine] = None
+        self._llm_classifier = None
+
+        # 缓存大小来自配置（默认 10000），分类结果缓存减半以省内存
+        cache_size = 10000
+        if isinstance(config, dict):
+            try:
+                cache_size = int((config.get("ai_settings") or {}).get("cache_size", 10000))
+            except (TypeError, ValueError):
+                cache_size = 10000
+        self.feature_cache: CacheManager[BookmarkFeatures] = CacheManager(max_size=cache_size, strategy="lru")
+        self.classification_cache: CacheManager[ClassificationResult] = CacheManager(max_size=max(cache_size // 2, 100), strategy="lru")
+
+        # stats 由多线程（_classify_batch）并发更新，需要锁保护
+        self._stats_lock = threading.Lock()
+
+        self.stats = {
+            "total_classified": 0,
+            "rule_engine": 0,
+            "fallback": 0,
+            "cache_hits": 0,
+            "average_confidence": 0.0,
+            "llm": 0,
+        }
+
+    @property
+    def config(self) -> Dict:
+        if self._config is None:
+            self._config = self._load_config()
+        return self._config
+
+    @property
+    def rule_engine(self) -> RuleEngine:
+        if self._rule_engine is None:
+            self._rule_engine = RuleEngine(self.config)
+        return self._rule_engine
+
+    @property
+    def llm_classifier(self):
+        if self._llm_classifier is None and LLMClassifier is not None:
+            try:
+                self._llm_classifier = LLMClassifier(self.config_path)
+            except Exception as e:
+                self.logger.warning(f"LLM 分类器初始化失败: {e}")
+        return self._llm_classifier
+
+    def _load_config(self) -> Dict:
+        config, _, _ = load_json_config(self.config_path)
+        normalized = normalize_category_config(config)
+        if not isinstance(normalized.get("category_rules"), dict) or not normalized.get("category_rules"):
+            raise ValueError(f"配置缺少有效的 category_rules: {self.config_path}")
+        return normalized
+
+    def extract_features(self, url: str, title: str) -> BookmarkFeatures:
+        cache_key = f"{url}::{title}"
+
+        def _extract():
+            content_type = self._detect_content_type(url, title)
+            language = detect_language(title)
+            return BookmarkFeatures.from_url_title(url, title, content_type, language)
+
+        return self.feature_cache.get_or_compute(cache_key, _extract)
+
+    def classify(self, url: str, title: str) -> ClassificationResult:
+        start_time = datetime.now()
+        cache_key = hashlib.md5(f"{url}::{title}".encode()).hexdigest()
+        cached = self.classification_cache.get(cache_key)
+        if cached is not None:
+            with self._stats_lock:
+                self.stats["cache_hits"] += 1
+            cached.processing_time = (datetime.now() - start_time).total_seconds()
+            return cached
+
+        features = self.extract_features(url, title)
+
+        # 1) 规则引擎 - 确定性优先
+        rule_result = self.rule_engine.classify(features)
+
+        # 2) LLM（可选）- 规则未命中时兜底，命中时补充子分类
+        llm_result = None
+        if self.llm_classifier and self.llm_classifier.enabled():
+            try:
+                llm_result = self.llm_classifier.classify(
+                    url, title,
+                    context={"domain": features.domain, "content_type": features.content_type, "language": features.language},
+                )
+            except Exception as e:
+                self.logger.warning(f"LLM 分类调用失败: {e}")
+
+        confidence_threshold = self.config.get("ai_settings", {}).get("confidence_threshold", 0.7)
+        final_result = self._cascade_fuse(
+            rule_result=rule_result,
+            llm_result=llm_result,
+            features=features,
+            confidence_threshold=float(confidence_threshold),
+        )
+
+        with self._stats_lock:
+            if "rule_engine" in final_result.method:
+                self.stats["rule_engine"] += 1
+            if "llm" in final_result.method:
+                self.stats["llm"] += 1
+            if final_result.method == "fallback":
+                self.stats["fallback"] += 1
+
+        final_result.processing_time = (datetime.now() - start_time).total_seconds()
+        self._update_stats(final_result)
+        self.classification_cache.put(cache_key, final_result)
+        return final_result
+
+    def _cascade_fuse(
+        self,
+        rule_result,
+        llm_result,
+        features: BookmarkFeatures,
+        confidence_threshold: float,
+    ) -> ClassificationResult:
+        """级联决策：规则命中即采用规则主分类，LLM 补子分类/facets；规则未命中才走 LLM"""
+        if rule_result is not None:
+            result = self._to_classification_result(rule_result)
+            # 1) 配置的 category_hierarchy 标题匹配
+            if result.subcategory is None:
+                result.subcategory = self._determine_subcategory(result.category, features)
+            # 2) LLM 补充子分类/facets/理由
+            if llm_result is not None:
+                llm = self._to_classification_result(llm_result)
+                if result.subcategory is None and llm.subcategory:
+                    result.subcategory = llm.subcategory
+                # LLM 输出不可控，facets 可能是非 dict（如列表/字符串），防御性处理
+                llm_facets = llm.facets if isinstance(llm.facets, dict) else {}
+                for k, v in llm_facets.items():
+                    if v and k not in (result.facets or {}):
+                        result.facets[k] = v
+                result.reasoning.extend(llm.reasoning or [])
+        elif llm_result is not None:
+            result = self._to_classification_result(llm_result)
+        else:
+            return ClassificationResult(
+                category="未分类", confidence=0.0,
+                reasoning=["没有匹配到任何分类规则"], method="fallback",
+            )
+
+        if result.category != "未分类" and result.confidence < confidence_threshold:
+            result.reasoning.append(
+                f"最终置信度 {result.confidence:.2f} 低于阈值 {confidence_threshold:.2f}，标记为未分类"
+            )
+            return ClassificationResult(
+                category="未分类", subcategory=None,
+                confidence=result.confidence,
+                reasoning=result.reasoning,
+                alternatives=result.alternatives[:3],
+                method=result.method, facets=result.facets,
+            )
+        return result
+
+    @staticmethod
+    def _to_classification_result(raw) -> ClassificationResult:
+        if isinstance(raw, ClassificationResult):
+            return raw
+        if isinstance(raw, dict):
+            # LLM 输出不可控，facets 可能是非 dict（如列表/字符串），统一防御
+            facets = raw.get("facets", {})
+            if not isinstance(facets, dict):
+                facets = {}
+            return ClassificationResult(
+                category=raw.get("category", "未分类"),
+                confidence=float(raw.get("confidence", 0.0)),
+                subcategory=raw.get("subcategory"),
+                reasoning=raw.get("reasoning", []),
+                alternatives=raw.get("alternatives", []),
+                processing_time=float(raw.get("processing_time", 0.0)),
+                method=raw.get("method", "unknown"),
+                facets=facets,
+            )
+        raise TypeError(f"Unexpected classification result type: {type(raw)}")
+
+    def _determine_subcategory(self, category: str, features: BookmarkFeatures) -> Optional[str]:
+        hierarchy = self.config.get("category_hierarchy", {})
+        if not isinstance(hierarchy, dict):
+            return None
+        # 规则引擎的 category 可能是 '主类/子类' 格式，按主类查 hierarchy
+        main = category.split("/", 1)[0].strip()
+        subs = hierarchy.get(category) or hierarchy.get(main)
+        if not isinstance(subs, list):
+            return None
+        title_lower = features.title.lower()
+        for sub in subs:
+            if str(sub).lower() in title_lower:
+                return sub
+        return None
+
+    def _detect_content_type(self, url: str, title: str) -> str:
+        url_lower = url.lower()
+        title_lower = title.lower()
+        if is_video_url(url):
+            return "video"
+        if any(d in url_lower for d in ["github.com", "gitlab.com"]):
+            return "code_repository"
+        if any(p in url_lower for p in ["docs.", "documentation", "wiki"]):
+            return "documentation"
+        if any(d in url_lower for d in ["arxiv.org", "acm.org", "ieee.org"]):
+            return "academic_paper"
+        if any(k in title_lower for k in ["news", "新闻", "breaking"]):
+            return "news"
+        if any(k in title_lower for k in ["tool", "工具", "online", "generator"]):
+            return "online_tool"
+        return "webpage"
+
+    def _update_stats(self, result: ClassificationResult):
+        with self._stats_lock:
+            self.stats["total_classified"] += 1
+            total = self.stats["total_classified"]
+            old_avg = self.stats["average_confidence"]
+            self.stats["average_confidence"] = (old_avg * (total - 1) + result.confidence) / total
+
+    def get_statistics(self) -> Dict:
+        total_predictions = (
+            self.stats["rule_engine"] + self.stats["llm"] + self.stats["fallback"]
+        )
+        # total_classified 只在缓存未命中时 +1，分母 = 命中 + 未命中 = 总尝试数
+        total_attempts = self.stats["cache_hits"] + self.stats["total_classified"]
+        return {
+            "total_classified": self.stats["total_classified"],
+            "cache_hits": self.stats["cache_hits"],
+            "cache_hit_rate": self.stats["cache_hits"] / max(total_attempts, 1),
+            "average_confidence": self.stats["average_confidence"],
+            "classification_methods": {
+                "rule_engine": self.stats["rule_engine"],
+                "llm": self.stats["llm"],
+                "unclassified (fallback)": self.stats["fallback"],
+                "total": total_predictions,
+            },
+            "llm_enabled": self.llm_classifier is not None and self.llm_classifier.enabled(),
+        }