@lyra_wang/chrome-frequent-pages 0.0.0-stage → 0.4.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +145 -2
- package/bin/cli.js +13 -0
- package/config.example.json +35 -0
- package/install.sh +30 -0
- package/main.py +417 -0
- package/package.json +15 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Wang Ying (LyraWang)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,146 @@
|
|
|
1
|
-
#
|
|
1
|
+
# chrome-frequent-pages
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**把 Chrome 里高频访问的网页收录进飞书「常用网页」表,并生成可导入 Chrome 的书签 HTML。**
|
|
4
|
+
|
|
5
|
+
只读 Chrome History(复制到临时目录后再读,不锁 Chrome),归一化 URL 后按
|
|
6
|
+
「域名 + 路径」聚合访问次数,过滤登录页 / 搜索页 / 本地地址。
|
|
7
|
+
|
|
8
|
+
## 安装(npm,推荐)
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
npm install -g --prefix ~/.local @lyra_wang/chrome-frequent-pages
|
|
12
|
+
chrome-frequent-pages --dry-run
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## 用法(仓库目录直接跑)
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
# 配置(默认 ~/.config/chrome-frequent-pages/config.json)
|
|
19
|
+
cp config.example.json ~/.config/chrome-frequent-pages/config.json
|
|
20
|
+
|
|
21
|
+
python3 main.py --dry-run # 只看结果
|
|
22
|
+
python3 main.py --write # 写/更新飞书表
|
|
23
|
+
python3 main.py --html ~/Desktop/常用网页.html # 导出可导入 Charter 的书签 HTML
|
|
24
|
+
python3 main.py --days 30 --min-visits 5 # 临时调窗口与阈值
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
## 统计窗口
|
|
28
|
+
|
|
29
|
+
`days` 决定"最近多少天内访问过的页面才参与统计":
|
|
30
|
+
|
|
31
|
+
| 值 | 效果 |
|
|
32
|
+
|---|---|
|
|
33
|
+
| 7 | 只看最近一周(最灵敏,候选最少)|
|
|
34
|
+
| **10** | **当前设置** |
|
|
35
|
+
| 30 | 一个月 |
|
|
36
|
+
| 90 | 一个季度(旧默认)|
|
|
37
|
+
|
|
38
|
+
改法:编辑 `~/.config/chrome-frequent-pages/config.json` 的 `days`,
|
|
39
|
+
或临时用 `python3 main.py --days 7 --dry-run`。
|
|
40
|
+
|
|
41
|
+
## 配置字段
|
|
42
|
+
|
|
43
|
+
| 字段 | 说明 |
|
|
44
|
+
|---|---|
|
|
45
|
+
| `chrome_profile_dir` | Chrome profile 目录(默认 `~/Library/Application Support/Google/Chrome/Default`)|
|
|
46
|
+
| `days` | 统计窗口(可配置;当前设为 **10 天**)|
|
|
47
|
+
| `min_visits` | 最少访问次数(当前设为 **5**)|
|
|
48
|
+
| `limit` | 最多收录条数(默认 60,`0` = 不限)|
|
|
49
|
+
| `exclude_hosts` | 排除的域名(邮箱、登录等)|
|
|
50
|
+
| `base_token` / `table_id` | 目标飞书表坐标 |
|
|
51
|
+
| `identity` | `user`(默认)或 `bot` |
|
|
52
|
+
| `ignored_urls_file` | 忽略清单(一行一个 URL);表里删掉的行会写进这里,避免下次重新收录 |
|
|
53
|
+
| `rules.drop_hosts` | 整站丢弃(娱乐视频站等)|
|
|
54
|
+
| `rules.drop_url_contains` | URL 含指定片段即丢弃(开发者后台、PR/Releases、OAuth、支付等)|
|
|
55
|
+
| `rules.root_only_hosts` | 只保留站点根域名(如 `github.com` 只留首页)|
|
|
56
|
+
| `rules.drop_titles` | 标题命中即丢(失效/无权限文档)|
|
|
57
|
+
| `rules.keep_urls` | 白名单,优先级最高 |
|
|
58
|
+
| `rules.one_per_host` | 每个域名只保留访问最多的一页 |
|
|
59
|
+
|
|
60
|
+
## 规则(0.4.0:根域名级)
|
|
61
|
+
|
|
62
|
+
```
|
|
63
|
+
最近 10 天 → 按「根域名」加总访问次数 → 超过 5 次 → 只排除影视娱乐与本地页 → 一行一个根域名
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
| 项 | 当前值 |
|
|
67
|
+
|---|---|
|
|
68
|
+
| `days` | 10 |
|
|
69
|
+
| `min_visits` | 5(**严格超过**,即 6 次起)|
|
|
70
|
+
| 统计口径 | 该根域名下**所有页面**的访问次数之和(不是单页)|
|
|
71
|
+
| 协议 | 只统计 `https://` |
|
|
72
|
+
| 排除 | 影视娱乐(yfsp / iyf / yifan / tv.cctv)+ 本地页(localhost / 127.0.0.1 / file)|
|
|
73
|
+
| 输出 | `https://<根域名>/` 一行;标题取 `domain_labels`,缺省用域名 |
|
|
74
|
+
| 黑名单 | 无(只有影视娱乐与本地页被排除)|
|
|
75
|
+
|
|
76
|
+
## 表结构(域名级)
|
|
77
|
+
|
|
78
|
+
0.3.1 起写入适配「一行一个域名」的表:
|
|
79
|
+
|
|
80
|
+
| 字段 | 写入内容 |
|
|
81
|
+
|---|---|
|
|
82
|
+
| `域名` | host(判重键)|
|
|
83
|
+
| `标题` | 取 `domain_labels` 里的中文说明;没有配置时退回域名 |
|
|
84
|
+
| `访问次数` | 该域名所有页面的访问次数之和 |
|
|
85
|
+
| `状态` | `已收录` |
|
|
86
|
+
|
|
87
|
+
`收录时间`、`分类` 由表自己维护,脚本不写。
|
|
88
|
+
|
|
89
|
+
## 写入策略
|
|
90
|
+
|
|
91
|
+
- 以 `URL` 去重:已存在则更新访问次数/最近访问,不存在才新建;
|
|
92
|
+
- 同一页面不同 query/fragment 会归一化成同一条(飞书 `/base/<token>`、`/wiki/<token>` 按 token 聚合);
|
|
93
|
+
- 状态默认「已收录」,方便你之后标「已加入书签」或「忽略」;
|
|
94
|
+
- 忽略清单:`~/.config/chrome-frequent-pages/ignored_urls.txt`,一行一个 URL,命中即跳过;
|
|
95
|
+
在表里删掉的页面记得同步进这个文件(或让工具写)。
|
|
96
|
+
|
|
97
|
+
## 书签替换(安全流程)
|
|
98
|
+
|
|
99
|
+
1. 先备份 Chrome 当前书签:
|
|
100
|
+
`cp "~/Library/Application Support/Google/Chrome/Default/Bookmarks" ~/Desktop/Bookmarks-backup-$(date +%F).json`
|
|
101
|
+
2. 跑 `--html` 生成新书签 HTML;
|
|
102
|
+
3. Chrome → 书签管理器 → 删除旧书签 → 导入 HTML;
|
|
103
|
+
4. 不建议直接改 `Bookmarks` JSON:Chrome 运行时会覆盖,写坏会丢全部书签。
|
|
104
|
+
|
|
105
|
+
## 定时(可选)
|
|
106
|
+
|
|
107
|
+
```bash
|
|
108
|
+
bash install.sh # 生成 ~/.local/bin/chrome-frequent-pages 并注册每周一 09:30 的 launchd 任务
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
## 中文标签(domain_labels)
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
"domain_labels": {
|
|
115
|
+
"meeting.bamamei.online": "组队会议分析",
|
|
116
|
+
"github.com": "GitHub"
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
表里的 `标题` 列与导出的书签名称都优先用这里的文字;没配置的域名退回域名本身。
|
|
121
|
+
|
|
122
|
+
## 书签:按分类子文件夹导出(0.4.1)
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
python3 main.py --bookmarks-from-table ~/Desktop/常用网页书签.html
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
按表里的 `分类` 列生成**同级**文件夹(都在书签栏一层):
|
|
129
|
+
|
|
130
|
+
```
|
|
131
|
+
书签栏
|
|
132
|
+
├── 常用网页 (全部 27 条,平铺)
|
|
133
|
+
├── 工作 (9 条)
|
|
134
|
+
├── 学习 (3 条)
|
|
135
|
+
└── 工具 (15 条)
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
没填分类的域名会进「其他」。
|
|
139
|
+
|
|
140
|
+
## 书签:按「域名」导出
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
python3 main.py --bookmarks-from-table ~/Desktop/常用网页书签.html
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
读取飞书表每个唯一「域名」,生成 `https://<域名>/` 书签 HTML(Chrome 书签管理器导入)。
|
package/bin/cli.js
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// npm bin shim: run the Python collector that ships in this package.
|
|
3
|
+
import { spawn } from 'node:child_process';
|
|
4
|
+
import { dirname, join } from 'node:path';
|
|
5
|
+
import { fileURLToPath } from 'node:url';
|
|
6
|
+
|
|
7
|
+
const here = dirname(fileURLToPath(import.meta.url));
|
|
8
|
+
const script = join(here, '..', 'main.py');
|
|
9
|
+
const child = spawn('/usr/bin/python3', [script, ...process.argv.slice(2)], { stdio: 'inherit' });
|
|
10
|
+
child.on('exit', (code, signal) => {
|
|
11
|
+
if (signal) process.kill(process.pid, signal);
|
|
12
|
+
else process.exit(code ?? 0);
|
|
13
|
+
});
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
{
|
|
2
|
+
"chrome_profile_dir": "~/Library/Application Support/Google/Chrome/Default",
|
|
3
|
+
"days": 10,
|
|
4
|
+
"min_visits": 5,
|
|
5
|
+
"limit": 60,
|
|
6
|
+
"exclude_hosts": [
|
|
7
|
+
"mail.google.com",
|
|
8
|
+
"accounts.google.com",
|
|
9
|
+
"localhost",
|
|
10
|
+
"127.0.0.1"
|
|
11
|
+
],
|
|
12
|
+
"base_token": "ZNjDb9TrpalLuCsw0macCeVdnld",
|
|
13
|
+
"table_id": "tblDejemWsgqKSA2",
|
|
14
|
+
"identity": "user",
|
|
15
|
+
"rules": {
|
|
16
|
+
"drop_hosts": [
|
|
17
|
+
"yfsp.tv",
|
|
18
|
+
"iyf.tv",
|
|
19
|
+
"yifan.tv",
|
|
20
|
+
"tv.cctv.com"
|
|
21
|
+
],
|
|
22
|
+
"drop_url_contains": [],
|
|
23
|
+
"root_only_hosts": [],
|
|
24
|
+
"drop_titles": [],
|
|
25
|
+
"keep_urls": [],
|
|
26
|
+
"one_per_host": false
|
|
27
|
+
},
|
|
28
|
+
"ignored_urls_file": "",
|
|
29
|
+
"domain_labels": {
|
|
30
|
+
"meeting.bamamei.online": "组队会议分析",
|
|
31
|
+
"eos.douyin.com": "抖音生活服务 · 直播",
|
|
32
|
+
"github.com": "GitHub",
|
|
33
|
+
"platform.deepseek.com": "DeepSeek 开放平台"
|
|
34
|
+
}
|
|
35
|
+
}
|
package/install.sh
ADDED
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
#!/bin/bash
|
|
2
|
+
set -eu
|
|
3
|
+
HERE="$(cd "$(dirname "$0")" && pwd)"
|
|
4
|
+
BIN_DIR="$HOME/.local/bin"
|
|
5
|
+
DATA_DIR="$HOME/.local/share/chrome-frequent-pages"
|
|
6
|
+
mkdir -p "$BIN_DIR" "$DATA_DIR" "$HOME/.config/chrome-frequent-pages"
|
|
7
|
+
# launchd 读不了 ~/Documents(TCC),所以把脚本复制到不受保护的目录再跑
|
|
8
|
+
cp "$HERE/main.py" "$DATA_DIR/main.py"
|
|
9
|
+
chmod +x "$DATA_DIR/main.py"
|
|
10
|
+
ln -sf "$DATA_DIR/main.py" "$BIN_DIR/chrome-frequent-pages"
|
|
11
|
+
chmod +x "$HERE/main.py"
|
|
12
|
+
[ -f "$HOME/.config/chrome-frequent-pages/config.json" ] || cp "$HERE/config.example.json" "$HOME/.config/chrome-frequent-pages/config.json"
|
|
13
|
+
PLIST="$HOME/Library/LaunchAgents/com.lyrawang.chrome-frequent-pages.plist"
|
|
14
|
+
cat > "$PLIST" <<EOF
|
|
15
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
16
|
+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
17
|
+
<plist version="1.0"><dict>
|
|
18
|
+
<key>Label</key><string>com.lyrawang.chrome-frequent-pages</string>
|
|
19
|
+
<key>ProgramArguments</key><array>
|
|
20
|
+
<string>/usr/bin/python3</string><string>$DATA_DIR/main.py</string><string>--write</string>
|
|
21
|
+
</array>
|
|
22
|
+
<key>StartCalendarInterval</key><dict><key>Weekday</key><integer>1</integer><key>Hour</key><integer>9</integer><key>Minute</key><integer>30</integer></dict>
|
|
23
|
+
<key>StandardOutPath</key><string>$HOME/.config/chrome-frequent-pages/run.log</string>
|
|
24
|
+
<key>StandardErrorPath</key><string>$HOME/.config/chrome-frequent-pages/run.log</string>
|
|
25
|
+
</dict></plist>
|
|
26
|
+
EOF
|
|
27
|
+
launchctl bootout "gui/$(id -u)/com.lyrawang.chrome-frequent-pages" 2>/dev/null || true
|
|
28
|
+
launchctl bootstrap "gui/$(id -u)" "$PLIST" 2>/dev/null || launchctl load "$PLIST" 2>/dev/null || true
|
|
29
|
+
echo "✓ 已安装:~/.local/bin/chrome-frequent-pages"
|
|
30
|
+
echo "✓ 已注册每周一 09:30 定时任务(launchd)"
|
package/main.py
ADDED
|
@@ -0,0 +1,417 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""把 Chrome 的高频访问页面收录进飞书「常用网页」表,并可导出书签 HTML。
|
|
3
|
+
|
|
4
|
+
只读 Chrome History(复制到临时目录后再读,不锁 Chrome),归一化 URL 后
|
|
5
|
+
按域名 + 路径聚合访问次数,过滤噪声与隐私站点。
|
|
6
|
+
|
|
7
|
+
用法:
|
|
8
|
+
python3 main.py --dry-run # 只看结果,不写表
|
|
9
|
+
python3 main.py --write # 写/更新飞书表
|
|
10
|
+
python3 main.py --html ~/Desktop/常用网页.html # 导出可导入 Chrome 的书签 HTML
|
|
11
|
+
"""
|
|
12
|
+
import argparse
|
|
13
|
+
import html
|
|
14
|
+
import json
|
|
15
|
+
import os
|
|
16
|
+
import shutil
|
|
17
|
+
import sqlite3
|
|
18
|
+
import subprocess
|
|
19
|
+
import sys
|
|
20
|
+
import tempfile
|
|
21
|
+
import urllib.parse
|
|
22
|
+
from collections import defaultdict
|
|
23
|
+
from datetime import datetime, timedelta, timezone
|
|
24
|
+
|
|
25
|
+
HOME = os.path.expanduser("~")
|
|
26
|
+
CONFIG_PATH = os.environ.get("CHROME_PAGES_CONFIG") or os.path.join(HOME, ".config", "chrome-frequent-pages", "config.json")
|
|
27
|
+
DEFAULT_CONFIG = {
|
|
28
|
+
"chrome_profile_dir": os.path.join(HOME, "Library", "Application Support", "Google", "Chrome", "Default"),
|
|
29
|
+
"days": 90,
|
|
30
|
+
"min_visits": 8,
|
|
31
|
+
"limit": 60,
|
|
32
|
+
"exclude_hosts": ["mail.google.com", "accounts.google.com", "localhost", "127.0.0.1"],
|
|
33
|
+
"base_token": "",
|
|
34
|
+
"table_id": "",
|
|
35
|
+
"identity": "user",
|
|
36
|
+
"ignored_urls_file": os.path.join(HOME, ".config", "chrome-frequent-pages", "ignored_urls.txt"),
|
|
37
|
+
"domain_labels": {},
|
|
38
|
+
"rules": {
|
|
39
|
+
# 只保留一条:娱乐影视整站丢弃;其余精细规则已按需求移除
|
|
40
|
+
"drop_hosts": ["yfsp.tv", "iyf.tv", "yifan.tv", "tv.cctv.com"],
|
|
41
|
+
"drop_url_contains": [],
|
|
42
|
+
"root_only_hosts": [],
|
|
43
|
+
"drop_titles": [],
|
|
44
|
+
"keep_urls": [],
|
|
45
|
+
"one_per_host": True,
|
|
46
|
+
},
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
# chrome time: microseconds since 1601-01-01
|
|
50
|
+
CHROME_EPOCH_OFFSET = 11644473600
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
def load_config():
|
|
54
|
+
cfg = dict(DEFAULT_CONFIG)
|
|
55
|
+
if os.path.exists(CONFIG_PATH):
|
|
56
|
+
with open(CONFIG_PATH, encoding="utf-8") as fh:
|
|
57
|
+
cfg.update(json.load(fh))
|
|
58
|
+
else:
|
|
59
|
+
print(f"[提示] 没有配置文件,使用默认值;可用 CHROME_PAGES_CONFIG 或 {CONFIG_PATH} 覆盖", file=sys.stderr)
|
|
60
|
+
for key in ("chrome_profile_dir",):
|
|
61
|
+
if isinstance(cfg.get(key), str):
|
|
62
|
+
cfg[key] = os.path.expanduser(cfg[key])
|
|
63
|
+
return cfg
|
|
64
|
+
|
|
65
|
+
|
|
66
|
+
def copy_history(profile_dir):
|
|
67
|
+
if not os.path.isdir(profile_dir):
|
|
68
|
+
raise SystemExit(f"找不到 Chrome profile:{profile_dir}")
|
|
69
|
+
tmp = tempfile.mkdtemp(prefix="chrome-history-")
|
|
70
|
+
for name in ("History", "History-wal", "History-shm"):
|
|
71
|
+
src = os.path.join(profile_dir, name)
|
|
72
|
+
if os.path.exists(src):
|
|
73
|
+
shutil.copy2(src, os.path.join(tmp, name))
|
|
74
|
+
target = os.path.join(tmp, "History")
|
|
75
|
+
if not os.path.exists(target):
|
|
76
|
+
raise SystemExit("复制 History 失败")
|
|
77
|
+
return tmp, target
|
|
78
|
+
|
|
79
|
+
|
|
80
|
+
def normalize(url):
|
|
81
|
+
parsed = urllib.parse.urlsplit(url)
|
|
82
|
+
host = (parsed.hostname or "").lower()
|
|
83
|
+
if host.startswith("www."):
|
|
84
|
+
host = host[4:]
|
|
85
|
+
path = parsed.path.rstrip("/")
|
|
86
|
+
# 飞书文档/表格按 token 聚合:只认路径开头的 /base/<token> 之类,
|
|
87
|
+
# 避免把 open.feishu.cn/document/.../docs/bitable-v1 这种开发文档误折叠。
|
|
88
|
+
import re as _re
|
|
89
|
+
match = _re.match(r"^/(base|wiki|docx|sheets|docs|mindnote|slides)/([A-Za-z0-9]+)", path)
|
|
90
|
+
if match:
|
|
91
|
+
return host, f"/{match.group(1)}/{match.group(2)}"
|
|
92
|
+
return host, path or "/"
|
|
93
|
+
|
|
94
|
+
|
|
95
|
+
def is_noise(url, host, exclude_hosts):
|
|
96
|
+
# 只看 https;http、file://、localhost 等本地页面一律不看
|
|
97
|
+
if not url.startswith("https://"):
|
|
98
|
+
return True
|
|
99
|
+
if not host:
|
|
100
|
+
return True
|
|
101
|
+
if host in exclude_hosts:
|
|
102
|
+
return True
|
|
103
|
+
low = url.lower()
|
|
104
|
+
for bad in ("chrome://", "file://", "chrome-extension://", "/search?q=", "login", "signin", "accounts."):
|
|
105
|
+
if bad in low:
|
|
106
|
+
return True
|
|
107
|
+
return False
|
|
108
|
+
|
|
109
|
+
|
|
110
|
+
def load_ignored(cfg):
|
|
111
|
+
path = cfg.get("ignored_urls_file") or ""
|
|
112
|
+
if not path:
|
|
113
|
+
return set()
|
|
114
|
+
path = os.path.expanduser(path)
|
|
115
|
+
if not os.path.exists(path):
|
|
116
|
+
return set()
|
|
117
|
+
with open(path, encoding="utf-8") as fh:
|
|
118
|
+
return {line.strip() for line in fh if line.strip()}
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
def apply_rules(url, host, title, rules):
|
|
122
|
+
"""Return None to drop, or the url to keep (root_only may rewrite it)."""
|
|
123
|
+
keep_urls = set(rules.get("keep_urls") or [])
|
|
124
|
+
if url in keep_urls:
|
|
125
|
+
return url
|
|
126
|
+
low = url.lower()
|
|
127
|
+
for needle in (rules.get("drop_url_contains") or []):
|
|
128
|
+
if needle.lower() in low:
|
|
129
|
+
return None
|
|
130
|
+
for dropped in (rules.get("drop_hosts") or []):
|
|
131
|
+
if host == dropped.lower() or host.endswith("." + dropped.lower()):
|
|
132
|
+
return None
|
|
133
|
+
clean_title = (title or "").strip()
|
|
134
|
+
for dead in (rules.get("drop_titles") or []):
|
|
135
|
+
if clean_title == dead:
|
|
136
|
+
return None
|
|
137
|
+
root_only = {h.lower() for h in (rules.get("root_only_hosts") or [])}
|
|
138
|
+
if host in root_only:
|
|
139
|
+
from urllib.parse import urlsplit
|
|
140
|
+
path = urlsplit(url).path or "/"
|
|
141
|
+
if path.rstrip("/") != "":
|
|
142
|
+
return None
|
|
143
|
+
return "https://" + host + "/"
|
|
144
|
+
return url
|
|
145
|
+
|
|
146
|
+
|
|
147
|
+
def collect(cfg):
|
|
148
|
+
ignored = load_ignored(cfg)
|
|
149
|
+
rules = cfg.get("rules") or {}
|
|
150
|
+
tmp, db = copy_history(cfg["chrome_profile_dir"])
|
|
151
|
+
try:
|
|
152
|
+
conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
|
153
|
+
cutoff_us = int((datetime.now().timestamp() - cfg["days"] * 86400 + CHROME_EPOCH_OFFSET) * 1_000_000)
|
|
154
|
+
rows = conn.execute(
|
|
155
|
+
"SELECT url, title, visit_count, last_visit_time FROM urls WHERE last_visit_time > ? AND url LIKE 'http%'",
|
|
156
|
+
(cutoff_us,),
|
|
157
|
+
).fetchall()
|
|
158
|
+
conn.close()
|
|
159
|
+
finally:
|
|
160
|
+
shutil.rmtree(tmp, ignore_errors=True)
|
|
161
|
+
|
|
162
|
+
# 按根域名聚合:访问次数 = 该域名下所有页面之和
|
|
163
|
+
aggregated = {}
|
|
164
|
+
for url, title, visits, last_us in rows:
|
|
165
|
+
host, path = normalize(url)
|
|
166
|
+
if is_noise(url, host, set(cfg["exclude_hosts"])):
|
|
167
|
+
continue
|
|
168
|
+
kept = apply_rules(url, host, title, rules)
|
|
169
|
+
if kept is None:
|
|
170
|
+
continue
|
|
171
|
+
key = host
|
|
172
|
+
last_ts = last_us / 1_000_000 - CHROME_EPOCH_OFFSET
|
|
173
|
+
entry = aggregated.get(key)
|
|
174
|
+
if entry is None:
|
|
175
|
+
aggregated[key] = {
|
|
176
|
+
"key": key,
|
|
177
|
+
# 输出根域名,不带路径
|
|
178
|
+
"url": f"https://{host}/",
|
|
179
|
+
"title": (title or "").strip(),
|
|
180
|
+
"host": host,
|
|
181
|
+
"visits": visits,
|
|
182
|
+
"last": last_ts,
|
|
183
|
+
}
|
|
184
|
+
else:
|
|
185
|
+
entry["visits"] += visits
|
|
186
|
+
if last_ts > entry["last"]:
|
|
187
|
+
entry["last"] = last_ts
|
|
188
|
+
if not entry["title"] and title:
|
|
189
|
+
entry["title"] = title.strip()
|
|
190
|
+
|
|
191
|
+
# 规则:根域名访问次数「超过」min_visits(严格大于)
|
|
192
|
+
items = [e for e in aggregated.values() if e["visits"] > cfg["min_visits"] and e["url"] not in ignored]
|
|
193
|
+
items.sort(key=lambda e: (e["visits"], e["last"]), reverse=True)
|
|
194
|
+
if False and rules.get("one_per_host"):
|
|
195
|
+
seen_hosts = set()
|
|
196
|
+
collapsed = []
|
|
197
|
+
for item in items:
|
|
198
|
+
if item["host"] in seen_hosts:
|
|
199
|
+
continue
|
|
200
|
+
seen_hosts.add(item["host"])
|
|
201
|
+
collapsed.append(item)
|
|
202
|
+
items = collapsed
|
|
203
|
+
if cfg["limit"]:
|
|
204
|
+
items = items[: cfg["limit"]]
|
|
205
|
+
labels = cfg.get("domain_labels") or {}
|
|
206
|
+
for item in items:
|
|
207
|
+
item["last_iso"] = datetime.fromtimestamp(item["last"]).strftime("%Y-%m-%d %H:%M")
|
|
208
|
+
item["title"] = labels.get(item["host"]) or item["host"]
|
|
209
|
+
return items
|
|
210
|
+
|
|
211
|
+
|
|
212
|
+
def lark(args):
|
|
213
|
+
result = subprocess.run(["lark-cli"] + args + ["--as", "user", "--format", "json"], capture_output=True, text=True, timeout=120)
|
|
214
|
+
if result.returncode != 0:
|
|
215
|
+
raise SystemExit("lark-cli 失败:" + (result.stderr or result.stdout).strip()[:400])
|
|
216
|
+
payload = json.loads(result.stdout or "{}")
|
|
217
|
+
if payload.get("ok") is False:
|
|
218
|
+
err = payload.get("error") or {}
|
|
219
|
+
raise SystemExit("lark-cli 返回失败:" + str(err.get("message") or err)[:400])
|
|
220
|
+
return payload
|
|
221
|
+
|
|
222
|
+
|
|
223
|
+
def existing_domains(cfg):
|
|
224
|
+
# 表结构是「一行一个域名」;读失败必须中止,避免重复插入。
|
|
225
|
+
data = lark(["base", "+record-list", "--base-token", cfg["base_token"], "--table-id", cfg["table_id"], "--limit", "200"])
|
|
226
|
+
meta = data.get("data") or {}
|
|
227
|
+
cols = meta.get("fields") or []
|
|
228
|
+
rows = meta.get("data") or []
|
|
229
|
+
ids = meta.get("record_id_list") or []
|
|
230
|
+
if "域名" not in cols:
|
|
231
|
+
raise SystemExit("表里没有「域名」列;请检查表结构")
|
|
232
|
+
idx = {name: i for i, name in enumerate(cols)}
|
|
233
|
+
out = {}
|
|
234
|
+
for rid, row in zip(ids, rows):
|
|
235
|
+
host = str(row[idx["域名"]] or "").strip()
|
|
236
|
+
if host:
|
|
237
|
+
out[host] = rid
|
|
238
|
+
return out
|
|
239
|
+
|
|
240
|
+
|
|
241
|
+
def aggregate_by_host(items, labels=None):
|
|
242
|
+
"""把页面级结果压成域名级:访问次数求和,标题用中文标签(缺省退回域名)。"""
|
|
243
|
+
labels = labels or {}
|
|
244
|
+
grouped = {}
|
|
245
|
+
for item in items:
|
|
246
|
+
host = item["host"]
|
|
247
|
+
rec = grouped.get(host)
|
|
248
|
+
if rec is None:
|
|
249
|
+
grouped[host] = {
|
|
250
|
+
"域名": host,
|
|
251
|
+
"访问次数": item["visits"],
|
|
252
|
+
"标题": labels.get(host) or host,
|
|
253
|
+
"_top": item["visits"],
|
|
254
|
+
}
|
|
255
|
+
else:
|
|
256
|
+
rec["访问次数"] += item["visits"]
|
|
257
|
+
if item["visits"] > rec["_top"]:
|
|
258
|
+
rec["_top"] = item["visits"]
|
|
259
|
+
rec["标题"] = labels.get(host) or host
|
|
260
|
+
rows = sorted(grouped.values(), key=lambda r: r["访问次数"], reverse=True)
|
|
261
|
+
for row in rows:
|
|
262
|
+
row.pop("_top", None)
|
|
263
|
+
return rows
|
|
264
|
+
|
|
265
|
+
|
|
266
|
+
def write_table(cfg, items):
|
|
267
|
+
if not cfg["base_token"] or not cfg["table_id"]:
|
|
268
|
+
raise SystemExit("配置里缺 base_token / table_id")
|
|
269
|
+
known = existing_domains(cfg)
|
|
270
|
+
create, update = [], []
|
|
271
|
+
for row in aggregate_by_host(items, cfg.get("domain_labels") or {}):
|
|
272
|
+
record = {
|
|
273
|
+
"域名": row["域名"],
|
|
274
|
+
"标题": row["标题"],
|
|
275
|
+
"访问次数": row["访问次数"],
|
|
276
|
+
"状态": ["已收录"],
|
|
277
|
+
}
|
|
278
|
+
if row["域名"] in known:
|
|
279
|
+
update.append((known[row["域名"]], record))
|
|
280
|
+
else:
|
|
281
|
+
create.append(record)
|
|
282
|
+
|
|
283
|
+
if create:
|
|
284
|
+
lark(["base", "+record-batch-create", "--base-token", cfg["base_token"], "--table-id", cfg["table_id"],
|
|
285
|
+
"--json", json.dumps({"create_records": create}, ensure_ascii=False)])
|
|
286
|
+
for record_id, record in update:
|
|
287
|
+
lark(["base", "+record-upsert", "--base-token", cfg["base_token"], "--table-id", cfg["table_id"],
|
|
288
|
+
"--record-id", record_id, "--json", json.dumps(record, ensure_ascii=False)])
|
|
289
|
+
return len(create), len(update)
|
|
290
|
+
|
|
291
|
+
|
|
292
|
+
CATEGORY_ORDER = ["工作", "学习", "工具", "内容", "其他"]
|
|
293
|
+
|
|
294
|
+
|
|
295
|
+
def bookmarks_from_table(cfg, path):
|
|
296
|
+
"""按飞书表导出书签:常用网页/<分类>/<域名书签>。"""
|
|
297
|
+
data = lark(["base", "+record-list", "--base-token", cfg["base_token"], "--table-id", cfg["table_id"], "--limit", "200"])
|
|
298
|
+
meta = data.get("data") or {}
|
|
299
|
+
cols = meta.get("fields") or []
|
|
300
|
+
rows = meta.get("data") or []
|
|
301
|
+
if "域名" not in cols:
|
|
302
|
+
raise SystemExit("表里没有「域名」列")
|
|
303
|
+
idx = {name: i for i, name in enumerate(cols)}
|
|
304
|
+
labels = cfg.get("domain_labels") or {}
|
|
305
|
+
groups = {}
|
|
306
|
+
for row in rows:
|
|
307
|
+
host = str(row[idx["域名"]] or "").strip()
|
|
308
|
+
if not host:
|
|
309
|
+
continue
|
|
310
|
+
cat = "其他"
|
|
311
|
+
if "分类" in idx:
|
|
312
|
+
raw = row[idx["分类"]]
|
|
313
|
+
if isinstance(raw, list) and raw:
|
|
314
|
+
cat = str(raw[0])
|
|
315
|
+
elif isinstance(raw, str) and raw.strip():
|
|
316
|
+
cat = raw.strip()
|
|
317
|
+
groups.setdefault(cat, []).append({
|
|
318
|
+
"host": host,
|
|
319
|
+
"name": str(row[idx["标题"]]).strip() if "标题" in idx and row[idx["标题"]] else labels.get(host) or host,
|
|
320
|
+
})
|
|
321
|
+
|
|
322
|
+
lines = [
|
|
323
|
+
"<!DOCTYPE NETSCAPE-Bookmark-file-1>",
|
|
324
|
+
'<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">',
|
|
325
|
+
"<TITLE>Bookmarks</TITLE>",
|
|
326
|
+
"<H1>Bookmarks</H1>",
|
|
327
|
+
"<DL><p>",
|
|
328
|
+
]
|
|
329
|
+
ordered = [c for c in CATEGORY_ORDER if c in groups] + sorted(c for c in groups if c not in CATEGORY_ORDER)
|
|
330
|
+
|
|
331
|
+
def emit_folder(name, entries):
|
|
332
|
+
lines.append(f" <DT><H3>{html.escape(name)}</H3>")
|
|
333
|
+
lines.append(" <DL><p>")
|
|
334
|
+
for entry in sorted(entries, key=lambda e: e["name"]):
|
|
335
|
+
lines.append(f' <DT><A HREF="https://{html.escape(entry["host"])}/">{html.escape(entry["name"])}</A>')
|
|
336
|
+
lines.append(" </DL><p>")
|
|
337
|
+
|
|
338
|
+
# 同级结构:常用网页(全部) + 各分类文件夹,都在书签栏同一层
|
|
339
|
+
emit_folder("常用网页", [e for cat in ordered for e in groups[cat]])
|
|
340
|
+
for cat in ordered:
|
|
341
|
+
emit_folder(cat, groups[cat])
|
|
342
|
+
lines.append("</DL><p>")
|
|
343
|
+
with open(os.path.expanduser(path), "w", encoding="utf-8") as fh:
|
|
344
|
+
fh.write("\n".join(lines) + "\n")
|
|
345
|
+
return ordered, groups
|
|
346
|
+
|
|
347
|
+
|
|
348
|
+
def write_html(items, path):
|
|
349
|
+
lines = [
|
|
350
|
+
"<!DOCTYPE NETSCAPE-Bookmark-file-1>",
|
|
351
|
+
'<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">',
|
|
352
|
+
"<TITLE>Bookmarks</TITLE>",
|
|
353
|
+
"<H1>Bookmarks</H1>",
|
|
354
|
+
"<DL><p>",
|
|
355
|
+
' <DT><H3>常用网页(chrome-frequent-pages)</H3>',
|
|
356
|
+
" <DL><p>",
|
|
357
|
+
]
|
|
358
|
+
for item in items:
|
|
359
|
+
title = html.escape(item["title"] or item["host"])
|
|
360
|
+
lines.append(f' <DT><A HREF="{html.escape(item["url"])}">{title}</A>')
|
|
361
|
+
lines += [" </DL><p>", "</DL><p>"]
|
|
362
|
+
with open(os.path.expanduser(path), "w", encoding="utf-8") as fh:
|
|
363
|
+
fh.write("\n".join(lines) + "\n")
|
|
364
|
+
|
|
365
|
+
|
|
366
|
+
def print_table(items):
|
|
367
|
+
print(f"{'访问':>6} {'最近访问':<16} 标题 / URL")
|
|
368
|
+
print("-" * 78)
|
|
369
|
+
for item in items:
|
|
370
|
+
print(f"{item['visits']:>6} {item['last_iso']:<16} {item['title'][:30]}")
|
|
371
|
+
print(f"{'':>6} {'':<16} {item['url']}")
|
|
372
|
+
|
|
373
|
+
|
|
374
|
+
def main():
|
|
375
|
+
parser = argparse.ArgumentParser()
|
|
376
|
+
parser.add_argument("--dry-run", action="store_true", help="只打印结果")
|
|
377
|
+
parser.add_argument("--write", action="store_true", help="写入飞书表")
|
|
378
|
+
parser.add_argument("--html", metavar="PATH", help="导出可导入 Chrome 的书签 HTML(按页面链接)")
|
|
379
|
+
parser.add_argument("--bookmarks-from-table", metavar="PATH", help="按飞书表的「域名」列导出书签 HTML")
|
|
380
|
+
parser.add_argument("--limit", type=int)
|
|
381
|
+
parser.add_argument("--days", type=int)
|
|
382
|
+
parser.add_argument("--min-visits", type=int)
|
|
383
|
+
args = parser.parse_args()
|
|
384
|
+
|
|
385
|
+
cfg = load_config()
|
|
386
|
+
if args.bookmarks_from_table:
|
|
387
|
+
ordered, groups = bookmarks_from_table(cfg, args.bookmarks_from_table)
|
|
388
|
+
total = sum(len(v) for v in groups.values())
|
|
389
|
+
print(f"已导出分类书签:{os.path.expanduser(args.bookmarks_from_table)}({total} 个域名 / {len(ordered)} 个分类)")
|
|
390
|
+
for cat in ordered:
|
|
391
|
+
print(f" [{cat}]")
|
|
392
|
+
for entry in sorted(groups[cat], key=lambda e: e["name"]):
|
|
393
|
+
print(" -", entry["name"], "->", "https://" + entry["host"] + "/")
|
|
394
|
+
return 0
|
|
395
|
+
for key in ("limit", "days", "min_visits"):
|
|
396
|
+
if getattr(args, key) is not None:
|
|
397
|
+
cfg[key] = getattr(args, key)
|
|
398
|
+
|
|
399
|
+
items = collect(cfg)
|
|
400
|
+
if not items:
|
|
401
|
+
print("没有满足条件的页面(可调低 --min-visits 或调大 --days)")
|
|
402
|
+
return 0
|
|
403
|
+
print(f"共 {len(items)} 个高频页面(窗口 {cfg['days']} 天,最少 {cfg['min_visits']} 次访问)\n")
|
|
404
|
+
print_table(items)
|
|
405
|
+
if args.html:
|
|
406
|
+
write_html(items, args.html)
|
|
407
|
+
print(f"\n已导出书签 HTML:{os.path.expanduser(args.html)}")
|
|
408
|
+
if args.write:
|
|
409
|
+
created, updated = write_table(cfg, items)
|
|
410
|
+
print(f"\n飞书写入完成:新增 {created} 条,更新 {updated} 条")
|
|
411
|
+
if not args.write and not args.html:
|
|
412
|
+
print("\n(dry-run:没有写表;加 --write 写飞书,--html <路径> 导出书签)")
|
|
413
|
+
return 0
|
|
414
|
+
|
|
415
|
+
|
|
416
|
+
if __name__ == "__main__":
|
|
417
|
+
sys.exit(main())
|
package/package.json
CHANGED
|
@@ -1,6 +1,17 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lyra_wang/chrome-frequent-pages",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.4.3",
|
|
4
|
+
"description": "Chrome 高频网页 → 飞书「常用网页」表 + 按分类导出 Chrome 书签(根域名 / 10 天 / >5 次)",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"type": "module",
|
|
7
|
+
"keywords": ["chrome", "history", "bookmarks", "feishu", "lark", "cli"],
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/LyraWang6688/ai-capability-store.git",
|
|
11
|
+
"directory": "external-capabilities/cli/chrome-frequent-pages"
|
|
12
|
+
},
|
|
13
|
+
"bin": { "chrome-frequent-pages": "bin/cli.js" },
|
|
14
|
+
"files": ["main.py", "bin/", "config.example.json", "README.md", "LICENSE", "install.sh"],
|
|
15
|
+
"publishConfig": { "access": "public" },
|
|
16
|
+
"engines": { "node": ">=20" }
|
|
17
|
+
}
|