nonebot-plugin-randpic 2.0.1__tar.gz → 2.1.0__tar.gz

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: nonebot-plugin-randpic
-Version: 2.0.1
+Version: 2.1.0
 Summary: 一个通过指令随机发送你存储的图片的nonebot2插件
 License: MIT
 License-File: LICENSE
@@ -66,10 +66,22 @@ _✨ 一个发送指令就能让你的 bot 发出对应指令的图片的插件
 
 首次启动会创建数据库，并扫描配置的图片文件夹，根据实际图片建立索引，发送次数从 0 开始。数据库不存在时也可由图片文件重新建立索引。
 
-之后每次启动都会核对文件变化，并保留已有图片的发送次数、消息引用关系和同步状态。这些运行数据无法仅凭图片内容恢复，因此不要在正常使用期间删除数据库。
+之后每次启动都会核对文件变化，并保留已有图片的发送次数和消息引用关系。这些运行数据无法仅凭图片内容恢复，因此不要在正常使用期间删除数据库。
 
 实现说明见 [架构与数据一致性](./docs/architecture.md)。
 
+### 数据库
+
+数据库只保存三类业务数据：
+
+| 表 | 用途 |
+| --- | --- |
+| `randpic_images` | 图片身份、指令、文件名、内容哈希、感知哈希、发送次数、文件大小和删除状态 |
+| `randpic_messages` | Bot／群／消息 ID 与图片的对应关系，用于引用删除 |
+| `randpic_sequences` | 每个指令的最大文件编号，防止删除后复用编号 |
+
+网页发布进度仅保存在内存，启动时重新生成；OSS 状态直接从远端获取，不建立数据库镜像或任务表。
+
 ### 本地存储和离线加图
 
 `randpic_store_dir_path` 指定的路径就是存储根目录，**不会额外追加一层 `randpic`**。未设置时，由 `nonebot-plugin-localstore` 提供默认目录。
@@ -84,8 +96,8 @@ data/randpic/
 │   │   └── randpic_capoo_0000000001.png
 │   └── 马哥/
 ├── database/
-│   ├── data.db            # 图片、发送次数、历史消息、待同步任务
-│   └── ...                # 文件恢复日志、网页上传缓存等内部文件
+│   ├── data.db            # 图片、发送次数、历史消息、文件编号
+│   └── ...                # 文件恢复日志、可选 OSS 页面缓存等内部文件
 └── public/                # 自动生成的本地静态网页与图片副本，请勿手动维护
 ```
 
@@ -105,7 +117,7 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 
 使用 FastAPI 驱动时，Bot 启动后即可通过 `http://<host>:<port>/randpic/` 查看图库，无需配置 OSS。默认每页 100 张图片，可通过 `randpic_web_page_size` 调整。图片使用 `loading="lazy"`，按需加载。
 
-增删图片后本地页面立即重新发布；已打开的网页在可见时每 15 秒检查一次目录版本，发现变化会自动刷新。没有 FastAPI 服务的驱动仍可使用聊天指令和 OSS 同步。
+增删图片后本地页面立即重新发布，浏览器手动刷新即可查看最新内容。没有 FastAPI 服务的驱动仍可使用聊天指令和 OSS 同步。
 
 #### 阿里云 OSS
 
@@ -114,7 +126,7 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 - `randpic_endpoint`：绑定到 Bucket 的完整网页域名，如 `https://gallery.example.com`。
 - `randpic_bucket`：Bucket 名称。
 - `randpic_region`：地域 ID，如 `cn-beijing`，可查阅 [OSS 地域和访问域名](https://help.aliyun.com/zh/oss/regions-and-endpoints)。
-- AccessKey：需要上传和删除本插件对象的权限。
+- AccessKey：需要 `oss:ListObjects`、`oss:GetObject`、`oss:PutObject` 和 `oss:DeleteObject` 权限，用于列举、核对、上传和删除图库对象。
 
 请配置网页访问权限、域名及静态网站首页 `index.html`，示意如下：
 
@@ -122,7 +134,11 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 
 启动、添加和删除后会自动同步；`上传oss` 可手动全量重新发布、重试未完成任务。默认最多 16 个并发请求，普通终端中以单行动态进度显示完成数、百分比和成功／失败数，最多每秒刷新 5 次；其他插件的默认 NoneBot 日志显示在进度行上方。同步结束后清除动态行，只保留一条结果汇总，错误详情仍单独记录。输出重定向、非交互控制台或已移除 NoneBot 默认日志处理器时仅输出汇总。
 
-失败任务存入数据库，后台约每 30 秒重试，重启后继续处理。OSS 的图片、页面和目录版本依次同步，网络故障期间云端可能暂时落后于本地图。`randpic_oss_no_upload_list` 只影响 OSS，本地网页仍显示所有配置的分类；此前已同步的分类被排除后，其受本插件跟踪的云端对象会清理。未被数据库跟踪的历史对象不保证自动清理。
+OSS 同步使用 Python SDK 分页列举远端对象，与本地发布文件比较大小和校验值，仅上传变化的文件，并清理不在本地发布清单中的云端图库文件。普通对象直接比较列表中的 ETag 与本地 MD5；遇到不一致的同大小对象，再查询 SHA-256 自定义元数据。无需安装 `ossutil`，数据库不保存云端状态或同步任务。
+
+后台只在内容变化或失败待重试时同步，失败后约每 30 秒重新比较；重启后重新检查远端，继续完成差异同步。网络故障期间云端可能暂时落后于本地图。`randpic_oss_no_upload_list` 只影响 OSS，本地网页仍显示所有配置的分类；被排除或移除的分类，其云端图库文件会在比较后清理。
+
+Bucket 根目录的 `index.html`、`<指令>/index.html`、分页及 `randpic_<指令>_<编号>.<图片后缀>` 均由插件维护。其他命名的文件不参与自动删除；请不要与其他站点共用上述路径。
 
 ## 💿 安装
 
@@ -222,8 +238,8 @@ randpic_oss_no_upload_list=["capoo"]
 
 群管理员或群主引用本插件发送的图库图片，再发送 `删除` 即可。添加成功时 Bot 回复的图片也支持引用删除。
 
-- 删除会移除原图和网页副本，并将数据库中的图片标记为已删除，并在启用 OSS 时删除云端对象、更新页面；影响所有群共用的图库，不会撤回已经发送的 QQ 消息。
-- 消息与图片的对应关系会持久化，重启 Bot 后仍可引用删除。升级前发送、尚未建立对应记录的历史消息无法直接删除，请让 Bot 重新发送对应图片后再操作。
+- 删除会移除原图和网页副本，并将数据库中的图片标记为已删除，在启用 OSS 时删除云端对象、更新页面；影响所有群共用的图库，不会撤回已经发送的 QQ 消息。
+- 消息与图片的对应关系会持久化，重启 Bot 后仍可引用删除。没有对应记录的消息无法直接删除，请让 Bot 重新发送对应图片后再操作。
 - 删除过的图片再次引用会提示已删除；若网页或 OSS 同步失败，再次引用原消息发送 `删除` 可重试，`上传oss` 也会重试未完成的删除同步。
 
 ### 效果图

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/README.md

@@ -35,10 +35,22 @@ _✨ 一个发送指令就能让你的 bot 发出对应指令的图片的插件
 
 首次启动会创建数据库，并扫描配置的图片文件夹，根据实际图片建立索引，发送次数从 0 开始。数据库不存在时也可由图片文件重新建立索引。
 
-之后每次启动都会核对文件变化，并保留已有图片的发送次数、消息引用关系和同步状态。这些运行数据无法仅凭图片内容恢复，因此不要在正常使用期间删除数据库。
+之后每次启动都会核对文件变化，并保留已有图片的发送次数和消息引用关系。这些运行数据无法仅凭图片内容恢复，因此不要在正常使用期间删除数据库。
 
 实现说明见 [架构与数据一致性](./docs/architecture.md)。
 
+### 数据库
+
+数据库只保存三类业务数据：
+
+| 表 | 用途 |
+| --- | --- |
+| `randpic_images` | 图片身份、指令、文件名、内容哈希、感知哈希、发送次数、文件大小和删除状态 |
+| `randpic_messages` | Bot／群／消息 ID 与图片的对应关系，用于引用删除 |
+| `randpic_sequences` | 每个指令的最大文件编号，防止删除后复用编号 |
+
+网页发布进度仅保存在内存，启动时重新生成；OSS 状态直接从远端获取，不建立数据库镜像或任务表。
+
 ### 本地存储和离线加图
 
 `randpic_store_dir_path` 指定的路径就是存储根目录，**不会额外追加一层 `randpic`**。未设置时，由 `nonebot-plugin-localstore` 提供默认目录。
@@ -53,8 +65,8 @@ data/randpic/
 │   │   └── randpic_capoo_0000000001.png
 │   └── 马哥/
 ├── database/
-│   ├── data.db            # 图片、发送次数、历史消息、待同步任务
-│   └── ...                # 文件恢复日志、网页上传缓存等内部文件
+│   ├── data.db            # 图片、发送次数、历史消息、文件编号
+│   └── ...                # 文件恢复日志、可选 OSS 页面缓存等内部文件
 └── public/                # 自动生成的本地静态网页与图片副本，请勿手动维护
 ```
 
@@ -74,7 +86,7 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 
 使用 FastAPI 驱动时，Bot 启动后即可通过 `http://<host>:<port>/randpic/` 查看图库，无需配置 OSS。默认每页 100 张图片，可通过 `randpic_web_page_size` 调整。图片使用 `loading="lazy"`，按需加载。
 
-增删图片后本地页面立即重新发布；已打开的网页在可见时每 15 秒检查一次目录版本，发现变化会自动刷新。没有 FastAPI 服务的驱动仍可使用聊天指令和 OSS 同步。
+增删图片后本地页面立即重新发布，浏览器手动刷新即可查看最新内容。没有 FastAPI 服务的驱动仍可使用聊天指令和 OSS 同步。
 
 #### 阿里云 OSS
 
@@ -83,7 +95,7 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 - `randpic_endpoint`：绑定到 Bucket 的完整网页域名，如 `https://gallery.example.com`。
 - `randpic_bucket`：Bucket 名称。
 - `randpic_region`：地域 ID，如 `cn-beijing`，可查阅 [OSS 地域和访问域名](https://help.aliyun.com/zh/oss/regions-and-endpoints)。
-- AccessKey：需要上传和删除本插件对象的权限。
+- AccessKey：需要 `oss:ListObjects`、`oss:GetObject`、`oss:PutObject` 和 `oss:DeleteObject` 权限，用于列举、核对、上传和删除图库对象。
 
 请配置网页访问权限、域名及静态网站首页 `index.html`，示意如下：
 
@@ -91,7 +103,11 @@ Bot 运行期间请通过指令增删图片，直接修改文件需重启才能
 
 启动、添加和删除后会自动同步；`上传oss` 可手动全量重新发布、重试未完成任务。默认最多 16 个并发请求，普通终端中以单行动态进度显示完成数、百分比和成功／失败数，最多每秒刷新 5 次；其他插件的默认 NoneBot 日志显示在进度行上方。同步结束后清除动态行，只保留一条结果汇总，错误详情仍单独记录。输出重定向、非交互控制台或已移除 NoneBot 默认日志处理器时仅输出汇总。
 
-失败任务存入数据库，后台约每 30 秒重试，重启后继续处理。OSS 的图片、页面和目录版本依次同步，网络故障期间云端可能暂时落后于本地图。`randpic_oss_no_upload_list` 只影响 OSS，本地网页仍显示所有配置的分类；此前已同步的分类被排除后，其受本插件跟踪的云端对象会清理。未被数据库跟踪的历史对象不保证自动清理。
+OSS 同步使用 Python SDK 分页列举远端对象，与本地发布文件比较大小和校验值，仅上传变化的文件，并清理不在本地发布清单中的云端图库文件。普通对象直接比较列表中的 ETag 与本地 MD5；遇到不一致的同大小对象，再查询 SHA-256 自定义元数据。无需安装 `ossutil`，数据库不保存云端状态或同步任务。
+
+后台只在内容变化或失败待重试时同步，失败后约每 30 秒重新比较；重启后重新检查远端，继续完成差异同步。网络故障期间云端可能暂时落后于本地图。`randpic_oss_no_upload_list` 只影响 OSS，本地网页仍显示所有配置的分类；被排除或移除的分类，其云端图库文件会在比较后清理。
+
+Bucket 根目录的 `index.html`、`<指令>/index.html`、分页及 `randpic_<指令>_<编号>.<图片后缀>` 均由插件维护。其他命名的文件不参与自动删除；请不要与其他站点共用上述路径。
 
 ## 💿 安装
 
@@ -191,8 +207,8 @@ randpic_oss_no_upload_list=["capoo"]
 
 群管理员或群主引用本插件发送的图库图片，再发送 `删除` 即可。添加成功时 Bot 回复的图片也支持引用删除。
 
-- 删除会移除原图和网页副本，并将数据库中的图片标记为已删除，并在启用 OSS 时删除云端对象、更新页面；影响所有群共用的图库，不会撤回已经发送的 QQ 消息。
-- 消息与图片的对应关系会持久化，重启 Bot 后仍可引用删除。升级前发送、尚未建立对应记录的历史消息无法直接删除，请让 Bot 重新发送对应图片后再操作。
+- 删除会移除原图和网页副本，并将数据库中的图片标记为已删除，在启用 OSS 时删除云端对象、更新页面；影响所有群共用的图库，不会撤回已经发送的 QQ 消息。
+- 消息与图片的对应关系会持久化，重启 Bot 后仍可引用删除。没有对应记录的消息无法直接删除，请让 Bot 重新发送对应图片后再操作。
 - 删除过的图片再次引用会提示已删除；若网页或 OSS 同步失败，再次引用原消息发送 `删除` 可重试，`上传oss` 也会重试未完成的删除同步。
 
 ### 效果图

nonebot_plugin_randpic-2.1.0/nonebot_plugin_randpic/ali_oss.py

@@ -0,0 +1,132 @@
+"""复用异步 OSS 客户端；不持有图库或数据库的锁。"""
+
+import asyncio
+import hashlib
+import mimetypes
+import re
+from pathlib import Path, PurePosixPath
+from typing import Dict, Optional
+
+import alibabacloud_oss_v2 as oss
+import alibabacloud_oss_v2.aio as oss_aio
+from nonebot.log import logger
+
+from .config import Config
+from .models import PublishedFile, RandpicError, RemoteObject
+from .utils import SUPPORTED_EXTENSIONS, parse_index, run_blocking, valid_command
+
+
+def is_gallery_key(key: str) -> bool:
+    """只处理插件自己的网页和规范命名图片，不清理 Bucket 内任意文件。"""
+    if key == 'index.html':
+        return True
+    parts = PurePosixPath(key).parts
+    if len(parts) != 2 or key != '/'.join(parts) or not valid_command(parts[0]):
+        return False
+    command, filename = parts
+    if filename == 'index.html' or re.fullmatch(r'page-[1-9][0-9]*\.html', filename):
+        return True
+    path = PurePosixPath(filename)
+    return path.suffix.lower() in SUPPORTED_EXTENSIONS and parse_index(path.stem, command) is not None
+
+
+class OSSUploaderV2:
+    def __init__(self, config: Config):
+        credentials = oss.credentials.StaticCredentialsProvider(
+            access_key_id=config.randpic_oss_access_key_id,
+            access_key_secret=config.randpic_oss_access_key_secret,
+        )
+        self.cfg = oss.config.load_default()
+        self.cfg.credentials_provider = credentials
+        self.cfg.region = config.randpic_region
+        self.bucket = config.randpic_bucket
+        self.client = None
+
+    async def open(self) -> None:
+        if self.client is None:
+            self.client = oss_aio.AsyncClient(self.cfg)
+
+    async def close(self) -> None:
+        if self.client is not None:
+            await self.client.close()
+            self.client = None
+
+    async def list_files(self) -> Dict[str, RemoteObject]:
+        """完整分页列举成功后才交给同步器，避免部分列表造成错误删除。"""
+        await self.open()
+        objects: Dict[str, RemoteObject] = {}
+        token = None
+        seen_tokens = set()
+        while True:
+            result = await asyncio.wait_for(self.client.list_objects_v2(oss.ListObjectsV2Request(
+                bucket=self.bucket, max_keys=1000, continuation_token=token,
+            )), timeout=60)
+            if not 200 <= result.status_code < 300:
+                raise RandpicError(f'列举 OSS 对象失败：HTTP {result.status_code}')
+            for item in result.contents or []:
+                if item.key is not None and is_gallery_key(item.key):
+                    objects[item.key] = RemoteObject(int(item.size), str(item.etag or '').strip('"').lower())
+            if not result.is_truncated:
+                return objects
+            next_token = result.next_continuation_token
+            if not next_token or next_token in seen_tokens:
+                raise RandpicError('OSS 分页标记无效，本轮停止同步。')
+            seen_tokens.add(next_token)
+            token = next_token
+
+    async def matches_file(self, key: str, file: PublishedFile, remote: RemoteObject) -> bool:
+        if remote.size != file.size:
+            return False
+        if remote.etag == file.etag:
+            return True
+        # 分片上传或加密对象的 ETag 不保证为 MD5，按本插件的 SHA-256 元数据核对。
+        await self.open()
+        try:
+            result = await asyncio.wait_for(self.client.head_object(oss.HeadObjectRequest(
+                bucket=self.bucket, key=key,
+            )), timeout=60)
+        except oss.exceptions.ServiceError as exc:
+            if exc.code == 'NoSuchKey':
+                return False
+            raise
+        if not 200 <= result.status_code < 300:
+            raise RandpicError(f'读取 OSS 对象属性失败：{key} (HTTP {result.status_code})')
+        return (result.content_length == file.size
+                and (result.metadata or {}).get('randpic-sha256') == file.content_hash)
+
+    async def upload_file(self, local_file_path: str, oss_key: str, expected_hash: Optional[str] = None) -> bool:
+        try:
+            await self.open()
+            data = await run_blocking(Path(local_file_path).read_bytes)
+            content_hash = hashlib.sha256(data).hexdigest()
+            if expected_hash is not None and content_hash != expected_hash:
+                return False  # 任务已过期，后续版本会重新上传，不发送旧内容。
+            result = await asyncio.wait_for(self.client.put_object(oss.PutObjectRequest(
+                bucket=self.bucket, key=oss_key, body=data,
+                metadata={'randpic-sha256': content_hash},
+                content_type=mimetypes.guess_type(oss_key)[0] or 'application/octet-stream',
+                cache_control='no-cache' if oss_key.endswith('.html') else 'public, max-age=31536000, immutable',
+            )), timeout=60)
+            if 200 <= result.status_code < 300:
+                return True
+            logger.error(f'OSS 上传失败：{oss_key} (HTTP {result.status_code})')
+        except Exception as exc:
+            logger.error(f'OSS 上传失败：{oss_key} ({exc})')
+        return False
+
+    async def delete_file(self, oss_key: str) -> bool:
+        try:
+            await self.open()
+            result = await asyncio.wait_for(self.client.delete_object(
+                oss.DeleteObjectRequest(bucket=self.bucket, key=oss_key)
+            ), timeout=60)
+            if 200 <= result.status_code < 300:
+                return True
+            logger.error(f'OSS 删除失败：{oss_key} (HTTP {result.status_code})')
+        except oss.exceptions.ServiceError as exc:
+            if exc.code == 'NoSuchKey':
+                return True
+            logger.error(f'OSS 删除失败：{oss_key} ({exc})')
+        except Exception as exc:
+            logger.error(f'OSS 删除失败：{oss_key} ({exc})')
+        return False

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/library.py

@@ -38,6 +38,7 @@ class ImageLibrary:
         self.settings = CommandSettings(self.root)
         self.journal = FileJournal(self.root, self.repository)
         self.store_lock = StoreLock(self.root / 'database' / 'randpic.lock')
+        self.revision = 0
 
     def start(self) -> None:
         for directory in (self.root, self.image_root, self.root / 'database'):
@@ -61,7 +62,7 @@ class ImageLibrary:
     def _candidates(self, command: str, existing: List[ImageRecord]) -> List[_Candidate]:
         directory = safe_path(self.image_root, f'{command}/placeholder').parent
         directory.mkdir(parents=True, exist_ok=True)
-        cached = {image.content_hash: image for image in existing if image.validated}
+        cached = {image.content_hash: image for image in existing}
         candidates = []
         for source in sorted(directory.iterdir()):
             if source.is_symlink() or not source.is_file() or source.name.startswith('.randpic-'):
@@ -161,15 +162,15 @@ class ImageLibrary:
                             self.journal.write(image, candidate.source, candidate.payload)
                         destination = self.image_root / image.key
                         stat = destination.stat()
-                        image = replace(image, byte_size=stat.st_size, mtime_ns=stat.st_mtime_ns)
+                        image = replace(image, byte_size=stat.st_size)
                         self.repository.upsert_image(db, image)
                         records.append(image)
                     self.repository.set_sequence(db, command, maximum)
                     old_state = {(i.id, i.filename, i.content_hash) for i in existing}
                     new_state = {(i.id, i.filename, i.content_hash) for i in records}
                     changed = changed or old_state != new_state
-                if changed:
-                    self.repository.mark_dirty(db)
+            if changed:
+                self.revision += 1
         finally:
             # 事务已经提交时清理备份，事务失败时还原文件。
             self.journal.recover()
@@ -197,12 +198,12 @@ class ImageLibrary:
                 payload = self.journal.stage(processed.data)
                 self.journal.write(image, None, payload)
                 stat = (directory / image.filename).stat()
-                image = replace(image, byte_size=stat.st_size, mtime_ns=stat.st_mtime_ns)
+                image = replace(image, byte_size=stat.st_size)
                 self.repository.upsert_image(db, image)
                 self.repository.set_sequence(db, command, index)
-                self.repository.mark_dirty(db)
         finally:
             self.journal.recover()
+        self.revision += 1
         saved = self.settings.add(command)
         return MutationResult(image, settings_saved=saved)
 
@@ -216,11 +217,11 @@ class ImageLibrary:
                 if path.exists() and (not path.is_file() or digest_file(path) != image.content_hash):
                     raise RandpicError('这条消息对应的文件已被替换，未执行删除。请重新发送当前图片后再操作。')
                 self.journal.delete(image)
-                db.execute('UPDATE randpic_images SET deleted = 1 WHERE id = ?', (image.id,))
-                self.repository.mark_dirty(db)
+                self.repository.mark_deleted(db, image.id)
                 result = MutationResult(replace(image, deleted=True))
         finally:
             self.journal.recover()
+        self.revision += 1
         return result
 
     def pick(self, command: str) -> Optional[ImageSnapshot]:
@@ -237,6 +238,6 @@ class ImageLibrary:
             except FileNotFoundError:
                 # 清理在运行中意外消失的文件，并继续挑选其他图片。
                 with self.repository.connection(write=True) as db:
-                    db.execute('UPDATE randpic_images SET deleted = 1 WHERE id = ?', (image.id,))
-                    self.repository.mark_dirty(db)
+                    self.repository.mark_deleted(db, image.id)
+                self.revision += 1
                 logger.warning(f'图片文件已丢失，已从图库索引移除：{path}')

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/models.py

@@ -25,9 +25,7 @@ class ImageRecord:
     phash: str
     send_count: int = 0
     byte_size: int = 0
-    mtime_ns: int = 0
     deleted: bool = False
-    validated: bool = True
 
     @property
     def key(self) -> str:
@@ -49,12 +47,17 @@ class MutationResult:
 
 
 @dataclass(frozen=True)
-class SyncJob:
-    key: str
-    action: str
-    local_path: Optional[str]
+class PublishedFile:
+    path: str
     content_hash: str
-    revision: int
+    etag: str
+    size: int
+
+
+@dataclass(frozen=True)
+class RemoteObject:
+    size: int
+    etag: str
 
 
 @dataclass(frozen=True)

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/repository.py

@@ -4,41 +4,40 @@ import sqlite3
 from contextlib import contextmanager
 from dataclasses import fields
 from pathlib import Path
-from typing import Iterator, List, Mapping, Optional, Tuple
+from typing import Iterator, List, Optional
 
-from .models import ImageRecord, RandpicError, SyncJob
+from .models import ImageRecord, RandpicError
 
 
 IMAGE_COLUMNS = ', '.join(field.name for field in fields(ImageRecord))
 SCHEMA = '''
-CREATE TABLE IF NOT EXISTS randpic_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
 CREATE TABLE IF NOT EXISTS randpic_images (
     id TEXT PRIMARY KEY, command TEXT NOT NULL, filename TEXT NOT NULL,
-    content_hash TEXT NOT NULL, phash TEXT NOT NULL, send_count INTEGER NOT NULL DEFAULT 0 CHECK(send_count >= 0),
-    byte_size INTEGER NOT NULL DEFAULT 0, mtime_ns INTEGER NOT NULL DEFAULT 0,
-    deleted INTEGER NOT NULL DEFAULT 0 CHECK(deleted IN (0, 1)),
-    validated INTEGER NOT NULL DEFAULT 1 CHECK(validated IN (0, 1))
-);
-CREATE UNIQUE INDEX IF NOT EXISTS randpic_image_path ON randpic_images(command, filename) WHERE deleted = 0;
-CREATE INDEX IF NOT EXISTS randpic_image_command ON randpic_images(command, deleted, id);
+    content_hash TEXT NOT NULL, phash TEXT NOT NULL,
+    send_count INTEGER NOT NULL DEFAULT 0 CHECK(send_count >= 0),
+    byte_size INTEGER NOT NULL DEFAULT 0 CHECK(byte_size >= 0),
+    deleted INTEGER NOT NULL DEFAULT 0 CHECK(deleted IN (0, 1))
+) WITHOUT ROWID;
+CREATE UNIQUE INDEX IF NOT EXISTS randpic_image_path
+    ON randpic_images(command, filename) WHERE deleted = 0;
+CREATE INDEX IF NOT EXISTS randpic_image_command
+    ON randpic_images(command, id) WHERE deleted = 0;
+CREATE INDEX IF NOT EXISTS randpic_image_hash
+    ON randpic_images(command, content_hash) WHERE deleted = 0;
 CREATE TABLE IF NOT EXISTS randpic_messages (
     bot_id TEXT NOT NULL, group_id INTEGER NOT NULL, message_id INTEGER NOT NULL,
-    image_id TEXT NOT NULL REFERENCES randpic_images(id), PRIMARY KEY(bot_id, group_id, message_id)
-);
-CREATE TABLE IF NOT EXISTS randpic_sequences (command TEXT PRIMARY KEY, last_index INTEGER NOT NULL);
-CREATE TABLE IF NOT EXISTS randpic_remote (key TEXT PRIMARY KEY, content_hash TEXT NOT NULL);
-CREATE TABLE IF NOT EXISTS randpic_sync_jobs (
-    key TEXT PRIMARY KEY, action TEXT NOT NULL CHECK(action IN ('upload', 'delete')),
-    local_path TEXT, content_hash TEXT NOT NULL, revision INTEGER NOT NULL,
-    attempts INTEGER NOT NULL DEFAULT 0, last_error TEXT
-);
+    image_id TEXT NOT NULL REFERENCES randpic_images(id),
+    PRIMARY KEY(bot_id, group_id, message_id)
+) WITHOUT ROWID;
+CREATE TABLE IF NOT EXISTS randpic_sequences (
+    command TEXT PRIMARY KEY, last_index INTEGER NOT NULL CHECK(last_index >= 0)
+) WITHOUT ROWID;
 '''
 
 
 def _image(row: sqlite3.Row) -> ImageRecord:
     values = dict(row)
     values['deleted'] = bool(values['deleted'])
-    values['validated'] = bool(values['validated'])
     return ImageRecord(**values)
 
 
@@ -74,33 +73,16 @@ class Repository:
         finally:
             db.close()
 
-    @staticmethod
-    def meta(db: sqlite3.Connection, key: str, default: str = '0') -> str:
-        row = db.execute('SELECT value FROM randpic_meta WHERE key = ?', (key,)).fetchone()
-        return row[0] if row else default
-
-    @staticmethod
-    def set_meta(db: sqlite3.Connection, key: str, value: str) -> None:
-        db.execute('INSERT OR REPLACE INTO randpic_meta VALUES (?, ?)', (key, value))
-
-    def mark_dirty(self, db: sqlite3.Connection) -> None:
-        self.set_meta(db, 'catalog_revision', str(int(self.meta(db, 'catalog_revision')) + 1))
-
     def initialize(self) -> None:
         self.path.parent.mkdir(parents=True, exist_ok=True)
         with self.connection() as db:
-            names = {row[0] for row in db.execute("SELECT name FROM sqlite_master WHERE type = 'table'")}
-            version = int(self.meta(db, 'schema_version')) if 'randpic_meta' in names else 0
-            if version > 2:
-                raise RandpicError('数据库来自更新版本的 Randpic，不能用 2.0.0 覆盖。')
             db.execute('PRAGMA journal_mode = WAL')
             db.execute('PRAGMA synchronous = FULL')
             db.executescript(SCHEMA)
-        with self.connection(write=True) as db:
-            if int(self.meta(db, 'schema_version')) == 0:
-                self.set_meta(db, 'schema_version', '2')
-                self.set_meta(db, 'published_revision', '-1')
-                self.mark_dirty(db)
+
+    @staticmethod
+    def mark_deleted(db: sqlite3.Connection, image_id: str) -> None:
+        db.execute('UPDATE randpic_images SET deleted = 1 WHERE id = ?', (image_id,))
 
     @staticmethod
     def upsert_image(db: sqlite3.Connection, image: ImageRecord) -> None:
@@ -145,9 +127,14 @@ class Repository:
 
     @staticmethod
     def duplicate(db: sqlite3.Connection, command: str, content_hash: str, phash: str) -> bool:
-        return db.execute('''SELECT 1 FROM randpic_images WHERE command = ? AND deleted = 0
-            AND (content_hash = ? OR hamming_distance(phash, ?) <= 5) LIMIT 1''',
-                          (command, content_hash, phash)).fetchone() is not None
+        exact = db.execute('''SELECT 1 FROM randpic_images
+            WHERE command = ? AND deleted = 0 AND content_hash = ? LIMIT 1''',
+                           (command, content_hash)).fetchone()
+        if exact is not None:
+            return True
+        return db.execute('''SELECT 1 FROM randpic_images
+            WHERE command = ? AND deleted = 0 AND hamming_distance(phash, ?) <= 5 LIMIT 1''',
+                          (command, phash)).fetchone() is not None
 
     @staticmethod
     def sequence(db: sqlite3.Connection, command: str) -> int:
@@ -179,53 +166,3 @@ class Repository:
         if row is None:
             raise RandpicError('找不到这条消息对应的图库图片，请引用本插件发送的图片。')
         return _image(row)
-
-    def revision(self) -> Tuple[int, int]:
-        with self.connection() as db:
-            return int(self.meta(db, 'catalog_revision')), int(self.meta(db, 'published_revision', '-1'))
-
-    def published(self, revision: int) -> None:
-        with self.connection(write=True) as db:
-            self.set_meta(db, 'published_revision', str(revision))
-
-    def reconcile_remote(self, manifest: Mapping[str, Tuple[str, str]], *, force: bool = False) -> None:
-        with self.connection(write=True) as db:
-            known = {row['key']: row['content_hash'] for row in db.execute('SELECT * FROM randpic_remote')}
-            jobs = {row['key']: row for row in db.execute('SELECT * FROM randpic_sync_jobs')}
-            revision = int(self.meta(db, 'sync_revision'))
-
-            def enqueue(key: str, action: str, path: Optional[str], content_hash: str) -> None:
-                nonlocal revision
-                old = jobs.get(key)
-                if old and (old['action'], old['local_path'], old['content_hash']) == (action, path, content_hash) and not force:
-                    return
-                revision += 1
-                db.execute('''INSERT OR REPLACE INTO randpic_sync_jobs
-                    (key, action, local_path, content_hash, revision) VALUES (?, ?, ?, ?, ?)''',
-                           (key, action, path, content_hash, revision))
-
-            for key, (path, content_hash) in manifest.items():
-                if force or known.get(key) != content_hash or key in jobs:
-                    enqueue(key, 'upload', path, content_hash)
-            for key in (set(known) | set(jobs)) - set(manifest):
-                enqueue(key, 'delete', None, '')
-            self.set_meta(db, 'sync_revision', str(revision))
-
-    def sync_jobs(self) -> List[SyncJob]:
-        with self.connection() as db:
-            return [SyncJob(row['key'], row['action'], row['local_path'], row['content_hash'], row['revision'])
-                    for row in db.execute('SELECT * FROM randpic_sync_jobs ORDER BY revision')]
-
-    def complete_job(self, job: SyncJob) -> None:
-        with self.connection(write=True) as db:
-            if job.action == 'delete':
-                db.execute('DELETE FROM randpic_remote WHERE key = ?', (job.key,))
-            else:
-                db.execute('INSERT OR REPLACE INTO randpic_remote VALUES (?, ?)', (job.key, job.content_hash))
-            # 老请求完成时，只能确认自己的版本，不能覆盖期间新加入的删除/上传任务。
-            db.execute('DELETE FROM randpic_sync_jobs WHERE key = ? AND revision = ?', (job.key, job.revision))
-
-    def fail_job(self, job: SyncJob, error: str) -> None:
-        with self.connection(write=True) as db:
-            db.execute('''UPDATE randpic_sync_jobs SET attempts = attempts + 1, last_error = ?
-                WHERE key = ? AND revision = ?''', (error[:500], job.key, job.revision))

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/service.py

@@ -1,9 +1,9 @@
-"""异步应用服务：业务串行提交，网页及时发布，远端任务独立重试。"""
+"""异步应用服务：业务串行提交，网页及时发布，按远端实际状态同步。"""
 
 import asyncio
 from dataclasses import replace
 from pathlib import Path
-from typing import Any, Optional
+from typing import Any, Dict, Optional
 from urllib.parse import urlsplit
 
 import httpx
@@ -12,7 +12,7 @@ from nonebot.log import logger
 from .ali_oss import OSSUploaderV2
 from .config import Config
 from .library import ImageLibrary
-from .models import ImageRecord, ImageSnapshot, MutationResult, RandpicError, SyncResult
+from .models import ImageRecord, ImageSnapshot, MutationResult, PublishedFile, RandpicError, SyncResult
 from .progress import SyncProgress
 from .utils import run_blocking
 from .web import GalleryPublisher
@@ -35,6 +35,10 @@ class RandpicService:
         self._worker: Optional[asyncio.Task] = None
         self.ready = False
         self._web_dirty = True
+        self._published_revision = -1
+        self._published_remote_enabled = False
+        self._manifest: Dict[str, PublishedFile] = {}
+        self._remote_dirty = True
         self._closed = False
 
     @property
@@ -93,19 +97,23 @@ class RandpicService:
                 finally:
                     await run_blocking(self.library.close)
 
-    async def _refresh_locked(self, *, force: bool = False, force_upload: bool = False) -> Optional[str]:
+    async def _refresh_locked(self, *, force: bool = False) -> Optional[str]:
         try:
-            revision, published = await run_blocking(self.repository.revision)
-            if not force and not force_upload and not self._web_dirty and revision == published:
+            revision = self.library.revision
+            remote_enabled = self.uploader is not None
+            if (not force and not self._web_dirty and revision == self._published_revision
+                    and remote_enabled == self._published_remote_enabled):
                 return None
             images = await run_blocking(self.repository.list_images)
             manifest = await run_blocking(
-                self.publisher.publish, images, list(self.commands), set(self.config.randpic_oss_no_upload_list)
+                self.publisher.publish, images, list(self.commands),
+                set(self.config.randpic_oss_no_upload_list), remote_enabled=remote_enabled,
             )
-            if self.uploader is not None:
-                await run_blocking(self.repository.reconcile_remote, manifest, force=force_upload)
-            await run_blocking(self.repository.published, revision)
+            self._manifest = manifest
+            self._published_revision = revision
+            self._published_remote_enabled = remote_enabled
             self._web_dirty = False
+            self._remote_dirty = remote_enabled
             return None
         except Exception as exc:
             self._web_dirty = True
@@ -156,7 +164,7 @@ class RandpicService:
             self._require_ready()
             snapshot = await run_blocking(self.library.pick, command)
             error = await self._refresh_locked()
-        if error:
+        if error or self._remote_dirty:
             self._wake.set()
         return snapshot
 
@@ -184,61 +192,100 @@ class RandpicService:
                 return SyncResult()
             async with self._mutation_lock:
                 self._require_ready()
-                error = await self._refresh_locked(force_upload=force)
+                error = await self._refresh_locked(force=force)
+                manifest = dict(self._manifest)
             if error:
-                return SyncResult(web_error=error)
-            if self.uploader is None:
+                return SyncResult(pending=1, web_error=error)
+            if self.uploader is None or (not force and not self._remote_dirty):
                 return SyncResult()
-            jobs = await run_blocking(self.repository.sync_jobs)
-            if not jobs:
-                return SyncResult()
-            succeeded = failed = completed = 0
+            self._remote_dirty = True
             started = asyncio.get_running_loop().time()
+            try:
+                remote = await self.uploader.list_files()
+            except Exception as exc:
+                logger.error(f'列举 OSS 文件失败，本轮未执行上传或删除，将自动重试：{exc}')
+                return SyncResult(failed=1, pending=1)
+
+            # 普通 PutObject 的 ETag 与 MD5 相同；大多数对象只需分页列表即可确认。
+            jobs = []
+            comparisons = []
+            failed_keys = set()
+            for key, file in manifest.items():
+                old = remote.get(key)
+                if force or old is None or old.size != file.size:
+                    jobs.append((key, file))
+                elif old.etag != file.etag:
+                    comparisons.append((key, file, old))
+            candidates = iter(comparisons)
+
+            async def compare_worker():
+                for key, file, old in candidates:
+                    try:
+                        if not await self.uploader.matches_file(key, file, old):
+                            jobs.append((key, file))
+                    except Exception as exc:
+                        failed_keys.add(key)
+                        logger.error(f'核对 OSS 文件失败，将自动重试：{key} ({exc})')
+
+            limit = self.config.randpic_upload_concurrency
+            await asyncio.gather(*(compare_worker() for _ in range(min(limit, len(comparisons)))))
+            jobs.extend((key, None) for key in remote.keys() - manifest.keys())
+            succeeded = completed = 0
 
-            with SyncProgress(len(jobs)) as progress:
-                async def phase(items):
-                    queue = asyncio.Queue()
-                    for job in items:
-                        queue.put_nowait(job)
-
-                    async def worker():
-                        nonlocal succeeded, failed, completed
-                        while not queue.empty():
-                            job = queue.get_nowait()
-                            try:
-                                if job.action == 'delete':
-                                    ok = await self.uploader.delete_file(job.key)
-                                else:
-                                    ok = await self.uploader.upload_file(job.local_path, job.key, job.content_hash)
-                                if ok:
-                                    await run_blocking(self.repository.complete_job, job)
-                                    succeeded += 1
-                                else:
-                                    await run_blocking(self.repository.fail_job, job, '远端请求失败或任务已过期')
-                                    failed += 1
-                            except Exception as exc:
-                                logger.error(f'同步 OSS 任务失败，已保留以便重试：{job.key} ({exc})')
-                                failed += 1
-                            finally:
-                                queue.task_done()
-                                completed += 1
-                                progress.update(completed, succeeded, failed)
-
-                    await asyncio.gather(*(worker() for _ in range(min(self.config.randpic_upload_concurrency, len(items)))))
-
-                # 先同步图片，再同步页面，最后更新用于浏览器自动刷新的目录版本。
-                await phase([job for job in jobs if not job.key.endswith(('.html', '.json'))])
-                await phase([job for job in jobs if job.key.endswith('.html')])
-                await phase([job for job in jobs if job.key.endswith('.json')])
-            pending = len(await run_blocking(self.repository.sync_jobs))
-            elapsed = asyncio.get_running_loop().time() - started
-            summary = (f'OSS 同步完成：成功 {succeeded}，失败 {failed}，待处理 {pending}，'
-                       f'用时 {elapsed:.1f} 秒')
-            if failed:
-                logger.warning(summary + '；失败任务将自动重试')
-            else:
-                logger.info(summary)
-            return SyncResult(succeeded=succeeded, failed=failed, pending=pending)
+            if jobs:
+                with SyncProgress(len(jobs)) as progress:
+                    async def phase(items):
+                        remaining = iter(items)
+
+                        async def worker():
+                            nonlocal succeeded, completed
+                            for key, file in remaining:
+                                try:
+                                    async with self._mutation_lock:
+                                        # 跳过被同步期间新操作替代的上传或删除。
+                                        if self._manifest.get(key) != file:
+                                            continue
+                                    if file is None:
+                                        ok = await self.uploader.delete_file(key)
+                                    else:
+                                        ok = await self.uploader.upload_file(file.path, key, file.content_hash)
+                                    if ok:
+                                        succeeded += 1
+                                    else:
+                                        failed_keys.add(key)
+                                except Exception as exc:
+                                    failed_keys.add(key)
+                                    logger.error(f'同步 OSS 文件失败，将自动重试：{key} ({exc})')
+                                finally:
+                                    completed += 1
+                                    progress.update(completed, succeeded, len(failed_keys))
+
+                        await asyncio.gather(*(worker() for _ in range(min(limit, len(items)))))
+
+                    # 先处理图片，再更新网页；网络期间不占用图库修改锁。
+                    await phase([job for job in jobs if not job[0].endswith('.html')])
+                    await phase([job for job in jobs if job[0].endswith('.html')])
+
+            async with self._mutation_lock:
+                pending_keys = failed_keys | {
+                    key for key in manifest.keys() | self._manifest.keys()
+                    if manifest.get(key) != self._manifest.get(key)
+                }
+                if self._web_dirty or self.library.revision != self._published_revision:
+                    pending_keys.add('<网页待发布>')
+                self._remote_dirty = bool(pending_keys)
+                changed = manifest != self._manifest
+            if changed:
+                self._wake.set()
+            if jobs or failed_keys:
+                elapsed = asyncio.get_running_loop().time() - started
+                summary = (f'OSS 同步完成：成功 {succeeded}，失败 {len(failed_keys)}，'
+                           f'待处理 {len(pending_keys)}，用时 {elapsed:.1f} 秒')
+                if failed_keys:
+                    logger.warning(summary + '；将重新比较远端内容后重试')
+                else:
+                    logger.info(summary)
+            return SyncResult(succeeded=succeeded, failed=len(failed_keys), pending=len(pending_keys))
 
     async def _retry_worker(self) -> None:
         while True:
@@ -249,6 +296,9 @@ class RandpicService:
             self._wake.clear()
             if self._closed:
                 return
+            if (not self._web_dirty and self.library.revision == self._published_revision
+                    and (self.uploader is None or not self._remote_dirty)):
+                continue
             try:
                 await self.sync()
             except Exception as exc:

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/utils.py

@@ -49,8 +49,8 @@ def safe_path(root: Path, key: str) -> Path:
     return path
 
 
-def digest_file(path: Path) -> str:
-    digest = hashlib.sha256()
+def digest_file(path: Path, algorithm: str = 'sha256') -> str:
+    digest = hashlib.new(algorithm, usedforsecurity=False)
     with path.open('rb') as file:
         for block in iter(lambda: file.read(128 * 1024), b''):
             digest.update(block)

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/nonebot_plugin_randpic/web.py

@@ -1,14 +1,13 @@
-"""静态网页发布：原子替换文件、分页、懒加载，以及增删后的自动刷新。"""
+"""静态网页发布：原子替换文件、分页与图片懒加载。"""
 
 import hashlib
-import json
 import math
 from html import escape
 from pathlib import Path
-from typing import Dict, Iterable, List, Mapping, Sequence, Set, Tuple
+from typing import Dict, Iterable, List, Mapping, Sequence, Set
 from urllib.parse import quote
 
-from .models import ImageRecord, RandpicError
+from .models import ImageRecord, PublishedFile, RandpicError
 from .utils import atomic_write, digest_file, parse_index, safe_path
 
 GALLERY_STYLE = '''
@@ -67,8 +66,7 @@ def _url(image: ImageRecord) -> str:
     return quote(image.filename, safe='') + '?v=' + image.content_hash[:16]
 
 
-def _shell(title: str, home: str, version: str, command: str, page: int, content: str) -> str:
-    state = json.dumps({'version': version, 'command': command, 'page': page}, ensure_ascii=False).replace('<', '\\u003c')
+def _shell(title: str, home: str, content: str) -> str:
     return f'''<!doctype html><html lang="zh-CN"><head><meta charset="utf-8">
 <meta name="viewport" content="width=device-width,initial-scale=1"><meta name="theme-color" content="#f5f6f2">
 <title>{escape(title)} · Randpic</title><style>{GALLERY_STYLE}
@@ -77,22 +75,13 @@ def _shell(title: str, home: str, version: str, command: str, page: int, content
 </style></head><body><div class="shell">
 <nav class="topbar"><a class="brand" href="{home}"><span class="brand-mark">✳</span><span>RANDPIC / 图片收藏馆</span></a><span class="top-note">捕捉喜欢的瞬间</span></nav>
 {content}<footer>Randpic · 收藏每一个喜欢的瞬间</footer></div>
-<script>(()=>{{const state={state}; const home={json.dumps(home)};
-setInterval(async()=>{{if(document.hidden)return;try{{
-const response=await fetch(home+'catalog.json',{{cache:'no-store'}});if(!response.ok)return;
-const next=await response.json();if(next.version===state.version)return;
-if(state.command && (!(state.command in next.pages)||state.page>next.pages[state.command])){{location.href=state.command in next.pages?'./':home;return;}}
-location.reload();}}catch(error){{}}}},15000);}})();</script></body></html>'''
+</body></html>'''
 
 
 def render_catalog(groups: Mapping[str, Sequence[ImageRecord]], page_size: int) -> Dict[str, bytes]:
-    """相同图库总是产生相同 HTML；发送次数变化不会导致网页重复刷新。"""
-    identity = [(command, [(image.id, image.filename, image.content_hash) for image in images])
-                for command, images in sorted(groups.items())]
-    version = hashlib.sha256(json.dumps(identity, ensure_ascii=False).encode()).hexdigest()
+    """生成纯静态 HTML，只让内容实际变化的页面产生同步任务。"""
     pages = {command: max(1, math.ceil(len(images) / page_size)) for command, images in groups.items()}
-    catalog = json.dumps({'version': version, 'pages': pages}, ensure_ascii=False, sort_keys=True).encode()
-    files = {'catalog.json': catalog}
+    files: Dict[str, bytes] = {}
     total = sum(len(images) for images in groups.values())
     cards = []
     for command, images in sorted(groups.items()):
@@ -106,7 +95,7 @@ def render_catalog(groups: Mapping[str, Sequence[ImageRecord]], page_size: int)
     home = f'''<header class="hero"><p class="eyebrow">YOUR IMAGE LIBRARY</p><h1>每一张，都值得<br>被好好收藏。</h1>
 <p class="hero-copy">在这里慢慢翻看收藏的每一份灵感。</p></header><div class="section-head"><h2>全部分类</h2>
 <span class="count">{len(groups)} 个分类　·　{total} 张图片</span></div><main class="folder-grid">{''.join(cards)}</main>'''
-    files['index.html'] = _shell('图片收藏馆', './', version, '', 1, home).encode()
+    files['index.html'] = _shell('图片收藏馆', './', home).encode()
     for command, images in groups.items():
         for page in range(1, pages[command] + 1):
             cards = []
@@ -128,7 +117,7 @@ def render_catalog(groups: Mapping[str, Sequence[ImageRecord]], page_size: int)
 <header class="gallery-title"><p class="eyebrow">COLLECTION</p><h1>{escape(command)}</h1><p class="hero-copy">本分类共收藏 {len(images)} 张图片。</p></header>
 <main class="image-grid">{''.join(cards)}</main><nav class="pagination">{''.join(navigation)}</nav>'''
             filename = 'index.html' if page == 1 else f'page-{page}.html'
-            files[f'{command}/{filename}'] = _shell(command, '../', version, command, page, content).encode()
+            files[f'{command}/{filename}'] = _shell(command, '../', content).encode()
     return files
 
 
@@ -140,6 +129,7 @@ class GalleryPublisher:
         self.remote_directory = root / 'database' / 'oss_pages'
         self.page_size = page_size
         self._asset_hashes: Dict[str, str] = {}
+        self._asset_md5: Dict[str, str] = {}
 
     @staticmethod
     def _write_pages(directory: Path, pages: Mapping[str, bytes]) -> None:
@@ -163,8 +153,8 @@ class GalleryPublisher:
                 except OSError:
                     pass
 
-    def publish(self, images: Sequence[ImageRecord], commands: Iterable[str], excluded: Set[str]) -> Dict[str, Tuple[str, str]]:
-        for directory in (self.local_directory, self.remote_directory):
+    def publish(self, images: Sequence[ImageRecord], commands: Iterable[str], excluded: Set[str], *, remote_enabled: bool = False) -> Dict[str, PublishedFile]:
+        for directory in ((self.local_directory, self.remote_directory) if remote_enabled else (self.local_directory,)):
             if directory.is_symlink():
                 raise RandpicError('网页输出目录不能是符号链接。')
             directory.mkdir(parents=True, exist_ok=True)
@@ -175,6 +165,8 @@ class GalleryPublisher:
         for command in groups:
             groups[command].sort(key=lambda image: (parse_index(Path(image.filename).stem, command) or 0, image.filename))
         assets: Dict[str, str] = {}
+        checksums: Dict[str, str] = {}
+        files: Dict[str, PublishedFile] = {}
         for group in groups.values():
             for image in group:
                 path = safe_path(self.local_directory, image.key)
@@ -186,16 +178,26 @@ class GalleryPublisher:
                             raise RandpicError(f'图片文件在运行时被修改：{image.key}')
                         atomic_write(path, data)
                 assets[image.key] = image.content_hash
+                if remote_enabled and image.command not in excluded:
+                    checksum = self._asset_md5.get(image.key) if known else None
+                    if checksum is None:
+                        checksum = digest_file(path, 'md5')
+                    checksums[image.key] = checksum
+                    files[image.key] = PublishedFile(str(path), image.content_hash, checksum, image.byte_size)
         local_pages = render_catalog(groups, self.page_size)
         self._write_pages(self.local_directory, local_pages)
         self._prune(self.local_directory, set(local_pages) | set(assets))
+        self._asset_hashes, self._asset_md5 = assets, checksums
+        if not remote_enabled:
+            return {}
         remote_groups = {command: group for command, group in groups.items() if command not in excluded}
         remote_pages = render_catalog(remote_groups, self.page_size)
         self._write_pages(self.remote_directory, remote_pages)
         self._prune(self.remote_directory, set(remote_pages))
-        self._asset_hashes = assets
-        manifest = {key: (str(self.remote_directory / key), hashlib.sha256(data).hexdigest())
-                    for key, data in remote_pages.items()}
-        manifest.update({image.key: (str(self.local_directory / image.key), image.content_hash)
+        manifest = {key: PublishedFile(
+            str(self.remote_directory / key), hashlib.sha256(data).hexdigest(),
+            hashlib.md5(data, usedforsecurity=False).hexdigest(), len(data),
+        ) for key, data in remote_pages.items()}
+        manifest.update({image.key: files[image.key]
                          for group in remote_groups.values() for image in group})
         return manifest

{nonebot_plugin_randpic-2.0.1 → nonebot_plugin_randpic-2.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "nonebot-plugin-randpic"
-version = "2.0.1"
+version = "2.1.0"
 description = "一个通过指令随机发送你存储的图片的nonebot2插件"
 authors = ["HuParry <hu_parry@qq.com>"]
 license = "MIT"

nonebot_plugin_randpic-2.0.1/nonebot_plugin_randpic/ali_oss.py

@@ -1,71 +0,0 @@
-"""复用异步 OSS 客户端；不持有图库或数据库的锁。"""
-
-import asyncio
-import hashlib
-import mimetypes
-from pathlib import Path
-from typing import Optional
-
-import alibabacloud_oss_v2 as oss
-import alibabacloud_oss_v2.aio as oss_aio
-from nonebot.log import logger
-
-from .config import Config
-from .utils import run_blocking
-
-
-class OSSUploaderV2:
-    def __init__(self, config: Config):
-        credentials = oss.credentials.StaticCredentialsProvider(
-            access_key_id=config.randpic_oss_access_key_id,
-            access_key_secret=config.randpic_oss_access_key_secret,
-        )
-        self.cfg = oss.config.load_default()
-        self.cfg.credentials_provider = credentials
-        self.cfg.region = config.randpic_region
-        self.bucket = config.randpic_bucket
-        self.client = None
-
-    async def open(self) -> None:
-        if self.client is None:
-            self.client = oss_aio.AsyncClient(self.cfg)
-
-    async def close(self) -> None:
-        if self.client is not None:
-            await self.client.close()
-            self.client = None
-
-    async def upload_file(self, local_file_path: str, oss_key: str, expected_hash: Optional[str] = None) -> bool:
-        try:
-            await self.open()
-            data = await run_blocking(Path(local_file_path).read_bytes)
-            if expected_hash is not None and hashlib.sha256(data).hexdigest() != expected_hash:
-                return False  # 任务已过期，后续版本会重新上传，不发送旧内容。
-            result = await asyncio.wait_for(self.client.put_object(oss.PutObjectRequest(
-                bucket=self.bucket, key=oss_key, body=data,
-                content_type=mimetypes.guess_type(oss_key)[0] or 'application/octet-stream',
-                cache_control='no-cache' if oss_key.endswith(('.html', '.json')) else 'public, max-age=31536000, immutable',
-            )), timeout=60)
-            if 200 <= result.status_code < 300:
-                return True
-            logger.error(f'OSS 上传失败：{oss_key} (HTTP {result.status_code})')
-        except Exception as exc:
-            logger.error(f'OSS 上传失败：{oss_key} ({exc})')
-        return False
-
-    async def delete_file(self, oss_key: str) -> bool:
-        try:
-            await self.open()
-            result = await asyncio.wait_for(self.client.delete_object(
-                oss.DeleteObjectRequest(bucket=self.bucket, key=oss_key)
-            ), timeout=60)
-            if 200 <= result.status_code < 300:
-                return True
-            logger.error(f'OSS 删除失败：{oss_key} (HTTP {result.status_code})')
-        except oss.exceptions.ServiceError as exc:
-            if exc.code == 'NoSuchKey':
-                return True
-            logger.error(f'OSS 删除失败：{oss_key} ({exc})')
-        except Exception as exc:
-            logger.error(f'OSS 删除失败：{oss_key} ({exc})')
-        return False