qentimart-sdk 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of qentimart-sdk might be problematic. Click here for more details.
- qentimart_sdk-0.1.0/PKG-INFO +89 -0
- qentimart_sdk-0.1.0/README.md +80 -0
- qentimart_sdk-0.1.0/pyproject.toml +19 -0
- qentimart_sdk-0.1.0/qentimart/__init__.py +7 -0
- qentimart_sdk-0.1.0/qentimart/_errors.py +24 -0
- qentimart_sdk-0.1.0/qentimart/_http.py +121 -0
- qentimart_sdk-0.1.0/qentimart/_localfs.py +201 -0
- qentimart_sdk-0.1.0/qentimart/_plan.py +195 -0
- qentimart_sdk-0.1.0/qentimart/_state.py +104 -0
- qentimart_sdk-0.1.0/qentimart/_static.py +132 -0
- qentimart_sdk-0.1.0/qentimart/_sync.py +214 -0
- qentimart_sdk-0.1.0/qentimart/_wire.py +169 -0
- qentimart_sdk-0.1.0/qentimart/cli.py +110 -0
- qentimart_sdk-0.1.0/qentimart/client.py +343 -0
- qentimart_sdk-0.1.0/qentimart_sdk.egg-info/PKG-INFO +89 -0
- qentimart_sdk-0.1.0/qentimart_sdk.egg-info/SOURCES.txt +18 -0
- qentimart_sdk-0.1.0/qentimart_sdk.egg-info/dependency_links.txt +1 -0
- qentimart_sdk-0.1.0/qentimart_sdk.egg-info/entry_points.txt +2 -0
- qentimart_sdk-0.1.0/qentimart_sdk.egg-info/top_level.txt +1 -0
- qentimart_sdk-0.1.0/setup.cfg +4 -0
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: qentimart-sdk
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: QentimArt 下游 SDK:拉取游戏资源版本,支持块级增量更新,无需本地对象缓存
|
|
5
|
+
Author: Pimeng
|
|
6
|
+
License: MIT
|
|
7
|
+
Requires-Python: >=3.8
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
|
|
10
|
+
# qentimart-sdk(Python)
|
|
11
|
+
|
|
12
|
+
QentimArt 的 Python 下游客户端,仅使用 Python 标准库,要求 Python ≥ 3.8。本地 `.qentimart/` 状态目录格式与 [Node.js SDK](../nodejs/README.md)、[Java SDK](../java/README.md) 通用,几个 SDK 可以读写同一个素材目录。
|
|
13
|
+
|
|
14
|
+
- 按数据块增量更新,文件移动或大文件局部修改时直接从本地旧文件复用数据块
|
|
15
|
+
- 不保留本地块缓存,同步完成后磁盘上只有业务素材和 `.qentimart/` 元数据(实测总占用约为素材体积的 100.04%)
|
|
16
|
+
- 多线程复用 HTTP 长连接并发拉取,落盘前逐块校验 SHA-256,写完临时文件后原子替换
|
|
17
|
+
|
|
18
|
+
## 安装
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
pip install ./sdk/python
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## 快速开始
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
from qentimart import QentimArtClient
|
|
28
|
+
|
|
29
|
+
with QentimArtClient("http://assets.internal:8080", "mygame", token="<read token>") as client:
|
|
30
|
+
r = client.sync("./assets", on_progress=lambda p: print(p["phase"], p["done"], p["total"]))
|
|
31
|
+
print(r["version"], r["storage"]["ratio"]) # 例如 1.2.0 1.0004
|
|
32
|
+
client.sync("./assets", version="1.0.0") # 切换到指定版本
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
更多用法见 [examples/plugin_example.py](examples/plugin_example.py)
|
|
36
|
+
|
|
37
|
+
## 静态模式
|
|
38
|
+
|
|
39
|
+
当素材通过静态模式托管在对象存储、CDN 或 Cloudflare Pages 上时,传入站点地址 `base_url` 即可,不需要 `server` 和 `token`:
|
|
40
|
+
|
|
41
|
+
```python
|
|
42
|
+
from qentimart import QentimArtClient
|
|
43
|
+
|
|
44
|
+
with QentimArtClient(base_url="https://mygame.pages.dev") as client:
|
|
45
|
+
client.sync("./assets")
|
|
46
|
+
client.sync("./assets", version="1.0.0")
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`sync`、`plan`、`status`、`verify`、`diff`、`storage_report` 的调用方式与连接服务端时相同
|
|
50
|
+
|
|
51
|
+
## API
|
|
52
|
+
|
|
53
|
+
| 方法 | 说明 |
|
|
54
|
+
|---|---|
|
|
55
|
+
| `QentimArtClient(server, project, token=None, concurrency=8)` | 创建连接服务端的客户端(支持 `with` 自动关闭连接) |
|
|
56
|
+
| `QentimArtClient(base_url=..., concurrency=8)` | 创建连接静态托管源的只读客户端 |
|
|
57
|
+
| `list_versions()` / `get_version(v="latest")` / `diff(from_, to)` | 查询版本列表、单版本元数据与版本间差异 |
|
|
58
|
+
| `sync(dir, **opts)` | 将目录同步到目标版本(默认最新正式版) |
|
|
59
|
+
| `plan(dir, **opts)` | 仅预估待更新、删除的文件列表及网络下载量,不改写磁盘 |
|
|
60
|
+
| `status(dir)` | 离线检查本地被修改、缺失以及多出的文件 |
|
|
61
|
+
| `verify(dir)` | 重新计算全部受管文件的 SHA-256,排查文件损坏 |
|
|
62
|
+
| `QentimArtClient.storage_report(dir)` | 返回磁盘占用统计 `{"asset_bytes", "overhead_bytes", "ratio"}` |
|
|
63
|
+
|
|
64
|
+
### `sync` 参数
|
|
65
|
+
|
|
66
|
+
| 参数 | 默认值 | 说明 |
|
|
67
|
+
|---|---|---|
|
|
68
|
+
| `version` | `"latest"` | 目标版本号 |
|
|
69
|
+
| `local_changes` | `"overwrite"` | `overwrite` 还原本地改动并删除多余文件;`keep` 保留本地改动与新增文件;`fail` 遇到冲突抛出 `SyncConflictError`,不修改磁盘 |
|
|
70
|
+
| `delete_untracked_on_first_sync` | `False` | 从未同步过的目录首次执行 `overwrite` 时,默认不删除目录里原有的其他文件;设为 `True` 才会清理 |
|
|
71
|
+
| `prerelease` | `False` | 允许 `latest` 匹配预发布版本 |
|
|
72
|
+
| `verify_all` | `False` | 忽略 `size + mtime` 快速检查,对所有本地文件重算哈希 |
|
|
73
|
+
| `verify_paths` | `None` | 对指定的相对路径列表强制算哈希校验(以 `/` 结尾表示目录前缀),其余文件仍按 `size + mtime` 快速判断 |
|
|
74
|
+
| `on_progress` | `None` | 进度回调,接收字典 `{"phase", "done", "total", "downloaded_bytes", "bytes_per_second"}` |
|
|
75
|
+
| `cancel_event` | `None` | 传入 `threading.Event`,外部调用 `set()` 可中途取消同步 |
|
|
76
|
+
|
|
77
|
+
## 命令行
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
qentimart-py sync --server http://127.0.0.1:8080 --project mygame --token $TOKEN --dir ./assets
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
qentimart-py status --dir ./assets --server http://127.0.0.1:8080 --project mygame
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
qentimart-py report --dir ./assets
|
|
89
|
+
```
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# qentimart-sdk(Python)
|
|
2
|
+
|
|
3
|
+
QentimArt 的 Python 下游客户端,仅使用 Python 标准库,要求 Python ≥ 3.8。本地 `.qentimart/` 状态目录格式与 [Node.js SDK](../nodejs/README.md)、[Java SDK](../java/README.md) 通用,几个 SDK 可以读写同一个素材目录。
|
|
4
|
+
|
|
5
|
+
- 按数据块增量更新,文件移动或大文件局部修改时直接从本地旧文件复用数据块
|
|
6
|
+
- 不保留本地块缓存,同步完成后磁盘上只有业务素材和 `.qentimart/` 元数据(实测总占用约为素材体积的 100.04%)
|
|
7
|
+
- 多线程复用 HTTP 长连接并发拉取,落盘前逐块校验 SHA-256,写完临时文件后原子替换
|
|
8
|
+
|
|
9
|
+
## 安装
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pip install ./sdk/python
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## 快速开始
|
|
16
|
+
|
|
17
|
+
```python
|
|
18
|
+
from qentimart import QentimArtClient
|
|
19
|
+
|
|
20
|
+
with QentimArtClient("http://assets.internal:8080", "mygame", token="<read token>") as client:
|
|
21
|
+
r = client.sync("./assets", on_progress=lambda p: print(p["phase"], p["done"], p["total"]))
|
|
22
|
+
print(r["version"], r["storage"]["ratio"]) # 例如 1.2.0 1.0004
|
|
23
|
+
client.sync("./assets", version="1.0.0") # 切换到指定版本
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
更多用法见 [examples/plugin_example.py](examples/plugin_example.py)
|
|
27
|
+
|
|
28
|
+
## 静态模式
|
|
29
|
+
|
|
30
|
+
当素材通过静态模式托管在对象存储、CDN 或 Cloudflare Pages 上时,传入站点地址 `base_url` 即可,不需要 `server` 和 `token`:
|
|
31
|
+
|
|
32
|
+
```python
|
|
33
|
+
from qentimart import QentimArtClient
|
|
34
|
+
|
|
35
|
+
with QentimArtClient(base_url="https://mygame.pages.dev") as client:
|
|
36
|
+
client.sync("./assets")
|
|
37
|
+
client.sync("./assets", version="1.0.0")
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
`sync`、`plan`、`status`、`verify`、`diff`、`storage_report` 的调用方式与连接服务端时相同
|
|
41
|
+
|
|
42
|
+
## API
|
|
43
|
+
|
|
44
|
+
| 方法 | 说明 |
|
|
45
|
+
|---|---|
|
|
46
|
+
| `QentimArtClient(server, project, token=None, concurrency=8)` | 创建连接服务端的客户端(支持 `with` 自动关闭连接) |
|
|
47
|
+
| `QentimArtClient(base_url=..., concurrency=8)` | 创建连接静态托管源的只读客户端 |
|
|
48
|
+
| `list_versions()` / `get_version(v="latest")` / `diff(from_, to)` | 查询版本列表、单版本元数据与版本间差异 |
|
|
49
|
+
| `sync(dir, **opts)` | 将目录同步到目标版本(默认最新正式版) |
|
|
50
|
+
| `plan(dir, **opts)` | 仅预估待更新、删除的文件列表及网络下载量,不改写磁盘 |
|
|
51
|
+
| `status(dir)` | 离线检查本地被修改、缺失以及多出的文件 |
|
|
52
|
+
| `verify(dir)` | 重新计算全部受管文件的 SHA-256,排查文件损坏 |
|
|
53
|
+
| `QentimArtClient.storage_report(dir)` | 返回磁盘占用统计 `{"asset_bytes", "overhead_bytes", "ratio"}` |
|
|
54
|
+
|
|
55
|
+
### `sync` 参数
|
|
56
|
+
|
|
57
|
+
| 参数 | 默认值 | 说明 |
|
|
58
|
+
|---|---|---|
|
|
59
|
+
| `version` | `"latest"` | 目标版本号 |
|
|
60
|
+
| `local_changes` | `"overwrite"` | `overwrite` 还原本地改动并删除多余文件;`keep` 保留本地改动与新增文件;`fail` 遇到冲突抛出 `SyncConflictError`,不修改磁盘 |
|
|
61
|
+
| `delete_untracked_on_first_sync` | `False` | 从未同步过的目录首次执行 `overwrite` 时,默认不删除目录里原有的其他文件;设为 `True` 才会清理 |
|
|
62
|
+
| `prerelease` | `False` | 允许 `latest` 匹配预发布版本 |
|
|
63
|
+
| `verify_all` | `False` | 忽略 `size + mtime` 快速检查,对所有本地文件重算哈希 |
|
|
64
|
+
| `verify_paths` | `None` | 对指定的相对路径列表强制算哈希校验(以 `/` 结尾表示目录前缀),其余文件仍按 `size + mtime` 快速判断 |
|
|
65
|
+
| `on_progress` | `None` | 进度回调,接收字典 `{"phase", "done", "total", "downloaded_bytes", "bytes_per_second"}` |
|
|
66
|
+
| `cancel_event` | `None` | 传入 `threading.Event`,外部调用 `set()` 可中途取消同步 |
|
|
67
|
+
|
|
68
|
+
## 命令行
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
qentimart-py sync --server http://127.0.0.1:8080 --project mygame --token $TOKEN --dir ./assets
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
qentimart-py status --dir ./assets --server http://127.0.0.1:8080 --project mygame
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
```bash
|
|
79
|
+
qentimart-py report --dir ./assets
|
|
80
|
+
```
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["setuptools>=61"]
|
|
3
|
+
build-backend = "setuptools.build_meta"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "qentimart-sdk"
|
|
7
|
+
version = "0.1.0"
|
|
8
|
+
description = "QentimArt 下游 SDK:拉取游戏资源版本,支持块级增量更新,无需本地对象缓存"
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
requires-python = ">=3.8"
|
|
11
|
+
license = { text = "MIT" }
|
|
12
|
+
authors = [{ name = "Pimeng" }]
|
|
13
|
+
dependencies = []
|
|
14
|
+
|
|
15
|
+
[project.scripts]
|
|
16
|
+
qentimart-py = "qentimart.cli:main"
|
|
17
|
+
|
|
18
|
+
[tool.setuptools]
|
|
19
|
+
packages = ["qentimart"]
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
"""QentimArt 下游 SDK(Python,仅依赖标准库)。"""
|
|
2
|
+
from ._errors import QentimArtError, SyncConflictError
|
|
3
|
+
from ._wire import STATE_DIR, Manifest, ManifestFile, Chunk
|
|
4
|
+
from .client import QentimArtClient
|
|
5
|
+
|
|
6
|
+
__all__ = ["QentimArtClient", "QentimArtError", "SyncConflictError", "STATE_DIR", "Manifest", "ManifestFile", "Chunk"]
|
|
7
|
+
__version__ = "0.1.0"
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
from typing import List, Optional
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
class QentimArtError(Exception):
|
|
7
|
+
"""基础异常。服务端有响应时,``status`` 为 HTTP 状态码。"""
|
|
8
|
+
|
|
9
|
+
def __init__(self, message: str, status: Optional[int] = None, code: Optional[str] = None):
|
|
10
|
+
super().__init__(message)
|
|
11
|
+
self.status = status
|
|
12
|
+
self.code = code
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
class SyncConflictError(QentimArtError):
|
|
16
|
+
"""在 ``local_changes='fail'`` 下,本地改动阻止同步时抛出。"""
|
|
17
|
+
|
|
18
|
+
def __init__(self, conflicts: List[str]):
|
|
19
|
+
head = ", ".join(conflicts[:10]) + (", ..." if len(conflicts) > 10 else "")
|
|
20
|
+
super().__init__(
|
|
21
|
+
"%d locally modified/untracked file(s) block the sync (local_changes='fail'): %s" % (len(conflicts), head),
|
|
22
|
+
code="ECONFLICT",
|
|
23
|
+
)
|
|
24
|
+
self.conflicts = conflicts
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
import http.client
|
|
4
|
+
import json
|
|
5
|
+
import ssl
|
|
6
|
+
import threading
|
|
7
|
+
import time
|
|
8
|
+
from typing import Any, Callable, Dict, Optional, Tuple
|
|
9
|
+
from urllib.parse import quote, urlsplit
|
|
10
|
+
|
|
11
|
+
from ._errors import QentimArtError
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
class HttpClient:
|
|
15
|
+
"""线程安全的 HTTP 客户端:每个线程各自保持一个持久连接。"""
|
|
16
|
+
|
|
17
|
+
def __init__(self, server: str, project: str, token: Optional[str], timeout: float = 60.0, retries: int = 3):
|
|
18
|
+
u = urlsplit(server)
|
|
19
|
+
if u.scheme not in ("http", "https") or not u.hostname:
|
|
20
|
+
raise QentimArtError("invalid server URL %r" % server)
|
|
21
|
+
self.https = u.scheme == "https"
|
|
22
|
+
self.host = u.hostname
|
|
23
|
+
self.port = u.port
|
|
24
|
+
self.prefix = u.path.rstrip("/") + "/api/v1/projects/" + quote(project, safe="")
|
|
25
|
+
self.token = token
|
|
26
|
+
self.timeout = timeout
|
|
27
|
+
self.retries = retries
|
|
28
|
+
self._local = threading.local()
|
|
29
|
+
self._lock = threading.Lock()
|
|
30
|
+
self._conns = set()
|
|
31
|
+
|
|
32
|
+
def _conn(self) -> http.client.HTTPConnection:
|
|
33
|
+
c = getattr(self._local, "conn", None)
|
|
34
|
+
if c is None:
|
|
35
|
+
if self.https:
|
|
36
|
+
c = http.client.HTTPSConnection(self.host, self.port, timeout=self.timeout, context=ssl.create_default_context())
|
|
37
|
+
else:
|
|
38
|
+
c = http.client.HTTPConnection(self.host, self.port, timeout=self.timeout)
|
|
39
|
+
self._local.conn = c
|
|
40
|
+
with self._lock:
|
|
41
|
+
self._conns.add(c)
|
|
42
|
+
return c
|
|
43
|
+
|
|
44
|
+
def drop_conn(self) -> None:
|
|
45
|
+
c = getattr(self._local, "conn", None)
|
|
46
|
+
if c is not None:
|
|
47
|
+
self._local.conn = None
|
|
48
|
+
with self._lock:
|
|
49
|
+
self._conns.discard(c)
|
|
50
|
+
try:
|
|
51
|
+
c.close()
|
|
52
|
+
except Exception:
|
|
53
|
+
pass
|
|
54
|
+
|
|
55
|
+
def close(self) -> None:
|
|
56
|
+
with self._lock:
|
|
57
|
+
conns = list(self._conns)
|
|
58
|
+
self._conns.clear()
|
|
59
|
+
for c in conns:
|
|
60
|
+
try:
|
|
61
|
+
c.close()
|
|
62
|
+
except Exception:
|
|
63
|
+
pass
|
|
64
|
+
|
|
65
|
+
def request(self, method: str, path: str, body: Optional[bytes] = None, headers: Optional[Dict[str, str]] = None) -> http.client.HTTPResponse:
|
|
66
|
+
"""发送请求;返回尚未读取的响应(status < 300)。"""
|
|
67
|
+
h = dict(headers or {})
|
|
68
|
+
if self.token:
|
|
69
|
+
h["Authorization"] = "Bearer " + self.token
|
|
70
|
+
conn = self._conn()
|
|
71
|
+
try:
|
|
72
|
+
conn.request(method, self.prefix + path, body=body, headers=h)
|
|
73
|
+
resp = conn.getresponse()
|
|
74
|
+
except Exception:
|
|
75
|
+
self.drop_conn()
|
|
76
|
+
raise
|
|
77
|
+
if resp.status >= 300:
|
|
78
|
+
data = resp.read()
|
|
79
|
+
msg = data.decode("utf-8", "replace")
|
|
80
|
+
try:
|
|
81
|
+
msg = json.loads(msg).get("error", msg)
|
|
82
|
+
except Exception:
|
|
83
|
+
pass
|
|
84
|
+
raise QentimArtError("%s %s: HTTP %d: %s" % (method, path, resp.status, msg), status=resp.status)
|
|
85
|
+
return resp
|
|
86
|
+
|
|
87
|
+
def buffer(self, method: str, path: str, body: Optional[bytes] = None) -> Tuple[bytes, Any]:
|
|
88
|
+
def go(_attempt: int) -> Tuple[bytes, Any]:
|
|
89
|
+
resp = self.request(method, path, body)
|
|
90
|
+
try:
|
|
91
|
+
return resp.read(), resp.headers
|
|
92
|
+
except Exception:
|
|
93
|
+
self.drop_conn()
|
|
94
|
+
raise
|
|
95
|
+
|
|
96
|
+
return self.with_retry(go)
|
|
97
|
+
|
|
98
|
+
def json(self, method: str, path: str) -> Any:
|
|
99
|
+
data, _ = self.buffer(method, path)
|
|
100
|
+
return json.loads(data.decode("utf-8"))
|
|
101
|
+
|
|
102
|
+
def with_retry(self, fn: Callable[[int], Any], cancel: Optional[threading.Event] = None) -> Any:
|
|
103
|
+
"""对网络错误、被截断的流以及 5xx 响应进行重试。"""
|
|
104
|
+
last: Optional[BaseException] = None
|
|
105
|
+
for attempt in range(self.retries + 1):
|
|
106
|
+
if cancel is not None and cancel.is_set():
|
|
107
|
+
raise QentimArtError("sync cancelled", code="ECANCELED")
|
|
108
|
+
try:
|
|
109
|
+
return fn(attempt)
|
|
110
|
+
except QentimArtError as e:
|
|
111
|
+
if (e.status is not None and e.status < 500) or e.code in ("ECANCELED", "ECONFLICT"):
|
|
112
|
+
raise
|
|
113
|
+
last = e
|
|
114
|
+
except (OSError, http.client.HTTPException) as e:
|
|
115
|
+
last = e
|
|
116
|
+
self.drop_conn()
|
|
117
|
+
if attempt < self.retries:
|
|
118
|
+
time.sleep(0.3 * (2 ** attempt))
|
|
119
|
+
if isinstance(last, QentimArtError):
|
|
120
|
+
raise last
|
|
121
|
+
raise QentimArtError("request failed after %d attempts: %s" % (self.retries + 1, last), code="ENETWORK")
|
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
from __future__ import annotations
|
|
2
|
+
|
|
3
|
+
import errno
|
|
4
|
+
import hashlib
|
|
5
|
+
import os
|
|
6
|
+
import threading
|
|
7
|
+
import time
|
|
8
|
+
from concurrent.futures import ThreadPoolExecutor
|
|
9
|
+
from typing import Callable, Dict, Iterable, NamedTuple, Set, Tuple, TypeVar
|
|
10
|
+
|
|
11
|
+
from ._wire import STATE_DIR
|
|
12
|
+
|
|
13
|
+
O_BINARY = getattr(os, "O_BINARY", 0)
|
|
14
|
+
T = TypeVar("T")
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
class LocalFile(NamedTuple):
|
|
18
|
+
size: int
|
|
19
|
+
mtime_ms: int
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
def run_pool(items: Iterable[T], workers: int, fn: Callable[[T], None]) -> None:
|
|
23
|
+
"""用线程池对 items 执行 fn;重新抛出第一个错误。"""
|
|
24
|
+
items = list(items)
|
|
25
|
+
if not items:
|
|
26
|
+
return
|
|
27
|
+
if workers <= 1 or len(items) == 1:
|
|
28
|
+
for it in items:
|
|
29
|
+
fn(it)
|
|
30
|
+
return
|
|
31
|
+
with ThreadPoolExecutor(max_workers=min(workers, len(items))) as ex:
|
|
32
|
+
futures = [ex.submit(fn, it) for it in items]
|
|
33
|
+
first = None
|
|
34
|
+
for f in futures:
|
|
35
|
+
try:
|
|
36
|
+
f.result()
|
|
37
|
+
except BaseException as e: # noqa: BLE001 - 在下方重新抛出
|
|
38
|
+
if first is None:
|
|
39
|
+
first = e
|
|
40
|
+
for g in futures:
|
|
41
|
+
g.cancel()
|
|
42
|
+
if first is not None:
|
|
43
|
+
raise first
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
def scan_dir(root: str) -> Tuple[Dict[str, LocalFile], Set[str]]:
|
|
47
|
+
"""列出 root 下的普通文件(跳过状态目录)以及所有目录。"""
|
|
48
|
+
files: Dict[str, LocalFile] = {}
|
|
49
|
+
dirs: Set[str] = set()
|
|
50
|
+
stack = [(root, "")]
|
|
51
|
+
while stack:
|
|
52
|
+
abs_dir, rel = stack.pop()
|
|
53
|
+
try:
|
|
54
|
+
it = os.scandir(abs_dir)
|
|
55
|
+
except FileNotFoundError:
|
|
56
|
+
continue
|
|
57
|
+
with it:
|
|
58
|
+
for e in it:
|
|
59
|
+
r = rel + "/" + e.name if rel else e.name
|
|
60
|
+
if not rel and e.name == STATE_DIR:
|
|
61
|
+
continue
|
|
62
|
+
if e.is_dir(follow_symlinks=False):
|
|
63
|
+
dirs.add(r)
|
|
64
|
+
stack.append((e.path, r))
|
|
65
|
+
elif e.is_file(follow_symlinks=False):
|
|
66
|
+
st = e.stat(follow_symlinks=False)
|
|
67
|
+
files[r] = LocalFile(st.st_size, st.st_mtime_ns // 1_000_000)
|
|
68
|
+
return files, dirs
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def mtime_ms(path: str) -> int:
|
|
72
|
+
return os.stat(path).st_mtime_ns // 1_000_000
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
def hash_file(path: str) -> str:
|
|
76
|
+
h = hashlib.sha256()
|
|
77
|
+
with open(path, "rb", buffering=0) as f:
|
|
78
|
+
buf = bytearray(1 << 20)
|
|
79
|
+
mv = memoryview(buf)
|
|
80
|
+
while True:
|
|
81
|
+
n = f.readinto(buf)
|
|
82
|
+
if not n:
|
|
83
|
+
break
|
|
84
|
+
h.update(mv[:n])
|
|
85
|
+
return h.hexdigest()
|
|
86
|
+
|
|
87
|
+
|
|
88
|
+
def replace_retry(src: str, dst: str, attempts: int = 8) -> None:
|
|
89
|
+
"""带重试的 os.replace,用于应对 Windows 上短暂的共享冲突。
|
|
90
|
+
|
|
91
|
+
重试面与 Node(EPERM/EBUSY/EACCES)对齐:除 PermissionError 外,也重试
|
|
92
|
+
Windows 共享/占用(winerror 5/32)与 EBUSY/EACCES/EPERM。"""
|
|
93
|
+
for i in range(attempts + 1):
|
|
94
|
+
try:
|
|
95
|
+
os.replace(src, dst)
|
|
96
|
+
return
|
|
97
|
+
except OSError as e:
|
|
98
|
+
transient = isinstance(e, PermissionError) \
|
|
99
|
+
or getattr(e, "winerror", None) in (5, 32) \
|
|
100
|
+
or e.errno in (errno.EACCES, errno.EBUSY, errno.EPERM)
|
|
101
|
+
if not transient or i >= attempts:
|
|
102
|
+
raise
|
|
103
|
+
time.sleep(0.05 * (2 ** min(i, 5)))
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
def write_atomic(path: str, data: bytes) -> None:
|
|
107
|
+
tmp = "%s.%d.tmp" % (path, os.getpid())
|
|
108
|
+
with open(tmp, "wb") as f:
|
|
109
|
+
f.write(data)
|
|
110
|
+
replace_retry(tmp, path)
|
|
111
|
+
|
|
112
|
+
|
|
113
|
+
def dir_bytes(path: str) -> int:
|
|
114
|
+
n = 0
|
|
115
|
+
for base, _dirs, names in os.walk(path):
|
|
116
|
+
for name in names:
|
|
117
|
+
try:
|
|
118
|
+
n += os.path.getsize(os.path.join(base, name))
|
|
119
|
+
except OSError:
|
|
120
|
+
pass
|
|
121
|
+
return n
|
|
122
|
+
|
|
123
|
+
|
|
124
|
+
def prune_empty_dirs(root: str, rel: str) -> None:
|
|
125
|
+
d = os.path.dirname(rel.replace("/", os.sep))
|
|
126
|
+
while d:
|
|
127
|
+
try:
|
|
128
|
+
os.rmdir(os.path.join(root, d))
|
|
129
|
+
except OSError:
|
|
130
|
+
return
|
|
131
|
+
d = os.path.dirname(d)
|
|
132
|
+
|
|
133
|
+
|
|
134
|
+
class RandomAccessFile:
|
|
135
|
+
"""支持线程安全定位读写的文件描述符
|
|
136
|
+
(可用时使用 os.pread/os.pwrite,否则(如 Windows)使用锁 + seek)。"""
|
|
137
|
+
|
|
138
|
+
def __init__(self, path: str, write: bool, size: int = 0):
|
|
139
|
+
flags = (os.O_RDWR | os.O_CREAT | os.O_TRUNC) if write else os.O_RDONLY
|
|
140
|
+
self.fd = os.open(path, flags | O_BINARY, 0o644)
|
|
141
|
+
self.lock = threading.Lock()
|
|
142
|
+
if write and size > 0:
|
|
143
|
+
os.ftruncate(self.fd, size)
|
|
144
|
+
|
|
145
|
+
def write_at(self, offset: int, data: bytes) -> None:
|
|
146
|
+
if hasattr(os, "pwrite"):
|
|
147
|
+
view = memoryview(data)
|
|
148
|
+
while view:
|
|
149
|
+
n = os.pwrite(self.fd, view, offset)
|
|
150
|
+
view = view[n:]
|
|
151
|
+
offset += n
|
|
152
|
+
return
|
|
153
|
+
with self.lock:
|
|
154
|
+
os.lseek(self.fd, offset, os.SEEK_SET)
|
|
155
|
+
view = memoryview(data)
|
|
156
|
+
while view:
|
|
157
|
+
n = os.write(self.fd, view)
|
|
158
|
+
view = view[n:]
|
|
159
|
+
|
|
160
|
+
def read_at(self, offset: int, size: int) -> bytes:
|
|
161
|
+
if hasattr(os, "pread"):
|
|
162
|
+
return os.pread(self.fd, size, offset)
|
|
163
|
+
with self.lock:
|
|
164
|
+
os.lseek(self.fd, offset, os.SEEK_SET)
|
|
165
|
+
parts = []
|
|
166
|
+
left = size
|
|
167
|
+
while left:
|
|
168
|
+
b = os.read(self.fd, left)
|
|
169
|
+
if not b:
|
|
170
|
+
break
|
|
171
|
+
parts.append(b)
|
|
172
|
+
left -= len(b)
|
|
173
|
+
return b"".join(parts)
|
|
174
|
+
|
|
175
|
+
def close(self) -> None:
|
|
176
|
+
if self.fd >= 0:
|
|
177
|
+
os.close(self.fd)
|
|
178
|
+
self.fd = -1
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
def pid_alive(pid: int) -> bool:
|
|
182
|
+
if pid <= 0:
|
|
183
|
+
return False
|
|
184
|
+
if os.name == "nt":
|
|
185
|
+
import ctypes
|
|
186
|
+
|
|
187
|
+
kernel32 = ctypes.windll.kernel32 # type: ignore[attr-defined]
|
|
188
|
+
handle = kernel32.OpenProcess(0x1000, False, pid) # PROCESS_QUERY_LIMITED_INFORMATION
|
|
189
|
+
if not handle:
|
|
190
|
+
return False
|
|
191
|
+
code = ctypes.c_ulong()
|
|
192
|
+
ok = kernel32.GetExitCodeProcess(handle, ctypes.byref(code))
|
|
193
|
+
kernel32.CloseHandle(handle)
|
|
194
|
+
return bool(ok) and code.value == 259 # STILL_ACTIVE
|
|
195
|
+
try:
|
|
196
|
+
os.kill(pid, 0)
|
|
197
|
+
except ProcessLookupError:
|
|
198
|
+
return False
|
|
199
|
+
except PermissionError:
|
|
200
|
+
return True
|
|
201
|
+
return True
|