dpe-hash 0.1.5__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- dpe_hash-0.1.5/.gitignore +16 -0
- dpe_hash-0.1.5/LICENSE +21 -0
- dpe_hash-0.1.5/PKG-INFO +77 -0
- dpe_hash-0.1.5/README.md +60 -0
- dpe_hash-0.1.5/pyproject.toml +36 -0
- dpe_hash-0.1.5/src/dpe_hash/__init__.py +86 -0
- dpe_hash-0.1.5/src/dpe_hash/constants.py +68 -0
- dpe_hash-0.1.5/src/dpe_hash/contract1.py +377 -0
- dpe_hash-0.1.5/src/dpe_hash/errors.py +61 -0
- dpe_hash-0.1.5/src/dpe_hash/jcs.py +146 -0
- dpe_hash-0.1.5/src/dpe_hash/models.py +96 -0
- dpe_hash-0.1.5/src/dpe_hash/py.typed +0 -0
dpe_hash-0.1.5/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 TuringFocus Team
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
dpe_hash-0.1.5/PKG-INFO
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: dpe-hash
|
|
3
|
+
Version: 0.1.5
|
|
4
|
+
Summary: DPE hash 契约的独立 hash 核心:零运行时依赖、纯 Python
|
|
5
|
+
Project-URL: Homepage, https://github.com/A2C-SMCP/dpe-protocol
|
|
6
|
+
Project-URL: Documentation, https://doc.turingfocus.cn/dpe/
|
|
7
|
+
Project-URL: Repository, https://github.com/A2C-SMCP/dpe-protocol
|
|
8
|
+
Author-email: TuringFocus Team <dev@turingfocus.cn>
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
13
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
14
|
+
Classifier: Typing :: Typed
|
|
15
|
+
Requires-Python: >=3.11
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# dpe-hash
|
|
19
|
+
|
|
20
|
+
[DPE(Document / Page / Element)协议](https://doc.turingfocus.cn/dpe/)的独立 hash 核心:按 [hash 契约 1](https://doc.turingfocus.cn/dpe/latest/spec/hash-contract-1/) 计算元素、页、文档三层 hash(`dpe1:` 前缀),并导出契约常量。
|
|
21
|
+
|
|
22
|
+
- 零运行时依赖、纯 Python(≥ 3.11),带类型标记(`py.typed`);
|
|
23
|
+
- 逐字节通过规范仓库的一致性向量;
|
|
24
|
+
- 规范与源码:<https://github.com/A2C-SMCP/dpe-protocol>。
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
pip install "dpe-hash>=0.1.5,<0.2.0"
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## 版本约定
|
|
31
|
+
|
|
32
|
+
hash 契约升级对应 minor 版本(`0.2.0` = 契约 2),不影响 hash 行为的修订走 patch。依赖时锁定 minor:`dpe-hash>=0.1.5,<0.2.0`。
|
|
33
|
+
|
|
34
|
+
## API
|
|
35
|
+
|
|
36
|
+
```python
|
|
37
|
+
from dpe_hash import content_hash, page_hash, doc_hash, object_hash, document_hashes
|
|
38
|
+
|
|
39
|
+
h = content_hash({"category": "NarrativeText", "text": "Hello, DPE."}) # 元素对象
|
|
40
|
+
p = page_hash({"title": "p1"}, [h]) # 页字段 + 已有 content_hash
|
|
41
|
+
d = doc_hash({"file_type": "md", "title": "报告"}, [p]) # 文档字段 + 已有 page_hash
|
|
42
|
+
assert d == object_hash({"file_type": "md", "title": "报告", "pages": [p]}, "document") # 线上原像
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
| 入口 | 用途 |
|
|
46
|
+
| --- | --- |
|
|
47
|
+
| `content_hash(element, contract=CONTRACT)` | 元素对象的 `content_hash` |
|
|
48
|
+
| `page_hash(page, element_hashes, contract)` | 页自身字段 + 已有的 content_hash 列表;`page` 带 `elements` 即报错 |
|
|
49
|
+
| `doc_hash(document, page_hashes, contract)` | 文档自身字段 + 已有的 page_hash 列表,不需加载元素;`document` 带 `pages` 即报错 |
|
|
50
|
+
| `object_hash(obj, kind, contract)` | 线上原像(页带 `elements`、文档带 `pages`),`kind` 为 `"element"` / `"page"` / `"document"` |
|
|
51
|
+
| `document_hashes(expanded, contract)` | 展开视图一次算出三层,结果同向量的 `expected` |
|
|
52
|
+
| `children(obj, kind, contract)` | 原像引用的下一层:页 → content_hash,文档 → page_hash,元素 → blob 引用 |
|
|
53
|
+
| `parse_hash(s, contracts=SUPPORTED_CONTRACTS)` / `parse_blob_ref(s)` | 校验并拆分 `dpe1:<hex>` / `sha256:<hex>` |
|
|
54
|
+
| `blob_ref(data)` | blob 字节的引用 `sha256:<hex>` |
|
|
55
|
+
| `jcs(value)` | RFC 8785 规范化字符串 |
|
|
56
|
+
|
|
57
|
+
逐层入口的参数是 TypedDict,便于在编译期发现字段拼错;从线上反序列化、未加类型的 JSON(`Any`)可以直接传入,已标注为 `dict[str, Any]` 的数据请用 `object_hash`(接受 `Mapping[str, Any]`),或先 `cast`。
|
|
58
|
+
|
|
59
|
+
所有入口都对封闭 schema 做完整校验,未定义字段、未知 category / file_type、超过 2^53−1 的整数、NaN 等一律拒绝。
|
|
60
|
+
|
|
61
|
+
**原位重算**(契约 1 §6):各入口的 `contract` 参数决定按哪个契约计算。`DRILL_CONTRACT`(`dpe2`)是仅用于升级演练的假想契约,只在显式传入时接受,不在 `SUPPORTED_CONTRACTS` 中,`parse_hash` 默认拒绝它。
|
|
62
|
+
|
|
63
|
+
## 常量与类型
|
|
64
|
+
|
|
65
|
+
- `CONTRACT`(`"dpe1"`)、`SUPPORTED_CONTRACTS`(服务端 capabilities 的 `hash_contracts`)、`DRILL_CONTRACT`;
|
|
66
|
+
- `FILE_TYPES`:file_type 封闭枚举,按规范顺序;`CATEGORY_CONTENT_FIELDS`:category → 允许的内容字段(只读映射)。两者与规范 `vectors/manifest.json` 一致,消费方直接 import,不维护副本;
|
|
67
|
+
- TypedDict:`ElementObject`、`PageObject`、`DocumentObject`(线上原像),`PageFields`、`DocumentFields`(不含子列表),`ExpandedDocument` / `ExpandedPage`(展开视图),`DocumentHashes` / `PageHashes`(结果)。
|
|
68
|
+
|
|
69
|
+
## 异常
|
|
70
|
+
|
|
71
|
+
全部继承 `DpeHashError`(它是 `ValueError` 的子类)。每个异常带两个属性:`.code` 是规范错误码,`.path` 是出错位置的 RFC 6901 JSON Pointer(相对于传入的对象),可直接映射为 problem 体。
|
|
72
|
+
|
|
73
|
+
| 异常 | `code` |
|
|
74
|
+
| --- | --- |
|
|
75
|
+
| `ValidationError`,及其子类 `UndefinedFieldError`、`FileTypeUnknownError`、`IntegerOutOfRangeError` | `DPE_VALIDATION` |
|
|
76
|
+
| `CategoryUnknownError` | `DPE_CATEGORY_UNKNOWN` |
|
|
77
|
+
| `ContractUnsupportedError` | `DPE_CONTRACT_UNSUPPORTED` |
|
dpe_hash-0.1.5/README.md
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
# dpe-hash
|
|
2
|
+
|
|
3
|
+
[DPE(Document / Page / Element)协议](https://doc.turingfocus.cn/dpe/)的独立 hash 核心:按 [hash 契约 1](https://doc.turingfocus.cn/dpe/latest/spec/hash-contract-1/) 计算元素、页、文档三层 hash(`dpe1:` 前缀),并导出契约常量。
|
|
4
|
+
|
|
5
|
+
- 零运行时依赖、纯 Python(≥ 3.11),带类型标记(`py.typed`);
|
|
6
|
+
- 逐字节通过规范仓库的一致性向量;
|
|
7
|
+
- 规范与源码:<https://github.com/A2C-SMCP/dpe-protocol>。
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
pip install "dpe-hash>=0.1.5,<0.2.0"
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## 版本约定
|
|
14
|
+
|
|
15
|
+
hash 契约升级对应 minor 版本(`0.2.0` = 契约 2),不影响 hash 行为的修订走 patch。依赖时锁定 minor:`dpe-hash>=0.1.5,<0.2.0`。
|
|
16
|
+
|
|
17
|
+
## API
|
|
18
|
+
|
|
19
|
+
```python
|
|
20
|
+
from dpe_hash import content_hash, page_hash, doc_hash, object_hash, document_hashes
|
|
21
|
+
|
|
22
|
+
h = content_hash({"category": "NarrativeText", "text": "Hello, DPE."}) # 元素对象
|
|
23
|
+
p = page_hash({"title": "p1"}, [h]) # 页字段 + 已有 content_hash
|
|
24
|
+
d = doc_hash({"file_type": "md", "title": "报告"}, [p]) # 文档字段 + 已有 page_hash
|
|
25
|
+
assert d == object_hash({"file_type": "md", "title": "报告", "pages": [p]}, "document") # 线上原像
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
| 入口 | 用途 |
|
|
29
|
+
| --- | --- |
|
|
30
|
+
| `content_hash(element, contract=CONTRACT)` | 元素对象的 `content_hash` |
|
|
31
|
+
| `page_hash(page, element_hashes, contract)` | 页自身字段 + 已有的 content_hash 列表;`page` 带 `elements` 即报错 |
|
|
32
|
+
| `doc_hash(document, page_hashes, contract)` | 文档自身字段 + 已有的 page_hash 列表,不需加载元素;`document` 带 `pages` 即报错 |
|
|
33
|
+
| `object_hash(obj, kind, contract)` | 线上原像(页带 `elements`、文档带 `pages`),`kind` 为 `"element"` / `"page"` / `"document"` |
|
|
34
|
+
| `document_hashes(expanded, contract)` | 展开视图一次算出三层,结果同向量的 `expected` |
|
|
35
|
+
| `children(obj, kind, contract)` | 原像引用的下一层:页 → content_hash,文档 → page_hash,元素 → blob 引用 |
|
|
36
|
+
| `parse_hash(s, contracts=SUPPORTED_CONTRACTS)` / `parse_blob_ref(s)` | 校验并拆分 `dpe1:<hex>` / `sha256:<hex>` |
|
|
37
|
+
| `blob_ref(data)` | blob 字节的引用 `sha256:<hex>` |
|
|
38
|
+
| `jcs(value)` | RFC 8785 规范化字符串 |
|
|
39
|
+
|
|
40
|
+
逐层入口的参数是 TypedDict,便于在编译期发现字段拼错;从线上反序列化、未加类型的 JSON(`Any`)可以直接传入,已标注为 `dict[str, Any]` 的数据请用 `object_hash`(接受 `Mapping[str, Any]`),或先 `cast`。
|
|
41
|
+
|
|
42
|
+
所有入口都对封闭 schema 做完整校验,未定义字段、未知 category / file_type、超过 2^53−1 的整数、NaN 等一律拒绝。
|
|
43
|
+
|
|
44
|
+
**原位重算**(契约 1 §6):各入口的 `contract` 参数决定按哪个契约计算。`DRILL_CONTRACT`(`dpe2`)是仅用于升级演练的假想契约,只在显式传入时接受,不在 `SUPPORTED_CONTRACTS` 中,`parse_hash` 默认拒绝它。
|
|
45
|
+
|
|
46
|
+
## 常量与类型
|
|
47
|
+
|
|
48
|
+
- `CONTRACT`(`"dpe1"`)、`SUPPORTED_CONTRACTS`(服务端 capabilities 的 `hash_contracts`)、`DRILL_CONTRACT`;
|
|
49
|
+
- `FILE_TYPES`:file_type 封闭枚举,按规范顺序;`CATEGORY_CONTENT_FIELDS`:category → 允许的内容字段(只读映射)。两者与规范 `vectors/manifest.json` 一致,消费方直接 import,不维护副本;
|
|
50
|
+
- TypedDict:`ElementObject`、`PageObject`、`DocumentObject`(线上原像),`PageFields`、`DocumentFields`(不含子列表),`ExpandedDocument` / `ExpandedPage`(展开视图),`DocumentHashes` / `PageHashes`(结果)。
|
|
51
|
+
|
|
52
|
+
## 异常
|
|
53
|
+
|
|
54
|
+
全部继承 `DpeHashError`(它是 `ValueError` 的子类)。每个异常带两个属性:`.code` 是规范错误码,`.path` 是出错位置的 RFC 6901 JSON Pointer(相对于传入的对象),可直接映射为 problem 体。
|
|
55
|
+
|
|
56
|
+
| 异常 | `code` |
|
|
57
|
+
| --- | --- |
|
|
58
|
+
| `ValidationError`,及其子类 `UndefinedFieldError`、`FileTypeUnknownError`、`IntegerOutOfRangeError` | `DPE_VALIDATION` |
|
|
59
|
+
| `CategoryUnknownError` | `DPE_CATEGORY_UNKNOWN` |
|
|
60
|
+
| `ContractUnsupportedError` | `DPE_CONTRACT_UNSUPPORTED` |
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
[project]
|
|
2
|
+
name = "dpe-hash"
|
|
3
|
+
version = "0.1.5"
|
|
4
|
+
description = "DPE hash 契约的独立 hash 核心:零运行时依赖、纯 Python"
|
|
5
|
+
readme = "README.md"
|
|
6
|
+
requires-python = ">=3.11"
|
|
7
|
+
license = "MIT"
|
|
8
|
+
# 仓库根 LICENSE 的副本(构建后端只收录包目录内的文件),测试断言两者一致
|
|
9
|
+
license-files = ["LICENSE"]
|
|
10
|
+
authors = [
|
|
11
|
+
{name = "TuringFocus Team", email = "dev@turingfocus.cn"}
|
|
12
|
+
]
|
|
13
|
+
classifiers = [
|
|
14
|
+
"Programming Language :: Python :: 3",
|
|
15
|
+
"Programming Language :: Python :: 3.11",
|
|
16
|
+
"Programming Language :: Python :: 3.12",
|
|
17
|
+
"Typing :: Typed",
|
|
18
|
+
]
|
|
19
|
+
# 零运行时依赖是硬约束(内核直接依赖本包),tools/check_packages.py 在 CI 中断言
|
|
20
|
+
dependencies = []
|
|
21
|
+
|
|
22
|
+
[project.urls]
|
|
23
|
+
Homepage = "https://github.com/A2C-SMCP/dpe-protocol"
|
|
24
|
+
Documentation = "https://doc.turingfocus.cn/dpe/"
|
|
25
|
+
Repository = "https://github.com/A2C-SMCP/dpe-protocol"
|
|
26
|
+
|
|
27
|
+
[build-system]
|
|
28
|
+
requires = ["hatchling>=1.27"]
|
|
29
|
+
build-backend = "hatchling.build"
|
|
30
|
+
|
|
31
|
+
[tool.hatch.build.targets.wheel]
|
|
32
|
+
packages = ["src/dpe_hash"]
|
|
33
|
+
|
|
34
|
+
# 测试依赖 workspace 根的 conftest 与仓库 vectors/,不随 sdist 分发
|
|
35
|
+
[tool.hatch.build.targets.sdist]
|
|
36
|
+
exclude = ["tests"]
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
"""dpe-hash:DPE hash 契约的独立 hash 核心。
|
|
2
|
+
|
|
3
|
+
零运行时依赖、纯 Python;内核与 dpe-sdk 都直接依赖本包,不另行维护 hash 实现。
|
|
4
|
+
实现 hash 契约 1(spec/hash-contract-1.md),并导出契约常量、三层对象类型与带规范错误码的异常。
|
|
5
|
+
"""
|
|
6
|
+
|
|
7
|
+
from importlib.metadata import version
|
|
8
|
+
|
|
9
|
+
from dpe_hash.constants import (
|
|
10
|
+
CATEGORY_CONTENT_FIELDS,
|
|
11
|
+
CONTRACT,
|
|
12
|
+
DRILL_CONTRACT,
|
|
13
|
+
FILE_TYPES,
|
|
14
|
+
SUPPORTED_CONTRACTS,
|
|
15
|
+
)
|
|
16
|
+
from dpe_hash.contract1 import (
|
|
17
|
+
blob_ref,
|
|
18
|
+
children,
|
|
19
|
+
content_hash,
|
|
20
|
+
doc_hash,
|
|
21
|
+
document_hashes,
|
|
22
|
+
object_hash,
|
|
23
|
+
page_hash,
|
|
24
|
+
parse_blob_ref,
|
|
25
|
+
parse_hash,
|
|
26
|
+
)
|
|
27
|
+
from dpe_hash.errors import (
|
|
28
|
+
CategoryUnknownError,
|
|
29
|
+
ContractUnsupportedError,
|
|
30
|
+
DpeHashError,
|
|
31
|
+
FileTypeUnknownError,
|
|
32
|
+
IntegerOutOfRangeError,
|
|
33
|
+
UndefinedFieldError,
|
|
34
|
+
ValidationError,
|
|
35
|
+
)
|
|
36
|
+
from dpe_hash.jcs import jcs
|
|
37
|
+
from dpe_hash.models import (
|
|
38
|
+
DocumentFields,
|
|
39
|
+
DocumentHashes,
|
|
40
|
+
DocumentObject,
|
|
41
|
+
ElementObject,
|
|
42
|
+
ExpandedDocument,
|
|
43
|
+
ExpandedPage,
|
|
44
|
+
ObjectKind,
|
|
45
|
+
PageFields,
|
|
46
|
+
PageHashes,
|
|
47
|
+
PageObject,
|
|
48
|
+
)
|
|
49
|
+
|
|
50
|
+
__version__ = version("dpe-hash")
|
|
51
|
+
|
|
52
|
+
__all__ = [
|
|
53
|
+
"CATEGORY_CONTENT_FIELDS",
|
|
54
|
+
"CONTRACT",
|
|
55
|
+
"DRILL_CONTRACT",
|
|
56
|
+
"FILE_TYPES",
|
|
57
|
+
"SUPPORTED_CONTRACTS",
|
|
58
|
+
"CategoryUnknownError",
|
|
59
|
+
"ContractUnsupportedError",
|
|
60
|
+
"DocumentFields",
|
|
61
|
+
"DocumentHashes",
|
|
62
|
+
"DocumentObject",
|
|
63
|
+
"DpeHashError",
|
|
64
|
+
"ElementObject",
|
|
65
|
+
"ExpandedDocument",
|
|
66
|
+
"ExpandedPage",
|
|
67
|
+
"FileTypeUnknownError",
|
|
68
|
+
"IntegerOutOfRangeError",
|
|
69
|
+
"ObjectKind",
|
|
70
|
+
"PageFields",
|
|
71
|
+
"PageHashes",
|
|
72
|
+
"PageObject",
|
|
73
|
+
"UndefinedFieldError",
|
|
74
|
+
"ValidationError",
|
|
75
|
+
"__version__",
|
|
76
|
+
"blob_ref",
|
|
77
|
+
"children",
|
|
78
|
+
"content_hash",
|
|
79
|
+
"doc_hash",
|
|
80
|
+
"document_hashes",
|
|
81
|
+
"jcs",
|
|
82
|
+
"object_hash",
|
|
83
|
+
"page_hash",
|
|
84
|
+
"parse_blob_ref",
|
|
85
|
+
"parse_hash",
|
|
86
|
+
]
|
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
"""契约常量:消费方(内核、服务端、dpe-sdk)直接 import,不自行维护副本。
|
|
2
|
+
|
|
3
|
+
测试断言它们与 ``vectors/manifest.json`` 一致(``file_types`` / ``category_content_fields``)。
|
|
4
|
+
"""
|
|
5
|
+
|
|
6
|
+
from __future__ import annotations
|
|
7
|
+
|
|
8
|
+
from collections.abc import Mapping
|
|
9
|
+
from types import MappingProxyType
|
|
10
|
+
|
|
11
|
+
__all__ = [
|
|
12
|
+
"CATEGORY_CONTENT_FIELDS",
|
|
13
|
+
"CONTRACT",
|
|
14
|
+
"DRILL_CONTRACT",
|
|
15
|
+
"FILE_TYPES",
|
|
16
|
+
"SUPPORTED_CONTRACTS",
|
|
17
|
+
]
|
|
18
|
+
|
|
19
|
+
#: 当前契约版本(hash 值前缀),各 hash 函数的默认契约。
|
|
20
|
+
CONTRACT = "dpe1"
|
|
21
|
+
|
|
22
|
+
#: 本包支持的真实契约,供服务端 capabilities 的 ``hash_contracts`` 使用。
|
|
23
|
+
SUPPORTED_CONTRACTS: tuple[str, ...] = ("dpe1",)
|
|
24
|
+
|
|
25
|
+
#: 假想契约 ``dpe2``,**仅用于契约升级演练**(vectors/README.md):摘要输入为 ASCII ``dpe2``
|
|
26
|
+
#: 后接 JCS 原像字节。不在 ``SUPPORTED_CONTRACTS`` 中:只有调用方显式传 ``contract="dpe2"``
|
|
27
|
+
#: 时 hash 函数才接受它,``parse_hash`` 默认拒绝。
|
|
28
|
+
DRILL_CONTRACT = "dpe2"
|
|
29
|
+
|
|
30
|
+
#: category 封闭枚举 → 允许的内容字段(契约 1 §4.1,字段按表中顺序)。
|
|
31
|
+
#: 元素对象另有必有的 ``category`` 与可选的 ``metadata``,不在此列。
|
|
32
|
+
CATEGORY_CONTENT_FIELDS: Mapping[str, tuple[str, ...]] = MappingProxyType(
|
|
33
|
+
{
|
|
34
|
+
**{
|
|
35
|
+
c: ("text",)
|
|
36
|
+
for c in (
|
|
37
|
+
"UncategorizedText",
|
|
38
|
+
"CheckBox",
|
|
39
|
+
"CompositeElement",
|
|
40
|
+
"FigureCaption",
|
|
41
|
+
"NarrativeText",
|
|
42
|
+
"ListItem",
|
|
43
|
+
"Title",
|
|
44
|
+
"Address",
|
|
45
|
+
"EmailAddress",
|
|
46
|
+
"PageBreak",
|
|
47
|
+
"TableChunk",
|
|
48
|
+
"Header",
|
|
49
|
+
"Footer",
|
|
50
|
+
"CodeSnippet",
|
|
51
|
+
"PageNumber",
|
|
52
|
+
"FormKeysValues",
|
|
53
|
+
"tfchat",
|
|
54
|
+
)
|
|
55
|
+
},
|
|
56
|
+
"Table": ("text", "text_as_html"),
|
|
57
|
+
"Formula": ("text", "text_as_html"),
|
|
58
|
+
"Image": ("text", "image_blob", "image_mime_type"),
|
|
59
|
+
}
|
|
60
|
+
)
|
|
61
|
+
|
|
62
|
+
#: file_type 封闭枚举(core.md §2.5,按规范顺序)。
|
|
63
|
+
FILE_TYPES: tuple[str, ...] = (
|
|
64
|
+
"bmp", "csv", "doc", "docx", "eml", "epub", "heic", "html", "jpg", "json", "md", "msg",
|
|
65
|
+
"ndjson", "odt", "org", "pdf", "png", "ppt", "pptx", "rst", "rtf", "tiff", "tsv", "txt",
|
|
66
|
+
"wav", "xls", "xlsx", "xml", "zip", "java_repo", "python_repo", "javascript_repo",
|
|
67
|
+
"typescript_repo", "unk", "empty", "tfchat", "jira_project", "jira_issue",
|
|
68
|
+
) # fmt: skip
|
|
@@ -0,0 +1,377 @@
|
|
|
1
|
+
"""hash 契约 1(``dpe1:``)——spec/hash-contract-1.md 的实现。
|
|
2
|
+
|
|
3
|
+
契约 1 是三层同构的 tree:元素、页、文档三层对象统一为
|
|
4
|
+
|
|
5
|
+
H(obj) = "dpe1:" + hex(sha256(utf8(JCS(norm(obj)))))
|
|
6
|
+
|
|
7
|
+
``norm``:递归删除值为 null 的键(缺省 ≡ null);三层 metadata 字段缺省视同 ``{}``(§3.2)。
|
|
8
|
+
三层对象都是封闭 schema,未定义字段一律拒绝(§3.4)。页没有页号,顺序只由上层数组表达。
|
|
9
|
+
|
|
10
|
+
本模块是唯一的规范化与校验实现,全部公开函数都只是它的不同入口:
|
|
11
|
+
|
|
12
|
+
- ``content_hash`` / ``page_hash`` / ``doc_hash``:逐层,用已有的子 hash 上溯(内核按页增量维护);
|
|
13
|
+
- ``object_hash``:直接接受线上原像(页带 ``elements``、文档带 ``pages``,服务端校验用);
|
|
14
|
+
- ``document_hashes``:展开视图一次算出三层;
|
|
15
|
+
- ``children``:对象引用的下一层(缺失清单)。
|
|
16
|
+
|
|
17
|
+
``contract`` 参数用于契约 1 §6 的原位重算;``dpe2`` 只在显式传入时接受(见 ``DRILL_CONTRACT``)。
|
|
18
|
+
"""
|
|
19
|
+
|
|
20
|
+
from __future__ import annotations
|
|
21
|
+
|
|
22
|
+
import hashlib
|
|
23
|
+
import re
|
|
24
|
+
from collections.abc import Collection, Mapping, Sequence
|
|
25
|
+
from typing import Any
|
|
26
|
+
|
|
27
|
+
from dpe_hash.constants import (
|
|
28
|
+
CATEGORY_CONTENT_FIELDS,
|
|
29
|
+
CONTRACT,
|
|
30
|
+
DRILL_CONTRACT,
|
|
31
|
+
FILE_TYPES,
|
|
32
|
+
SUPPORTED_CONTRACTS,
|
|
33
|
+
)
|
|
34
|
+
from dpe_hash.errors import (
|
|
35
|
+
CategoryUnknownError,
|
|
36
|
+
ContractUnsupportedError,
|
|
37
|
+
FileTypeUnknownError,
|
|
38
|
+
UndefinedFieldError,
|
|
39
|
+
ValidationError,
|
|
40
|
+
)
|
|
41
|
+
from dpe_hash.jcs import canonical, pointer
|
|
42
|
+
from dpe_hash.models import (
|
|
43
|
+
DocumentFields,
|
|
44
|
+
DocumentHashes,
|
|
45
|
+
ElementObject,
|
|
46
|
+
ExpandedDocument,
|
|
47
|
+
ObjectKind,
|
|
48
|
+
PageFields,
|
|
49
|
+
PageHashes,
|
|
50
|
+
)
|
|
51
|
+
|
|
52
|
+
__all__ = [
|
|
53
|
+
"blob_ref",
|
|
54
|
+
"children",
|
|
55
|
+
"content_hash",
|
|
56
|
+
"doc_hash",
|
|
57
|
+
"document_hashes",
|
|
58
|
+
"object_hash",
|
|
59
|
+
"page_hash",
|
|
60
|
+
"parse_blob_ref",
|
|
61
|
+
"parse_hash",
|
|
62
|
+
]
|
|
63
|
+
|
|
64
|
+
#: 各契约的摘要前缀字节:dpe2 = ASCII "dpe2" ‖ 原像(vectors/README.md)
|
|
65
|
+
_SALT = {"dpe1": b"", DRILL_CONTRACT: b"dpe2"}
|
|
66
|
+
_FILE_TYPES = frozenset(FILE_TYPES)
|
|
67
|
+
_HEX64 = re.compile(r"[0-9a-f]{64}")
|
|
68
|
+
#: 本包认识的全部契约(真实契约 + 演练契约);hash 值的前缀不在其中即 DPE_CONTRACT_UNSUPPORTED
|
|
69
|
+
_KNOWN_CONTRACTS = frozenset({*SUPPORTED_CONTRACTS, DRILL_CONTRACT})
|
|
70
|
+
_BLOB_PREFIX = "sha256:"
|
|
71
|
+
|
|
72
|
+
_PAGE_FIELDS = frozenset({"title", "page_metadata"})
|
|
73
|
+
_DOCUMENT_FIELDS = frozenset({"file_type", "title", "doc_metadata"})
|
|
74
|
+
|
|
75
|
+
|
|
76
|
+
# ---------------------------------------------------------------------------
|
|
77
|
+
# 值格式(§1)
|
|
78
|
+
# ---------------------------------------------------------------------------
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def _salt(contract: str) -> bytes:
|
|
82
|
+
if isinstance(contract, str) and (
|
|
83
|
+
contract in SUPPORTED_CONTRACTS or contract == DRILL_CONTRACT
|
|
84
|
+
):
|
|
85
|
+
return _SALT[contract]
|
|
86
|
+
raise ContractUnsupportedError(f"不支持的 hash 契约:{contract!r}")
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def _digest(preimage: str, contract: str, salt: bytes) -> str:
|
|
90
|
+
return f"{contract}:{hashlib.sha256(salt + preimage.encode('utf-8')).hexdigest()}"
|
|
91
|
+
|
|
92
|
+
|
|
93
|
+
def _split_hash(value: object, accepted: frozenset[str], at: str) -> tuple[str, str]:
|
|
94
|
+
if not isinstance(value, str):
|
|
95
|
+
raise ValidationError(f"hash 值必须是字符串,实际为 {type(value).__name__}", at)
|
|
96
|
+
prefix, sep, hexpart = value.partition(":")
|
|
97
|
+
if not sep:
|
|
98
|
+
raise ContractUnsupportedError("hash 值缺少契约前缀", at)
|
|
99
|
+
if prefix not in accepted:
|
|
100
|
+
raise ContractUnsupportedError(f"未知或不支持的契约前缀:{prefix!r}", at)
|
|
101
|
+
if not _HEX64.fullmatch(hexpart):
|
|
102
|
+
raise ValidationError("hash 值的摘要部分必须是 64 位小写 hex", at)
|
|
103
|
+
return prefix, hexpart
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
def _blob_hex(value: object, at: str) -> str:
|
|
107
|
+
if not isinstance(value, str):
|
|
108
|
+
raise ValidationError(f"blob 引用必须是字符串,实际为 {type(value).__name__}", at)
|
|
109
|
+
if not value.startswith(_BLOB_PREFIX) or not _HEX64.fullmatch(value[len(_BLOB_PREFIX) :]):
|
|
110
|
+
raise ValidationError("blob 引用必须是 sha256:<64 位小写 hex>", at)
|
|
111
|
+
return value[len(_BLOB_PREFIX) :]
|
|
112
|
+
|
|
113
|
+
|
|
114
|
+
def parse_hash(value: str, contracts: Collection[str] = SUPPORTED_CONTRACTS) -> tuple[str, str]:
|
|
115
|
+
"""校验并拆分 hash 值,返回 ``(契约, 64 位 hex)``。
|
|
116
|
+
|
|
117
|
+
无前缀或前缀不在 ``contracts`` 内抛 ``ContractUnsupportedError``(契约 1 §1);摘要格式不对抛
|
|
118
|
+
``ValidationError``。默认只接受 ``SUPPORTED_CONTRACTS``,升级演练需显式传入
|
|
119
|
+
``(*SUPPORTED_CONTRACTS, DRILL_CONTRACT)``。``contracts`` 是契约的集合,不能是单个字符串;
|
|
120
|
+
其中出现本包不认识的契约抛 ``ContractUnsupportedError``。
|
|
121
|
+
"""
|
|
122
|
+
if isinstance(contracts, str):
|
|
123
|
+
raise TypeError('contracts 必须是契约的集合(如 ("dpe1",)),不能是单个字符串')
|
|
124
|
+
accepted = frozenset(contracts)
|
|
125
|
+
unknown = accepted - _KNOWN_CONTRACTS
|
|
126
|
+
if unknown:
|
|
127
|
+
raise ContractUnsupportedError(f"不支持的 hash 契约:{sorted(unknown)!r}")
|
|
128
|
+
return _split_hash(value, accepted, "")
|
|
129
|
+
|
|
130
|
+
|
|
131
|
+
def parse_blob_ref(value: str) -> str:
|
|
132
|
+
"""校验 blob 引用 ``sha256:<64 位小写 hex>``,返回 hex 部分;不合法抛 ``ValidationError``。"""
|
|
133
|
+
return _blob_hex(value, "")
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
def blob_ref(data: bytes | bytearray | memoryview) -> str:
|
|
137
|
+
"""blob 原始字节的引用 ``"sha256:<hex>"``(契约 1 §1)。"""
|
|
138
|
+
return _BLOB_PREFIX + hashlib.sha256(data).hexdigest()
|
|
139
|
+
|
|
140
|
+
|
|
141
|
+
# ---------------------------------------------------------------------------
|
|
142
|
+
# 三层对象的校验与规范化(§3–§5),返回 JCS 原像
|
|
143
|
+
# ---------------------------------------------------------------------------
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
def _mapping(value: object, at: str, what: str) -> Mapping[str, Any]:
|
|
147
|
+
if not isinstance(value, Mapping):
|
|
148
|
+
raise ValidationError(f"{what}必须是 JSON 对象,实际为 {type(value).__name__}", at)
|
|
149
|
+
return value
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def _closed(obj: Mapping[str, Any], allowed: frozenset[str], at: str, what: str) -> None:
|
|
153
|
+
extra = sorted(str(k) for k in obj if k not in allowed)
|
|
154
|
+
if extra:
|
|
155
|
+
raise UndefinedFieldError(f"{what}不允许字段 {extra[0]!r}", at + pointer(extra[0]))
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
def _opt_str(obj: Mapping[str, Any], field: str, at: str) -> str | None:
|
|
159
|
+
value = obj.get(field)
|
|
160
|
+
if value is not None and not isinstance(value, str):
|
|
161
|
+
raise ValidationError(
|
|
162
|
+
f"{field} 必须是字符串或 null,实际为 {type(value).__name__}", at + pointer(field)
|
|
163
|
+
)
|
|
164
|
+
return value
|
|
165
|
+
|
|
166
|
+
|
|
167
|
+
def _metadata(obj: Mapping[str, Any], field: str, at: str) -> Mapping[str, Any]:
|
|
168
|
+
value = obj.get(field)
|
|
169
|
+
if value is None:
|
|
170
|
+
return {} # 缺省视同 {}(§3.2)
|
|
171
|
+
return _mapping(value, at + pointer(field), f"{field} ")
|
|
172
|
+
|
|
173
|
+
|
|
174
|
+
def _hash_list(value: object, contract: str, at: str) -> list[str]:
|
|
175
|
+
"""子对象 hash 列表:每项必须是同一契约下的合法 hash 值(契约 1 §5)。
|
|
176
|
+
|
|
177
|
+
前缀不是受支持的契约 → ``ContractUnsupportedError``;是受支持的契约、但与本对象的契约不同
|
|
178
|
+
(契约混用)→ ``ValidationError``。「受支持」= ``SUPPORTED_CONTRACTS`` 加上本次调用显式选择的
|
|
179
|
+
契约:演练契约 dpe2 只在被选中时才算受支持,因此 dpe1 对象引用 dpe2 子 hash 与生产服务端一样
|
|
180
|
+
得到 ``DPE_CONTRACT_UNSUPPORTED``。
|
|
181
|
+
"""
|
|
182
|
+
if isinstance(value, str) or not isinstance(value, (list, tuple)):
|
|
183
|
+
raise ValidationError(f"必须是 hash 值数组,实际为 {type(value).__name__}", at)
|
|
184
|
+
accepted = frozenset({*SUPPORTED_CONTRACTS, contract})
|
|
185
|
+
for i, item in enumerate(value):
|
|
186
|
+
prefix, _ = _split_hash(item, accepted, at + pointer(i))
|
|
187
|
+
if prefix != contract:
|
|
188
|
+
raise ValidationError(
|
|
189
|
+
f"子对象 hash 的契约 {prefix} 与本对象的契约 {contract} 不一致", at + pointer(i)
|
|
190
|
+
)
|
|
191
|
+
return list(value)
|
|
192
|
+
|
|
193
|
+
|
|
194
|
+
def _element_preimage(element: object, at: str) -> str:
|
|
195
|
+
el = _mapping(element, at, "元素对象")
|
|
196
|
+
if "category" not in el:
|
|
197
|
+
raise ValidationError("元素对象缺少 category", at)
|
|
198
|
+
category = el["category"]
|
|
199
|
+
if not isinstance(category, str):
|
|
200
|
+
raise ValidationError("category 必须是字符串", at + pointer("category"))
|
|
201
|
+
allowed = CATEGORY_CONTENT_FIELDS.get(category)
|
|
202
|
+
if allowed is None: # 封闭枚举,禁止退化为 text-only(§4)
|
|
203
|
+
raise CategoryUnknownError(f"未知 category:{category!r}", at + pointer("category"))
|
|
204
|
+
_closed(el, frozenset({"category", "metadata", *allowed}), at, f"{category} 元素")
|
|
205
|
+
obj: dict[str, Any] = {"category": category}
|
|
206
|
+
for field in allowed:
|
|
207
|
+
value = _opt_str(el, field, at)
|
|
208
|
+
if value is not None:
|
|
209
|
+
obj[field] = value
|
|
210
|
+
if "image_blob" in obj:
|
|
211
|
+
_blob_hex(obj["image_blob"], at + pointer("image_blob"))
|
|
212
|
+
obj["metadata"] = _metadata(el, "metadata", at)
|
|
213
|
+
return canonical(obj, strip_nulls=True, at=at)
|
|
214
|
+
|
|
215
|
+
|
|
216
|
+
def _page_preimage(
|
|
217
|
+
page: Mapping[str, Any], element_hashes: object, contract: str, at: str, *, with_children: bool
|
|
218
|
+
) -> str:
|
|
219
|
+
"""``with_children``:``page`` 自带 ``elements`` 键(线上原像或展开视图),其值由调用方处理。"""
|
|
220
|
+
allowed = _PAGE_FIELDS | {"elements"} if with_children else _PAGE_FIELDS
|
|
221
|
+
if not with_children and "elements" in page:
|
|
222
|
+
raise UndefinedFieldError(
|
|
223
|
+
"page_hash 的页字段不得带 elements(子 hash 单独传入;线上原像请用 object_hash)",
|
|
224
|
+
at + pointer("elements"),
|
|
225
|
+
)
|
|
226
|
+
_closed(page, allowed, at, "页对象")
|
|
227
|
+
obj: dict[str, Any] = {}
|
|
228
|
+
title = _opt_str(page, "title", at)
|
|
229
|
+
if title is not None:
|
|
230
|
+
obj["title"] = title
|
|
231
|
+
obj["page_metadata"] = _metadata(page, "page_metadata", at)
|
|
232
|
+
obj["elements"] = _hash_list(element_hashes, contract, at + pointer("elements"))
|
|
233
|
+
return canonical(obj, strip_nulls=True, at=at)
|
|
234
|
+
|
|
235
|
+
|
|
236
|
+
def _document_preimage(
|
|
237
|
+
document: Mapping[str, Any], page_hashes: object, contract: str, at: str, *, with_children: bool
|
|
238
|
+
) -> str:
|
|
239
|
+
allowed = _DOCUMENT_FIELDS | {"pages"} if with_children else _DOCUMENT_FIELDS
|
|
240
|
+
if not with_children and "pages" in document:
|
|
241
|
+
raise UndefinedFieldError(
|
|
242
|
+
"doc_hash 的文档字段不得带 pages(子 hash 单独传入;线上原像请用 object_hash)",
|
|
243
|
+
at + pointer("pages"),
|
|
244
|
+
)
|
|
245
|
+
_closed(document, allowed, at, "文档对象")
|
|
246
|
+
if "file_type" not in document:
|
|
247
|
+
raise ValidationError("文档对象缺少 file_type", at)
|
|
248
|
+
file_type = document["file_type"]
|
|
249
|
+
if not isinstance(file_type, str):
|
|
250
|
+
raise ValidationError("file_type 必须是字符串", at + pointer("file_type"))
|
|
251
|
+
if file_type not in _FILE_TYPES:
|
|
252
|
+
raise FileTypeUnknownError(f"未知 file_type:{file_type!r}", at + pointer("file_type"))
|
|
253
|
+
obj: dict[str, Any] = {"file_type": file_type}
|
|
254
|
+
title = _opt_str(document, "title", at)
|
|
255
|
+
if title is not None:
|
|
256
|
+
obj["title"] = title
|
|
257
|
+
obj["doc_metadata"] = _metadata(document, "doc_metadata", at)
|
|
258
|
+
obj["pages"] = _hash_list(page_hashes, contract, at + pointer("pages"))
|
|
259
|
+
return canonical(obj, strip_nulls=True, at=at)
|
|
260
|
+
|
|
261
|
+
|
|
262
|
+
def _required(obj: Mapping[str, Any], field: str, at: str, what: str) -> Any:
|
|
263
|
+
if field not in obj:
|
|
264
|
+
raise ValidationError(f"{what}缺少 {field}", at)
|
|
265
|
+
return obj[field]
|
|
266
|
+
|
|
267
|
+
|
|
268
|
+
def _preimage(obj: object, kind: ObjectKind, contract: str) -> str:
|
|
269
|
+
"""线上原像的 JCS 原像(``object_hash`` / ``children`` 共用)。"""
|
|
270
|
+
if kind == "element":
|
|
271
|
+
return _element_preimage(obj, "")
|
|
272
|
+
if kind == "page":
|
|
273
|
+
page = _mapping(obj, "", "页对象")
|
|
274
|
+
elements = _required(page, "elements", "", "页对象")
|
|
275
|
+
return _page_preimage(page, elements, contract, "", with_children=True)
|
|
276
|
+
if kind == "document":
|
|
277
|
+
document = _mapping(obj, "", "文档对象")
|
|
278
|
+
pages = _required(document, "pages", "", "文档对象")
|
|
279
|
+
return _document_preimage(document, pages, contract, "", with_children=True)
|
|
280
|
+
raise ValueError(f"未知的对象层级:{kind!r}")
|
|
281
|
+
|
|
282
|
+
|
|
283
|
+
# ---------------------------------------------------------------------------
|
|
284
|
+
# 公开入口
|
|
285
|
+
# ---------------------------------------------------------------------------
|
|
286
|
+
|
|
287
|
+
|
|
288
|
+
def content_hash(element: ElementObject, contract: str = CONTRACT) -> str:
|
|
289
|
+
"""元素对象的 ``content_hash``(契约 1 §4)。"""
|
|
290
|
+
salt = _salt(contract)
|
|
291
|
+
return _digest(_element_preimage(element, ""), contract, salt)
|
|
292
|
+
|
|
293
|
+
|
|
294
|
+
def page_hash(page: PageFields, element_hashes: Sequence[str], contract: str = CONTRACT) -> str:
|
|
295
|
+
"""``page_hash``(契约 1 §5):页自身字段 + 已有的 content_hash 列表(按页内顺序)。
|
|
296
|
+
|
|
297
|
+
``page`` 带 ``elements`` 键即报错,避免两份元素列表的歧义;``element_hashes`` 中的错误
|
|
298
|
+
路径记为 ``/elements/<i>``。
|
|
299
|
+
"""
|
|
300
|
+
salt = _salt(contract)
|
|
301
|
+
pre = _page_preimage(
|
|
302
|
+
_mapping(page, "", "页对象"), element_hashes, contract, "", with_children=False
|
|
303
|
+
)
|
|
304
|
+
return _digest(pre, contract, salt)
|
|
305
|
+
|
|
306
|
+
|
|
307
|
+
def doc_hash(document: DocumentFields, page_hashes: Sequence[str], contract: str = CONTRACT) -> str:
|
|
308
|
+
"""``doc_hash``(契约 1 §5):文档自身字段 + 已有的 page_hash 列表(按页序),不需加载元素。
|
|
309
|
+
|
|
310
|
+
``document`` 带 ``pages`` 键即报错,避免两份页序的歧义;``page_hashes`` 中的错误路径记为
|
|
311
|
+
``/pages/<i>``。
|
|
312
|
+
"""
|
|
313
|
+
salt = _salt(contract)
|
|
314
|
+
pre = _document_preimage(
|
|
315
|
+
_mapping(document, "", "文档对象"), page_hashes, contract, "", with_children=False
|
|
316
|
+
)
|
|
317
|
+
return _digest(pre, contract, salt)
|
|
318
|
+
|
|
319
|
+
|
|
320
|
+
def object_hash(obj: Mapping[str, Any], kind: ObjectKind, contract: str = CONTRACT) -> str:
|
|
321
|
+
"""线上原像的 hash:一步完成封闭 schema 校验、规范化、JCS 与摘要。
|
|
322
|
+
|
|
323
|
+
页对象须带 ``elements``、文档对象须带 ``pages``(子 hash 列表,契约须与 ``contract`` 一致)。
|
|
324
|
+
对同一文档,结果与 ``content_hash`` / ``page_hash`` / ``doc_hash`` 逐层相等。
|
|
325
|
+
"""
|
|
326
|
+
salt = _salt(contract)
|
|
327
|
+
return _digest(_preimage(obj, kind, contract), contract, salt)
|
|
328
|
+
|
|
329
|
+
|
|
330
|
+
def children(obj: Mapping[str, Any], kind: ObjectKind, contract: str = CONTRACT) -> list[str]:
|
|
331
|
+
"""线上原像引用的下一层(先按 ``object_hash`` 的规则完整校验)。
|
|
332
|
+
|
|
333
|
+
页 → ``elements`` 的 content_hash;文档 → ``pages`` 的 page_hash;元素 → ``image_blob``
|
|
334
|
+
引用(没有则为空)。按出现顺序返回,重复保留,去重由调用方决定。
|
|
335
|
+
"""
|
|
336
|
+
_salt(contract)
|
|
337
|
+
_preimage(obj, kind, contract)
|
|
338
|
+
if kind == "element":
|
|
339
|
+
blob = obj.get("image_blob")
|
|
340
|
+
return [blob] if isinstance(blob, str) else []
|
|
341
|
+
return list(obj["elements" if kind == "page" else "pages"])
|
|
342
|
+
|
|
343
|
+
|
|
344
|
+
def document_hashes(document: ExpandedDocument, contract: str = CONTRACT) -> DocumentHashes:
|
|
345
|
+
"""展开视图(vectors/README.md)的三层 hash。
|
|
346
|
+
|
|
347
|
+
结果形如 ``{"doc_hash", "pages": [{"page_hash", "elements"}…]}``,同向量的 ``expected``。
|
|
348
|
+
|
|
349
|
+
页序即 ``pages`` 数组顺序,页内元素序即 ``elements`` 数组顺序。用于整篇计算与契约 1 §6
|
|
350
|
+
的原位重算(按任一受支持契约重算已存的骨架与内容,结果按位置一一对应)。
|
|
351
|
+
"""
|
|
352
|
+
salt = _salt(contract)
|
|
353
|
+
doc = _mapping(document, "", "文档")
|
|
354
|
+
pages = _required(doc, "pages", "", "文档")
|
|
355
|
+
if not isinstance(pages, (list, tuple)):
|
|
356
|
+
raise ValidationError("pages 必须是数组", pointer("pages"))
|
|
357
|
+
out_pages: list[PageHashes] = []
|
|
358
|
+
page_hashes: list[str] = []
|
|
359
|
+
for i, raw_page in enumerate(pages):
|
|
360
|
+
at = pointer("pages", i)
|
|
361
|
+
page = _mapping(raw_page, at, "页")
|
|
362
|
+
elements = _required(page, "elements", at, "页")
|
|
363
|
+
if not isinstance(elements, (list, tuple)):
|
|
364
|
+
raise ValidationError("elements 必须是数组", at + pointer("elements"))
|
|
365
|
+
element_hashes = [
|
|
366
|
+
_digest(_element_preimage(el, at + pointer("elements", j)), contract, salt)
|
|
367
|
+
for j, el in enumerate(elements)
|
|
368
|
+
]
|
|
369
|
+
ph = _digest(
|
|
370
|
+
_page_preimage(page, element_hashes, contract, at, with_children=True), contract, salt
|
|
371
|
+
)
|
|
372
|
+
page_hashes.append(ph)
|
|
373
|
+
out_pages.append({"page_hash": ph, "elements": element_hashes})
|
|
374
|
+
dh = _digest(
|
|
375
|
+
_document_preimage(doc, page_hashes, contract, "", with_children=True), contract, salt
|
|
376
|
+
)
|
|
377
|
+
return {"doc_hash": dh, "pages": out_pages}
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
"""公开异常:调用方按类型或 ``code`` 分派,不解析消息字符串。
|
|
2
|
+
|
|
3
|
+
``code`` 取规范错误码(core.md §6),服务端可直接映射为 problem 体;``path`` 是出错位置的
|
|
4
|
+
RFC 6901 JSON Pointer,相对于传入函数的那个对象(如 ``/pages/0/elements/2/text_as_html``),
|
|
5
|
+
出错位置就是对象本身时为空串。
|
|
6
|
+
"""
|
|
7
|
+
|
|
8
|
+
from __future__ import annotations
|
|
9
|
+
|
|
10
|
+
from typing import ClassVar
|
|
11
|
+
|
|
12
|
+
__all__ = [
|
|
13
|
+
"CategoryUnknownError",
|
|
14
|
+
"ContractUnsupportedError",
|
|
15
|
+
"DpeHashError",
|
|
16
|
+
"FileTypeUnknownError",
|
|
17
|
+
"IntegerOutOfRangeError",
|
|
18
|
+
"UndefinedFieldError",
|
|
19
|
+
"ValidationError",
|
|
20
|
+
]
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
class DpeHashError(ValueError):
|
|
24
|
+
"""dpe-hash 全部异常的共同基类。"""
|
|
25
|
+
|
|
26
|
+
code: ClassVar[str]
|
|
27
|
+
|
|
28
|
+
def __init__(self, message: str, path: str = "") -> None:
|
|
29
|
+
super().__init__(f"{message}(位置 {path})" if path else message)
|
|
30
|
+
self.message = message
|
|
31
|
+
self.path = path
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
class ValidationError(DpeHashError):
|
|
35
|
+
"""输入不合法(``DPE_VALIDATION``):类型错误、非法 hash / blob 引用、契约混用等。"""
|
|
36
|
+
|
|
37
|
+
code = "DPE_VALIDATION"
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
class UndefinedFieldError(ValidationError):
|
|
41
|
+
"""封闭 schema 违例:对象出现规范未定义(或其 category 未允许)的字段(core.md §2)。"""
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
class FileTypeUnknownError(ValidationError):
|
|
45
|
+
"""file_type 不在封闭枚举内(core.md §2.5)。"""
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
class IntegerOutOfRangeError(ValidationError):
|
|
49
|
+
"""整数字面量绝对值超过 2^53−1(契约 1 §3.3、core.md §2.6)。"""
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
class CategoryUnknownError(DpeHashError):
|
|
53
|
+
"""category 不在契约的封闭枚举内(``DPE_CATEGORY_UNKNOWN``,契约 1 §4)。"""
|
|
54
|
+
|
|
55
|
+
code = "DPE_CATEGORY_UNKNOWN"
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
class ContractUnsupportedError(DpeHashError):
|
|
59
|
+
"""hash 契约不被支持,或 hash 值缺少 / 带未知契约前缀(``DPE_CONTRACT_UNSUPPORTED``)。"""
|
|
60
|
+
|
|
61
|
+
code = "DPE_CONTRACT_UNSUPPORTED"
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
"""RFC 8785 JCS 序列化(spec/hash-contract-1.md §3.3)。
|
|
2
|
+
|
|
3
|
+
按规范文本独立实现,与 ``scripts/gen_vectors.py`` 的一致由向量逐字节校验,而不是共享代码。
|
|
4
|
+
|
|
5
|
+
输入是 JSON 值:``None`` / ``bool`` / ``int`` / ``float`` / ``str``,对象为 ``Mapping``(键必须是
|
|
6
|
+
``str``),数组为 ``list`` 或 ``tuple``。其余类型、NaN / Infinity、超过 2^53−1 的整数、含孤立代理项
|
|
7
|
+
的字符串一律拒绝,异常带出错位置的 JSON Pointer。
|
|
8
|
+
"""
|
|
9
|
+
|
|
10
|
+
from __future__ import annotations
|
|
11
|
+
|
|
12
|
+
import math
|
|
13
|
+
import re
|
|
14
|
+
from collections.abc import Mapping
|
|
15
|
+
from json.encoder import encode_basestring
|
|
16
|
+
from typing import Any
|
|
17
|
+
|
|
18
|
+
from dpe_hash.errors import DpeHashError, IntegerOutOfRangeError, ValidationError
|
|
19
|
+
|
|
20
|
+
__all__ = ["jcs"]
|
|
21
|
+
|
|
22
|
+
_MAX_SAFE_INTEGER = 2**53 - 1
|
|
23
|
+
_SURROGATE = re.compile("[\ud800-\udfff]")
|
|
24
|
+
|
|
25
|
+
|
|
26
|
+
class _Invalid(Exception):
|
|
27
|
+
"""序列化中途的错误:逐层回溯时收集路径分量,最外层再转换为公开异常。"""
|
|
28
|
+
|
|
29
|
+
def __init__(self, error: type[DpeHashError], message: str) -> None:
|
|
30
|
+
self.error = error
|
|
31
|
+
self.message = message
|
|
32
|
+
self.segments: list[str | int] = []
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
def pointer(*segments: str | int) -> str:
|
|
36
|
+
"""把路径分量拼成 RFC 6901 JSON Pointer。"""
|
|
37
|
+
return "".join(
|
|
38
|
+
"/" + (str(s) if isinstance(s, int) else s.replace("~", "~0").replace("/", "~1"))
|
|
39
|
+
for s in segments
|
|
40
|
+
)
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
def _number(x: int | float) -> str:
|
|
44
|
+
"""ECMAScript ``Number::toString``(base 10)。"""
|
|
45
|
+
if isinstance(x, int):
|
|
46
|
+
if abs(x) > _MAX_SAFE_INTEGER:
|
|
47
|
+
raise _Invalid(IntegerOutOfRangeError, f"整数超出 ±(2^53−1):{int.__repr__(x)}")
|
|
48
|
+
return int.__repr__(x) # 不走子类(如 IntEnum)的 __str__
|
|
49
|
+
if math.isnan(x) or math.isinf(x):
|
|
50
|
+
raise _Invalid(ValidationError, "JCS 不允许 NaN / Infinity")
|
|
51
|
+
if x == 0:
|
|
52
|
+
return "0" # 含 -0.0
|
|
53
|
+
sign = x < 0
|
|
54
|
+
# float.__repr__ 即最短往返十进制(如 "0.1"、"1e-07"、"1.5e+22")。直接解析该字符串,
|
|
55
|
+
# 不经 Decimal:Decimal 运算受线程全局的 decimal 上下文(精度)影响,会静默改变 hash。
|
|
56
|
+
mantissa, _, exponent = float.__repr__(abs(x)).partition("e")
|
|
57
|
+
int_part, _, frac_part = mantissa.partition(".")
|
|
58
|
+
digits = int_part + frac_part
|
|
59
|
+
n = len(int_part) + int(exponent or "0") # value = 0.digits × 10^n
|
|
60
|
+
stripped = digits.lstrip("0")
|
|
61
|
+
n -= len(digits) - len(stripped)
|
|
62
|
+
s = stripped.rstrip("0")
|
|
63
|
+
k = len(s)
|
|
64
|
+
if k <= n <= 21:
|
|
65
|
+
body = s + "0" * (n - k)
|
|
66
|
+
elif 0 < n <= 21:
|
|
67
|
+
body = s[:n] + "." + s[n:]
|
|
68
|
+
elif -6 < n <= 0:
|
|
69
|
+
body = "0." + "0" * (-n) + s
|
|
70
|
+
else:
|
|
71
|
+
mantissa = s[0] + ("." + s[1:] if k > 1 else "")
|
|
72
|
+
e = n - 1
|
|
73
|
+
body = f"{mantissa}e{'+' if e >= 0 else '-'}{abs(e)}"
|
|
74
|
+
return ("-" if sign else "") + body
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def _string(s: str) -> str:
|
|
78
|
+
if _SURROGATE.search(s):
|
|
79
|
+
raise _Invalid(ValidationError, "字符串含孤立代理项,不是合法的 Unicode")
|
|
80
|
+
# 标准库(ensure_ascii=False 时)的转义恰好是 JCS 要求的集合:
|
|
81
|
+
# " \ \b \f \n \r \t 与其余 < 0x20 的小写 \u00XX,其他字符原样输出
|
|
82
|
+
return encode_basestring(s)
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
def _utf16_key(key: str) -> bytes:
|
|
86
|
+
return key.encode("utf-16-be") # 大端字节序比较 = UTF-16 码元序
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
def _serialize(value: Any, strip_nulls: bool) -> str:
|
|
90
|
+
if value is None:
|
|
91
|
+
return "null"
|
|
92
|
+
if value is True:
|
|
93
|
+
return "true"
|
|
94
|
+
if value is False:
|
|
95
|
+
return "false"
|
|
96
|
+
if isinstance(value, str):
|
|
97
|
+
return _string(value)
|
|
98
|
+
if isinstance(value, (int, float)):
|
|
99
|
+
return _number(value)
|
|
100
|
+
if isinstance(value, (dict, Mapping)): # dict 在前:常见情形免走 ABC 检查
|
|
101
|
+
for key in value:
|
|
102
|
+
if not isinstance(key, str):
|
|
103
|
+
raise _Invalid(
|
|
104
|
+
ValidationError, f"对象的键必须是字符串,实际为 {type(key).__name__}"
|
|
105
|
+
)
|
|
106
|
+
if _SURROGATE.search(key):
|
|
107
|
+
err = _Invalid(ValidationError, "键含孤立代理项,不是合法的 Unicode")
|
|
108
|
+
err.segments.append(key)
|
|
109
|
+
raise err
|
|
110
|
+
keys = sorted(
|
|
111
|
+
(k for k, v in value.items() if not (strip_nulls and v is None)), key=_utf16_key
|
|
112
|
+
)
|
|
113
|
+
parts = []
|
|
114
|
+
for key in keys:
|
|
115
|
+
try:
|
|
116
|
+
parts.append(f"{_string(key)}:{_serialize(value[key], strip_nulls)}")
|
|
117
|
+
except _Invalid as err:
|
|
118
|
+
err.segments.append(key)
|
|
119
|
+
raise
|
|
120
|
+
return "{" + ",".join(parts) + "}"
|
|
121
|
+
if isinstance(value, (list, tuple)):
|
|
122
|
+
items = []
|
|
123
|
+
for i, item in enumerate(value): # 数组元素不受 null 删除影响
|
|
124
|
+
try:
|
|
125
|
+
items.append(_serialize(item, strip_nulls))
|
|
126
|
+
except _Invalid as err:
|
|
127
|
+
err.segments.append(i)
|
|
128
|
+
raise
|
|
129
|
+
return "[" + ",".join(items) + "]"
|
|
130
|
+
raise _Invalid(ValidationError, f"不是 JSON 值:{type(value).__name__}")
|
|
131
|
+
|
|
132
|
+
|
|
133
|
+
def canonical(value: Any, *, strip_nulls: bool = False, at: str = "") -> str:
|
|
134
|
+
"""JCS 规范化字符串;``strip_nulls`` 为 True 时同时递归删除值为 null 的键(契约 1 §3.2)。
|
|
135
|
+
|
|
136
|
+
``at`` 是 ``value`` 自身的 JSON Pointer,用作异常路径的前缀。
|
|
137
|
+
"""
|
|
138
|
+
try:
|
|
139
|
+
return _serialize(value, strip_nulls)
|
|
140
|
+
except _Invalid as err:
|
|
141
|
+
raise err.error(err.message, at + pointer(*reversed(err.segments))) from None
|
|
142
|
+
|
|
143
|
+
|
|
144
|
+
def jcs(value: Any) -> str:
|
|
145
|
+
"""把 JSON 值序列化为 RFC 8785 规范化字符串(不做 null 键删除等任何 DPE 规范化)。"""
|
|
146
|
+
return canonical(value)
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
"""三层对象的类型(TypedDict,标准库实现):供 mypy strict 下的调用方在编译期发现字段拼错。
|
|
2
|
+
|
|
3
|
+
运行时校验(封闭 schema、各 category 允许的字段)由 hash 函数完成,与类型无关。
|
|
4
|
+
TypedDict 在类型系统中是开放的:带多余键的字典(如带 ``elements`` 的页)仍可通过
|
|
5
|
+
``PageFields`` 的类型检查。封闭 TypedDict(PEP 728)需要第三方的 ``typing_extensions``,与本包
|
|
6
|
+
零依赖冲突,因此这类误用只在运行时拒绝(``page_hash`` / ``doc_hash`` 抛 ``UndefinedFieldError``)。
|
|
7
|
+
|
|
8
|
+
- 线上原像(契约 1 §4–§5):``ElementObject``、``PageObject``(带 ``elements``)、
|
|
9
|
+
``DocumentObject``(带 ``pages``);
|
|
10
|
+
- 不含子对象列表的自身字段:``PageFields``、``DocumentFields``,配合 ``page_hash`` /
|
|
11
|
+
``doc_hash`` 用已存的子 hash 上溯;
|
|
12
|
+
- 展开视图(vectors/README.md):``ExpandedDocument`` / ``ExpandedPage``,配合
|
|
13
|
+
``document_hashes``;其输出为 ``DocumentHashes`` / ``PageHashes``。
|
|
14
|
+
"""
|
|
15
|
+
|
|
16
|
+
from __future__ import annotations
|
|
17
|
+
|
|
18
|
+
from typing import Any, Literal, NotRequired, Required, TypedDict
|
|
19
|
+
|
|
20
|
+
__all__ = [
|
|
21
|
+
"DocumentFields",
|
|
22
|
+
"DocumentHashes",
|
|
23
|
+
"DocumentObject",
|
|
24
|
+
"ElementObject",
|
|
25
|
+
"ExpandedDocument",
|
|
26
|
+
"ExpandedPage",
|
|
27
|
+
"ObjectKind",
|
|
28
|
+
"PageFields",
|
|
29
|
+
"PageHashes",
|
|
30
|
+
"PageObject",
|
|
31
|
+
]
|
|
32
|
+
|
|
33
|
+
#: ``object_hash`` / ``children`` 的对象层级
|
|
34
|
+
ObjectKind = Literal["element", "page", "document"]
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
class ElementObject(TypedDict):
|
|
38
|
+
"""元素对象(core.md §2.3)。内容字段是否允许取决于 ``category``(契约 1 §4.1)。"""
|
|
39
|
+
|
|
40
|
+
category: str
|
|
41
|
+
text: NotRequired[str | None]
|
|
42
|
+
text_as_html: NotRequired[str | None]
|
|
43
|
+
image_blob: NotRequired[str | None]
|
|
44
|
+
image_mime_type: NotRequired[str | None]
|
|
45
|
+
metadata: NotRequired[dict[str, Any] | None]
|
|
46
|
+
|
|
47
|
+
|
|
48
|
+
class PageFields(TypedDict, total=False):
|
|
49
|
+
"""页对象除 ``elements`` 之外的字段。"""
|
|
50
|
+
|
|
51
|
+
title: str | None
|
|
52
|
+
page_metadata: dict[str, Any] | None
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
class PageObject(PageFields):
|
|
56
|
+
"""页对象(core.md §2.2):``elements`` 为 content_hash 列表。"""
|
|
57
|
+
|
|
58
|
+
elements: Required[list[str]]
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
class DocumentFields(TypedDict):
|
|
62
|
+
"""文档对象除 ``pages`` 之外的字段。"""
|
|
63
|
+
|
|
64
|
+
file_type: str
|
|
65
|
+
title: NotRequired[str | None]
|
|
66
|
+
doc_metadata: NotRequired[dict[str, Any] | None]
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
class DocumentObject(DocumentFields):
|
|
70
|
+
"""文档对象(core.md §2.1):``pages`` 为 page_hash 列表。"""
|
|
71
|
+
|
|
72
|
+
pages: list[str]
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
class ExpandedPage(PageFields):
|
|
76
|
+
"""展开视图中的页:``elements`` 为元素对象本身。"""
|
|
77
|
+
|
|
78
|
+
elements: Required[list[ElementObject]]
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
class ExpandedDocument(DocumentFields):
|
|
82
|
+
"""展开视图中的文档:``pages`` 为展开的页。"""
|
|
83
|
+
|
|
84
|
+
pages: list[ExpandedPage]
|
|
85
|
+
|
|
86
|
+
|
|
87
|
+
class PageHashes(TypedDict):
|
|
88
|
+
page_hash: str
|
|
89
|
+
elements: list[str]
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
class DocumentHashes(TypedDict):
|
|
93
|
+
"""``document_hashes`` 的结果,形状同向量的 ``expected``。"""
|
|
94
|
+
|
|
95
|
+
doc_hash: str
|
|
96
|
+
pages: list[PageHashes]
|
|
File without changes
|