@antprofuse/saddle-islands-tools 0.1.4 → 0.1.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/tools/docs/iteration.md +48 -19
- package/tools/modules/workflow/iteration.py +163 -90
- package/tools/modules/workflow/registry.py +151 -0
package/package.json
CHANGED
package/tools/docs/iteration.md
CHANGED
|
@@ -1,34 +1,63 @@
|
|
|
1
|
-
#
|
|
1
|
+
# 迭代管理与阶段交付
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
统一公开入口:`python3 tools/cli/main.py iteration <action>`。所有能力只管理文件、校验与渲染,不调用CoTask、不授予启动许可。以下绝对路径为调用方提供的实际位置。
|
|
4
|
+
|
|
5
|
+
## 持久元数据
|
|
6
|
+
|
|
7
|
+
发起者填写workflow包 `resources/iteration.schema.json` 定义的v2实例。`persistentRoot` 必须等于 `--registry/<iteration.id>`,不得位于临时共享根内。Skill及工具定位、来源版本等元数据不全量注入执行者。
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
python3 tools/cli/main.py iteration init --registry /workspace/iterations \
|
|
11
|
+
--input /workspace/prepared-meta.json --workflow /packages/workflow
|
|
12
|
+
python3 tools/cli/main.py iteration list --registry /workspace/iterations
|
|
13
|
+
python3 tools/cli/main.py iteration update --registry /workspace/iterations \
|
|
14
|
+
--input /workspace/updated-meta.json --workflow /packages/workflow \
|
|
15
|
+
--expected-sha256 <current-metadata-sha256>
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
init拒绝覆盖。update校验当前摘要、保旧元数据历史,再原子替换元数据和刷新索引。操作以目录锁串行;单文件写入原子,多文件不构成数据库事务,索引可由list扫描真实元数据恢复。list只读现有台账,不要求已结束任务的临时共享区仍存在。更新完整元数据须通过schema与结构校验,但不要求已回收的共享区仍存在;不能靠改变状态自动放行任务。
|
|
19
|
+
|
|
20
|
+
工具支持随包schema使用的受限JSON Schema词汇;未知字段和未知校验关键词拒绝。还核绝对/相对路径、独立持久位置、写入范围、记录输入禁止、分支名、阶段ID与依赖环。渲染成功不证明分支已推送、全部输入实际可读或业务已通过;Owner须在放行前确认。
|
|
21
|
+
|
|
22
|
+
## 任务渲染
|
|
4
23
|
|
|
5
24
|
```bash
|
|
6
|
-
python3 tools/cli/main.py iteration render \
|
|
7
|
-
--
|
|
8
|
-
--input /absolute/path/to/iteration.json \
|
|
9
|
-
--output /absolute/path/to/new-review.json
|
|
25
|
+
python3 tools/cli/main.py iteration render --workflow /packages/workflow \
|
|
26
|
+
--input /workspace/iterations/example/iteration.meta.json --output /workspace/rendered.json
|
|
10
27
|
```
|
|
11
28
|
|
|
12
|
-
|
|
29
|
+
生成公共及执行/检查点Markdown,还有结构化Git权限与依赖。使用context原文,不把JSON塞进任务正文。不覆盖已有输出。localInputs/localOutputs按内容分配,可多个;recordDirectory只写执行记录,不能被其他阶段列为输入。inputCommit允许草稿阶段暂缺,交接后须绑定精确提交。
|
|
13
30
|
|
|
14
|
-
|
|
31
|
+
## 阶段门禁
|
|
15
32
|
|
|
16
|
-
|
|
33
|
+
Owner生成阶段最小配置并通过执行环境索引提供:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
python3 tools/cli/main.py iteration scope --workflow /packages/workflow \
|
|
37
|
+
--input /workspace/iterations/example/iteration.meta.json --stage-id D \
|
|
38
|
+
--output /shared/environment/scopes/design.json
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
执行者无需访问Owner元数据,只读取该冻结范围文件:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
python3 tools/cli/main.py iteration stage-gate --workflow /packages/workflow \
|
|
45
|
+
--input /shared/environment/scopes/design.json --repository /work/business \
|
|
46
|
+
--head <full-candidate-commit> --output /shared/execution-records/design/gate.md
|
|
47
|
+
```
|
|
17
48
|
|
|
18
|
-
|
|
49
|
+
范围来自stage类型及冻结stage-meta,基线使用scope.inputCommit;scope不接受任意路径白名单。scope必须经Owner准备并冻结,门禁本身不验证分发者身份。公开工具只读Git,比较基线至候选;不扫描未提交内容、不自动推送。完整提交、祖先关系和路径权限通过返回0,越界返回3并输出差异Markdown,读取或参数错误返回2。重命名检查两端,空diff允许。未提交修改仍须按Skill处理,不能仅有门禁绿就声称完整交付。
|
|
19
50
|
|
|
51
|
+
Owner集成审阅仍可用 `iteration diff-review --input <meta> --workflow <package> --repository <repo> --base <sha> --head <sha> --stage-id D [--stage-id MT] --output <new.md>` 检查已接受阶段权限并集。它只检查路径,不扩展成内容或文件类型门禁。
|
|
20
52
|
|
|
21
|
-
##
|
|
53
|
+
## 报告与记录归档
|
|
22
54
|
|
|
23
55
|
```bash
|
|
24
|
-
python3 tools/cli/main.py iteration
|
|
25
|
-
--
|
|
26
|
-
--
|
|
27
|
-
--repository /absolute/path/to/business-checkout \
|
|
28
|
-
--base <full-base-commit> --head <full-candidate-commit> \
|
|
29
|
-
--stage-id D --output /absolute/path/to/new-diff-review.md
|
|
56
|
+
python3 tools/cli/main.py iteration archive \
|
|
57
|
+
--input /workspace/iterations/example/iteration.meta.json \
|
|
58
|
+
--source /shared/report-content --kind reports --name minifish-r1
|
|
30
59
|
```
|
|
31
60
|
|
|
32
|
-
|
|
61
|
+
`kind`为reports或records。目标在持久迭代目录下,以新内容版本命名,不覆盖原件。逐文件摘要校验复制一致,拒绝符号链接,检查静态HTML/CSS引用是否缺失或逃出归档目录;保存归档清单到records/archives。不会生成ZIP,不删除源目录。源需已停止写入;先后摘要不同则拒绝交付。
|
|
33
62
|
|
|
34
|
-
|
|
63
|
+
不执行或解析JavaScript生成的动态URL、不下载外部HTTP资源,也不证明图片能够解码或预览服务可达。最终检查点须实际打开用户预览链接、检查截图;动态资源在浏览器中核验。服务与持久文件分别管理。结束CoTask前归档并核验,不能结束后才尝试复制已回收文件。
|
|
@@ -57,119 +57,192 @@ def branch(value):
|
|
|
57
57
|
raise Invalid('Invalid branch name: ' + value)
|
|
58
58
|
|
|
59
59
|
|
|
60
|
-
def render(config, workflow):
|
|
60
|
+
def render(config, workflow, require_shared=True):
|
|
61
61
|
workflow = Path(workflow)
|
|
62
|
-
|
|
62
|
+
validate(config, json.loads((workflow / 'resources/iteration.schema.json').read_text()))
|
|
63
63
|
meta = json.loads((workflow / 'resources/stage-meta.json').read_text())
|
|
64
|
-
validate(config, schema)
|
|
65
64
|
if meta['version'] != config['metaVersion']:
|
|
66
65
|
raise Invalid('Stage metadata version mismatch')
|
|
67
|
-
relative(config['bizdir'])
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
shared =
|
|
74
|
-
if not shared.is_dir():
|
|
75
|
-
raise Invalid('Shared root must exist
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
66
|
+
relative(config['bizdir']); branch(config['git']['branch'])
|
|
67
|
+
def absolute(value):
|
|
68
|
+
p = Path(value)
|
|
69
|
+
if not p.is_absolute() or '..' in p.parts:
|
|
70
|
+
raise Invalid('Expected normalized absolute path: ' + value)
|
|
71
|
+
return p.resolve()
|
|
72
|
+
shared = absolute(config['sharedRoot'])
|
|
73
|
+
if require_shared and not shared.is_dir():
|
|
74
|
+
raise Invalid('Shared root must exist')
|
|
75
|
+
persistent = absolute(config['persistentRoot'])
|
|
76
|
+
if persistent == shared or persistent.is_relative_to(shared) or shared.is_relative_to(persistent):
|
|
77
|
+
raise Invalid('Persistent iteration storage must be independent of temporary shared storage')
|
|
78
|
+
absolute(config['source']['root']); absolute(config['environmentIndex'])
|
|
79
|
+
for artifact in config['provenance']:
|
|
80
|
+
absolute(artifact['path'])
|
|
81
|
+
def field(k, v): return f'- {k}:`{v}`'
|
|
82
|
+
def local(k, v): return field(k, 'localPath:' + v)
|
|
83
|
+
records = shared / config['bizdir'] / 'execution-records'
|
|
84
|
+
nodes, ids, branches, writes = [], set(), set(), []
|
|
85
|
+
def output_path(value):
|
|
86
|
+
p = absolute(value)
|
|
87
|
+
if p == shared or not p.is_relative_to(shared):
|
|
88
|
+
raise Invalid('Shared output escapes its root')
|
|
89
|
+
if any(p == old or p.is_relative_to(old) or old.is_relative_to(p) for old in writes):
|
|
90
|
+
raise Invalid('Overlapping shared writers')
|
|
91
|
+
writes.append(p)
|
|
92
|
+
return p
|
|
93
93
|
for stage in config['stages']:
|
|
94
94
|
if stage['id'] in ids or stage['workBranch'] in branches:
|
|
95
95
|
raise Invalid('Duplicate stage ID or work branch')
|
|
96
96
|
ids.add(stage['id']); branches.add(stage['workBranch']); branch(stage['workBranch'])
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
97
|
+
rec = output_path(stage['recordDirectory'])
|
|
98
|
+
if rec == records or not rec.is_relative_to(records):
|
|
99
|
+
raise Invalid('Execution records must be inside the business execution-records area')
|
|
100
|
+
for item in stage['localInputs']:
|
|
101
|
+
p = absolute(item['path'])
|
|
102
|
+
if p == records or p.is_relative_to(records) or records.is_relative_to(p):
|
|
103
|
+
raise Invalid('Execution records cannot be a stage input')
|
|
104
|
+
for item in stage['localOutputs']:
|
|
105
|
+
p = output_path(item['path'])
|
|
106
|
+
if p == records or p.is_relative_to(records):
|
|
107
|
+
raise Invalid('Use recordDirectory for execution records')
|
|
106
108
|
m = meta['stages'][stage['stage']]
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
if not relative(
|
|
109
|
+
def expand(value):
|
|
110
|
+
value = value.replace('{bizdir}', config['bizdir'])
|
|
111
|
+
if '{' in value or '}' in value:
|
|
112
|
+
raise Invalid('Unknown path placeholder')
|
|
113
|
+
if not relative(value).is_relative_to(relative(config['bizdir'])) or value == config['bizdir']:
|
|
112
114
|
raise Invalid('Output outside business directory')
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
lines += [field(k, 'gitBranchPath:' + expand(v
|
|
121
|
-
lines
|
|
122
|
-
lines
|
|
115
|
+
return value
|
|
116
|
+
directories = [expand(v) for _, v in m['outputDirectories']]
|
|
117
|
+
files = [expand(v) for _, v in m['outputFiles']]
|
|
118
|
+
lines = ['### 输入', '', field('工作分支', stage['workBranch'])]
|
|
119
|
+
lines += [local(i['label'], i['path']) for i in stage['localInputs']]
|
|
120
|
+
if 'group' in stage: lines.append(field('业务分组', stage['group']))
|
|
121
|
+
lines += ['', '### 执行', '', f'使用 Skill `{m["skill"]}` 完成{m["label"]}。', '', '### 交付', '']
|
|
122
|
+
lines += [field(k, 'gitBranchPath:' + expand(v)) for k, v in m['outputDirectories'] + m['outputFiles']]
|
|
123
|
+
lines += [local(i['label'], i['path']) for i in stage['localOutputs']]
|
|
124
|
+
lines.append(local('任务执行记录目录', stage['recordDirectory']))
|
|
123
125
|
nodes.append({'id':stage['id'], 'stage':stage['stage'], 'context':'\n'.join(lines),
|
|
124
126
|
'workBranch':stage['workBranch'], 'outputDirectories':directories,
|
|
125
|
-
'outputFiles':files, '
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
127
|
+
'outputFiles':files, 'recordDirectory':stage['recordDirectory'],
|
|
128
|
+
'dependencies':[i+'-review' for i in stage['dependencies']]})
|
|
129
|
+
reserved = {i+'-review' for i in ids} | {'delivery'}
|
|
130
|
+
if ids & reserved: raise Invalid('Stage ID conflicts with a checkpoint ID')
|
|
131
|
+
dependencies = {s['id']:s['dependencies'] for s in config['stages']}
|
|
132
|
+
visited, active = set(), set()
|
|
133
|
+
def visit(key):
|
|
134
|
+
if key not in ids: raise Invalid('Unknown dependency: ' + key)
|
|
135
|
+
if key in active: raise Invalid('Cyclic stage dependency')
|
|
136
|
+
if key in visited: return
|
|
137
|
+
active.add(key)
|
|
138
|
+
for dep in dependencies[key]: visit(dep)
|
|
139
|
+
active.remove(key); visited.add(key)
|
|
140
|
+
for key in ids: visit(key)
|
|
141
|
+
for entry in config['reviews']:
|
|
142
|
+
if entry['stageId'] not in ids: raise Invalid('Unknown reviewed stage')
|
|
143
|
+
absolute(entry['record'])
|
|
144
|
+
common = '\n'.join(['### 公共执行信息', '', field('Git仓库',config['git']['url']),
|
|
145
|
+
field('业务目录',config['bizdir']),local('Skill与工具入口',config['environmentIndex'])])
|
|
146
|
+
checkpoints=[]
|
|
147
|
+
for n in nodes:
|
|
148
|
+
lines=['### 输入','',field('工作分支',n['workBranch']),local('任务执行记录目录',n['recordDirectory']),
|
|
149
|
+
'', '### 审核','',meta['checkpoint']['review'],'','### 交付','',
|
|
150
|
+
local('审核记录目录',str(records/'checkpoints'/Path(n['recordDirectory']).relative_to(records)))]
|
|
151
|
+
checkpoints.append({'id':n['id']+'-review','dependencies':[n['id']],'context':'\n'.join(lines)})
|
|
152
|
+
used={dep for deps in dependencies.values() for dep in deps}
|
|
153
|
+
final={'id':'delivery','dependencies':[n['id']+'-review' for n in nodes if n['id'] not in used],
|
|
154
|
+
'context':'\n'.join(['### 输入','']+[field(n['stage']+'工作分支',n['workBranch']) for n in nodes if n['id'] not in used]+[
|
|
155
|
+
local('任务执行记录目录',str(records)),'','### 审核','',meta['delivery']['review'],'','### 交付','',
|
|
156
|
+
local('持久迭代目录',str(persistent))])}
|
|
157
|
+
return {'schemaVersion':2,'metaVersion':meta['version'],'commonContext':common,'nodes':nodes,
|
|
142
158
|
'checkpoints':checkpoints,'delivery':final}
|
|
143
159
|
|
|
144
160
|
|
|
161
|
+
def stage_scope(config, rendered, stage_id):
|
|
162
|
+
stage = next((s for s in config['stages'] if s['id'] == stage_id), None)
|
|
163
|
+
if stage is None: raise Invalid('Unknown stage ID')
|
|
164
|
+
if 'inputCommit' not in stage: raise Invalid('Stage input commit has not been accepted yet')
|
|
165
|
+
return {'schemaVersion':1,'id':stage_id,'stage':stage['stage'],'bizdir':config['bizdir'],
|
|
166
|
+
'workBranch':stage['workBranch'],'inputCommit':stage['inputCommit'],
|
|
167
|
+
'metaVersion':rendered['metaVersion']}
|
|
168
|
+
|
|
169
|
+
|
|
170
|
+
def scope_nodes(scope, workflow):
|
|
171
|
+
text={'type':'string','minLength':1,'pattern':r'^[^\x00-\x1f`{}]+$'}
|
|
172
|
+
props={'schemaVersion':{'const':1},'id':text,'stage':text,'bizdir':text,'workBranch':text,
|
|
173
|
+
'inputCommit':{'type':'string','pattern':'^[0-9a-f]{40}$'},'metaVersion':{'const':2}}
|
|
174
|
+
validate(scope,{'type':'object','additionalProperties':False,'properties':props,'required':list(props)})
|
|
175
|
+
relative(scope['bizdir']); branch(scope['workBranch'])
|
|
176
|
+
meta=json.loads((Path(workflow)/'resources/stage-meta.json').read_text())
|
|
177
|
+
if meta['version'] != scope['metaVersion']: raise Invalid('Stage metadata version mismatch')
|
|
178
|
+
if scope['stage'] not in meta['stages']: raise Invalid('Unknown stage type')
|
|
179
|
+
stage=meta['stages'][scope['stage']]
|
|
180
|
+
result={}
|
|
181
|
+
for key in ('outputDirectories','outputFiles'):
|
|
182
|
+
paths=[v.replace('{bizdir}',scope['bizdir']) for _,v in stage[key]]
|
|
183
|
+
for p in paths:
|
|
184
|
+
if not relative(p).is_relative_to(relative(scope['bizdir'])) or p == scope['bizdir']:
|
|
185
|
+
raise Invalid('Invalid stage output path')
|
|
186
|
+
result[key]=paths
|
|
187
|
+
return [result]
|
|
188
|
+
|
|
189
|
+
|
|
145
190
|
def main(argv=None):
|
|
146
191
|
parser = argparse.ArgumentParser(description=__doc__)
|
|
147
|
-
parser.add_argument('action', choices=['render',
|
|
148
|
-
parser.add_argument('--input',
|
|
149
|
-
parser.add_argument('--workflow',
|
|
150
|
-
parser.add_argument('--output'
|
|
192
|
+
parser.add_argument('action', choices=['render','diff-review','scope','stage-gate','init','update','list','archive'])
|
|
193
|
+
parser.add_argument('--input', help='Owner metadata or stage scope, depending on command')
|
|
194
|
+
parser.add_argument('--workflow', help='Frozen workflow Skill directory')
|
|
195
|
+
parser.add_argument('--output')
|
|
151
196
|
parser.add_argument('--repository')
|
|
152
197
|
parser.add_argument('--base')
|
|
153
198
|
parser.add_argument('--head')
|
|
154
199
|
parser.add_argument('--stage-id', action='append')
|
|
200
|
+
parser.add_argument('--registry', help='Initiator workspace persistent iterations directory')
|
|
201
|
+
parser.add_argument('--expected-sha256')
|
|
202
|
+
parser.add_argument('--source')
|
|
203
|
+
parser.add_argument('--kind', choices=['reports','records'])
|
|
204
|
+
parser.add_argument('--name')
|
|
155
205
|
args = parser.parse_args(argv)
|
|
206
|
+
def require(*names):
|
|
207
|
+
for name in names:
|
|
208
|
+
if not getattr(args,name): raise Invalid('Missing --'+name.replace('_','-'))
|
|
156
209
|
try:
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
210
|
+
from . import registry
|
|
211
|
+
if args.action == 'list':
|
|
212
|
+
require('registry')
|
|
213
|
+
print(json.dumps(registry.entries(args.registry),ensure_ascii=False,indent=2)); return 0
|
|
214
|
+
require('input')
|
|
215
|
+
config=json.loads(Path(args.input).read_text())
|
|
216
|
+
if args.action == 'archive':
|
|
217
|
+
require('source','kind','name')
|
|
218
|
+
result=registry.archive(args.input,args.source,args.kind,args.name)
|
|
219
|
+
print(json.dumps({'directory':result['directory'],'files':len(result['files'])},ensure_ascii=False)); return 0
|
|
220
|
+
require('workflow')
|
|
221
|
+
if args.action in ('init','update'):
|
|
222
|
+
require('registry')
|
|
223
|
+
if args.action == 'update': require('expected_sha256')
|
|
224
|
+
elif args.expected_sha256: raise Invalid('init does not accept an update hash')
|
|
225
|
+
print(json.dumps(registry.save(config,args.workflow,args.registry,args.expected_sha256),ensure_ascii=False)); return 0
|
|
226
|
+
require('output')
|
|
227
|
+
target=Path(args.output)
|
|
228
|
+
if target.exists(): raise Invalid('Output already exists; choose a new artifact')
|
|
229
|
+
if args.action == 'stage-gate':
|
|
230
|
+
require('repository','head')
|
|
231
|
+
from .diff_review import review
|
|
232
|
+
checked=review(args.repository,config['inputCommit'],args.head,scope_nodes(config,args.workflow))
|
|
233
|
+
target.write_text(checked['markdown']); return 0 if checked['passed'] else 3
|
|
234
|
+
result=render(config,args.workflow)
|
|
235
|
+
if args.action == 'scope':
|
|
236
|
+
if not args.stage_id or len(args.stage_id)!=1: raise Invalid('scope requires exactly one stage-id')
|
|
237
|
+
result=stage_scope(config,result,args.stage_id[0])
|
|
238
|
+
elif args.action == 'diff-review':
|
|
239
|
+
require('repository','base','head','stage_id')
|
|
240
|
+
selected=[n for n in result['nodes'] if n['id'] in args.stage_id]
|
|
241
|
+
if {n['id'] for n in selected} != set(args.stage_id): raise Invalid('Unknown stage ID')
|
|
167
242
|
from .diff_review import review
|
|
168
|
-
checked
|
|
169
|
-
target.write_text(checked['markdown'])
|
|
170
|
-
|
|
171
|
-
target.write_text(json.dumps(result, ensure_ascii=False, indent=2) + '\n')
|
|
243
|
+
checked=review(args.repository,args.base,args.head,selected)
|
|
244
|
+
target.write_text(checked['markdown']); return 0 if checked['passed'] else 3
|
|
245
|
+
target.write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n')
|
|
172
246
|
except (ValueError, KeyError, OSError) as error:
|
|
173
|
-
print(str(error),
|
|
174
|
-
return 2
|
|
247
|
+
print(str(error),file=sys.stderr); return 2
|
|
175
248
|
return 0
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
"""Initiator-owned iteration registry and durable report storage; never drives tasks."""
|
|
2
|
+
from contextlib import contextmanager
|
|
3
|
+
from hashlib import sha256
|
|
4
|
+
from html.parser import HTMLParser
|
|
5
|
+
from pathlib import Path
|
|
6
|
+
from urllib.parse import unquote, urlsplit
|
|
7
|
+
import fcntl
|
|
8
|
+
import json
|
|
9
|
+
import os
|
|
10
|
+
import re
|
|
11
|
+
import shutil
|
|
12
|
+
import tempfile
|
|
13
|
+
|
|
14
|
+
|
|
15
|
+
def digest(path):
|
|
16
|
+
return sha256(Path(path).read_bytes()).hexdigest()
|
|
17
|
+
|
|
18
|
+
|
|
19
|
+
def atomic_json(path, value):
|
|
20
|
+
path = Path(path)
|
|
21
|
+
path.parent.mkdir(parents=True, exist_ok=True)
|
|
22
|
+
fd, tmp = tempfile.mkstemp(dir=path.parent, prefix='.pending-')
|
|
23
|
+
try:
|
|
24
|
+
with os.fdopen(fd, 'w') as out:
|
|
25
|
+
json.dump(value, out, ensure_ascii=False, indent=2)
|
|
26
|
+
out.write('\n'); out.flush(); os.fsync(out.fileno())
|
|
27
|
+
os.replace(tmp, path)
|
|
28
|
+
finally:
|
|
29
|
+
if os.path.exists(tmp): os.unlink(tmp)
|
|
30
|
+
|
|
31
|
+
|
|
32
|
+
@contextmanager
|
|
33
|
+
def lock(root):
|
|
34
|
+
root = Path(root).resolve()
|
|
35
|
+
root.mkdir(parents=True, exist_ok=True)
|
|
36
|
+
with (root / '.registry.lock').open('a') as stream:
|
|
37
|
+
fcntl.flock(stream, fcntl.LOCK_EX)
|
|
38
|
+
yield root
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
def entries(root):
|
|
42
|
+
result = []
|
|
43
|
+
for path in sorted(Path(root).glob('*/iteration.meta.json')):
|
|
44
|
+
value = json.loads(path.read_text())
|
|
45
|
+
result.append({'id':value['iteration']['id'], 'goal':value['iteration']['goal'],
|
|
46
|
+
'state':value['iteration']['state'], 'metadata':str(path), 'sha256':digest(path)})
|
|
47
|
+
return result
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
def save(config, workflow, root, expected=None):
|
|
51
|
+
from .iteration import render, Invalid
|
|
52
|
+
render(config, workflow, require_shared=False)
|
|
53
|
+
with lock(root) as root:
|
|
54
|
+
target = root / config['iteration']['id']
|
|
55
|
+
if Path(config['persistentRoot']).resolve() != target.resolve():
|
|
56
|
+
raise Invalid('persistentRoot must equal registry root / iteration ID')
|
|
57
|
+
path = target / 'iteration.meta.json'
|
|
58
|
+
if path.exists():
|
|
59
|
+
if not expected or digest(path) != expected:
|
|
60
|
+
raise Invalid('Metadata exists or changed; supply its current SHA256 for update')
|
|
61
|
+
old = json.loads(path.read_text())
|
|
62
|
+
if old['iteration']['id'] != config['iteration']['id']:
|
|
63
|
+
raise Invalid('Cannot change iteration identity')
|
|
64
|
+
history = target / 'records' / 'metadata-history' / (digest(path)+'.json')
|
|
65
|
+
atomic_json(history, old)
|
|
66
|
+
elif expected:
|
|
67
|
+
raise Invalid('Cannot update missing iteration metadata')
|
|
68
|
+
for subdir in ('records','reports'):
|
|
69
|
+
(target/subdir).mkdir(parents=True, exist_ok=True)
|
|
70
|
+
atomic_json(path, config)
|
|
71
|
+
atomic_json(root/'index.json', {'schemaVersion':1,'iterations':entries(root)})
|
|
72
|
+
return {'metadata':str(path),'sha256':digest(path)}
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
class References(HTMLParser):
|
|
76
|
+
def __init__(self):
|
|
77
|
+
super().__init__(); self.values=[]
|
|
78
|
+
def handle_starttag(self, tag, attrs):
|
|
79
|
+
attrs=dict(attrs)
|
|
80
|
+
if tag == 'base':
|
|
81
|
+
raise ValueError('Report must not override its relative base URL')
|
|
82
|
+
for key in ('src','href','poster','data'):
|
|
83
|
+
if key in attrs: self.values.append(attrs[key])
|
|
84
|
+
if attrs.get('srcset'):
|
|
85
|
+
value=attrs['srcset']
|
|
86
|
+
if not value.startswith('data:'):
|
|
87
|
+
self.values.extend(x.strip().split()[0] for x in value.split(',') if x.strip())
|
|
88
|
+
if attrs.get('style'):
|
|
89
|
+
self.values.extend(css_refs(attrs['style']))
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def css_refs(text):
|
|
93
|
+
return re.findall(r'url\(\s*[\"\']?([^\"\')]+)[\"\']?\s*\)',text,flags=re.I) + re.findall(r'@import\s+[\"\']([^\"\']+)[\"\']',text,flags=re.I)
|
|
94
|
+
|
|
95
|
+
|
|
96
|
+
def check_assets(root):
|
|
97
|
+
"""Check static HTML/CSS dependencies. JS-generated URLs require browser review."""
|
|
98
|
+
root=Path(root).resolve(); files={}; errors=[]
|
|
99
|
+
for path in sorted(root.rglob('*')):
|
|
100
|
+
if path.is_symlink():
|
|
101
|
+
raise ValueError('Archive source contains a symlink: '+str(path))
|
|
102
|
+
if not path.is_file(): continue
|
|
103
|
+
files[path.relative_to(root).as_posix()]=digest(path)
|
|
104
|
+
refs=[]
|
|
105
|
+
if path.suffix.lower() in ('.html','.htm'):
|
|
106
|
+
text=path.read_text(); parser=References(); parser.feed(text)
|
|
107
|
+
refs=parser.values+css_refs(text)
|
|
108
|
+
elif path.suffix.lower()=='.css': refs=css_refs(path.read_text())
|
|
109
|
+
for ref in refs:
|
|
110
|
+
url=urlsplit(ref.strip())
|
|
111
|
+
if url.scheme in ('http','https','data','mailto','tel') or url.netloc or not url.path: continue
|
|
112
|
+
if url.scheme or url.path.startswith('/'):
|
|
113
|
+
errors.append(f'{path.relative_to(root)}: nonportable reference {ref}'); continue
|
|
114
|
+
linked=(path.parent/unquote(url.path)).resolve()
|
|
115
|
+
if not linked.is_relative_to(root) or not linked.exists():
|
|
116
|
+
errors.append(f'{path.relative_to(root)}: missing/outside reference {ref}')
|
|
117
|
+
if errors: raise ValueError('\n'.join(errors))
|
|
118
|
+
if not files: raise ValueError('Archive contains no files')
|
|
119
|
+
return files
|
|
120
|
+
|
|
121
|
+
|
|
122
|
+
def archive(metadata, source, kind, name):
|
|
123
|
+
from .iteration import Invalid
|
|
124
|
+
if not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_-]*',name): raise Invalid('Invalid archive name')
|
|
125
|
+
if kind not in ('reports','records'): raise Invalid('Unknown archive kind')
|
|
126
|
+
metadata=Path(metadata).resolve()
|
|
127
|
+
config=json.loads(metadata.read_text())
|
|
128
|
+
persistent=Path(config['persistentRoot']).resolve()
|
|
129
|
+
if metadata != persistent/'iteration.meta.json': raise Invalid('Not a registered metadata path')
|
|
130
|
+
source=Path(source)
|
|
131
|
+
if source.is_symlink() or not source.is_dir(): raise Invalid('Archive source must be a real directory')
|
|
132
|
+
source=source.resolve()
|
|
133
|
+
if source == persistent or source.is_relative_to(persistent) or persistent.is_relative_to(source):
|
|
134
|
+
raise Invalid('Source and persistent archive must be disjoint')
|
|
135
|
+
before=check_assets(source)
|
|
136
|
+
with lock(persistent.parent):
|
|
137
|
+
target=persistent/kind/name
|
|
138
|
+
if target.exists(): raise Invalid('Archive already exists; use a new content version')
|
|
139
|
+
pending=Path(tempfile.mkdtemp(prefix='.archive-',dir=persistent))
|
|
140
|
+
try:
|
|
141
|
+
shutil.copytree(source,pending/'content')
|
|
142
|
+
after=check_assets(pending/'content')
|
|
143
|
+
if before != after: raise Invalid('Archive source changed while copying')
|
|
144
|
+
target.parent.mkdir(parents=True,exist_ok=True)
|
|
145
|
+
os.replace(pending/'content',target)
|
|
146
|
+
record={'schemaVersion':1,'kind':kind,'name':name,'directory':str(target),
|
|
147
|
+
'files':after,'staticAssetsVerified':True,'browserReview':'required-for-user-preview'}
|
|
148
|
+
atomic_json(persistent/'records'/'archives'/(kind+'-'+name+'.json'),record)
|
|
149
|
+
return record
|
|
150
|
+
finally:
|
|
151
|
+
shutil.rmtree(pending)
|