@mcptoolshop/style-dataset-lab 2.2.1 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.es.md +76 -144
- package/README.fr.md +75 -143
- package/README.hi.md +75 -143
- package/README.it.md +76 -144
- package/README.ja.md +75 -143
- package/README.pt-BR.md +75 -143
- package/README.zh.md +76 -144
- package/SECURITY.md +3 -1
- package/bin/sdlab.js +21 -21
- package/lib/errors.js +82 -0
- package/lib/log.js +36 -0
- package/lib/snapshot.js +5 -2
- package/lib/split.js +5 -2
- package/package.json +3 -4
- package/scripts/snapshot.js +27 -23
- package/scripts/split.js +3 -1
package/README.zh.md
CHANGED
|
@@ -11,196 +11,128 @@
|
|
|
11
11
|
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue" alt="MIT License"></a>
|
|
12
12
|
</p>
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
编写您的视觉规则。生成艺术作品。根据这些规则评估每张图像。将结果作为版本控制的、可审计的训练数据发布。
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
Style Dataset Lab 将您关于艺术风格的描述与实际用于训练的数据集联系起来。您定义一套规范,包括轮廓规则、调色板限制、材质语言,以及对您的项目而言重要的任何内容。流水线生成候选作品,根据这些规则对其进行评分,并将通过的成果打包成可重复的数据集,其中每个记录都解释了它被包含的原因。
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
循环闭合:训练一个模型,生成新的输出,根据相同的标准对其进行评分,并将通过的作品重新导入。数据集不断增长,规则始终有效。
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
流水线会生成四个产出物:
|
|
23
|
-
|
|
24
|
-
| 文物。 | 它是什么。 |
|
|
25
|
-
|----------|-----------|
|
|
26
|
-
| **Snapshot** | 经过筛选,符合条件的记录已冻结,并进行了指纹识别。所有被选入的记录都有明确的纳入理由。 |
|
|
27
|
-
| **Split** | 安全的训练/验证/测试数据集划分,防止数据泄露。共享相同主题的数据记录始终会被划分到同一个子集。 |
|
|
28
|
-
| **Export package** | 自包含的数据集:包含清单、元数据、图像、数据集划分、数据集描述文档以及校验和。 |
|
|
29
|
-
| **Eval pack** | 符合佳能标准的验证任务包括:车道覆盖、禁止漂移、锚点/黄金区域、主体连续性。 |
|
|
30
|
-
|
|
31
|
-
流水线中的每个资产都包含以下三个要素:
|
|
32
|
-
|
|
33
|
-
1. **来源信息** -- 完整的生成历史记录(检查点、LoRA、种子、采样器、配置参数、生成时间)。
|
|
34
|
-
2. **合规性** -- 指示该资产是否符合相关规定,以及是否完全符合、不符合或部分符合。
|
|
35
|
-
3. **质量评估** -- 采用每维度的评分标准,对质量进行评估,结果分为:通过/未通过/临界。
|
|
36
|
-
|
|
37
|
-
主要从事游戏美术、角色设计、生物设计、建筑设计、车辆/机械概念设计等工作,以及任何需要保持视觉风格一致性的领域。
|
|
38
|
-
|
|
39
|
-
## 快速入门
|
|
20
|
+
## 流水线
|
|
40
21
|
|
|
41
22
|
```bash
|
|
42
|
-
#
|
|
43
|
-
npm install -g @mcptoolshop/style-dataset-lab
|
|
44
|
-
|
|
45
|
-
# Scaffold a new project
|
|
23
|
+
# Write your canon. Scaffold the project.
|
|
46
24
|
sdlab init my-project --domain character-design
|
|
47
25
|
|
|
48
|
-
#
|
|
49
|
-
sdlab
|
|
50
|
-
|
|
26
|
+
# Generate candidates via ComfyUI, then review them
|
|
27
|
+
sdlab generate inputs/prompts/wave1.json --project my-project
|
|
28
|
+
sdlab curate <id> approved "Strong silhouette, correct faction palette"
|
|
51
29
|
|
|
52
|
-
|
|
30
|
+
# Bind approved work to constitution rules
|
|
31
|
+
sdlab bind --project my-project
|
|
53
32
|
|
|
54
|
-
|
|
33
|
+
# Freeze a versioned dataset
|
|
34
|
+
sdlab snapshot create --project my-project
|
|
35
|
+
sdlab split build
|
|
36
|
+
sdlab export build
|
|
55
37
|
|
|
56
|
-
|
|
57
|
-
sdlab
|
|
58
|
-
sdlab
|
|
59
|
-
|
|
60
|
-
sdlab
|
|
61
|
-
sdlab generate:identity <packet> # Named-subject identity images
|
|
62
|
-
sdlab generate:controlnet # ControlNet-guided generation
|
|
63
|
-
sdlab generate:ipadapter # IP-Adapter reference-guided
|
|
64
|
-
|
|
65
|
-
sdlab curate <id> <status> <explanation> # Record review judgment
|
|
66
|
-
sdlab compare <a> <b> <winner> <reason> # Pairwise A-vs-B comparison
|
|
67
|
-
sdlab bind [--project <name>] # Bind records to constitution rules
|
|
68
|
-
sdlab painterly [--project <name>] # Post-processing style pass
|
|
69
|
-
|
|
70
|
-
sdlab snapshot create [--profile <name>] # Create frozen dataset snapshot
|
|
71
|
-
sdlab snapshot list # List all snapshots
|
|
72
|
-
sdlab snapshot diff <a> <b> # Compare two snapshots
|
|
73
|
-
sdlab eligibility audit # Audit record training eligibility
|
|
74
|
-
sdlab split build [--snapshot <id>] # Build train/val/test split
|
|
75
|
-
sdlab split audit <id> # Audit split for leakage + balance
|
|
76
|
-
sdlab card generate # Generate dataset card (md + JSON)
|
|
77
|
-
sdlab export build [--snapshot <id>] # Build versioned export package
|
|
78
|
-
sdlab eval-pack build # Build canon-aware eval pack
|
|
38
|
+
# Build a training package and close the loop
|
|
39
|
+
sdlab training-manifest create --profile character-style-lora
|
|
40
|
+
sdlab training-package build
|
|
41
|
+
sdlab eval-run create && sdlab eval-run score <id> --outputs results.jsonl
|
|
42
|
+
sdlab reingest generated --source ./outputs --manifest <id>
|
|
79
43
|
```
|
|
80
44
|
|
|
81
|
-
|
|
45
|
+
那个最后的命令至关重要。生成的输出会经过与所有其他内容相同的审查流程。循环闭合。
|
|
82
46
|
|
|
83
|
-
##
|
|
47
|
+
## 它产生的内容
|
|
84
48
|
|
|
85
|
-
|
|
49
|
+
七个版本控制的、具有校验和的成果。每个成果都链接到其前身,因此您可以追溯任何训练记录,了解它是根据哪个规则被批准的。
|
|
86
50
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
records/ Per-asset JSON (provenance + judgment + canon)
|
|
97
|
-
inputs/prompts/ Prompt packs (JSON)
|
|
98
|
-
outputs/ Generated images (gitignored)
|
|
99
|
-
comparisons/ A-vs-B preference judgments
|
|
100
|
-
snapshots/ Frozen dataset snapshots
|
|
101
|
-
splits/ Train/val/test partitions
|
|
102
|
-
exports/ Versioned export packages
|
|
103
|
-
eval-packs/ Canon-aware eval instruments
|
|
104
|
-
```
|
|
51
|
+
| 成果 | 它的定义 |
|
|
52
|
+
|----------|-----------|
|
|
53
|
+
| **Snapshot** | 带有配置指纹的冻结记录选择。每个包含项都有明确的理由。 |
|
|
54
|
+
| **Split** | 训练/验证/测试数据集,其中主题类别之间绝不交叉。 |
|
|
55
|
+
| **Export package** | 自包含的数据集:清单、元数据、图像、数据集划分、数据集描述、校验和。 |
|
|
56
|
+
| **Eval pack** | 与规范相关的测试任务:覆盖范围、避免偏差、锚定/黄金标准、主题连续性。 |
|
|
57
|
+
| **Training package** | 通过适配器(`diffusers-lora`、`generic-image-caption`)提供可供训练器使用的布局。相同的真理,不同的格式。 |
|
|
58
|
+
| **Eval scorecard** | 每个任务的通过/失败结果,基于生成的输出与评估包的比较。 |
|
|
59
|
+
| **Implementation pack** | 提示示例、已知的失败案例、连续性测试以及重新导入指南。 |
|
|
105
60
|
|
|
106
|
-
##
|
|
61
|
+
## 存在的理由
|
|
107
62
|
|
|
108
|
-
|
|
109
|
-
canon → generate → curate → bind → snapshot → split → export → eval
|
|
110
|
-
| | | | | | | |
|
|
111
|
-
rules ComfyUI judgment rules frozen subject package verify
|
|
112
|
-
selection isolation
|
|
113
|
-
```
|
|
63
|
+
训练数据是任何视觉人工智能流水线中最具价值的资源。但大多数训练数据只是一个包含图像的文件夹,没有任何历史记录、评估记录,以及与它应该遵循的风格规则的联系。
|
|
114
64
|
|
|
115
|
-
|
|
116
|
-
2. **生成**:ComfyUI 生成具有完整溯源信息的候选结果。
|
|
117
|
-
3. **筛选**:根据每个维度的评分和失败模式,进行批准/拒绝。
|
|
118
|
-
4. **关联**:将每个资源与规范规则关联,并给出通过/失败/部分通过的判定。
|
|
119
|
-
5. **快照**:将符合条件的记录冻结为具有确定性和唯一标识的选择。
|
|
120
|
-
6. **划分**:划分为训练集/验证集/测试集,同时进行主题隔离和类别平衡。
|
|
121
|
-
7. **导出**:构建一个独立的软件包,包含清单、元数据、图像和校验和。
|
|
122
|
-
8. **评估**:生成符合规范要求的测试工具,用于模型验证。
|
|
65
|
+
Style Dataset Lab 明确地建立了这种联系。您的规范定义了规则。您的评分标准定义了评分维度。您的筛选记录记录了评估结果。您的规范证明了这种联系。您的数据集将所有这些信息以结构化、可查询、可重复的方式传递下去。
|
|
123
66
|
|
|
124
|
-
|
|
67
|
+
实际结果:当您的 LoRA 出现偏差时,您可以询问*原因*。当您的下一个训练轮需要更好的数据时,您清楚地知道哪些记录是接近理想的,以及它们违反了哪个规则。当新团队成员询问项目的视觉语言是什么时,答案不是一个 Figma 板,而是一份包含 1182 个已评分示例的可搜索规范。
|
|
125
68
|
|
|
126
|
-
##
|
|
69
|
+
## 五个领域,真实的规则
|
|
127
70
|
|
|
128
|
-
|
|
71
|
+
不是占位符模板。每个领域都包含生产级别的规范规则、定义、评分标准和词汇表。
|
|
129
72
|
|
|
130
|
-
|
|
|
131
|
-
|
|
132
|
-
| **game-art** |
|
|
133
|
-
| **character-design** |
|
|
134
|
-
| **creature-design** |
|
|
135
|
-
| **architecture** |
|
|
136
|
-
| **vehicle-mech** |
|
|
73
|
+
| 领域 | 定义 | 评估内容 |
|
|
74
|
+
|--------|-------|-----------------|
|
|
75
|
+
| **game-art** | 角色、环境、道具、UI、飞船、内部、设备 | 游戏场景中的轮廓、派系特征、磨损和老化 |
|
|
76
|
+
| **character-design** | 人像、全身照、旋转视图、表情图、动作姿势 | 比例、服装逻辑、个性、姿势清晰度 |
|
|
77
|
+
| **creature-design** | 概念图、正交视图、细节研究、动作、比例参考、栖息地 | 解剖结构、进化逻辑、轮廓区分 |
|
|
78
|
+
| **architecture** | 外部、内部、街道景观、结构细节、废墟、景观 | 结构、材质一致性、透视、时代一致性 |
|
|
79
|
+
| **vehicle-mech** | 外部、驾驶舱、组件、示意图、轮廓图、损坏变体 | 机械逻辑、设计语言、访问点、损坏叙述 |
|
|
137
80
|
|
|
138
|
-
##
|
|
81
|
+
## 项目结构
|
|
139
82
|
|
|
140
|
-
|
|
83
|
+
每个项目都是独立的。五个 JSON 配置文件定义了规则;其他所有内容都是数据。
|
|
141
84
|
|
|
142
85
|
```
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
86
|
+
projects/my-project/
|
|
87
|
+
project.json Identity + generation defaults
|
|
88
|
+
constitution.json Rules with rationale templates
|
|
89
|
+
lanes.json Subject lanes with detection patterns
|
|
90
|
+
rubric.json Scoring dimensions + thresholds
|
|
91
|
+
terminology.json Group vocabulary + detection order
|
|
92
|
+
records/ Per-asset JSON (provenance + judgment + canon)
|
|
93
|
+
snapshots/ Frozen dataset snapshots
|
|
94
|
+
splits/ Train/val/test partitions
|
|
95
|
+
exports/ Versioned export packages
|
|
96
|
+
training/ Profiles, manifests, packages, eval runs, implementations
|
|
151
97
|
```
|
|
152
98
|
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
**分割**:将记录分配到训练集/验证集/测试集,同时保证样本隔离(同一个样本家族不会出现在多个分割中),并实现通道平衡的分布。使用种子伪随机数生成器,确保从相同的种子生成相同的结果。
|
|
156
|
-
|
|
157
|
-
**导出包**:是自包含的:清单、metadata.jsonl、图像(符号链接或复制)、分割、数据集卡(Markdown + JSON)以及 BSD 格式的校验和。 包含重建数据集所需的所有内容。
|
|
99
|
+
## 信任属性
|
|
158
100
|
|
|
159
|
-
|
|
101
|
+
这些不是理想化的目标,而是强制执行的。
|
|
160
102
|
|
|
161
|
-
|
|
103
|
+
- **快照是不可变的。** 配置文件指纹(SHA-256)可以证明没有发生任何更改。
|
|
104
|
+
- **分片可以防止数据泄露。** 按照身份、血统或 ID 后缀划分的主体类别永远不会跨越分区边界。
|
|
105
|
+
- **清单是冻结的合同。** 导出哈希值 + 配置文件指纹。如果任何内容发生更改,请创建一个新的清单。
|
|
106
|
+
- **适配器不能修改原始数据。** 不同的布局,相同的记录。不允许添加、删除或重新分类。
|
|
107
|
+
- **生成的输出需要经过审核才能重新使用。** 不允许绕过。像其他内容一样,进行筛选和绑定。
|
|
162
108
|
|
|
163
|
-
## Star Freight
|
|
109
|
+
## Star Freight
|
|
164
110
|
|
|
165
|
-
|
|
111
|
+
该仓库包含一个完整的示例:1182 条记录,5 个派系,7 条路线,24 条宪法规则,892 个已批准的资源,2 个训练配置文件。这是一个充满科幻感的角色扮演游戏视觉素材库,经过完整的筛选。
|
|
166
112
|
|
|
167
113
|
```bash
|
|
168
114
|
git clone https://github.com/mcp-tool-shop-org/style-dataset-lab
|
|
169
115
|
cd style-dataset-lab
|
|
170
|
-
|
|
171
|
-
# Validate the project
|
|
172
116
|
sdlab project doctor --project star-freight
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
sdlab snapshot create --project star-freight # 839 eligible records
|
|
176
|
-
sdlab split build --project star-freight # ~80/10/10, zero leakage
|
|
177
|
-
sdlab export build --project star-freight # package with checksums
|
|
178
|
-
sdlab eval-pack build --project star-freight # 78 eval records
|
|
117
|
+
sdlab snapshot create --project star-freight # 839 eligible records
|
|
118
|
+
sdlab split build --project star-freight # zero subject leakage
|
|
179
119
|
```
|
|
180
120
|
|
|
181
|
-
##
|
|
121
|
+
## 下游格式
|
|
182
122
|
|
|
183
|
-
|
|
123
|
+
`sdlab` 拥有数据集。格式转换由 [`repo-dataset`](https://github.com/mcp-tool-shop-org/repo-dataset) 处理:支持 TRL、LLaVA、Qwen2-VL、JSONL、Parquet 等格式。`repo-dataset` 用于渲染,但不决定数据的包含与否。
|
|
184
124
|
|
|
185
|
-
|
|
186
|
-
# Rename your data directory
|
|
187
|
-
mv games projects
|
|
125
|
+
## 安装
|
|
188
126
|
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
sdlab bind --project star-freight # canonical form
|
|
127
|
+
```bash
|
|
128
|
+
npm install -g @mcptoolshop/style-dataset-lab
|
|
192
129
|
```
|
|
193
130
|
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
**仅本地。** 与运行在 `localhost:8188` 上的 ComfyUI 通信。 没有遥测,没有分析,没有外部请求。 图像保留在您的 GPU 和文件系统中。
|
|
131
|
+
需要 Node.js 20+ 以及安装在本地主机 8188 端口上的 [ComfyUI](https://github.com/comfyanonymous/ComfyUI) 才能进行生成。
|
|
197
132
|
|
|
198
|
-
##
|
|
133
|
+
## 安全性
|
|
199
134
|
|
|
200
|
-
|
|
201
|
-
- DreamShaper XL Turbo 检查点 + ClassipeintXL LoRA
|
|
202
|
-
- Node.js 20+
|
|
203
|
-
- [`@mcptoolshop/repo-dataset`](https://github.com/mcp-tool-shop-org/repo-dataset) 用于训练数据导出
|
|
135
|
+
仅在本地运行。没有遥测数据,没有分析,没有外部请求。图像保留在您的 GPU 和文件系统中。
|
|
204
136
|
|
|
205
137
|
## 许可证
|
|
206
138
|
|
package/SECURITY.md
CHANGED
package/bin/sdlab.js
CHANGED
|
@@ -7,6 +7,9 @@
|
|
|
7
7
|
* Each script exports run(argv) and is lazy-loaded on demand.
|
|
8
8
|
*/
|
|
9
9
|
|
|
10
|
+
import { enableDebug, handleCliError, inputError } from '../lib/errors.js';
|
|
11
|
+
import { setLogLevel } from '../lib/log.js';
|
|
12
|
+
|
|
10
13
|
const COMMANDS = {
|
|
11
14
|
'init': '../scripts/init.js',
|
|
12
15
|
'generate': '../scripts/generate.js',
|
|
@@ -99,18 +102,27 @@ function printHelp() {
|
|
|
99
102
|
console.log('Options:');
|
|
100
103
|
console.log(' --project <name> Project to operate on (default: star-freight)');
|
|
101
104
|
console.log(' --game <name> Deprecated alias for --project');
|
|
105
|
+
console.log(' --debug Show stack traces on error');
|
|
106
|
+
console.log(' --verbose Verbose output');
|
|
107
|
+
console.log(' --quiet Suppress non-essential output');
|
|
108
|
+
console.log(' --dry-run Preview changes without writing (where supported)');
|
|
102
109
|
console.log(' --help Show this help');
|
|
103
110
|
console.log('');
|
|
104
111
|
console.log('Examples:');
|
|
105
112
|
console.log(' sdlab init my-project --domain character-design');
|
|
106
|
-
console.log(' sdlab
|
|
107
|
-
console.log(' sdlab
|
|
108
|
-
console.log(' sdlab
|
|
113
|
+
console.log(' sdlab snapshot create --project star-freight');
|
|
114
|
+
console.log(' sdlab split build --dry-run --project star-freight');
|
|
115
|
+
console.log(' sdlab training-package build --debug --project star-freight');
|
|
109
116
|
}
|
|
110
117
|
|
|
111
118
|
async function main() {
|
|
112
119
|
const args = process.argv.slice(2);
|
|
113
120
|
|
|
121
|
+
// Global flags
|
|
122
|
+
if (args.includes('--debug')) enableDebug();
|
|
123
|
+
if (args.includes('--verbose')) setLogLevel('verbose');
|
|
124
|
+
if (args.includes('--quiet')) setLogLevel('quiet');
|
|
125
|
+
|
|
114
126
|
if (args.length === 0 || args[0] === '--help' || args[0] === '-h') {
|
|
115
127
|
printHelp();
|
|
116
128
|
process.exit(0);
|
|
@@ -123,30 +135,18 @@ async function main() {
|
|
|
123
135
|
if (command === 'project' && args[1] && PROJECT_COMMANDS[args[1]]) {
|
|
124
136
|
commandArgs = args.slice(2);
|
|
125
137
|
const modulePath = PROJECT_COMMANDS[args[1]];
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
await mod.run(commandArgs);
|
|
129
|
-
} catch (err) {
|
|
130
|
-
console.error(`\x1b[31mError:\x1b[0m ${err.message}`);
|
|
131
|
-
process.exit(1);
|
|
132
|
-
}
|
|
138
|
+
const mod = await import(modulePath);
|
|
139
|
+
await mod.run(commandArgs);
|
|
133
140
|
return;
|
|
134
141
|
}
|
|
135
142
|
|
|
136
143
|
const modulePath = COMMANDS[command];
|
|
137
144
|
if (!modulePath) {
|
|
138
|
-
|
|
139
|
-
console.error(`Run "sdlab --help" for available commands.`);
|
|
140
|
-
process.exit(1);
|
|
145
|
+
throw inputError('INPUT_UNKNOWN_COMMAND', `Unknown command: ${command}`, 'Run "sdlab --help" for available commands.');
|
|
141
146
|
}
|
|
142
147
|
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
await mod.run(commandArgs);
|
|
146
|
-
} catch (err) {
|
|
147
|
-
console.error(`\x1b[31mError:\x1b[0m ${err.message}`);
|
|
148
|
-
process.exit(1);
|
|
149
|
-
}
|
|
148
|
+
const mod = await import(modulePath);
|
|
149
|
+
await mod.run(commandArgs);
|
|
150
150
|
}
|
|
151
151
|
|
|
152
|
-
main();
|
|
152
|
+
main().catch(handleCliError);
|
package/lib/errors.js
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Structured error shape for Style Dataset Lab.
|
|
3
|
+
*
|
|
4
|
+
* Every user-facing error has: code, message, hint.
|
|
5
|
+
* Exit codes: 0 ok, 1 user error, 2 runtime error.
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
export class SdlabError extends Error {
|
|
9
|
+
/**
|
|
10
|
+
* @param {string} code — error code (e.g. INPUT_MISSING_FLAG)
|
|
11
|
+
* @param {string} message — human-readable description
|
|
12
|
+
* @param {Object} options
|
|
13
|
+
* @param {string} [options.hint] — actionable fix suggestion
|
|
14
|
+
* @param {Error} [options.cause] — original error
|
|
15
|
+
* @param {number} [options.exitCode] — 1 (user) or 2 (runtime)
|
|
16
|
+
*/
|
|
17
|
+
constructor(code, message, { hint, cause, exitCode } = {}) {
|
|
18
|
+
super(message);
|
|
19
|
+
this.name = 'SdlabError';
|
|
20
|
+
this.code = code;
|
|
21
|
+
this.hint = hint || null;
|
|
22
|
+
this.cause = cause || null;
|
|
23
|
+
this.exitCode = exitCode || 1;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
toJSON() {
|
|
27
|
+
return {
|
|
28
|
+
code: this.code,
|
|
29
|
+
message: this.message,
|
|
30
|
+
hint: this.hint,
|
|
31
|
+
};
|
|
32
|
+
}
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
// ─── Convenience constructors ──────────────────────────────────────
|
|
36
|
+
|
|
37
|
+
export function inputError(code, message, hint) {
|
|
38
|
+
return new SdlabError(code, message, { hint, exitCode: 1 });
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
export function runtimeError(code, message, hint, cause) {
|
|
42
|
+
return new SdlabError(code, message, { hint, cause, exitCode: 2 });
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
// ─── CLI error handler ─────────────────────────────────────────────
|
|
46
|
+
|
|
47
|
+
let debugMode = false;
|
|
48
|
+
|
|
49
|
+
export function enableDebug() {
|
|
50
|
+
debugMode = true;
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
export function isDebug() {
|
|
54
|
+
return debugMode;
|
|
55
|
+
}
|
|
56
|
+
|
|
57
|
+
/**
|
|
58
|
+
* Handle an error at the CLI boundary.
|
|
59
|
+
* Shows structured output by default, stack trace with --debug.
|
|
60
|
+
*/
|
|
61
|
+
export function handleCliError(err) {
|
|
62
|
+
if (err instanceof SdlabError) {
|
|
63
|
+
console.error(`\x1b[31mError [${err.code}]:\x1b[0m ${err.message}`);
|
|
64
|
+
if (err.hint) {
|
|
65
|
+
console.error(`\x1b[33mHint:\x1b[0m ${err.hint}`);
|
|
66
|
+
}
|
|
67
|
+
if (debugMode && err.stack) {
|
|
68
|
+
console.error('\n' + err.stack);
|
|
69
|
+
}
|
|
70
|
+
if (debugMode && err.cause) {
|
|
71
|
+
console.error('\nCaused by:', err.cause);
|
|
72
|
+
}
|
|
73
|
+
process.exit(err.exitCode);
|
|
74
|
+
} else {
|
|
75
|
+
// Unstructured error — runtime failure
|
|
76
|
+
console.error(`\x1b[31mError:\x1b[0m ${err.message || err}`);
|
|
77
|
+
if (debugMode && err.stack) {
|
|
78
|
+
console.error('\n' + err.stack);
|
|
79
|
+
}
|
|
80
|
+
process.exit(2);
|
|
81
|
+
}
|
|
82
|
+
}
|
package/lib/log.js
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Logging levels for CLI output.
|
|
3
|
+
*
|
|
4
|
+
* Levels: quiet < normal < verbose
|
|
5
|
+
* Set via --quiet or --verbose flags (parsed by caller).
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
let level = 'normal'; // quiet | normal | verbose
|
|
9
|
+
|
|
10
|
+
export function setLogLevel(l) {
|
|
11
|
+
level = l;
|
|
12
|
+
}
|
|
13
|
+
|
|
14
|
+
export function getLogLevel() {
|
|
15
|
+
return level;
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
/** Always shown (errors, final results). */
|
|
19
|
+
export function log(...args) {
|
|
20
|
+
console.log(...args);
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
/** Shown at normal and verbose levels (progress, summaries). */
|
|
24
|
+
export function info(...args) {
|
|
25
|
+
if (level !== 'quiet') console.log(...args);
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
/** Shown only at verbose level (per-record details, debug context). */
|
|
29
|
+
export function verbose(...args) {
|
|
30
|
+
if (level === 'verbose') console.log(...args);
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
/** Always shown on stderr. */
|
|
34
|
+
export function warn(...args) {
|
|
35
|
+
console.error('\x1b[33m⚠\x1b[0m', ...args);
|
|
36
|
+
}
|
package/lib/snapshot.js
CHANGED
|
@@ -53,7 +53,7 @@ export function computeConfigFingerprint(projectRoot) {
|
|
|
53
53
|
* @param {Object} profile — selection profile
|
|
54
54
|
* @returns {Promise<{snapshotId: string, included: number, excluded: number}>}
|
|
55
55
|
*/
|
|
56
|
-
export async function createSnapshot(projectRoot, profile) {
|
|
56
|
+
export async function createSnapshot(projectRoot, profile, options = {}) {
|
|
57
57
|
const config = loadProjectConfig(projectRoot);
|
|
58
58
|
const recordsDir = join(projectRoot, 'records');
|
|
59
59
|
const snapshotId = generateSnapshotId();
|
|
@@ -104,7 +104,10 @@ export async function createSnapshot(projectRoot, profile) {
|
|
|
104
104
|
}
|
|
105
105
|
}
|
|
106
106
|
|
|
107
|
-
// Write snapshot files
|
|
107
|
+
// Write snapshot files (skip if dry run)
|
|
108
|
+
if (options.dryRun) {
|
|
109
|
+
return { snapshotId, included: included.length, excluded: excluded.length };
|
|
110
|
+
}
|
|
108
111
|
await mkdir(snapshotDir, { recursive: true });
|
|
109
112
|
|
|
110
113
|
const manifest = {
|
package/lib/split.js
CHANGED
|
@@ -127,7 +127,7 @@ function buildAncestorMap(records) {
|
|
|
127
127
|
* @param {Object} profile — split profile (from loadSplitProfile)
|
|
128
128
|
* @returns {Promise<{splitId: string, train: number, val: number, test: number}>}
|
|
129
129
|
*/
|
|
130
|
-
export async function createSplit(projectRoot, snapshotId, profile) {
|
|
130
|
+
export async function createSplit(projectRoot, snapshotId, profile, options = {}) {
|
|
131
131
|
const config = loadProjectConfig(projectRoot);
|
|
132
132
|
const snapshot = await loadSnapshot(projectRoot, snapshotId);
|
|
133
133
|
const includedEntries = await loadSnapshotIncluded(projectRoot, snapshotId);
|
|
@@ -242,8 +242,11 @@ export async function createSplit(projectRoot, snapshotId, profile) {
|
|
|
242
242
|
records, trainRecords, valRecords, testRecords, config.lanes
|
|
243
243
|
);
|
|
244
244
|
|
|
245
|
-
// Write output
|
|
245
|
+
// Write output (skip if dry run)
|
|
246
246
|
const splitId = generateSplitId();
|
|
247
|
+
if (options.dryRun) {
|
|
248
|
+
return { splitId, train: trainRecords.length, val: valRecords.length, test: testRecords.length };
|
|
249
|
+
}
|
|
247
250
|
const splitDir = join(projectRoot, 'splits', splitId);
|
|
248
251
|
await mkdir(splitDir, { recursive: true });
|
|
249
252
|
|
package/package.json
CHANGED
|
@@ -1,20 +1,19 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mcptoolshop/style-dataset-lab",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.3.0",
|
|
4
4
|
"description": "Visual canon and dataset production pipeline — snapshots, splits, versioned exports, eval packs, and canon-bound curation for any image production domain",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
7
7
|
"sdlab": "./bin/sdlab.js"
|
|
8
8
|
},
|
|
9
9
|
"scripts": {
|
|
10
|
+
"verify": "node bin/sdlab.js --help && node bin/sdlab.js project doctor --project star-freight",
|
|
10
11
|
"generate": "node scripts/generate.js",
|
|
11
12
|
"curate": "node scripts/curate.js",
|
|
12
13
|
"compare": "node scripts/compare.js",
|
|
13
14
|
"canon-bind": "node scripts/canon-bind.js",
|
|
14
15
|
"painterly": "node scripts/painterly.js",
|
|
15
|
-
"generate:identity": "node scripts/generate-identity.js"
|
|
16
|
-
"export": "echo 'Usage: repo-dataset visual generate ./projects/<project> --format trl'",
|
|
17
|
-
"inspect": "echo 'Usage: repo-dataset visual inspect ./projects/<project>'"
|
|
16
|
+
"generate:identity": "node scripts/generate-identity.js"
|
|
18
17
|
},
|
|
19
18
|
"files": [
|
|
20
19
|
"bin/",
|
package/scripts/snapshot.js
CHANGED
|
@@ -32,39 +32,43 @@ export async function run(argv = process.argv.slice(2)) {
|
|
|
32
32
|
const profileIdx = args.indexOf('--profile');
|
|
33
33
|
const profileId = profileIdx >= 0 ? args[profileIdx + 1] : null;
|
|
34
34
|
const profile = loadSelectionProfile(projectRoot, profileId);
|
|
35
|
+
const dryRun = argv.includes('--dry-run');
|
|
35
36
|
|
|
36
37
|
console.log(`\x1b[1msdlab snapshot create\x1b[0m — ${projectName}`);
|
|
37
38
|
console.log(` Profile: ${profileId || 'training-default (built-in)'}`);
|
|
39
|
+
if (dryRun) console.log(' Mode: DRY RUN (no files written)');
|
|
38
40
|
console.log('');
|
|
39
41
|
|
|
40
|
-
const result = await createSnapshot(projectRoot, profile);
|
|
42
|
+
const result = await createSnapshot(projectRoot, profile, { dryRun });
|
|
41
43
|
|
|
42
44
|
console.log(` \x1b[32m✓\x1b[0m Snapshot: ${result.snapshotId}`);
|
|
43
45
|
console.log(` Included: ${result.included}`);
|
|
44
46
|
console.log(` Excluded: ${result.excluded}`);
|
|
45
47
|
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
}
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
}
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
48
|
+
if (!dryRun) {
|
|
49
|
+
// Read and display summary
|
|
50
|
+
const summary = JSON.parse(
|
|
51
|
+
await readFile(join(projectRoot, 'snapshots', result.snapshotId, 'summary.json'), 'utf-8')
|
|
52
|
+
);
|
|
53
|
+
console.log('\n Lane distribution:');
|
|
54
|
+
for (const [lane, count] of Object.entries(summary.lane_distribution).sort((a, b) => b[1] - a[1])) {
|
|
55
|
+
console.log(` ${lane}: ${count}`);
|
|
56
|
+
}
|
|
57
|
+
console.log('\n Faction distribution:');
|
|
58
|
+
for (const [f, c] of Object.entries(summary.faction_distribution).sort((a, b) => b[1] - a[1])) {
|
|
59
|
+
console.log(` ${f}: ${c}`);
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
// Read and categorize exclusions
|
|
63
|
+
const excludedRaw = await readFile(
|
|
64
|
+
join(projectRoot, 'snapshots', result.snapshotId, 'excluded.jsonl'), 'utf-8'
|
|
65
|
+
);
|
|
66
|
+
const excluded = excludedRaw.trim().split('\n').filter(Boolean).map(l => JSON.parse(l));
|
|
67
|
+
const categories = categorizeExclusions(excluded);
|
|
68
|
+
console.log('\n Exclusion reasons:');
|
|
69
|
+
for (const [cat, count] of Object.entries(categories)) {
|
|
70
|
+
if (count > 0) console.log(` ${cat}: ${count}`);
|
|
71
|
+
}
|
|
68
72
|
}
|
|
69
73
|
|
|
70
74
|
} else if (subcommand === 'list') {
|
package/scripts/split.js
CHANGED
|
@@ -42,6 +42,7 @@ export async function run(argv = process.argv.slice(2)) {
|
|
|
42
42
|
const profileIdx = args.indexOf('--profile');
|
|
43
43
|
const profileId = profileIdx >= 0 ? args[profileIdx + 1] : null;
|
|
44
44
|
const profile = loadSplitProfile(projectRoot, profileId);
|
|
45
|
+
const dryRun = argv.includes('--dry-run');
|
|
45
46
|
|
|
46
47
|
console.log(`\x1b[1msdlab split build\x1b[0m — ${projectName}`);
|
|
47
48
|
console.log(` Snapshot: ${snapshotId}`);
|
|
@@ -49,9 +50,10 @@ export async function run(argv = process.argv.slice(2)) {
|
|
|
49
50
|
console.log(` Strategy: ${profile.strategy}`);
|
|
50
51
|
console.log(` Ratios: ${profile.train_ratio}/${profile.val_ratio}/${profile.test_ratio}`);
|
|
51
52
|
console.log(` Seed: ${profile.seed}`);
|
|
53
|
+
if (dryRun) console.log(' Mode: DRY RUN (no files written)');
|
|
52
54
|
console.log('');
|
|
53
55
|
|
|
54
|
-
const result = await createSplit(projectRoot, snapshotId, profile);
|
|
56
|
+
const result = await createSplit(projectRoot, snapshotId, profile, { dryRun });
|
|
55
57
|
|
|
56
58
|
console.log(` \x1b[32m✓\x1b[0m Split: ${result.splitId}`);
|
|
57
59
|
console.log(` Train: ${result.train}`);
|